Name: How to control costs and show ROI for Claude Code on Google Cloud (APAC Rebroadcast)
Uploaded: 2026-09-30T02:16:09.328Z
Duration: 1 h 8 s
Description: How to control costs and show ROI for Claude Code on Google Cloud (APAC Rebroadcast)

Transcript for "How to control costs and show ROI for Claude Code on Google Cloud (APAC Rebroadcast)": Hello, everyone. Good morning, good afternoon, good evening. Thank you for joining us. This is how to manage cost and show ROI for Claude Code on Google Cloud. My name is Roy Arsan. I'm an applied AI architect here at Anthropic, and I am joined by Ivan Nardini from Google Cloud. Between the two of us, we spend a lot of time helping platform teams, roll out Claude Code on Google Cloud. And today is about the two questions that come up rightfully so, in many conversation. Before we dive in, some housekeeping items. Don't worry if you miss something during the webinar. We will send out a recording, via email within twenty four hours. And if you have any questions, don't wait till the end of the webinar. Just shoot them right away, and we'll try to answer as many as possible and as best as possible over the live chat. And lastly, please give us feedback from today's session. There's a short survey at the end. It really helps, and it shapes what we're gonna cover next. Now, what we're gonna cover in the next hour, first, we're gonna jump straight straight into the cost governance toolbox, what Google Cloud gives you natively and what Claude Code instrumentation gives you as well natively. Then we're gonna have a couple of demos. First, how to govern, and we're gonna dive into that what that means exactly. Number two, how to monitor, using OTEL and cloud monitoring. And then, Ivan is gonna walk us through how you can go from, all the measurements, to proving value, you know, how the usage turns into dollars and impact tracking, and then we're gonna close with another demo and, and final takeaways. Before we dive into this, let's talk about the problem we are trying to solve. When platform teams evaluate Claude Code at scale, the questions we hear most often is, can I bound the spend against a given budget? Can I predict what this cost? Can I control which models my teams use? Those are all versions of the job number one over to the left. It's controlling the spend. Right? If you don't have this set up correctly, you don't have a central view of who's using what until the bill arrives. You, you know, you don't have per team limits or even guidance, and you're gonna end up with a bill that it's hard to reconcile. Right? Now there's a second job, which is proving what the spend bought. Right? Six months in, someone from finance might ask you, hey. What did we get out of this? Right? And that's a fair question. If you only build for job one, you'll have a very precise record of spending, but nothing, you know, not much to show in terms of value. Today, we're gonna cover both jobs and the tool that you are most likely already using. So spend itself is not the problem. I just wanna put that out there. It's the number that nobody can explain. Right? So everything we're gonna show you today is designed designed around that idea. So two years ago, AI answered questions. Right? Today, Claude does real work, drafting documents, designing decks or UI, building products, building, UI, for example, and shipping. And naturally, cost follows. Right? We're giving it more tasks. It's more agentic, more autonomous. So, the good news is Claude and Google Cloud is consumption pricing, right, as you are already used to, cloud environments and cloud infrastructure. Capacity and cost follow actual usage. There's no fixed provisioning here. Of course, there's an exception with provisioned throughput, which we can talk about separately if you want, you know, dedicated capacity. But in general, nothing sits idle. Right? Adoption drives usage. Usage is what you pay for, and then if you do the second job right, usage is where the value shows up. Right? That's the good news, but also the challenge. Right? Your bill is a readout of how much Claude is doing, but you as a administrator, your job has changed. Right? It's how do I let people get the most out of this without surprises? That's a balance, cost governance on one side, but also not blocking the work that really matters on the other side. And the balance is is is always shifting. Right? And it moves as both usage and value evolve. So what actually moves that bill? So there are two main dials. Right? And this would not be, you know, a surprise to you. So it's how many tokens and which model you use. So tokens scale with the tax complexity. You know, as I mentioned, you know, chat is is a short q and a questions, the conversation is very short, but as the agentic service grows and we are giving more ambitious tasks to our, to Claude, the agentic tool like Claude Code, for example, is doing multi step work and is spending more token per task because it's actually doing more of the work off of your plate. If So let's jump into, you know, let's open up the toolbox and see what that looks like. Over to the left side is what Google Cloud already gives you, right, For a project billing, for example, you can have a dedicated project for Claude Code as a clean way for cost tracking. You have budgets and alerts, so you are notified before you hit the certain threshold, but also you can set up a cap, right, on the Google Cloud side. I mentioned provision throughput if you need guaranteed capacity. And, obviously, quota is something you wanna plan for so that, you know, you're not hit with quota limits, you know, early on in your usage and as you scale up scale up. On the right side, where we're gonna spend most of the time, in this session on, this is what Claude Code and Claude Apps Gateway provides. And we're gonna talk more about Claude Apps Gateway in a little bit. But what you want what you expect, what you should look from, Claude Code and a gateway, you wanna look for per user identity, on every request. Right? If you attribute user identity on every request, then you can do analytics for users, per team, across the org, and you can set up those controls. Right? What kind of controls? Model allow list, spend caps, per user telemetry, we're gonna spend more time on each of those. Right? Now a little bit about Claude Apps Gateway. Claude Apps Gateway is Anthropic's own self hosted gateway. This is service that you run-in your Google Cloud project. It sits between your developer's Claude Code and Claude on, Google Cloud's agent platform. Right? It's not a separate product. It's actually the same cloud binary that your developers already run, and you just run it, as a server mode. Now why am I mentioning this? Because this is what gives you the controls we're gonna talk about next. Now if you are using a separate LLM gateway or maybe you are sending hotels straight from your developer machines without a gateway, you wanna make sure those controls are available to you some way, one way or another. But the Claude apps gateway provide you these controls out of the box, and this is what we're gonna demonstrate, in this talk. So level one, the word that matters here when it comes to control is is really guiding. Right? We don't wanna restrict, you know, work knowledge workers or developers. High cost sessions are not necessarily bad. So in fact, they're actually usually high value, especially in Claude Code. Those are generally your power users that are you know, at the forefront that are using new workflows and and getting a lot of value and a lot of work done. Right? You wanna actually find out, learn more about this. And again, we're gonna talk you know, Ivan is gonna touch on that a little bit. The the waste, however, is when you have a long gamut, long middle of high spenders. Right? For example, they're using the wrong model for routine work, which we we we touched on that earlier, you know, using Opus 5, for example, for something, like a daily routine that they can use Sonnet 5 instead. Right? Or maybe they have forgotten some scheduled task. That's why setting up these controls, you know, are important. What are these controls? Right? First, you've got spend limits. The Claude app gateway gives you spend caps. You can set can spend caps at the user level, at the team level, at the org level. You can set it by different time period, day, week, or month. Number two, threshold alerts. And this is really cool. So developers see the warnings right inside Claude Code as they approach a cap. Right? On the admin side, you can also alert through cloud monitoring, but also as a developer, I don't wanna be suddenly cut out, right? Again, you wanna guide folks, and we're gonna cover this in a little bit, how you can automate the system where you can requesting raising these apps. Right? Number three is the model default and allow list. Actually, yeah, model default and allow list is, you know, this is where you wanna specify, for example, which models a group can use. Right? You may not wanna open up Fable 5 for the entire organization. You can set it up for identity provider group. Remember, model choice is the other spend dial. What we found that works really well is you wanna set model default. Right? In most cases, people will stick to the model default. Now if they wanna switch to a model, they usually are switching for a reason. That's a high intent user. So definitely consider model default in addition or instead of model allow list. So this is where end user education helps a lot in addition to what we're talking about in terms of spend gaps and model defaults and allow list. Last but not least, you have the gateway admin API. Now everything I just said is as I alluded to is actually scriptable. So you have a admin API spend limits endpoint that the Claude apps gateway provides. So this enforce at the server side spend caps per user, per team, per org. One organization whose own product actually is incidentally a corporate spend management tool that uses this approach. They set threshold target threshold across their user base. The users can see and request increases, so there's no really hard caps. Right? And we're gonna show you live how that looks like from a developer experience point of view. So enough theory. That was a lot. Let's just see how how this this work. So the Claude apps gateway as I alluded to runs in your own Google Cloud project, in your own VPC. This is what fronts all the requests from your Claude Code sessions to Claude models on platform. Right? We're not gonna go into the architecture details, but this is just to show you that it runs in your own infra. Nothing through the you know, inference traffic goes to Anthropic. Everything stays in your own top perimeter and that is by design. Right? And then it's important to notice it's important to showcase that it provides you not just the spend limits and access control, but and the telemetry, but also the identity identity control. So every request is is is going through agent platform via a short lived session token, after I authenticate with my identity provider. Right? In this case, we're gonna use Google Workspace. So every request carries a verified identity. So let's jump into the first demo that covers this first part, which is governing. Right? Governing the identity and the spend caps. So what you're looking at is right now is my terminal. I have Claude Code running, and it's I'm just logging in here for the first time as a developer. So I'm putting my developer hat here. And again, as I'm starting this out, it's already configured to point to a gateway URL. Right? So this is a gateway that is already hosted on our company's Google Cloud project, our own corporate network. I'm just gonna click yes, and I'm gonna obviously, I need to I'm prompted. Do I do I trust this? Because the gateway is actually going to push down manage settings. Right? And that's also exciting because now most of the settings that you would configure via MDM or you push down to your developers' instances, now the gateway can push those down. Now this talk is not about security controls and and locking down and hardening Claude Code, which is which is, which is what you can do also, obviously, with the manage settings, with permissions and and tools permissions. We're gonna focus, obviously, on the OTEL aspect, the telemetry, aspect of this, and also, the spend caps. So, again, the VAW gateway covers all of those. I'm gonna trust this gateway. It's gonna redirect me via my identity provider. As you guys are not gonna see this, but it's opening my browser for, you know, single sign on. I'm confirming that this is the exact code I'm seeing, and I'm just logging in right now with my identity provider, which, you know, happens to be Google Workspace in this case, and it's already authenticated. Right? Now it's taking a second here to load all the policy, the managed settings from the gateway, including the hotel export settings. So what's happening is that my Claude Code hotel configuration, which is a bunch of environment variables, are pushed down from the gateway. And because it's it's sensitive, right, you're actually sending, telemetry data to your, you know, hotel collector. In this case, the gateway is for forwarding them to your hotel collector. There's a prompt that asks you to trust these settings. Now we're getting into the weeds here, but this is just to show you the experience of as a developer, I'm already in. Right? I already logged in. So if I do, my model allow list, I can see what the organization has allowed me to use. Right? So for the sake of this example, I'm just gonna use open Opus 5. In fact, I can see that, Fable 5 is available. So, let me see that here. Actually, I could do this. Claude Fable 5. Let's see if my organization made it available. Okay. Spend limit reach. Let me switch gear here and uncap. So this is a good example of how caps can be set. So as an admin so as a user, I already reached my spend caps. Obviously, we don't want this experience. Right? So what the experience should be, and I just, on a different terminal, I just removed my cap again using the admin API, and you can see here I, you know, I can interact with Claude, which is actually running on Asian platform fronted by the gateway. I'm just gonna say, you know, which model are we using, for example. I just wanna generate some traffic to see how this goes. Generate a 200 line random file. Okay? Yeah. Okay. So what's happening here is now I have uncapped. So what I'm gonna do just to simulate the spend caps, actually, on the admin API panel right now, on the admin API side, side, I'm actually gonna enforce a user spend cap. So what's gonna happen here as I'm interacting with this, see, I already reached my limit. So let's see. What I okay. So what's interesting here is that there's a admin message that's telling me to lift or raise your cap, file a ticket at go slash cc increase. Right? Now this is this is fictional. Right? But this is just to show you how you can improve the experience. But what what I didn't show is the, warning. So let me try this one more time because I looks like I skipped I went over the threshold of my cap immediately. So let me do this. I'm gonna do a ping, and there we go. So it's telling me you've used 93% of your user's credit. Right? It resets it. Now for those who are used to the first party experience or your or your Claude subscription, you this is not this is not foreign to you. But this is exciting. Now it's available on, on on on third party like GCP. So as a developer, I know I'm approaching my limit. I have visibility. There are no surprises. Right? And then when I do hit the limit, it's gonna give me this message that is admin custom message from my admin from my IT team, on how to lift this cap. Right? Again, we don't wanna put restrictions against developers. We wanna guide them. Right? Like, you know, especially the power users. So this is it for, the first, you know, the first demo that is control. I'm just gonna jump back to the slide. That was that was control. Right? That was level level one. Right? You are controlling the spend gaps. You're controlling the model allow list. You're controlling, identity, right, making sure everything is verified by identity. Now the next lever is tracking, right. You need an overview of the usage and the cost, which skills and which connectors are used, right. The good news is that Claude Code OpenTelemetry, provides metrics and events out of the box, and they can land in tools you already operate, like cloud monitoring, as we're gonna watch live in in the next demo. The second, you know, part of tracking is you wanna have end user view. This one is important. Right? Developers see their own spend and usage warnings, which as you saw earlier before they hit the limit. We don't wanna have surprises on either side, not on the platform side, not on the developer, you know, midway through a task. And then third is all this data can you can export to your own existing systems. Right? Claude spend and Claude usage right with your cloud billing and BigQuery, as we gonna show you as Ivan is gonna show you in a little bit as well. The same telemetry obviously can flow to your existing observability stack. If you're using, you know, another observability stack, you can send it over hotel. And, as we've shown you earlier, the get the gateway admin API can feed, the gateway admin API. We haven't really shown you the API, but it's all documented in the gateway Claude as gateway docs, but you can feed those spend limits straight into your existing ticketing system. In fact, one large enterprise came to us with very tight per user caps, similar to basically what I demonstrated earlier where I skipped my cap really quickly, but you wanna give more buffer, and they had more model restrictions in place. After working through their actual usage data, you know, together with us, they actually removed the caps entirely. This is their choice, obviously, and shifted to high daily limits plus a weekly top spender review. Right? Because they wanted to actually, what they really wanted is not just necessarily shrinking the bill, but really understanding where, you know, the bill essentially. So this is what how Telemetry gets into Google Cloud. Let's talk about this in this upcoming slide. You know, Claude Code, which is running on the developer machines, or it could be on your cloud workstations. It emits hotel out of the box. There's a, you know, a list of metrics we're gonna show you earlier, but it's well documented in our reference docs, but we're talking sessions, active time, lines of code, broken down by user, team, and model. It also can be broken down by skill and plugin and agents, if you're running agents. So obviously, you don't see just people using Claude Code, but also the agents running. And then and then two attribution that makes this really, useful. Right? You can attribute to your cost centers, to your departments using custom attributes in hotel. Right? So how does this look like? Right? How does this all land in Cloud Monitoring or Grafana or Splunk if that's your shop? Let's take a look at that. So we're gonna jump into the second demo here. Okay. So what I'm gonna show you is this window, which was basically the single sign on that you didn't see earlier when I was signing in to Claude Code, and this is gateway redirecting me to, the after the redirect after the identity provider login screen, in this case, Google Workspace, to send me back to the gateway and it confirmed that I'm signed in. But let's go back to the actual demo here for part number two, which is, you know, tracking usage. Right? So what you're looking at here is the adoption view, number of active developers. Again, this is in my own demo environment, but you can see, you know, number of developers, number of sessions over the last twenty four hours, total number of token. You know, this is actual, you know, usage and spend. You don't want this is also the block request, so you can you saw the block request earlier, in the Claude Code session. This is where it's being tracked. You really wanna avoid this, obviously. So you wanna make sure you have reasonable caps and also reasonable alerts and and and ahead of time. And like I mentioned, you know, this slots in really well with your existing ticketing system. You know, if a developer, you know, would like to request an increase. Right? And then this is my daily cap for the organization. Again, this is, you know, good to set that to make sure, like, you're controlling, and making sure there's no runaway cost. And then, total cost for the day, I'm not gonna go through every single thing, but the point thing to show you here is that all of these are driven by Claude Code hotel metrics. So this is a token usage by user, over time and then token usage by type of token. So I can see my cache, you know, prompt caching, for example, it's working nicely. I'm leveraging all the prompt caching. And then, you know, session per users, line of code. Even though line of code is not a good productivity metrics, use you know, as as Ivan is gonna show us shortly, but it is available out of the box. Cost per hour by model, you can see what models are being used. So a lot of data available at your disposal. And, also, you can see, hey. How close are developers to hitting their cap? You can have, you know, alerts set up in cloud monitoring, in addition to the inline CLI warnings that developer sees. Right? This is what I wanted to show you in terms of tracking, and, you know, to let's let's put this all together. So going back to this line. So what we've seen so far is, you know, Claude Code, native telemetry. Right? So we have different metrics. We have users metric. We have cost metric. We're not gonna go through each one of them here, but this is what we have established so far. Right? We have the usage and the cost, you know, by user, by model. You can have also an idea of the cash efficiency right there if you're looking at the operational side of things. You know, there's nothing to reverse engineer. There's nothing custom to roll out. Right? And you can attribute usage, you know, by user. The one thing I wanna highlight here is that these cost figures are approximation. These are based on list prices. Right? There's a way in the gateway you can override these prices, but generally speaking, for the source of truth of actual bills, you wanna correlate this with your GCP bill. The GCP bill is your, you know, dollars of record. Right? The telemetry I showed you is helpful to attribute to users and to team to understand where things are going, but for your final bill, obviously, reference your Google Cloud bill. Again, in the gateway Claude apps gateway, you can override, you know, the price list based on your own discounts, or pricing. With that, actually, this is a good point to hand it off to Ivan who's gonna talk about how we can take this, you know, and not just visualize the spend, but also forecast the spend and understand where things are going. Thank you, thank you, Roy. So let me quickly share the slides here. Slides are coming. Okay. So here we are, at one of the most important questions, that we, Roy, will receive from customers. So the idea here in this, last part of the presentation, we want to answer how do you how you can measure the value of using Claude Code on Google Cloud. And so we will provide you a possible framework, that you can look at and, and use. But before to jump, into, the framework, let's provide some, definition. So everything that is worth, measuring in Claude Code is related to essentially three dimensions. One is, the adoption and the usage, which is essentially, say, if people are actually using it, using Claude Code to code. The second one is satisfaction. So how are engineers happy and, what, what can be changed in the experience of using Claude Code. And, third one is the impact. So using Claude Code is delivering, like, a measurable engineering outcomes inside the the organization. So if you think about these three dimension dimensions, like, the first one and the second one, we already show you how you can measure it. And, this is something that Roy just show in the demo. With respect to satisfaction, it's more about running, you know, a survey to the team that is using or is about to use Go code and establish a sort of a baseline. But what about the impact, which is the most important thing, that you want to, that you you want to consider as well? You can, the here, you have a set of, possible impact metrics that you can start looking at. And so we can divide them in three main groups. First of all, the first one is, time. So which is more about how quickly, work get shipped, with, with GloCode. And, this dimension is usually related to measure, like, the PR cycle time from when you open a new PR to when the PR get merged and, what is called, like, the review turnaround. The second dimension, or the second group is, related to the effort. So how much work, each change, takes in order to, you know, be implemented. And so here, you have dimensions of measures like, review iterations or the rework of a particular PR. And finally, you you want to measure the volume. And so which essentially represent what the free, capacity delivers, in using Claude Code inside your organization. And so here you have measures like PRs per engineer, deployment frequency, the name body of works, which is essentially the number of backlog items that your team actually shipped, with, with this free capacity. Now those, those those dimensions and those measures that we pair for you or metrics that we pair for you are directional reference points. So you don't have to commit to them, but they are a very good starting point in order to, like, figure it out how to measure the impact of using Claude Code on Google Cloud with respect to, your organization. And, as always, like, in order to measure the impact, you need to define a baseline. And those metrics actually allows you to reuse, your delivery story, in some sense because are only mainly based around this concept of a PR. And so you would be able to set a baseline and then, like, start measuring, them, with respect to the usage of, of, of of Claude Code. Just one thing, probably some of you, might raise the questions or may notice is that, like, what's what's the concept of the the PR? There are so many type of, PR. So which one, which one we consider? And here, as I said, there are so many type of PRs. So, a a typo fix, it can be a PR, as well as, you know, you can work on a PR for an entire week. So what what is a what is a PR? Essentially so when you start, counting for a PR, like, the idea is, that the unit that you consider is a a merged, PR, with respect to a particular type. So when you start coding with Claude, you, like, you do you you code like, you start coding. You open a a PR, and then a person based on the PR and the code that is generated with respect to that PR decide, if the the code can be shipped or not. And, usually, because the PR contains a lot of metadata, you can also associate some label. Like, for example, the, the workflow owner of that particular PR, the ticket ID, if Claude was involved or not. So you can leverage essentially all these metadata with respect to merge PR in order to calculate some of the metrics that, I mentioned I mentioned before. So this, it gives you so the the dimensions and the metrics that we shared, in the previous slide and this concept of merge PR and how to you can leverage the metadata associated to the PR itself. It should give you an idea of how you can, now measure the impact of using Claude Code on Google Cloud with respect to your, software development life cycle. But, without further ado, let's jump, directly into the demo. So I will show you how you can structure, a flow and what are the metrics that, you will land. Okay. So here we are in our, development, environment. In particular, I am on Cloud Shell, on Google Cloud. The scenario that we, we decide to use in this case is a simple, it's a simple, checkout application. So I I already did some sessions to accumulate data using Claude Code with respect to this app, But the app is very simple. As you can see, we define several class. One is catalog. The other one is the promo. So it's a very basic cart, app. So imagine that, you are a developer. You you have this, this, base app, and then you want to start, developing a new feature. So what we can do is just, ask Claude to do that. Now we're looking new, feature using the live demo s h. So I already, like, I already created a very simple script. That, like, using code here with respect to what just, Roy mentioned to you, a bunch of logs that would be automatically collected in the Google Cloud environment. So if we switch if we switch on the, logs, explorer view in cloud logging, you can see how along the last days, I collected a bunch of, events. And these are the raw data that, Callcode send, directly to, Google Cloud that can be used to calculate some of the metrics that, I was, discussing before. And thanks to the native integration between, Claude, cloud logging to BigQuery, you will automatically, like, collect these metrics inside, a dataset that it will look like this. So, here, you have a bunch of views which are, essentially SQL queries, that you build on top of the like, you build on top of the of the raw data, and, that are already run. And, those raw data that we I was showing you in cloud logging, they will probably land in a table like, this one, which essentially, collect all the event time, and then you can extract some, metrics from there, from some some parameters from there. But you have the metrics name, the the, email of the user that has been used in order to, you know, develop that, the role, the project associated, the model that busy has been used, the token type, and so on. So from that table, the this is just an example of the usage, metrics, raw data that you can collect. From that table, then, again, you can define, those views. So running those, queries on top of these raw data in a a way to, in a way to calculate some of those, those metrics that I was saying. So, for example, like, we can, we can calculate, the cost, the cost per model. And, here you see how, like, the very simple query that you can use in order to do that. This is one of the simplest, like, metrics that you can calculate starting from those data. But, if we look, for example, at the impact metric as well, here, it gives you an idea of the kind of, queries that you can define starting from those, those raw data or derived table of, of that data. So, for example, in this case, I have exactly the, one of the metrics, that I, showed you in the slide with respect to the volume, the time, and the effort. Now we don't want to waste time on, like, spend time on, looking how to calculate this directly. And the reason is because all the code that we are sharing in this presentation, it will be published, after the event. So you can go in the repository and study the the metrics. So what I want to show you is that at the end of the day, like, directly, using the integration between BigQuery and Looker, you will land on a dashboard that look like this. So this is kind of the, this is kind of the dashboard that at the end, you will deliver to the stakeholder that needs to analyze the impact of using Claude Code with respect different team. So let's spend some time navigating this, this dashboard. And the dashboard that's been designed to answer some of the specific questions that, stakeholder might have. So, for example, like, the first the simplest thing that you can calculate in this that you can see in this slide is the spent amount. So over five days of coding, essentially, I spent 43, dollars. And this is because the app that I build, it was very simple, and this is a simulation. Probably, you will spend more over five days of coding. But, this is just, this is just an example. And, but if we want to also know, okay, who spend this, like, this, like, how we can associate this cost to different, you know, roads, in the in the team. Then you can just go to this, to this view that essentially split that spending among different teams, like engineering, security, SRE, design, and, and product management. So in this case, like, you you can directly associate, like, the, aggregated cost to, you know, the cost, with respect to with respect to the team. And, this is, as Roy said, is completely based on the GCP billing, that, is automatically, like, generated, is automatically collected on, on the Google Cloud Google Cloud console. The other the other question that you want to answer is the adoption. So what are the, active developers that use Claude Code by role, and, how many, edits they accept automatically from our Claude Code. And so as you can see here, like, we have, deep like, the team is very small. We have one PM, one SRE, one security engineer, two SRE. So these are the user that are adopting, like, call to call. And here, starting from those raw data that, Roy also was mentioning, we can collect, like, okay, how many, AI edit each figure it accept. And so we as we can see, in this case, the security engineer is, the one that accepted the most AI edit, the AI edit with respect to what suggested by, Cloecode. But and then so now we have an idea of, how much we are spending, per team and, what's the adoption of Callcode, and, like, how this adoption, you know, relates with respect to the code that is generated. But is this adoption producing, producing something? And if it's some if if it is producing something, what is it producing? So in this case, we need to remember what I just, explained in the presentation. Right? The unit of work with respect to this dashboard is the merge PR. And so, in this case, like, if we look, at the cost per PR on top, we see how each PR on average, it cost us, like, 3 almost 4, $4. And, and, like, in this way, you can understand, okay, how much how much, accepting this, this suggestion from the, Claude Code is generating, is generating cost. And this is very this is very important because, like, it gives you also dimension of the, like, how, each features or fix that has been implemented cost using, Claude Code. And, the last dimension that we want to, one of the last dimension that we want to, measure is that, yeah, okay. I paid almost $54 for this, for a particular PR, but was it worth? Like, what's the payoff of doing this? And so here is, where, you want you have a measure like, the the the ROI one. That measure, which is in this case, it's around, like, 34%, is essentially say is answering this kind of question. Is the engine isn't the engineer time, Claude saved, on a on a particular PR worth more than, you know, the PR cost itself that we just, we just saw. And, the idea here is essentially you you can calculate this, with respect combining, you know, the the cost per PR, the amount of time that, Claude removed in developing that PR with respect to, the, the the cost, of the free time that the engineer, got with respect to that particular PR thanks to the use of, of, claw code. So in this, so in this, in this sense, like, that 38% again, it measure the engineering time that cloth, allows to save with respect to that, particular to a particular PR. The last thing that I want to mention, which is, what, Roy was also mentioning before is that, in order to, like, have a a a vision or a view of how much the team it can spend. You can also leverage one of the cool features of BigQuery, which is the forecasting, capabilities. So the, the AI forecasting capabilities embedded in BigQuery using BigQuery ML to, like, forecast the next period spend per team. So in this case, like, I stopped, coding. If I remember well, I I stopped coding on August 27, and I was able to forecast in for the for the coming, for the coming days. And so with us and so in this case, you can see that I I'm forecasting, more or less, $10 in the coming in the coming day with respect to the spent, that, has been, that has been generated already with a certain confidence level, which in this case was 90%. But this is just gives you, an idea of the kind of, metrics that you can calculate with respect to the dimension that, we mentioned we mentioned at the beginning. And, of course, like, you can you can also have view with respect to the cost of the models. But, again, this is give you an overview of, how you can measure the, the impact of using Claude Code in developing, code within your company on, on Google Cloud. So with that being said, let's go back on the slide, and I think I need to pass you back, Roy. Thank you, Ivan. That was a great overview of how to map the usage to ROI. So this is a slide that we are building on top of the previous one. So now we added contribution and productivity column in addition to the usage and cost. Right? So the metrics you're seeing here are, you know, out of the box metrics provided by Claude Code. Couple of things I wanna highlight based on Ivan's demo. You also saw that Ivan is also capturing user prompts and, you know, tool decisions. Those are actually captured in the hotel, events. So what you're looking at on this slide are only the metrics, but there's also the events, you know, wealth of data. User prompt in particular, you know, you also have to opt in in an environment variable. You have to specifically request to log the user prompt because those are not logged automatically because of the potentially sensitive content, but it's, it's a good way to understand what your user base are trying to build. Right? Because as Ivan alluded to, not everything is is, you know, a feature built. I'm using Claude Code right now as my demo assistant. Right? I literally ask it to, you know, generate real traffic, you know, of simulated traffic into Claude into Google, cloud telemetry, and that's what's powering the dashboard that you saw earlier. You know, actual real Autel metrics, but there's simulated traffic from a Claude Code session. Right? So is that a PR? No. Which actually goes back to the other point. Not every unit of work is necessarily a PR, so you wanna figure out what your users are doing and and how to to track that. Because for example, in this case, my session was able to allow me to, focus more on the slide and the content than actually, you know, generating traffic. Right? So this is a simple example, but also another thing that Ivan alluded to, cloth code is measuring the number of PR created, not necessarily what's merged. So you wanna correlate the, you know, your version control system like GitHub or GitLab and, your ticket management system in terms of the actual unit of work produced, right? And Merge PR is basically an accepted work. So again, that's a good signal of what actually shipped, right? So customers commonly correlate the data that you're looking at right now with their data in their ticket management system. And, obviously, you can have Claude Code generate those insights for you as well, as well as the dashboard that we have shown you. So here's what some organization have, achieved, for example, as an example of what impacts look like with Claude Code on Google Cloud. All of these customers are actually Claude Code on Google Cloud customers. Deliver Hero, for example, had, you know, they overhauled their entire design system across 100 UI components. So work that used to take weeks or more, now it's in minutes. Again, this is a huge ROI in terms of engineering time. We're talking orders of magnitude of ROI return. Claude Code did it in minutes. Right? Shopify's, you know, again, similar story where engineers are building faster, but not just, engineers, also not engineers. You know, we're talking about internal tooling, you know, designers, etcetera. So but you might be wondering, is it just about feature velocity, but what about quality? So, you know, if you look at the third example with fifth dimension, you know, every engineer there is using Claude Code on Google Cloud and combined, they're writing 300,000 plus lines of code a month with a 98% acceptance rate. Right? So the acceptance rate is is is a good idea to see, you know, it's just not just about generating code, but also the high quality, you know, having your engineers trust, you know, and and and and Claude Code and ship at much faster rate and an equal or better, bar. So you also wanna track, the quality as well. We realized that, you know, even though the metrics and the events and and, you know, the native billing on the Google side and the reporting, a lot of this is available, there's a lot of pieces here that we covered that you, it's on the platform team to roll out. So for example, we've mentioned the Claude App Gateway, we mentioned the Autel export. So that's why we're happy to provide these assets. There are documentation around the hotel reference docs, but also there's a deployment guide for the Claude apps gateway. In addition to that, we're happy to publish some of these assets that we demonstrated today in our joint repo. You're gonna you're gonna get the link as part of the resources in this webinar. Specifically, in that repo, you'll find configuration on how to you know, configure the gateway to have the proper hotel settings that it's being pushed down to the Claude Code developer machines. Right? You know, because, again, even though they're available out of the box, you need to configure them, turn them on, turn on the telemetry, turn on the logs, turn on the user prompt if that's what you're looking for. So we do package those configuration for you. And then the reports, as you saw, the cloud monitoring dashboard that I've shown and the local dashboard that Ivan shown, even though it's based on metrics that is, you know, natively being exported, again, you need to roll out those on your own. So we may we are making them available so you have a place to start from. Yeah. So, with that in mind, I I wanna keep you with one last piece of advice. You know, the system you're building right now is to notice, observe, behave, and notice behavior in your user base and adjust. It's more about adjusting than just setting some numbers. Right? So the loop is continuous. You know? You're setting this target thresholds in terms of caps. You're you're setting these model allow list. You're monitoring. You're surfacing, when users are coming close to this threshold. You're also correlating with the value, and you're adjusting. Right? And both of those are are changing, especially as the amount of work and the type of work we're delegating to Claude Code is becoming deeper and more diverse. Right? So, hopefully, with this with this session now, you feel comfortable with the toolbox you have to control the spend and to prove the value. Thank you for spending the time with us. I also wanna thank a whole a lot of team that, we are, you know, surfacing some of the work from customer success team, from partner engineering team at Google, DevRel at Google. So a lot of the assets you're gonna see published, there's a there's a lot of folks who are working on this. So, again, thank you for spending the time, and we're happy to answer, you know, q and a questions if this time time permits over the live chat. Thank you. Thank you. Bye bye.