Name: Tokenomics on AWS: Control and optimize your Claude spend
Uploaded: 2026-10-21T18:16:45.627Z
Duration: 1 h 12 s
Description: Tokenomics on AWS: Control and optimize your Claude spend

Transcript for "Tokenomics on AWS: Control and optimize your Claude spend": Welcome, everyone. Thanks so much for joining us today on this session, from Anthropic and AWS on tokenomics on AWS, how to control and optimize your Claude spend. My name is Scott Munson. I'm a member of the technical staff at Anthropic, and, along with me today is Ayan Ray. Ayan? Hey. Thanks, Scott. Ayan Ray. I'm the worldwide tech lead for the Anthropic partnership at AWS. Good housekeeping upfront. We are gonna provide a recording of this session. It will be distributed by email within twenty four hours, so, you'll have access to that. If you have any questions, please do submit them in the Q&A tab. That's the best way to reach us so that we can respond to your questions efficiently. And, also, we'd love your feedback. Please do rate this webinar. That survey gives us good information for us to pick the top... Next topics, in these sessions going forward. So we appreciate all the feedback you can you can share with us. And let's talk to the agenda. So today, we're gonna just do a brief bit of material upfront, talk to the concepts about what... Where costs come from, and what levers you have control over. Then we're gonna go to two, kinda more substantial sections really where Ayan gonna cover the Claude applications, Claude code, Claude enterprise, and Claude desktop, what, methods are available to optimize your costs with those platforms, and we'll go through those techniques in a very, like, practical way so you can see what that looks like in practice. And then I'll come back and cover, the Claude API services, both bedrock and AWS, talk through what is sort of helpful for you to understand, but then, again, take kind of a before agentic application, work with the cost optimized capability in Claude Code to improve your code and optimize for costs, and then look at kind of a before and after scenario there as well and a quick recap at the end. So really hoping this is gonna be a a practical session for all the folks in the audience. So to kick things off, I just wanna set the stage with, what you actually pay for. You know, there's the sticker price kind of is input tokens and output tokens. Generally speaking, these are, you know, what you send and what you get back from these models. And, if you think about the input tokens, it's kind of the one x. Output tokens are generally priced around five x. The thing that maybe gets less attention but is, critically important is cache, reads and writes. So when you write something to cache when you have this setting on, then the rate of input is 1.25 input. This is the kind of cache writing process, but the big opportunity here is cache reads. So point one x the input cost if... And sometimes much lower, in fact. We'll we'll go through some prices a little bit today, but huge opportunity for savings, especially for agentic applications. So we'll talk a little bit more about cash reads and writes later, but a big opportunity for cost optimization. And, another kind of useful level setting thing is to think in terms of cost per task and not strictly cost per token. You know, if you... If your tokens are inexpensive, but you have to do multiple tries to get the right answer, and you're actually burning a lot of tokens and ultimately more cost to get a successful task completion, it's sort of a false economy. So it's important to frame and think about these problems in terms of what does it cost to get to the right answer to get the task completed successfully. It's easy to compare those numbers side by side while on the... On kind of the rate card, but, really, the important part and with our more mature customers are doing is our our thinking in terms of cost per completed successful task. And this is where you wanna put your energy and time, so they're not you're not accidentally, measuring the wrong thing. Quick coverage of sort of the high level levers you control. Some of these you can control on both the app side as well as the API builder side. You know, the top of mind, of course, is model choice. You have different levels of intelligence, with different costs associated with them, both in price per token and cash rates. There's admins on the apps, builders. You're gonna wanna think through what the right intelligence level, and what model class to pick, to get your jobs done. Effort is the next, dial that you have access to. So how much thinking goes into each response? This is in terms of, you know, how thorough this intelligence is going to be. This is relevant for all audiences and a good one to to kinda develop some intuition, and there's also some controls over this to ensure that your teams are, building appropriately. Caching, we touched on briefly. Big opportunity here, and I'll go deeper on that later. Contact size, is another one to manage when you're working in, Claude code and Claude applications generally. And then there's also a lot of controls in the app side, especially in Claude code that admins can use. So default models, allow lists, sped limits, and alerts, these are all, great opportunities to make sure that you have predictability in your costs and can manage that at an org level. Also, I wanna highlight a pretty significant item here, which is that our list prices have actually gone down considerably. So at the large tier Opus level, it's 20% on the input output level and, 20¢ per million on cash reads, so down, 60%. In Sonnet we've gone down, 33% in input and output, and cash reads as well are at 10% per million. And then on Haiku, 90% reduction since our last launch of the Haiku model, up to a 100 k token prompts. Cash reads here are down to just 1¢ per million, down 90%. So huge opportunity if you're, picking the right model and model family and taking full advantage of cash. There's a a great, amount of savings to be had just in using our latest models and setting your, applications up correctly. And with that, I'm gonna invite Ayan back, to talk us through the Claude applications. Thanks, Scott. So, this part is for those who roll out Claude code, Claude enterprise, or Claude desktop to a team or to a whole company. You are usually not the one writing prompts. You are the one deciding what hundreds of people get by default, how far they go before someone gets a warning, or how will... You will see what it's costing and which habits you teach them. So couple of things. First up is the defaults and the model allow lists. Number one, pin the default model with the model setting. Here's what I see, with most of the customers. Most often, people use whatever they're given. This one matters on plot code with Bedrock's particularly because if you don't pin it, plot code on Bedrock defaults to Opus and fills at the Opus rate. But for every decoding, you might also want to start with Sonnet and then reach out to Opus for complex tasks. Second is to add an allow list with available models. So people can pick only models that you as an admin have approved and also turn on enforce available models to take this into effect. And also it covers the default models. So if you don't want your team to get access to Fable, this is the place where you omit it. Third is to cap the reasoning effort with max effort level. That works on any model provider including Bedrock. And also because these are managed settings, developers can override them. You set them once, and they apply, to everyone across the org. So defaults are the starting points. And next, we'll move to limits. Limits are usually the ceilings. And so there are a couple of ways to doing it. With Claude enterprise, you can set the spend limits at the organization, at the group, or at an individual user level. Admins get an alert when it is 7590% of the organization limit, and, also, users get their own notices at 70%... At seventy five and ninety five percent. With Claude desktop on Amazon Bedrock, the admin console is a... Which is in beta, is a great way to set token limits per user. And you can also use Claude apps gateway in between your harness that is Claude code and Claude desktop and bedrock to set daily, weekly, or monthly dollar caps for the whole organization or for a particular group or for individual users. And as I said, it works both with Claude code and also with Claude desktop. And you can also use AWS Budgets, which will alert on the total bill as a backstop. Limits the stops or prices, but you also want to know where the money is going. That's why the next topic is visibility. You want to see spend today and not when the invoice shows up. There are two views here, and you'll want both. The live view is open telemetry. It's one setting. Use Claude code enable telemetry setting in manage settings. Turn it on and Claude Code reports tokens and cost per user in near real time, send that to Amazon CloudWatch and label it by team or cost center. If you run on Amazon Bedrock, the billing view is the cost allocation tags. Give each team its own application inference profile in Bedrock, tag it, and that team spend shows up in its own AWS Cost Explorer. One view tells you what's happening right now. The other reconciles to the bill. So the techniques that I discussed until now are admin work. But the last one that I'll discuss now is for your developers. This one is about habits, and each one of these maps back to something Scott covered at the start. For example, the usage. This command shows what a session has costed so far and how well the cache is working. So it shows you the token cost input, output, cache write, cache rate, everything. The context command shows what's filling the context window. And remember, that's... Context is what gets sent on every single term, so it adds up. Compact summarizes a long session. So when your context window is getting full, use compact to summarize a long session, preferably do it as a natural break and not in the middle of a task. Clear starts fresh when you move to unrelated work, So the new task doesn't carry the old one. Effort turns reasoning down for routine tasks. So if you want to switch back the effort from high to medium or to low, this is the place where you do it. You can also start Claude Code with a particular model and send simple sub agent work like running tests or reading logs to a smaller model like Haikyu and keep the main session lean. So if you remember just one line from this slide, that would be keep the cache warm and keep the context small. So how do you prove this in your own environment? Pick one number, cost per developer per active day. Simple. Everyone understands and easy to track. Measure a baseline first. Give it a couple of weeks using the usage command and the OpenTelemetry cost metric we just saw, then apply the four techniques we discussed, defaults, limits, enable visibility, enable caching, and then compare compare it with your baseline. Then apply the four techniques, the... Set the default models, set an allow list to allow specific models, set the limits, spend limits at org team or individual user level, turn on visibility into the CloudWatch or or the other observability tools, and then compare. Give it a week and compare it with your baseline. And then measure again with the same team doing the same work. And the gap between those two bars is your serving. If you want a reference point, Anthropic's documentation puts the average for Claude code at about $13 per developer per active day with 90% of users under $30. Your your number is your own, and that's exactly why you measure it with your baseline. And now let me jump onto the demo. Okay. So here's Claude code on Amazon Bedrock. In the settings file, I've pinned Sonnet as the default, limited which models people can pick, and capped effort at high. Let's start Claude code and open the model picker. As you can see, it's pinned to summit just as you asked. And notice what's missing table. It's not in my allow list, so nobody can pick it by accident. Next effort is set to high by default, which is the ceiling we just defined. Now the part everyone wants to see, what is it costing? Usage breaks it down for me. Input tokens, output tokens, cash rates, cash rights. And look at this. 70% of my input tokens were served from cash. The same panel has tabs for model, config, and status, and status confirms the API provider is AWS WebRTC. Next context, this shows how much of the context window I'm using. Right now, it's 5% of a million tokens. Remember, everything in here goes back to the model in every single term. So let's tidy it up. Compact summarizes the conversation so far, and clear starts me fresh for the next task. Check again, and now it is down to 2%. And for routine work, I can always dial the effort down from high to medium, less thinking, fewer output tokens, and lower cost. That was the developer's view. Now let's look at it from the admin side in Claude enterprise. First, spend limits. At the top, I can set a monthly limit for the whole organization. Let's make it $10,001. And below that, I can set limits for each group and also for individual members. A member's own limit overrides their groups, and the organization limit is always the selling. So you can give one power user more room without raising it for everyone. And this view tells me how those limits are landing. Right now, zero users are blocked and zero are near the limit. If that number starts climbing, my limits are too tight. Next models. Here, I can allow or block each model and set the maximum effort it can reach. I'm turning Fable off because it's one of the most expensive models in the Claude family, and most day to day work doesn't need it. I can also choose the default model and the default effort of PIXONNET 5.5, and I can also set the default effort level so everyone starts in the same sensible place. And if different teams need different rules, I can create a custom role. Let's call it demo custom role. Inside it, I choose exactly which models this role can use, and I can choose... Switch capabilities on or off. For this role, I'm allowing chat and code execution and turning the rest off. Now let's switch to Claude code inside Claude desktop, signed in with Claude enterprise. The same commands work here. Status shows me how I'm connected, and usage shows what the session has cost. Exactly like the CLI on bedrock, so developers don't need to learn anything new. Here's one I really like is the adviser. My default model stays Sonnet and I set op... Opus 5.5 as the adviser. Sonnet does the work and only checks when... With Opus when it needs a second opinion. So you get Opus level judgment where it matters and mostly Sonnet prices. And admin gets observability built in. This is the Claude Code dashboard with reviews, productivity, usage, and value. The metric I'd like to highlight over here is cost per commit and even better cost per PR because that's the real question, not what a token costs, but what it costs to get a piece of work done. A merged PR is a finished task. So this is cost per task measured for you. Everything I've clicked so far, you can also do in code. Here, creating an API key that can read and write spend limits. I'll have it somewhere safe and then move over to the terminal. I'll now move move over to the terminal. I'll set the endpoint and the key. Let's look at the spend limits. You can see amount is 5,100 over here or 5,000. Everything is in cents. So, basically, it's $51 and $50. Now let's try to raise that spend limit for one of the user to $75. We'll say it as 75,000 because everything is in cents again. Now let's visualize again. And you can see the spend limit for this user ID has been changed to $75. So you can wear limits in your own tools, raise the limit when a project kicks off, or reset them every quarter. Now let's move to Claude desktop on AWS Bedrock. You get the admin console over there too. I set the inference provider as Bedrock. Turn on single sign on so people log in with with your account. When I add the models I want people to use, the first one in the list becomes the default. I set the default effort and tell it to always start with the default model. So every new sessions begin in the same cost sensible place. And finally, Claude apps gateway. If you run Claude code on bedrock and want, say, hard dollar caps per developer, this is how you get them. The gateway sits in your AWS account in front of Bedrock, and developers sign in through it with with with the company com... Company SSO. Here, I'm setting a spend limit through the gateway. Caps can be set at daily, weekly, or monthly. It can be set for the whole organization, a group, or one developer. And Claude Code warns developers at 75/95%. And at the limit, the gateway stops their requests until the period resets. So that's the full picture. Pick sensible defaults, set limits that warn before the block, and what's the numbers that matter, like cost per pull request. Set it once, and every developer starts in the right place. So that's about the demo. If you want a reference point, Anthropic documentation puts the average for plot code at about $13 per developer per active day with 90% of users under $30. But your numbers are your own. That's exactly why you measure it. So this is what you are saying and what we, did in the demo. Start with the baseline, apply the four techniques, defaults, allow lists, enable caching, do the spend limits, enable visibility, put in the right habits, and after that, you'll be able to see a significant cost savings into your overall cost. With that, I'll hand it over to Scott to cover the API side of things. Thanks, Ayan. Yeah. Let's dive into what, capabilities are available to help optimize the costs on the Claude API. This is, valid for both, the Claude, platform on AWS as well as Amazon FedRAMP. So, the first thing you wanna make sure you're doing with your applications is to match the model and the effort to the task. Right? So we touched on this briefly earlier, but, there's different levels of intelligence available from Anthropic, with different clog models. And any kind of simple high volume work are gonna be appropriate for the lowest, intelligence models. And when your evals, say you're not hitting the mark that you need, you'd wanna consider, higher tier, intelligence. And effort is, more about how thorough you're being with, the model. So the additional context, the amount of verification, all of that is wrapped up in the effort setting. Also wanna make sure, we go a little bit deeper on caching. So, we touched on this briefly earlier, but with the cache read and write, it needs to be a byte identical, write and read process in order to keep that cache and you need to grow it over time. On On an agentic workload, this is very important because it's often, multistep, and so you wanna make sure that you're preserving your cash over time. I wanna just talk through an example here. You know, your material that you input is made up of a number of things, not just the most recent message. So this includes the system prompt, any tools, any kind of documents and memory that are loaded in, the session state, and the conversation so far. All of that makes up, your cash. And so a new message when it's added onto that, if it's just that new message, you get to retain that cash and grow it over time. Again, significant cost savings on the read of that, cached content. Notably, there's an opportunity here to break that cache if you don't have it set up correctly. So one example that we see sometimes with customers is that, they might have a time stamp of some kind in a session state. If that is in the middle of your, information set, that can break the cash point. And, in doing that, you're gonna only get to keep the cash for that, material that precedes that time stamp and lose the cash afterward. Significant savings opportunity lost there. So that's just one example where cash can be, a big savings opportunity if you make sure to set it up correctly. Another scenario like this, that I wanna talk through is how routers work with cache, something to keep an eye on. Routers are great in concept certainly to route, a request to their appropriate level of intelligence on your behalf, but there is a bit of a a hidden cost here if you're not careful. I'm gonna talk through a scenario where, a router might actually cost you more than if you hadn't had a router in place because of cash. So let's just look at, you know, in this case, let's say we've got a a model a that you're sending a task to. You've got your system prompt, your tools, as well as that task. On step two, the router might send it to that same model a, and you would get to keep the cache that you've already got warmed up and just add that additional message on turn two. If you were to maybe ask a trivial follow-up, some lightweight simple question, as a as a follow-up, the router would probably route that next request to a different model, thereby breaking your cache and losing that, warmed up cache on model a, and now you've got a brand new request going into model b. That's a new cost you're incurring. The entire, inference, is is being... Is new to that model b, and you're gonna have a much higher cost for that call. Now if that next call is, again, a more sophisticated question or request, the router would push you back to a. And if that cache has gone cold, you would actually be doing a brand new call there as well. So while it sounds, like a like a... Like a frugal move to to use a router, you actually might end up costing much more. We've got a work example on the right here where, you know, 200 k token session, the first step will maybe be 4¢, but that switch to model b is a 12.5 x, hit on that, which is up to 50¢. And then by the time you come back to model a again, it's 25 x. What it would have been if you had just kept using the same model? So, without a router in place, that 10 step session could have cost 40¢. With one in place, you could end up as high as a dollar 82 for the same scenario. So make sure that you're being thoughtful about using any kind of router in, and make... Ensuring you're preserving your cache in the process. So I wanna talk a little bit more about the capability of the Claude API plugin in Claude Code. Basically, there's some really powerful capabilities in the Claude API, plugin. The way you access this, if you're in Claude Code, and I'll be demonstrating cost optimized in just a moment, in some detail, what that process looks like. So I'll show the cost optimized, but there's also a number of other that I won't have time for today, but I want... They're worth bringing attention to. So in Claude Code, you just type forward slash Claude dash API prompt audit, and, that can, help with a number of things, but it sort of audits that comp to make sure it's using caching and using, best practices for the model in question. We've got one that does migrate as well, which is also quite powerful for migrating models from older models to newer. And it will catch those kinds of, instructions and, can be used kind of without any additional eval setup. We also have the cost optimized, which I will walk through today, which helps you reduce your costs. It profiles your spend and then ranks a bunch of different items that you can do to reduce your costs of that application, that can be used with or without an eval. There's also the Claude API Hill Climb here, which go... Pairs very well with the build eval, which is another one of the capabilities that we have. So build eval lets you, pick an evaluation, and build out an eval based on your application. Once you have that eval in place, you can use hill climb, and that, with an eval paired with this hill climb, command, lets you kinda sweep through a variety of model, capabilities, effort levels, and, even the prompt itself to hill climb and improve the quality of your application, its performance, based against those eval outputs. Really quite powerful, very sophisticated. We're gonna do that with cost optimized today, and we'll do a sweep, and you'll be able to see what that looks like, when you have an eval in hand, what you can do to optimize the cost of your agentic application. Quick walk through of what we'll be seeing. The cost optimize command lets you, in Claude code, look through your repository. It will read your code and your usage data, so any logs you have in place, and it will evaluate and understand where these tokens are going. It will then look for opportunities to save, and it will rank those opportunities. You'll see this, in a in a moment, but that can be anywhere from prompt caching to batching to lots of other, options. And it'll identify which ones may have some trade offs. So you can kinda thoughtfully go, line by line to get the the improvements you think are are relevant or appropriate for your application. It will... You know, you can do these patches. It will write the code, for you, and then you can do... Apply those, and you can run tests against it to validate that it is still running at the quality that you require. And in those test runs, it actually asks you for a budget so that you're not, so you're aware and can control kind of costs you're spending on the test runs to, do these kind of, methodical evaluations on the optimizing of your application. And then you can apply those as commits just like you would any other software change, one change at a time. If the quality holds and is to your satisfaction, you can make the commit, and you're, you know, basically just, you know, improving your application's performance, alongside Claude Code. And just to reinforce, this is done in a way that's, you know, measured. We wanna make sure that our our customers are using this capability, against the baseline. You change one thing at a time. You remeasure again. Is the quality still there? Is it working the way we want it to work? If so, great. You can keep. If not, you revert, and you repeat that with the next lever. So you can really methodically go through and just optimize your application for cost in a thoughtful way, in a, irreversible way that, would be the kind of thing that, you know, you can build, in a in a really clear progress. So here's the the demo itself. We will start with an agent. It's a claims agent built for accuracy, not for cost. We'll establish the baseline. We're gonna run cost optimized in Claude code. We'll show what that looks like. We'll identify and find some savings opportunities, both the free wins and some trade offs, and then we will apply and remeasure and sweep through models and effort levels and find the right optimal settings, to improve batch cache as well as, the right, optimal model and effort. Alright. So let's jump to the demo. Let's walk through an example of how you can use the cost optimized feature in the Claude API plugin to reduce costs of the Claude agent without giving up quality. Quick note, this is a replay of a real run, So we can advance through the wait times and keep things moving. Our example is Acme Insurance. Acme is fictional and all the data is synthetic. It handles about a million auto claims a year and Claude agent decides each one. Adjuster opens a claim and awaits on screen. The agent pulls the claim, the policy, the history, and the fraud score, checks them against an underwriting manual, and ends with one of four decisions, Approve, deny, send to a supervisor, or refer to the fraud unit. Here's claim four. The agent refers it to the fraud unit. Three turns and about 23¢. Now I run a second job. At 2AM, it writes a short note on each new claim. Adjusters read those notes at eight. Both Joe's jobs log their own usage, so here's where we start. The agent scored 10 out of 10 on our test claims at about 29¢ a claim. The notes cost about 7¢ each. If you look at the cash column, 0%. We'll come back to that. At a million claims a year, this comes to about $366,000. So we start Claude code and on one command, forward slash Claude API cost optimize. It ships with Claude code, so there's nothing to install. It reads the code in the app's usage log, and it checks the current pricing and guidance. At this stage, it spends nothing on the API, and it changes nothing. The first pass only reports. And notice it drafts and tests its proposed changes in the scratch copy. So my repo stays untouched. About three and a half minutes later, here's the audit. The headlines at the top. Almost all the input tokens are sent at full price every time. There's a time stamp at the very start of the system prompt, so no two requests begin the same way, and nothing is marked for caching. That means a 13,000 token underwriting manual gets billed in full on every turn of the claim. So here's the starting point it measured. Straight from the logs, about 29 a claim for the agent and about 7¢ for triage. And here's the ranked list. It separates two kinds of change. A free win lowers the bill without changing the answer... Answers. And a trade off might change the answers, so it has to be tested. Number one here is prompt caching. And number two moves the overnight job to the batch API because nobody reads those notes until the morning, and the batch work is half price. Together, these two take us from about 37¢ to about 9¢ across both jobs, roughly 75 percent before we touch the model. Notice it calls the annual figure an extrapolation, and it's not not an actual measurement. Next, it explains the actual changes. They're written and tested, but not applied. It also spots a real operational risk that I didn't ask about. Batch is usually finished within an hour, but the guarantee is twenty four hours. So an 8AM, the deadline isn't guaranteed, and it suggests a cutoff. And I find this section also useful, what it recommends against and why. So tool search, compaction, moving the manual behind lookup. It's not throwing every feature at the problem straight away. And finally, it asks me to decide. It won't spend money or change code until I say so. So here's my reply. I'll go through, it as I type. And then, first, we'll apply the two free wins one at a time. And, we'll commit each one that it keeps and have it add the cutoff. I'm also gonna confirm that moving the time stamp is fine with the audit team since that was... It's one question for me. And then I set the rule that matters most. The configuration only stays if every trial score is 10 out of 10. I give a $25 budget, and I ask it to step down through effort levels and models. So I named Sonnet 5.5 in that list myself. It's worth saying that tool didn't propose it. And then I accepted 10 claims for this demo. I ask to see the table before it touches the config. It's an interaction here. And I'm able to to kind of guide it. The overnight notes keep their current effort, and we measure them on cost only. And now it gets to work. Now this first step takes a few minutes in real life, so we've skipped ahead. It does a really thorough investigation. And here's the same 10 claims with prompt cashing on. Same verdicts, 10 out of 10, twice. The cost per claim drops from about 29¢ to about 8¢. Huge win there. That's 72% off the agent, and no answer changed. Up to 97% of the input is now read from cash at a fraction of the price. And then it commits the change and moves on to the overnight job. And now the trade offs. This is where things get a little more interesting. It runs these comparisons in parallel. So for each model, it lowers the effort a step at a time with at least two trials each and stops a model when it hits the floor or fails. And here you can watch the prices. So Sonra 5.5 at high effort is about 2¢ a claim. Opens 5.5 at medium about 3¢. And the other setups keep coming back clean, and the costs keep falling. And then look at this. Opus five at low effort actually misses a claim. It sent nine claims to a supervisor when it should have gone to the fraud unit. Sent claim nine. Sorry. Nine out of 10, so that's setup is out. And that's why you test. On cost alone, it looks like a fine choice, but a wrong decision on a fraud case, costs more than the tokens you save. And so it's removed. And, it stops here before touching the config because I told it to show me the table first. So every row has caching on, and the one rejected row is the miss we just saw. It also caught a mistake in our own code. The price list had the wrong cache read price for Sonic five five, so those logged costs were actually too high. It checked the pricing page itself and recalculated. And looks like on these numbers, Sonic five five at medium looks like the winner. I just over sent a claim. It recommends that, and so did I. But I also tell it to run one more check on the final setup, and let's see what happens. So I approve it. It's not a five five meeting for the agent, and triage stays where it is. It also fixes the cache read price as its own commit. And I skipped the refusal fallbacks for now and asked them for... Asked for them in a report as a step before production. And then one fresh pass on the committed code. It only stays if it scores 10 out of 10. And notice how specific this is, the model, what stays put and what has to be true before a change stays. Everything lands in its... Has its own commit, so it's easy to review or undo. And now it runs that last pass, and it got nine out of 10. So claim eight should have gone to the supervisor, and this time, Sonnet approved it. It had passed three trials, then missed on the fourth. So under the rule I set, it comes back. And let's... Wanna be clear here. So it's the most kind of useful moment of the demo. It would have been easy to just ship this. The 10 claims is such a small test and tells us that two or three clean trials isn't much evidence. So it recommends testing the next candidates harder for about $2.70. And I agree. So we run four more trials on each of the three candidates for six trials each in total. Same rule. Every trial must score 10 out of 10. And if nothing passes, it stays on Opus five at high. And then it reapplies the split. It runs a final pass and writes the report with every result in it, including the failures. Off it goes. Skip ahead here again. Sonnet 5.5 at high effort misses a claim too, so it's out. And Opus 5.5 at medium, at low effort holds. Sorry. 10 out of 10, six times. Opus 5.5 at medium holds as well, but it costs a little more, so So what was the pick? Then the final pass in the committed code, 10 out of 10 at about 2.3¢ a claim. And this time, it stays. So here's where we land. The agent goes from about 29¢ a claim to about 2¢, down 92%. The overnight notes go from about 7¢ to about 2¢, down 75%. Together, And remember, you can run this on your own agent with one command that is already available in Claude code and, optimize your agentic application. Alright. So, just a quick recap on the demo. We had a chance to, baseline the application we're working with. We were running at about $366,000 annually. We went through the the claims. We found, a way to optimize that application. We used prompt caching. We batched for nightly jobs, and we went through a sequence of model, settings to make sure we found the right one and adjusted it to make sure that we still hit the quality that we needed with 10 out of 10. And as a result, we ended up with 89% saved at a 42,000 a year cost. And now let's do a quick, recap on, you know, kinda what we learned overall. We have really covered two different kind of categories of Claude usage. One is the applications. There are a lot of settings you can set up at the admin level, as well as, you know, techniques like, default settings, limits, visibility, and habits you can kinda set up within your teams as well to move from the baseline number of usage today, to kind of an outcome after the fact where you're having meaningful savings. And same as an opportunity on the API side. So builders out there, you can use a tool like cost optimize to identify any kind of application that's using, Claude models. How can you optimize your cost, reduce your your cost, through caching, through batch, and through model and effort selection? And, just, you know, keep in mind, the the price list is just the rate per token, and there's so much more to, a well tuned deployment of Claude applications and to well built AI applications that's not... That's beyond just the token price. And so, you know, you wanna measure. You wanna methodically go through and change these levers to set them the way that you want them and remeasure and keep what is working and holds the value, for your organization and reduce the costs. And we'll offer a few, next steps on where to go, with this information. We've got, you know, something small that you can do this week would just be to... You know, if you're on the admin side, pin a default model and an allow list in your manage settings, a simple starting point. You can turn on spend limits and alerts for a pilot team to kinda get comfortable with that and set that up. You wanna use telemetry and cost allocation tags for the platform team as well to make sure you're... You have eyes on the spend and understand, where that's flowing today in your organization. For the builders out there, update your kind of thinking to, you know, measuring cost per completed task for the agent, and then you can use tools like Claude API prompt audit, cost optimize, and that lets you really measure and methodically improve the application in a demonstrable way where you have data to back up that your application is running in an optimized way. And I wanna share some resources. There are some really good, articles that we're writing about these capabilities inside Claude code, techniques for, you know, best practices for Claude enterprise and Claude apps and Claude desktop and Claude code as well. A few blogs here. Please, put these to use and keep an eye out. We have these coming out all the time. Really wanna make sure that our, customers, put these in practice and have an easy way to to implement them in their environments. And with that, just wanna thank you all for being here today. Appreciate, you spending your time with us. Again, Scott Munson, from Anthropic. And... Yeah. Thank you, Scott. Thank you, everyone, for joining. Let us know your feedback, and let us know what sessions you want to see in future, and we'd be happy to address that. Alright. Thanks a lot. Thanks, everyone. Thank you.