Video: How the Claude Code team does code review | Duration: 2912s | Summary: How the Claude Code team does code review | Chapters: Introduction to Code Review (10s), AI Model Inflection Point (67.39s), Code Review Bottleneck (153.15s), Bot Chaos Problem (248.415s), Code Review Goals (333.855s), Agent Topology Experiments (453.875s), Agent Workflow Results (608.83s), Agent Convergence (759.42s), Deployment Pipeline (900.53s), Cloud Code GA (994.095s), Code Review Workflow (1048.555s), ReviewMD Configuration (1254.75s), Evals and Continuous Improvement (1480.29s), Risk-Based Code Approvals (1671.855s), Q&A Session (1930.775s), Bot Conflicts Q&A (2056.96s), Team Adoption Success (2212.195s), Tuning and Evaluation (2291.305s), Context Management (2392.88s), Code Review Quality (2480.31s), Closing Remarks (2704.565s), Closing Remarks (2831.415s)
Transcript for "How the Claude Code team does code review": Alright. Hey there, folks. My name is Morgan Lund. I'm a product manager on the cloud code team, and I'm here to talk with you today about code review and how we do code review at Anthropic. This has changed a lot over the last couple of months. We've we've learned a lot. We've experimented a lot. We've tried things that didn't work. We've tried things that did work, and I'm here to just, like, bring dump all of this to you all and hopefully answer some good questions at the end as well. Liz, I believe we are good to go. Right? Should I jump on in? Alright. I will jump on in. So, I'm gonna start this by telling you all kind of a story, the timeline of what code review has looked like at Anthropic in the last nine months to one year or so. If you zoom back one year, we looked a lot like any other company. We had humans reviewing pretty much every pull request. PRs were shipped at the rate at which humans could write code, and that's, you know, the way that we worked. We hit this funny inflection point, around nine months ago when the models got good enough to build features. And, this changed things quite a bit. We suddenly went from this world where, like, folks were sitting synchronously with their cloud sessions and using it as sort of a fancy auto complete or a kind of in the moment synchronous assistant to being able to say, hey, Claude, see Jira ticket XYZ123. Go do that. And it could somewhat reliably just kind of do that to a reasonably good degree of quality. This was great. It saved everyone a whole lot of time. It meant that we could do a lot more. But it also meant that the next part of our process, started to see some some bottlenecks. So here's like a really simplified view of of the SDLC, at least as I see it. Starting with, planning, writing code, reviewing, fixing, CI, approving, merging, deploying, and monitoring, and everyone has additional steps in between here, I'm I'm sure. Before we continue, I'm kind of curious to do a little poll here on today. Like, what step of the SDLC causes you the the most pain? We've got a little poll up here. We'll give it a second to, for folks to put some answers in here because I'm very curious to hear where where the pain is right now. I know what my answer is, and I'll share it with you after, after this in a second here. Alright. That's enough time because I don't want everyone to sit here in silence. Where the pain for me is today, it's kind of like over here in the in the right hand section. We built a lot of things internally to help solve this section, which we're gonna be talking through in a moment. But, yeah, as you can see, as time to write code has compressed, that makes the other things take longer. There's a Harvard study I came across a couple of days ago that basically, like, found that when companies adopt AI agents for coding, they saw an average increase of 56%, of time it takes to get APR reviewed. My guess, based on my reading of this paper, is that this is because there is such a volume of PRs and people, like, start to drown under them. There is so much to review. Things take longer to review, and your throughput is, like, throttled by by this bottleneck here. So we started drowning in code review internally at Anthropic six to nine months ago. It was really rough because everyone felt so productive. We felt like we could throw up PRs for this feature and that feature and this bug fix and that bug fix. But then our PRs just started to sit there in this giant review queue. And we would go harass each other and be like, hey, you know, Clark, I'll stamp your PR if you stamp mine. And we kinda got to this weird phase where people were kinda stamping based on reputation, based on a really high level skim read of the PR. Like, it's kinda like a theater in in some way. Like, that's not a good way to to make sure you're maintaining a high quality code base. It's kind of it's kinda fake. And we lived in this kind of fake world for for a little bit of time. Being anthropic and being very comfortable building agents and stuff, a lot of individuals started to build their own code review agents, where they tried to kind of encode, in in logic and in in, agent skills and stuff. The way that they approach code review as an individual, the things to look for, the architectural patterns that they want to maintain. And this worked reasonably well for for some repos, but we ended up in this funny world where a lot of people had the same good idea, And we had bots stomping on each other. We had, like, seven different review bots showing up on a PR disagreeing about things, making conflicting changes. One change that one bot wanted would piss off another bot. We'd end up with these ridiculous loops of, like, bots fighting each other and, like, making 50, a 100, 200 commits on the single PR, which causes intent to drift away from, like, what the author originally, wanted on the PR. So that, like, wasn't a super great solution either. But, hey, we're we're learning. Right? So in the midst of all of this, maybe, back in July or so, Boris, the the, the godfather of cloud code came to me and to my team and said, hey, guys. Could you guys solve code review for entropic? And that was a bit of a scary ask. I don't know what it means to solve code review for entropic, let alone for anybody. But we we decided to to give it a go and see see what we could do. So we we set our own goal of, like, what it means to solve code review. And our goal was, let's try to make it so that at least 90% of PRs within anthropic are able to merge without any additional human effort beyond the human that told Claude to do something. So we built out this kind of notion of zero versus one versus two human in the loop PRs. Zero human in the loop PR means that Claude autonomously finds something that needs to be fixed. It fixes it. It puts up the PR. It babysits it through to Birch, and no human was involved at all. We have some workflows like that running. One human in the loop is I, as a PM on cloud code, saw a customer demand for a feature, saw a bug that that needed to be fixed, and I work with Claude to put up a PR, and get that all the way through to merge. And I don't need to go bother anybody else for a stamp because the automated systems, say that it's okay to go through. And then for higher, complexity, higher sensitivity sort of things, we do two or more human in the loop where, we require someone in addition to the author to sign off on a PR before it goes through. This is typically stuff that touches, like, really sensitive billing paths, things with, like, user authentication, things with our application auto updaters, stuff that, like, we really, really, really don't wanna break. But, you know, that we try to keep that very narrowly scoped. So we set the goal of 90% of PRs merging with zero or one humans of... Humans in the loop. So I don't need to go bug anybody to go merge a change because that's, like, the biggest time suck here is waiting for someone to get around to looking at your PR, loading the mental context, understanding what's in it, and giving, like, an informed decision on it. That's what really takes time. Right? And the guardrail we agreed on was we don't want any increase in reverts, in fixed forwards, in incidents. We don't want this, automation that we're installing here to make us worse in any dimension. So this was kind of the the the high level plan, the objective that that we agreed to. I'm not a researcher. I am just a lowly PM that thinks code is fun. Fortunately, Anthropic is full of people much smarter than me, who have a lot of research skills, especially building agentic systems. And I was able to nerd snipe some of them into coming to help fix code review with me. And they had all of these brilliant ideas around, different agent topologies, different ways of building evals that we can, test different approaches for doing code review. What we ended up landing on was creating an eval built of all incidents that we have ever had within anthropic and the PRs that were determined to have caused those incidents. So instead of being like a synthetic dataset that has artificially introduced bugs and code and languages and features that may not be directly reflective of what we use internally at Anthropic, we went in mind our real world incidents and, built a dataset out of those. Like, this is things that actually happened on our code that actually caused problems in production. We had humans go spot check some of these and make sure that we are getting it right as we built out our rubric. And this gave us something really strong that we could, we could hill climb on as we try to make really, really strong code review. So the hill climbing involved a bunch of experiments as anything with with science and research does. Right? So we tried different prompts. We tried different lenses that we could instruct the agent to look through. We we could say, you know, look for these sorts of security vulnerabilities or these sorts of item potency issues or these sorts of, you know, what... Whatever it may be to to catch the stuff that we needed to catch given the the eval dataset that we had built. We tried different agent topologies since that's a fancy way of saying, like, agents fanning out agents fanning out agents in different structures. Different jobs for the agents. So some agents' jobs are to find issues. And if you only have finder agents, you end up with a giant list of of findings and they may not all, like, be good findings or may not all be, like, important findings. So we introduce verifier agents that challenge the finding agents and, try to enforce that the things that are found and that are listed and and presented to a human or presented to a system are actually legitimate findings. And we ran tens of thousands, hundreds of thousands of iterations of various configurations and hill climbed against this. Hill climbing is just simply means you try something and did it make results better? No? Okay. Throw it away. Yes. It did? Okay. Now adopt that into main line and try a new experiment. Did it make results better? No? Throw it away. Yes. You know, and just do that a zillion times. Cloud makes it really easy to run a lot of a lot of experiments in parallel now. So we did all of that, and we got to a point where we were relatively happy with with the quality. The general flow that that we aligned on is that agents goes fan out a bunch of finder agents, they have verifier agents go, adversarially check the findings that the that the finder agents had located. We rank the the issues based on severity. We remove duplicates, and then we post them as a as a GitHub comment effectively, on the PR. We'd... We found that we kind of needed to partition out findings that are, like, legit but not, like, really worth blocking a PR versus those that should block a PR. So things that are super irrelevant, like, those get cut during the the verification stage. But things that are, like, kind of important but maybe not launch blocking, we invented this category of a knit, to to put those under. So the way our internal flow works is that important issues will block a PR from from merging. Nits will not, but those are kinda there for us to go mine later to do a big sweep if we care to. And then preexisting issues were a very interesting category. Would you tell an agent of sufficient intelligence to find a problem? It's gonna find a problem. It may, you know, be a big stretch, it may be like a really weird silly edge case, but like, these things are so motivated to do what you ask them to do. You need to kind of reign them in to some degree to like maintain like a relationship with reality. Otherwise, you're gonna end up in this crazy world where you have 500 back and forth cycles of, like, code review and fix and code review and fix and that's weird and lame and and not very fun. So we've got these categories. We only block on important ones internally, but we do try to surface the other things. And and the end result of this that that we found was 54% of our PRs get, like, substantive review comments now. When I say substantive, I mean, like, an important finding that we probably do want to fix before merge. Before we had automated code review, 16% of our PRs got a substantive review finding. So we're finding more things and more legit things, like, not more, like, lame crap that doesn't matter things. So this makes me, you know, confident that, that the process is working. Right? That handing over the reins to the agent is actually not to our detriment. Not only does it help speed us up and help us ship faster and help my day to day be less annoying, so I don't have to go bug people for stamps, but it's also just measurably safer and and does perform better than human review did for us. I alluded to this earlier, but, the topic of convergence is something that, like, I spent three weeks, like, obsessing over for for better or worse. Agents really love to find issues if you tell them to find issues. Agents love to fix issues if you tell them to fix issues. If you have an agent find a bunch of issues, have another agent fix them, and then tell the the finder agent, okay, those should be fixed. How are we looking now? It's gonna keep finding increasingly, like, niche and mundane crap to complain about. So we spent quite a bit of work tuning our agents to try and, present all findings upfront and to not, you know, continue digging deeper, deeper, deeper into a hole, after the findings have been fixed. So, you know, we had a period of time where agents would go back and forth overnight for 50 or a 100 different, commits, finding and fixing and finding and fixing resulting in, like, drift from the initial intent of the user and generally being very hard to read and very hard to follow along. We've been able to reign that in quite a bit. I don't think we have the stat here, but, some 90 some percent of our PRs converge after, like, three review fix cycles right now. So we got a pretty good handle on this, but it took a little bit of, banging our heads against the prompting to to make it work right. So at the end of the day, ever since we automated our code review, our throughput in terms of merged PRs has doubled again. I feel like every time I give a talk like this, it's doubled since the last time I gave the talk. Continues to increase. And we've had a 40% drop in, in PRs that had to be reverted. So this stuff is working. Like, it's safer. Like, the the analogy I like to give is kinda like self driving cars. You don't need to be perfect. You just need to be better than all the humans out there. And our squishy brains aren't that good at a lot of stuff. And I think code review is one of the things they're not, you know, the best at. Just loading of context and being tired and being lazy and being overwhelmed with all the things to do. I think a well structured code review, is really better suited for for agents at at this point. For legal and compliance reasons and stuff, you may still need to have humans in the loop, and we can talk about that a bit in the q and a. I've got a bunch of, of of experience and thoughts there. But generally speaking, you know, this is how we ship and and how we encourage our customers to ship now. It's it's it's genuinely safer. So the six layers that we go through kind of end to end, not to talk just about code review, We have agents verify what they build, like, in the session where they are building the code, and I'll show this in a in a quick demo in a second here before PR is ever put put up. Then we do the agentic code review, through the cloud code review product. We have our own security review. We have some additional, like, just regular CI things, you know, linting and and other tests and, stuff like that. And then we have some human approval that we we only involve on, the PRs that really need it. And I'll get into the logic for that in in a moment here, and then things merge. We're very fortunate in that, because of the shape of our products, they're largely shipped, via web or via apps that we control. We're able to make gratuitous use of feature flags. So most features in cloud code, we're in anthropics cloud services in general, have feature flags that enable us to very quickly turn things on or off. So if we ship something into prod, usually, it defaults to feature flagged off, then then we can open it up to a small number of internal testers, then a bigger number of external testers before we make something publicly available. And the nice part about this is if something breaks or if something is working wrong, we can very quickly within minutes just disable it for everybody without having to push a code change while we go fix it on the back end. So that's a really nice way of working. It makes my life a lot easier. If you have a realistic way of doing that, I would suggest it. It's it's pretty nice. So Cloud Code Gevey went GA about a month ago, in mid October, as part of GA. So it had been in public preview since, I think March. And as part of GA, we entirely rebuilt it. Basically, I mean, we we rebased the public code review onto our internal thing that I just talked about the process for building. So what you use today, if you go use cloud code review, is pretty much the same thing that we use internally. Because models had advanced by this time, we were able to make it a lot cheaper than it was too. Each PR cost, like, $25 or so to review back in March when we first released the preview. As of GA, we have three modes that that cost between 5 and $25 depending on the depth of review that you would like to do. So it's gotten cheaper. It's gotten faster. It's gotten better. I expect it'll continue to do in the future. It's a good time to be, to be trying to build things on top of these agents. So, just to give a quick little demo of, like, exactly what the end to end looks like when I'm developing the feature. Here, I'm working on an orders API. I'm asking Claude to add a date filter, and so it is going to go and make the code change for me, and it's gonna run NPM test. Usually, if you are running, like, a very normal tech stack that's got lots of information about how to how to use it in the training data, Claude is just gonna kind of know to run NPM test or a bun test or what... You know, whatever it is for the language stack that you're using. Sometimes you might wanna create your own verify skill if you've got some, like, weird custom way of invoking this stuff so that Claude knows how how to do that synchronously in the loop. And and I really believe in, like, the Swiss cheese method of of preventing regressions and failures. So test your stuff before you put up the PR. Test it after you put up the PR. You know, look at it through various lenses. And if you've got, you know, that much, confidence that you built, then you can start to push things through much more and much more quickly. Right? So in this case, you could see a test, failed. And because this is the session where Claude is actually just creating the the the code change, it can go and fix it itself. And it'll even have a sub agent go and review with a fresh context to make sure it looks good. One quick note on our slash code review here that you can use in the CLI. The hosted cloud code review, that you turn on at the repo level in your cloud.ai admin settings today is a little different than the slash code review that you get in the CLI or in the desktop app. This is intentional. It's because when you're working on the CLI or the desktop app, presumably, you're working somewhat synchronously. You're sitting there with the agent. You're working with it, back and forth. You probably do not want to wait for a fifteen or twenty minute code review, to take you out of your flow. So we've got this kind of very quick and simple code review and the slash code review command that you can use. And then after you have your PR up, when you're a little less time constrained because it's kind of going through the the flow up on GitHub, that is when the hosted code review will kick off and and do a deeper review. So we we make use of both of those. If you do want to run the deeper review from your, client, from your CLI, or from your desktop app, you could do slash code review ultra or ultra review, and that kicks off the exact same workflow that we do in the, in the hosted code review offering. So that's checking work before a pull request. And then on a pull request, so this is the hosted code review. This is the thing that I've been talking about, the thing that actually were... Attaches to your PRs in GitHub and just fires off, automatically anytime the PR is put up. So you can see it's gonna run. It's gonna read the diff. It's gonna look beyond the diff at the rest of the content of the repo if it feels like it wants to. Fan out all those finders that we talked about, fan out the verifiers, and rank the findings by severity. Then it's gonna come back with, okay, I got an important issue. I got a net. I got a preexisting issue. And it's gonna print those in line for you in this nice diff you including its reasoning, like explaining act... Exactly what what the issue is if you wanna click into it and learn those details. And you can see here there's a knit just about the time stamps, format. Again, not the kind of thing you necessarily want to block a PR, but worth, worth mentioning. Someone approves the changes, and we are in business. Quick note on CloudMD versus ReviewMD files. So if you use our our code review product, you can drop a ReviewMD file in the root of your repo and tell the agent what to look for and what not to look for. Maybe there's particular sorts of problems that you are particularly sensitive to and you wanna make sure the agent does a deep pass on those. Maybe there's, like, generated source in your repo that you don't want the agent to waste time looking at. All that stuff goes in your review MD. That is guidance for the review agent. The Cloud MD should remain stuff that anyone contributing and working on this repo needs to be aware of and needs to pay attention to. So there is, some prompting in the code review to tell it, hey. Take a look at the Cloud MD and make sure that the standards there are adhered to in in the PR. This prevents a a drift from the intent in your quad MD, in your PR. But I would still focus on keeping the quad MD mostly around, like, you know... As a very stupid example, we use tabs instead of spaces or spaces instead of tabs and, you know, we like to to use layers of abstraction following these general principles and not those general principles or whatever it may be. Those standards you want kind of consistent in the repo, do belong in the Cloud MD still. Here's some examples of stuff you can put in the Review MD. A change that breaks behavior, leaks data, or blocks the rollback is a failure. That's somewhat obvious. You probably don't need to say that. But if you only want a certain number of nits, if you don't like your reports having, like, 5,000 nits in them, you can tell it to cap that. If you don't want stuff that CI enforces to to be mentioned in there, you can you can tell it to not pay attention to things like that, to linting things or whatever. And you can give it, like, some requirements that you have for citations if you have a particular format that you want the agent to print its findings so you can hand it back to a fixed agent or something. So here's just a quick little example of behavior before and after you add some stuff to a ReviewMD. You can see here I got one important finding and a bunch of nits. And now I can tell the agent, hey. Like, don't tell me style issues. My linter catches that. Please don't waste your tokens on that. That's silly. And don't worry about the auto generated stuff. Like, that's kind of irrelevant. It's gonna be generated how it's generated. And then when you go run it again, you can see, look, I've only got the important findings. I don't have this, this noise. I'll talk a little bit more in a minute about, like, recall versus precision, but but this is kind of a way to make your findings a little bit more, precise. Oh, look at that. Slides right here. So the two criteria that we really measure our code review efficacy on are recall and precision. Recall is, like, given a rubric of potential things that could be wrong in a PR. How many does your code review agent find? This is when we're doing evals. And precision is how many things did your review agent find outside the scope of that rubric. The reason we have both of these is that finding everything in a rubric is actually not that hard if you give an agent free rein to, like, print 500,000 findings. Like, odds are it will eventually just through, like, throwing enough darts, hit everything that is on the rubric. But that's not especially useful in a code review product where, you want the signal to not be drowned out by the noise. So we also measure precision, which is basically what did the agent find that was not on the rubric. And presumably, if it's not on the rubric, it's not a real finding or if it's... Or it's not of, like, the sufficient importance that it should be reported. So a lot of our eval training is is around, tuning these two values to a degree that works properly for us. Too high of recall, and you're finding shit that doesn't matter. Precision, you know, you... The precision lets you whittle down to to the things that do matter. You can improve this continually, on your repo. So we have routines set up to automatically look at human comments on PRs and tune review MDs based on them. So if I put up a PR, a code review happens, and it has a finding that I think is really dumb and does not matter, I can comment on the PR and say, hey, Claude, like, this is actually not an issue because blah blah blah. And we have weekly routines that will run and go look for human comments on PRs, which is not very hard because there's not that many of them these days, and see if there's something to extract from the human comment to go add to the review MD so that in the future, the behavior of the review agent is a little more dialed in with, with what I would like. So it's through a process like this that you can kind of continually evolve and improve your automated code review. So you build more trust in it and you're able to hand off more, more to it. I talked about evals quite a bit and a lot of people ask me, like, well, that sounds like scary and academic and, like, what is an eval? How do I do that? Like... So I'm I'm showing you here. It's not that complicated. So right here, we have got a PR that merged, that had to be rolled back because, some customers got the same invoice twice, as a result of this PR. So, Claude has identified... A human has identified, why... Where the issue was. It was on line 88 of this TypeScript function right here. So this I am adding now as an eval case and we have, you know, many thousands and thousands of eval cases, but this is what one looks like. It's got an ID. It's got, the the commit ID of the repo. It's got the pull request number. It's got the, file and line number where this issue was, and it's got an answer. So I talked about a rubric before. The rubric for the eval is just a whole bunch of these. So if you've got thousands of these little YAML files here, each is, an example of something that Claude, when pointed at a particular repo at a particular point in time, should be catching as a part of its code review process. So you can build a corpus of these, like, out of your own incidents. If you have an incident tracker, which most companies I talk to do, point cloud at the incident tracker, ask it to, like, extract the PR numbers that were determined to be guilty for causing incidents, and just, like, go build this. You could probably do it in a in a couple of days, and create your own eval data set that you can then hill climb against on your view MDs. Try a variety of code review products. We're we're super proud of ours and we rely heavily on it. I personally think it's really great and works really well, but I don't want you to, like, adopt a product based on my feelings. I would really feel better if folks build their own code review evals and and prove to yourselves that these things are are good and they work well and they're trustworthy. And that gives you the confidence to, remove human checks where you can, save yourself some time, and ultimately be able to to ship a little bit faster. Here's just an example of the of the eval running, across a a test example of a code review agent, and a review MD being tuned as well. So after code review happens, we do some risk tiering internally. So, basically, we have a bot that decides based on the files that are touched in a PR, the extent of the changes in those PRs, how many humans we need to loop in. So for something that's just like a very simple docs change with a low blast radius, you probably don't need additional human approval like that one just goes through. For ordinary changes, usually the human author that told Claude to do something is sufficient. And then for the higher risk stuff, again, the auto updater, things that do have to do with billing, we might involve two people. We did this initially with just code owners files, but we we found that to be kind of a blunt force tool where you specify, like, for files under the certain path or according to a certain glob. You need one human or two humans depending on what is touched. We wanted more fine grained control than that, so we layered on top, some l m based classifiers such that you can do things like four, four code in this path that has to do with billing. If there's a logic change, you need to involve another person for sign off. If there's not a logic change, you don't and it has to go through. So we try to get very, very precise about where we require human, incremental human effort. So these are kind of the rules that we generally look at. The tiers come from what is touched, sensitive paths need a person, lighter paths are are for trusted authors. There's things that I can do in the cloud code repo that someone else in Anthropic that's not in the cloud code team cannot do. We have a little bit of partitioning in in that way. And the person who asked for changes is responsible for it. One side thing that I I do wanna mention that's not in this deck, the really cool part about automating code review and evaluating it and getting it to a point that you have high confidence in it is that you can start to be confident that whatever... No matter what you put in the the left side of the pipe, nothing is gonna come out on the right hand side that is gonna cause you a problem. And so this lets you do really fun things, like, if a partner team is blocked on you and needs a a feature and needs a change in your code base, you can just let them put up a PR. You can just kind of encourage drive by PRs, and you can be confident that nothing is gonna make it through to the other side unless it, like, adheres to your team's standards and it's not gonna break stuff. So no longer, you know, are we blocked on other people and trying to get on people's road maps to to do something, we kinda just can go push drive by PRs to other teams' repos, maybe ping them a Slack message and give them a heads up. But things aren't gonna merge unless it's, like, good according to that team standard. So, this lets you not be blocked on partner teams so much. It's been kind of refreshing for me. And and another advantage to being able to work a little more quickly like this is you can just try more things. If I have, like, a theory that I want to experiment in terms of, I don't know, how we present a particular feature in cloud code or how something should work, I can spin up, you know, five different versions of of a concept, AB test them at scale across users, and depending on the metrics that I care about, ship the thing that actually matters. I also see this happening a lot in performance optimization. So Boris has, like, these routines that run every single day that look for opportunities to improve start up time and stuff on the cloud code CLI. They're just autonomously trying experiments and experiments and experiments, and the things that, measurably improve latencies, measurably improve startup time, and don't regress anything else get merged. And so we can have just kind of this continual pipeline of of clouds trying to evolve our software and feel confident that it's not gonna break anything, once we've got these guardrails in place. So, there... There's a lot of benefits other than just shipping faster. You can, like, do, like, some novel stuff, once you kinda put your your SDLC on rails like this. And this is an example of the various approval paths and what that may look like. So here we've got, like, a typo being fixed in the setup guide. That is definitely gonna merge without additional human review. Adding a date filter to an orders list that we take one human review because it touches orders. And then, changing the release workflow. Obviously, we need someone from the release team to to okay that because that's a higher blast radius. So this is legitimately kind of how it looks on the inside for us these days. After findings land, something has to fix them. A lot of our work is done with, like, Claude tag or people have written their own PR babysitter skills. So their own Claude session that, like, wrote a PR is kind of pulling, to GitHub to see if there's a code review finding or something that needs to be fixed. We've got a number of solutions kind of depending on how people interact with Claude for for doing this. We have some more stuff upcoming that y'all will learn about relatively soon. But I would strongly encourage everyone to explore various ways of, like, automatically fixing code review findings because you're not gonna see that acceleration if you're still gated on humans, being awake, being available to to fix something that's blocking merge. And then finally, of course, the decision stage that we talked about with automated risk classification. You can hopefully reduce the number of, pieces of your code base that require human sign off and get things through the pipe quicker so you can realize all these fun fantastical benefits of of, a more automated SDLC. Can do more experiments. You can min max on on the metrics that you care about. You can just get features out for your customers quicker. It's it's nice. I live in this world. It's really cool. I want everyone else to live in it too. So, yeah, thank you for letting me rant at you for half an hour. I'm happy to take questions for for the rest of the time we're here. I got Ching and Clark on my team who are gonna help out with questions. Alright. I hear your keyboard, I think. No voice. I think you may need to go on stage, Clark. Or, Ching, if you have any questions, Sandy, we can dive in. Yeah. I can, start off... Start us off. There was, one question, that I'm seeing here. What was the reason that bots had conflicting approaches to code review? It seems that code review would be a logical process, and one with kind of less conflict, for different bots. Yeah. It had to do with the, the different lenses that different folks cared about. So some individuals wanted to be very highly prescriptive around how, how an interface is designed or about how, we implement things in a in a particular bit of our code base, and some were less prescriptive. Really, like, I think we could have harmonized all of these bots, together. And to some extent, we did. We took... We kind of looked at how each of them were designed when we created our our main code review agent. But the the issue was that the bots didn't have, like, a shared context and a shared goal. So when one bot saw the code change not exactly according to what it thought should be changed, it would get angry and and find other, things to complain about. So, you can kind of have all the same lenses and areas of observation for code review encoded in a single bot that can kind of internally deduplicate and, defrag, if you will, its notion of findings. But having a bunch of separate adversarial bots is kind of the, too many coats in the kitchen situation where the noise drowned out the the signal. Yeah. And I'll add a little bit onto that myself. I think doing these personally, it's not very well calibrated. It really leads to, like, a lot of tribal knowledge and kind of, like, you know, a single point of failure where a lot of these should be kind of put into, like, a view MD, like you said. So the code review itself can kinda steer it so that people are kind of being able to spend their time and, and their brains kind of at a higher level thinking about what should be done rather than kind of commenting and looking for the same things over and over. Okay. Let me, find another question. And just to introduce Jing, he's the lead engineer on cloud code review. Like, I'm just the PM. Jing is actually, like, the real guy behind it, so, happy to have him join us here. Hey. Good. Hey, Steve. Sorry. Technical difficulties, but now I'm on with you. Yeah. Let's let's keep going. I'll jump back in, for the q and a. So I I think this was an interesting question, and Morgan touched on this some, but how did Anthropic itself roll out, you know, code review and the strategy across the team? Can you guys speak to some of that? Yeah. So I I think everyone with anthropic is very data driven. So once we got to the point where we could, like, run these evals against real life previous incidents and show individual teams within anthropic, Hey. Like, here's the efficacy that we're seeing. We think it's pretty good. I can actually, slice out the incidents that are relevant to to your team or your part of the code base and show you the results there. Folks build confidence relatively quickly. I think also a lot of folks got kind of sick of maintaining their own code review bots because that's like a thankless job that that, helps you and helps your team, but it's not like something that's like most people's day to day work. So so, Ching, tell me if you had another perception. But I... My take was everyone was, like, more than happy to just hand this over and be, like, thank god this is someone else's problem now. If there's a problem with code review, I can go tell them, and they'll fix it. I don't have to, like, get distracted from my day to day work. Yeah. Teams are super happy to have this kind of centralized, for us to kind of own this piece of their SDLC pipeline. Obviously, everyone has their own guardrail metrics from things like how many reverts and fixes they needed to get in, by kind of handing off the reins, the amount of incidents they had, whether reliability went down. And then the huge plus side was that, like, everyone's productivity went up. Great. I see a variety of questions, along the lines of, you know, variance between repos, fine tuning, any any insight you guys can provide on how we approach that and how, how all developers building with cloud code can can approach that? Yeah. So tuning your your review MD for a particular repo, again, this is where evals can be very helpful. You can run back tests on different versions of review MD, once you have a... An eval dataset that you're happy with and see how performance is. That's also, like, kind of heavy though. Like, I'm also a big believer in just, like, continuous improvement and incremental improvement. So if you have humans, you know, occasionally spot check PRs and drop a comment on GitHub when they disagree with the review bot, that is, like, gold. Like, literal, like, those are very, very useful pieces of feedback. Have developers do that once in a while if they really do disagree with the PR bot. And then you can go mine all those findings and have cloud update the review MD to to not make whatever mistakes were identified in in, next time. Yeah. I know some some folks are concerned, like, is the review, like, at its baseline, better than a human review? And there's, like, a really easy way to check that is you kind of, you know, run the code review against, like, that point in time. Basically, eval it against your previous review. So let's say, a human commented three things on there. Like, does code review find those things or more important things? And of the ones that maybe it didn't find, like, are those important and know, are... Or they maybe just knits or things that were just like, oh, take a look at this, but it end up being nothing. Great. I'm seeing... So there were also a a good variety of questions around context. You know, code review is not just purely looking at the diff. Can you guys speak to how code review, manages context across repos to ensure that it has has what it needs to to be most effective? Yeah. So as with many things with agents, progressive disclosure is incredibly useful. You want agents to be able to find the information that they need when they need it without overloading their context from the outset with stuff that may not be relevant. So by default, the code review agent first looks at the diff and only the diff, and then it has the ability if it thinks it needs to to zoom out and look at the full contents of the repo to understand the broader context in which this diff exists. For us, a lot of our stuff is done in sort of a monorepo, which means that the agent can go discover lots of things that, in terms of dependencies and interrelationships that may affect what's, being changed in the diff. Some customers have this stuff spread across multiple repos. Currently, code review is scoped, like, on a permissions and access basis to to a single repo. We're in some conversations with privacy to figure out ways that we can, allow code review access to multiple repos if it needs to. We're still working that out. If you have a mono repo, it's not gonna be a problem. If you have multiple repos, we'll get there. Excellent. Another question sort of on quality and how we approach ensuring customers get really good code review. Are you verifying intent of the change at the code review stage? What does the agent compare the code against in terms of what a good feature change looks like? Can you guys speak to that? Yeah. I can speak a little bit to that. So I think there's, like, different, three different levels of kind of, things that, we should think about doing a code review. So there's, like, the very low level, like, nits and, like, code quality and, you know, assuming that the intent is right, do, does it exactly kind of follow the pattern of clean code that you should go in? Code review is good at, like, the second level, which is a little bit higher, which is assuming the intent is correct, does it find the, does it find and report things that would block it, that would cause it to be wrong? So is it correct as written? And then what we kind of wanna shift left and back to humans is is is the intent... Is this feature something that we should be making? Is this change kind of in the right direction? So code review doesn't assume that it should be telling you what to build, but it should be telling you that if you're building this, is that correct? Is it safe? We try to make it not, like, overly paternalistic where it's gonna, like, block you from doing something that you tell it to do because that's really annoying. So it's still kind of up to you to make sure that someone isn't maliciously, like, putting dumb stuff into your repo. I would hope that everyone has good intents if they if they have access to your repo. Excellent. I'm seeing some other questions along the lines of context quality. Is there too much context? How much context is too much? And how do you deal with this, when a code, code review agent is running? Any thoughts there, guys? Yeah. I'll keep it, like, super simple is that, models these days are really intelligent, and we don't give them enough credit. We've tried so many experiments of giving, the code review access to more tools, to more context, like things like other repos, letting it actually run, some changes instead of being kind of more read only. And a lot of it was actually not necessary and didn't improve it at all. It added cost at added time. It added kind of, complexity and we had to make some, like, safety considerations when we tried those. But these days, a lot of what you can find, is just static. This is the exciting and annoying part of working on cloud code is we we take the model capabilities and we figure out where the edges of that are. And then we go build a product with some scaffolding and some additional prompting and harness, things to make the product to do things that you cannot just single shot with the prompt. And then the RL team takes all of that and they train the next round of models and all of a sudden you don't need that extra scaffolding and you don't need that extra harness anymore. And we need to go find a new problem to solve. The way I describe it working here is kinda like we build sand castles. We build these beautiful things that we're very proud of, and then the tide of model advancement comes in and washes the sand castle away. And you have to move, offshore a little bit and build your next sand castle. So it's, you know, it's fun and exciting in one way. But on the other hand, I wouldn't be surprised if code review was just a totally solved problem in in the near future, and me and Ching are gonna be working on something different to con... Continue to accelerate the the SDLC for everyone. Excellent. I think I think on that note, if you guys have any other final thoughts or questions you spotted in the in the, channel, otherwise, I'll I'll pass it back to Morgan, to kinda wrap this up. Yeah. Any other last ones you wanna grab, Ching, that you've seen come across here? Yeah. I'm taking a look. I know a few people here actually asked about, like, what model and what effort we think is the best for code review. And to that, I say, like, just following up on this previous answer, that will change very quickly as months go by. And our team is here to kind of continue to evolve, continue to fine tune, the hosted product and any skills and products that we offer so that they take advantage of the most recent generation of models that's, more cost efficient, faster, and and, higher quality and just make sure we, pass those along. Yeah. Keep... Keeping track of the current, like, meta for intelligence per dollar is so annoying even for me, and I work at Anthropic. Like, the the meta used to be use stable low for everything because even though it's a bit more expensive, it's gonna be smarter and it's just gonna get to a solution faster and just actually incur less cost in terms of tokens. That Opus five five dropped and the meta is use Opus five five medium for pretty much everything and that's gonna give you the best intelligence per dollar trade off. Every time we drop a new model, this changes. At the very least on our side, we run gratuitous evals every time there's a new model and so we'll tune, you know, what do the finders use, what do the verifiers use, what effort levels, what... You know, we we run a zillion experiments and then make the low, medium, high defaults on code review. The Pareto optimal relative to to what we see in our experiments. So, that's part of the value that we we try to bring is you don't need to, like, be doing the guesswork on this. Like, we kinda continually keep this pegged to a certain level of performance. We're not gonna let that drop. We will let the cost drop. But as the models advance, the the thing that I've seen is that intelligence per dollar continues to decrease. Anything else we should jump on here, or should we should we wrap up? Just taking a final look through. Lot of good questions, so thanks everyone for submitting. I think we've covered most of the common themes. Cool. Well, thanks everyone for coming. I appreciate your your time. If you couldn't tell, this is something that I'm super jazzed about. It's been a really fun journey trying to solve code review for Anthropic, and we feel like we've largely gotten there. And we did this poll earlier and I forgot to mention the results, but looking at them now, 77% of people say that, review is the most painful part of your SDLC. Totally did not bias this by, asking this question in a session called how cloud code does code review. But, like, yeah. It's real. I mean, this was our life six months ago. This Kabuki theater phase of, like, fake stamps and half looking at PRs. And it was broken and it was stupid. And I'm here to tell you, like, it doesn't have to be broken and stupid. Like, we we dug out of that hole. Encourage all of you to give our stuff a try. We think it works really well. And if it doesn't work really well, let us know, because that's really good data, and we would like to make it work better. So thanks again for your time. Have a good day, everyone.