Name: Long Running Agents: How Outtake built a cyber investigator on Claude
Uploaded: 2026-04-28T18:17:20.926Z
Duration: 47 min 32 s
Description: Long Running Agents: How Outtake built a cyber investigator on Claude

Transcript for "Long Running Agents: How Outtake built a cyber investigator on Claude": Alright. Hi, everyone. I'm Jake from the go to market team at Anthropic. We'll go ahead and get started here. First thing, thank you so much to everyone for joining today, for registering, for being here. I think this this is one of the most important topics that we can be talking about in AI right now, which is long running agents. And today, I'll be joined by the Outtake team where we'll be talking about how they built a long running agent that is a cyber investigator built on Claude. So before we jump in, let's go through the agenda a little bit so everyone knows what to expect. I will introduce you to the speakers. The outtake team will tell us a little bit about the new attack surface that, is being seen on the Internet today. Then the recon agent that they recently built, alongside Anthropic, as well as some of the learnings from that, and we'll have folks from our plat AI team joining as well. And then a q and a with the team going into a little bit more detail about some of the learnings, some of the mistakes, some of the successes that they found, in the in the early prototypes of this tool. So I wanna quickly share with everyone just a little bit of housekeeping. There'll be a recording of the session distributed within twenty four hours. You guys can submit questions through the Q&A. We wanna have some time at the end to go through those, so please put interesting stuff into the chat that you wanna know. And then same thing on the feedback side. We wanna make sure that we can do more of these, and that they're really interesting topics to you. So please give us some feedback when that survey gets sent out at the end. Alrighty. The exciting part. I wanna introduce you to the folks that are gonna be on the call. So I'm Jake Richards. I lead our cybersecurity work with, startups at Anthropic. So I help teams like Outtake build long running agents on on the API, as well as focusing on internal use cases. So making employees work smarter and faster through cloud code and cloud enterprise. I'm joined by Alex Dillon, the CEO and founder of Outtake. Alex spent five years at Palantir working on their moonshot team and reported directly to the CTO of Palantir. And something that he really learned in working through highly regulated industries is that a lot of these issues that were being seen on the Internet are very manual challenging and and time, intensive. And that's what led him to actually founding Outtake in 2023. Outtake has raised $60,000,000 in funding from a lot of notable investors, like Iconic, CRV, angels like Satya Nadella, Nikesh Arora, and Bill Ackman. And this is all to protect, you know, the leading AI labs, major hedge funds, as well as even some US federal agencies. So it's a real privilege to have Alex on the call today. Then Justin Young is, my right hand man that I work with a lot, in these, in these situations like Outtake. So Justin is on our applied AI team effectively sitting between research and go to market. So he'll come in, look at folks, agentic architecture, make these recommendations, and he's really a wealth of knowledge because he's seen so much across so many scale deployments, and and his patent recognition for this is really high. Before joining Anthropic, he was at Abnormal AI where he led a lot of their, threat detection and ML teamwork. And then Jack Hayford, is joining us from the Outtake team as well. He is the lead for the agent platform. Before Outtake, he was the lead engineer at Cortex, which is an internal developer tool that's used by companies like Spotify, Docker, and Notion. And today, he's the person who's actually leading in stress testing this recon agent that you'll see today in the session. And I just wanna quickly give a shout out to Jack. He's one of the most proficient builders that we've seen this year on the agent SDK as well as, has really pushed the limits of cloud code, and and that led to a lot of this work that we'll see today. So I wanna run through the journey a little bit of of how Outtake kind of worked within the cloud ecosystem and certainly how we're thinking about it as well. So firstly, starting at the base, we're the only, major model provider that's available on all three hyperscalers as well as having a native API. So we allow you to meet your customers where they are, your data requirements, etcetera. And then you wanna start thinking about, well, what's the model that I select on top of that and for which use case? So Claude Opus, obviously, our most intelligent model built really for long running agents, and situations where benchmarking and accuracy matters most. Claude SONNET is a well cost model, very good for production grade code, very good for RAG, and a couple of other use cases. And then Haiku is I like to think of this where, like, speed and cost really matters to you. So real time UX classification, summarization, things like that. The next layer above this, the platform, I personally think that this is becoming the most increasingly important part of our thinking engine, which is all of the primitives, the plumbing, the infrastructure that we're building so that you can all effectively go create these agents really quickly and spin out to create your new, you know, product feature, your next thing that drives revenue. So how out Outtake kind of came into this with the outcomes that we see is we noticed that the Outtake team was kind of hitting on this, like, smarter employees, one at the top left. So they were using Cloud Code very extensively internally. I've met with their GTM team a lot as well, and and they're very good at using, Cowork. So there was a bit of a culture kind of brewing internally of, well, how can we work better with these agents and particularly something that's like an agent economist like called code. And then that allowed them to actually speed up these faster processes. And we often see teams building internal Slack agents and tools like this, and that leads to transformative products. If you really understand the model, if you understand the infrastructure and the platforms that you're working with, then it gives you those ideas to go out and spin out this this next generation of the product. And that's what we're gonna talk about today is that journey with outtake. Last component here. I know that there's like a lot of news around, you know, anthropic and cybersecurity right now, and we're putting a lot of focus into this. So I wanna talk a little bit about our mission here. We believe that SafeAI is a security advantage. We would really like everyone in the industry to take this long term view of what does actually securing the ecosystem look like and allow us to do to walk towards something like machines of love and grace, like, really good outcomes. And the way that we're focusing on this in Anthropic is through battle tested intelligence. So we have one of the world's leading red teams that are constantly doing real well detection and vulnerability discovery. So recently, we discovered 500, you know, critical zero days, in a lot of open source projects and tools that had gone, you know, survived for decades. It's built by defenders. So we're actually using Claude in our own detection, investigation, and remediation walk. So everything that we learn about securing, you know, one of the world's leading labs is translating into our products, our primitives, our platform, etcetera. And then lastly, long horizon reliability. Claude is extremely good at planning, at orchestrating, at using tools, and each model generation that we have shipped. We have gotten better with its its ability to be steered and kind of stay within the guardrails that are necessary to be safe. So this is a little bit about how we're thinking about it. I'm really excited to bring Alex up on stage because he has some really unique, insights into what his team have been seeing, on the Internet and the new threat landscape. So, Alex, come join me. Hey, Jake. Thanks thanks everyone for having me. Super excited to be here. Where I wanna begin is with a very core insight, which is, we're on the cusp of an incredibly low trust Internet. And when we think about this from an attacker's perspective, a lot of modern tech is potentially being misused in ways that empower bad actors. Right? So everything that we see as a web app builder is equally something that someone can use to clone a website and build a login portal. What we see as Agentic go to market tooling is equally used for Agentic phishing. Things that we see as fantastic video generation or image generation models turn into core identity spoofers. And so if you put on the bad actor's hat, it's actually a pretty good time, to to be running attacks. And so these have, unfortunately, very real world consequences. We've seen the amount of attacks happening across the Fortune 500 to governments actually go exponential over the last three years. The number of phishing attacks back in 2024 were already at at about 65%. In 2026, the estimates are closer to 90 to 95%. The average attack is now costing on the order of 6 to $8,000,000 because once you're inside as an attacker, the amount of things you can access, thanks to AI, is significantly higher. And there's a very clear sequence, in terms of how bad actors, approach executing an attack. And so step one is maybe perhaps unintuitive. You actually start from the outside. You weaponize public data. You start with understanding what PII leaks have occurred, what you can understand about the organization, without actually being inside yet. You then take this data and you build impersonations. You say, how can I build a fake website, fake application, ads, profiles, etcetera? And then you use these as lures to sort of get inside and ultimately exploit systems, often the crown jewels inside the perimeter for any, cybersecurity organization. And so our stance at Outtake is we're kind of on the cusp of a digital trust crisis. AI is forging entities at scale. We're seeing, you know, everything from websites all the way to modern exotic impersonations like impersonating agents. And traditional tooling isn't necessarily prepared to handle this. Traditional cyber tools focus on one chunk of that trust attack at a time. You know, if you try to figure out what public data might be used against you, you might use threat intel tools, OSINT tools, PII tools. If you try to figure out how to get rid of impersonations, you might work on some sort of brand protection or digital risk protection tool. And then third and finally, if you're trying to look at how to, you know, protect your internal systems from compromise, you're looking at traditional cybersecurity tools like network cloud and endpoint. But it's not how attackers think. Attackers are trying to go through steps one through three, not pick off one vendor at a time. And so our theory at Outtake is that you need to connect these workflows into one single cohesive defense. And so what we've built at Outtake really starts from the very top. We say for any organization, commercial or government, you wanna first understand everything possible about you externally because after all, those are exactly the things that are gonna be used against you. And so what that means is indexing every mention of your organization, your executives, your employees, and your products across the public Internet that includes arbitrary websites, social media, the dark web, and more. Once you've indexed that, you've got an incredible data asset upon which you can build Agensic work security workflows. Now there's a ton of different workflows you can build, but today, we're gonna be focusing on one that focuses on the removal of impersonations. And so you wanna detect a specific impersonation, but you don't wanna stop there. You wanna be able to say, okay. I found something harmful. What else is it connected to? Is this part of a broader adversarial graph? And once I've discovered that graph, what do I do about it? How do I automate the actual removals the and the protections for my organization? And so it was in going through these exact workflows that Outtake as a company was inspired to build the reconnaissance agent. See, in order to quickly map out these graphs and then figure out what to do next, we realized we needed to be moving at machine speed, and thus, the recon agent was born. And so what we're gonna be walking through today and what Jack Hayford will be demoing for us is a very real example, anonymized from, one of the one of the that we've been working with. And we'll be focusing on a financial institution where we've seen not just one specific attack, but how that attack connects to a number of other impersonations, and then the the power of the reconnaissance agent for actually improving cybersecurity for that specific organization. And with that, I'm gonna pass over to Jack. Thanks, Alex. Great to see everyone. My name is Jack. I am an engineer here at Outtake. Excited to kind of present, the recon agent, which Alex has kind of described. And we're going to for this case, we're gonna drop away from the slide deck and hop into a real example. So let me just go ahead and pull up my other screen here. And that should be up now. And so in this case, again, this is a real investigation conducted by the Recon agent recently. For the purposes of this presentation, we fully anonymized the details of the investigation. But let's step in the shoes of Acme Bank, a very real, legitimate company that we actually made up for the purposes of this investigation. See, at Acme Bank, we are a, financial firm, very trusted. You know, we've kind of built out, here's our official website, acmebank.com. And, you know, we have built out this brand identity, this website. People trust us, to handle their money. And, you know, we have kind of a a large amount of things that we wanna protect here. But the case that we're gonna walk through for this particular investigation is this website, which is not Acme Bank. It looks darn near close. I mean, the the things that you can do with AI in terms of being able to create these photo replicated kind of websites, they're using those same brandings that inspire trust in our users, but we're actually looking at acmebank.onl. And so this is an impersonation website. You know, we might have, you know, victims that are reaching out to us saying, hey, I arrived at this website thinking it was you. I got an SMS text and I put in my credentials and I logged in only to find out it wasn't really, you know, the real deal. And what we wanna dive into is from here, as Acme Bank, I care about stopping this. I wanna stop this website. I wanna stop further users and partners and and, you know, other trusted members of Acme Bank to go into this and being misled. And this is exactly the most important moment, not just for stopping this, but actually for triggering what the recon agent does, which is really investigating this actual domain, to to identify what is the most important thing that we can do next. So here we are actually in the Antique platform. We're looking at the recon agent investigation of that original kind of impersonation domain. And I think that the most important thing to hop over into is really quickly the, graph view of what we're about to uncover with this investigation. So here is our investigation playback is going to represent the graph of what we understand. And here we are starting out with that original domain, that website of Acme Bank ONL. It's the same exact one from before, the the near replica that other people were being led to. So traditional tools would take this domain and they say, well, this is malicious. It is impersonating us. And they would try to take it down to stop other people from getting there. This is actually the opportunity for us to do a lot more than that with with the Outtake recon agent. Because if we just take this down, there are still questions that haven't been answered. You know, the simplest example would be, hey. You know, if if this is phishing or and, you know, people were logging in with their credentials, where are those credentials going? Is it going to somewhere else? You know, are there any other attack patterns that this particular operator is actually leveraging against the victims? Are they trying to, you know, get money from them, additional information? And then the more important one is, is this actually an isolated incident or is this part of a much bigger scheme? Is this the first time that this operator has created an impersonation of my company or is this just another instance along a long running campaign? And so what we can do here really quickly is actually allow this to play out, and we're gonna skip right to the end and show you that this is not an isolated incident. Your kind of agent has found a large campaign here, and this is actually where we're going to go back and say, how did we get there? Okay? So the first thing that we can do, and we'll kind of slow down the playback here, and we can play it out. And you can see that the recon agent is starting the investigate. You know, we've taken that initial domain. We're starting to classify it, gather evidence, and already we're identifying further leads, connected infrastructure, and other parts of this operation. As you watch it play out, you'll see the agents starting to take on those additional nodes and investigate those as well. And we're gonna give a a bit of a second here and then actually, we'll kind of probably try to pause and maybe take a peek at, you know, what we found here. So let's hop in here and we will look at so we're actually seeing up at the top, there's actually a Google search ad that's actually connected to this domain. This is a impersonation of the company as well, this fake advertisement. And what it's gonna try to do is it's gonna lead victims to this fake impersonating site. So now they've not only searched Acme, they've seen this ad, and now they're going to follow it directly into the clutches of the of the, the threat actor. So now we've actually understand, actually, how are they distributing this? How are they getting victims to their site? We can also see over here, there's another node where it's a a Telegram account for direct transfers. And here's one, support team Asia. And this is actually a common pattern that we see where Telegram accounts are used to kind of impersonate support channels, to not only further kind of add credibility to the original site, but also to, offer them an additional avenue for pulling important credentials and information and misleading the victim. Here we also have another node here, which is actually a Facebook profile, and that's actually an impersonation of an executive at Acme Bank, and that's also leading traffic to this original domain. So you can see that this is a much more sophisticated campaign. This adversarial network that's been built up by this actor has a lot more nuance to it. You know, there's multiple areas where it's adding more credibility, trying to attack the victims, and then many other things that are leading them in. And we'll just kind of let this play out and maybe draw it all the way to the finish here in terms of identifying the entire breadth of what's been captured to understand that this is quite a large, campaign and adversarial network. And this is in fact, maybe a little hard for me even to understand at this point, the true kind of extent of what the Recon agent has been able to uncover. So one of the things that we quickly then started building out for our customers is, what is a place where we can find all that intelligence in a more easily digestible place? And so that's where we have our report. So the report is kind of taking all of that those findings, the evidence that's been acquired throughout this investigation, and it's putting into a much cleaner, kind of more parsable, place to to identify what are the true takeaways, what what are the information that we know about the actor. And this is kind of my favorite. This is the investigation timeline. And this not only tells you, you know, okay, how did the investigation go, but also from what we uncovered, we can actually tell you the timeline that the operator, the actor, actor, the threat actor behind this took as they were setting these things up. You know, and already we can see this thing started at March 2025. So this is not a brand new campaign. This is something that's been going for a year. We talk about kind of identifying if it's an isolated case. This has been going on for a while. And one of the things that the evidence that drives these reports allows us to do is not only, you know, create, the answers to those questions, but the other thing is, the more important part is doing something about it. And that comes into kind of more of the remediation. And so with that evidence, we're able to package it up and move over to identifying all of the alerts, all of those, you know, low trust impersonating nodes within that entire graph. And we're actually able to say instead of just taking down that original website that was misleading people, we can actually take down the whole lot. You know, we've identified the shared infrastructure and we have the evidence to actually rip it out once and for all and really set their operation back. So zooming back out, here we are at our overview. You know, we have the summary of the the high level investigation findings. You know, we found those alerts. The other thing is that, you know, once we've taken it down, we can actually also investigate further. There are more opportunities to go deeper. Maybe finding out the operator more information about the operator and the agent's able to kinda come up with these leads and suggest them. We can take them. We can also guide it if we wanna really actually focus on more of those Google Ads. And then the final thing that we're able to do with that firm further investigation with all this information that we uncover is finally, how do we make our entire system smarter? How do we protect kind of what we do next? How do we protect against this particular actor? We can take these things and actually turn those into real improvements on the platform and outtake. So all the outtake's agents know, hey, let's watch out for this IP address if there's more activity so we can see if they are continuing to build. And then also this information is even shareable with, you know, kind of internal network protection. So maybe I can share with my security team at Acme Bank. I could say, hey, don't let anything come from this IP address. The recon agent was able to find that during an investigation. We know that that is part of attacker infrastructure. So that's essentially kind of a walkthrough of a real life, case that the recon agent investigated. And, you know, that's why it's very important to not just stop at the takedown, but actually to dive deep with the recon agent to find the entire adversarial network, to remediate the entire thing, and to bring all that intelligence back into a compounding defense solution. So I'll go ahead and stop there. Hey, Justin. Alright. Hey, everyone. Thanks so much, Jack, for that demo. I'm Justin from our team at Anthropic. I love that demo. I've worked in cybersecurity for a number of years, and I remember, first time I saw an early version of that, I just had a big smile on my face because it's, what I love about that demo is it's, like, very visual and clear, the complexity that the agent is working through. And so now I just wanna ask Jack, as one of the builders of this system, some questions about, like, what actually went into making this work. So just starting with, like, domain expertise. Jack, what does it mean to become a domain expert in the problem that you're solving when building an agent? And, like, how do you think about the importance of that? Yeah. So, I mean, you know, talking from the experience with the recon agent, the most important thing, that I believe about building long running agents is that you really have to understand what does good look like, what is the agent supposed to be doing. I often joke that you have to be the agent before you can build the agent. And what that meant in the case of the recon agent is that we were, hopping into these real life situations, these real investigations and doing them actually ourselves. We're pulling on domain expertise from also kind of our customers and design partners to really understand what does a really strong investigation look like. You know, what what's necessary to always drive to the the proper solution to to kind of organize this evidence. And so I guess that's why I think one of the big things is, like, you know, becoming the expert first, be the agent, and really understand what good looks like because you're gonna see that for the rest of this process, we're gonna just return to what is good like like and ensuring that the agent can do that every single time. Yeah. I I totally agree, and I love that framing. I feel like it demystifies some of building agents because it's actually much more similar to building other kinds of software than it might seem. Like, you have to get a prototype almost like a bullet tracer working before you can expand and generalize, but a lot of the same, like, iterative building principles still apply. So with that in mind, I think that's a good segue to, the the second part here, which is, I wanted to ask, like, you were already experimenting with other agent frameworks, and then we're we're using cloud code. Could you talk me through, like, that iterative process in cloud code and, like, how you landed on that framework as a good fit for this use case? Yeah. Totally. So, obviously, we we kinda started off by figuring out how to investigate these ourselves, and then we started slowly. As you say, kind of like same thing with traditional software development. Find the simple thing first and only scale the complexity when it makes sense to. So we were starting to automate more and more pieces of those investigations, and we were using traditional, agent tools at the time, agent frameworks to kinda set that up. But what we quickly realized is that in order for this, recon agent to be able to truly succeed at every investigation, this workload was not only just the the workload of an investigator. It also needed to be half coding agent. We we understood very quickly that every single investigation is so different and it's deeply technical. And we knew that the agent that succeeds at this every time needs to not only have a really strong muscle of handling code, reading it, analyzing it, but it also should be able to write code. It should be able to run code. It should be able to create tools to be able to bridge the gap for the next step of the investigation. It should be able to actually interact with these malicious domains, these these login pages, and actually, you know, try a login and find out where that goes. So we quickly knew that it needed that muscle and that capability and it was and and we also kind of, you know, are, have worked with Cloud Code a lot and definitely found that it was a strong initial harness to actually, you know, validate those assumptions, to start experimenting more and more. And we we saw a lot of, early success with those experiments that only kind of further prove that that hypothesis. Yeah. It it totally makes sense. I I I think there's something interesting here, which is there is an obvious tension between, like, giving Claude these tools, like, building these muscles, making the agent more effective, but then also, you know, keeping it constrained, giving it certain guardrails, limiting its autonomy, to to either make it, like, more predictable or more secure. Like, can you talk through how you thought about that process of iteratively giving the model more autonomy as you built it? Yeah. I think I think this is the the really tricky part. You know, you start off with these really deterministic workflows in in normal software, you know, if x then y. And when you go to agent world, everything is totally flipped on its head and and it's not deterministic. And we like that because that means that it can give us some really great outputs in the right situations. But if we don't really confine that, then, you know, there's a really good chance that, hey, instead of that thirty minute investigation that should have kind of gotten us all the way to this final, point b, you know, we went right off the rails and it, you know, didn't even save the evidence and, you know, it has totally kind of lost the the, track of what what meant a good investigation. So as these things kind of start becoming more long running, it's very important for us to kind of create constraints at the top level, the orchestration, but not allow those constraints to get really close into the low level. So, like, a quick example is let's make sure that whenever we're looking at a domain, we're always doing x, y, z. And, you know, we actually really liked to do items and task items, those kind of tools as those kind of smaller term, kind of controls. But then when it got into really identifying what is the most important piece of evidence or what are the leads here on, you know, this particular node or website, we want it to then open up and be able to improv. Well, based off of what I'm seeing, the most important next step is x. And so it's really about creating what is the least amount of framing that always gives you the deterministic results, but create space for the agent to to do that improv, and that's usually where you get the really good results. Yeah. It it totally makes sense. And I can almost picture, you know, your, in cloud code getting this to work and then doing more higher level testing. Like, at what moment do you think you had confidence that, you know, you are ready to graduate to, like, the agent SDK running this in headless mode where it's just completely autonomous, and then even, like, a step beyond that where you're ready to put it in front of a customer. Yeah. So, I mean, I think as soon as we saw the the early success, on those experiments going in cloud code and and kind of seeing that, you know, in those, isolated cases, it it it's very successful. But we very quickly saw that, you know, for larger investigations, for kind of more intense requirements, you know, it did go off the rails quite a lot. And what we needed was better control, access to kind of the lower level primitives, but we also weren't trying to build those ourselves. So we knew that we wanted kind of a framework. We also knew that we liked the patterns that, Cloud Code had introduced and with, you know, skills and tools. And so using kind of a harness like the, a harness framework like the agent SDK was a really natural fit for us, to kind of ensure that we didn't drop any velocity, and to also not focus on rebuilding the wheel in terms of agent loop and handling sessions. We want to focus more on, like, how are we handling memory over time and context and the file system, that entire thing where we just talked about in terms of how we keep this thing on track, through our Nuance kind of use cases. And so that's when we reached for the agent SDK, and we felt like it really gave us the right level of abstraction without getting in our way. And, again, you know, it's nice when you can just kinda pull over your skills and not have to rewrite all those. And so, yeah, we we we saw a lot of kind of benefits to moving to the agent SDK and gaining that control. Yeah. Totally makes sense. And I I wanna get even more specific here so this can just, you know, possibly be a pretty short, like, concrete answer. But I'd love to hear just in detail, like, what your iteration loop looks like when you were, you know, doing this development. Yeah. I mean, it definitely varies a lot based off of which sliver of time you're looking at. You know, I'll I'll I'll take the two ends of the spectrum. At the very beginning when we were doing those experiments, you know, on cloud code and, you know, identifying when it went off the rails, it forgot to save some evidence or it just leaked to this crazy conclusion on, you know, who the actor behind this entire campaign was. We had already built out a very early on process, like a a a command for doing this whole retro of basically identifying what went well, what didn't, and then also insert what, I as the developer thought. And we had a very kind of repeatable method there that we could actually slowly graduate over time. And what it does is it allows me to step back from the reading transcripts of thirty minute runs, which is just I couldn't do it. Like, you you very much tell. I think, like, Kapiti says all this all the time and and a lot of your research as well that, you know, we have to pull ourselves out of the loop. We are the bottleneck. And when you build these long complex agents, it's very important to be able to rise up. So now the the feedback loop, to answer your question, is a lot more of me saying, hey. We had a case where, you know, one of our agents reported it really would have been able to do a better investigation if it had an access to a new tool. And my coding agent can take a look at all those suggestions, create the tool, and actually create a scenario to test it out. And I'm actually just able to look at, hey, what was the result? You know, after the agent looked at that, how did it do? What did it do well? Did that tool actually give us better results? So it's a lot more automated and it's a lot faster, and it's also a lot more satisfying as a developer. I I totally agree. That that's been my experience too. And I I think everything, you just said is clear in, like, the the demo that we just saw, which is, again, just super cool. So thanks for going through that q and a, Jack. I'm gonna move on now to zooming out a little bit and talking about some more generalizable lessons for building long running agents. So the the main point that I want to make here is that I think the thing to focus on is changing very quickly. It used to be, you know, building workflows, putting these training wheels on the model. I think what we're seeing now is that the core agent loop, something like the agent SDK, is converging to something very simple. It is basically just a loop that calls tools and then, calls out to the anthropic API to do inference. What becomes really powerful, what enables you to build a powerful agent now is everything that you empower the model with. Its memory, its tools, its skills, it's everything that lets the agent do his job, and you just get out of the way. So with that, gonna pass it back to Jack to to talk about, some of his learnings from, building this. Yeah. I mean, it's it was definitely hard to to pick, from the whole list. There are many learnings, both painful and and not so painful. And we arrived at kind of a lot of that that framework that Justin pointed out. But so to point out a couple of other big learnings, that number one is like, file system and bash and giving those kind of extremely powerful open ended tools and capabilities to an agent, is a huge step change. You know, there are just a whole cast of issues, not so dissimilar from the the kind of, problems that just get solved by a better model. There are so many problems that just get solved by instead of only giving it, you know, an agent these very nuanced specific tools, If I give it a file system and I give it the ability to write and read and run code, then it is able to kind of get around a lot of obstacles. We have plenty of cases where we saw the agent have, you know, a tool that was failing due to some kind of network hiccup or something, And, you know, it would just find the workarounds. And because the rest of the the harness that we had built was strong enough and it had the opportunity for improv and these tools, it was still able to get to a successful result. I think the second thing to really point out is when when you're building these long running agents, like prompts are suggestions. A natural cadence when you're building these agents that get complicated over time is you run it, you find a case where, you know, it didn't do what you wanted, it didn't do what good looked like, And then the natural response is to kind of add it to the most, classic part of the agent, which is the prompts. Like, I might just slip something into the system prompt where when this happens, make sure you do blah. And maybe that works initially, but as you add complexity as this agent runs longer, every single word in that prompt eventually will probably be ignored. And you need to really ensure that you build around that and identify, you know, what are the core frameworks or the core bones of this agent that I should always do every time and pull that out of the prompt and put that into the harness, into the guardrails around the agent so the agent doesn't have to think about it anymore. Because that means that it's just yet more context space and attention that the agent can put towards the area where it can really thrive. Third, evals are for speed too. You know, I think that some, some, engineers kind of feel apprehensive about building evals because it's like this idea of building a perfect case and that they are for reliability. You know, you build evals so that you ensure that the agent is always doing what you kind of expect. But we built some version of evals in from the very beginning. You know, we built out that process for kind of being able to evaluate how the agent did. That becomes the most expensive part of the loop, I I think especially today with software development. And so it's even more important that when you're working on top of an agent and the output is, you know, a transcript of thirty minutes, that process for evaluating that output, if it was good, needs to be as fast and as automatable as possible such that you can also put an agent there to read that results rather than human. So evals will make you build that agent faster regardless of how, you know, official or perfect they are. And finally, security. You know, I think this is a big no for us for building the recon agent. We gave it a file system and bash and we're sending it to adversarial environments. So the most important thing that we had to do was also solve the problem of building essentially this blast box where, you know, you could try to hide your agent from the Internet with sandboxing, but we knew that that would actually hinder it. And so for our case, security meant building out a very specific and sophisticated system for ensuring that even if the agent is compromised, we aren't and it can't escape in any way. Not all agents are able to do that. So, you know, there's a lot of other tactics for sandboxing. In fact, we're even kind of building out something at Outtape where we're trying to kind of instill that intelligence at the area where agents are reaching out to the Internet so you can kind of, add in the context of, is this an impersonation? Is it kind of, malware? Is it even maybe trying to prompt inject, called Agent Armour? But I think that that could be probably another podcast or a webinar on its own. Cool. Thanks so much for that, Jack and Justin. So fun, like, reliving a lot of the work as well, and just, like, hearing you kinda dive into that. What I wanna do is turn it back to the audience. There's a lot of good questions, that have come up. So if if Alex and Justin could join, let's let's roll through this together. I think this is gonna be really fun. Okay. Shout outs to Shahan, the most upvoted question, and I think it's such a good one because when we met with the Outtake team, in New York a couple months ago, we we kind of talked about this, like, exactly as a group, and I wanna wanna pass this to Justin and and Jack. So once agents are established, how do you keep them up to date once deployed? I know that we talked a lot as a group about, like, hydration of skills and different data that can be used and things like that. So maybe I'll I'll kick it off with you, Jack. How do you think about this? So the question is kind of how we make sure that these agents, after they've deployed, are up to date. Is that kind of with changing requirements? Are we thinking, like, even, you know, the context of the Internet? Feels like there's two paths I could take. That's it. I think let's take a general approach of the audience, like, building agents. Obviously, like, the data itself and what we see in the landscape will change, but I think importantly, like, how maybe once the models change or the benchmarks even change or, like, the new tools and primitives change, like, how do you think about, like, those new skills and those loops? Yeah. I mean, I think that comes back to what you guys were were hammering home very early into the, the life cycle when we were kind of coordinating together is is the importance of those evals. Because at the point where we're at once we've gone through, building out the system for sitting back from the entire agent harness and everything around it, and being able to stress test it against, you know, a 100 scenarios all simultaneously, it gives you the flexibility to make very large sweeping changes. That sweeping change might be just a model upgrade or maybe you've actually decided that you want to fully reengineer your entire memory system. You know, we've already undergone a handful of major refactors on the Recon agent from those early learnings, because of what we saw that the agent needed to be able to do. And by building out that eval suite around it such that you can test all those scenarios with high confidence, you allow for your agentic coding to hop in and make very large sweeping changes to match whether or not it's a new capability or even if a model brings in a step change and you find that your harness is actually getting in the way of it. Yeah. And the, Justin. here too is that you can try to apply some of the, like, old school software engineering principles, you know, version control, only you know, just don't make changes in production. It's actually not that easy, though, especially if you're working in cybersecurity. The world is changing around you. It might even be changing in an adversarial way where attackers are saying, Outtake has this amazing new agent. I have to change my approach. So there are these second order effects. So I think the way you combat that is just, like, have very tight monitoring, evals like Jack said, and then just be very reactive to pushing out changes, as you can, which, you know, if you have good evals, you're empowering yourself to to move quickly like that. Cool. And then I think that this is a super interesting question for you, Alex. Especially considering that, like, you're selling into the enterprise and regulated industries and, like, you've done so much of this work at Palantir as well. So Andrew has a question. I work in a highly regulated industry. How have you all found confidence in using Claude for cyber threat analysis in regulated spaces that ensures compliance and effectiveness at the same time? Yeah. It's a really powerful question. When it when in cybersecurity, you're you're you're coming in with a pretty strong suggested action. Right? You might be saying, this is a piece of the Internet that we need to get removed. This is an IP that we address we need to block. And so having confidence in your conclusion really matters. The short answer here is you basically want your agent to reason across multiple sources. So, for example, when you're identifying something as fake, you don't just sort of have a one shot reasoning. Rather, what you do is a lot of what the recon agent does where it says, well, this one node is actually connected to let's say, the website's connected to that Telegram group. Well, most websites don't connect to Telegram group. So now I'm actually that that's ticked off a red flag for me. And so you basically you try to tie together a bunch of behavioral threat signals before you come in with the confidence to say that something is actually harmful. This is exactly where you sort of want the best reasoning models, implemented because, yeah, if if you if you sort of try to just, again, one shot it, it's not gonna sort of synthesize the information correctly. So this is the short answer is that trust needs to be earned in regulated industries. It also needs to be built on, thoughtful connections of multiple sources that reasoning models can review. Totally. And on on our side as well, I think, like, the more customers that we speak to, we're also noticing that, like, CEOs and CSOs are paying much, much more attention to things like alignment. And that's just interesting because it's, like, almost they're getting more into the lore of of AI, and making a lot of their determinations for how they sell their end products and purchase products back by by the work that we're doing in Anthropic. Jack, a quick one for you. Couple people, like, reference this is like, how long does this agent run? It really depends on the investigation. You know, I think our our median is around sixteen minutes, in terms of investigation run times, but we we definitely see plenty go up to forty five minutes to an hour. It's actually more of a kind of situation of, a, how much is there to find? You know, if it if it is kind of a smaller, maybe a less serious campaign, you know, we might be able to cover a lot faster. But for anything where we really have uncovered quite a lot, like the one that we showed today, it can very much go much further, forty five minutes to an hour. In fact, we actually stop it. Like, we can it can very much run longer than that. I think the longest clock time is maybe like an hour and thirty, maybe two hours. But even still, we we set, turn limits to be able to come back, return it home, summarize what did you find? You know, keep a list of all the leads. But, from there, we want to kind of actually come back also to users and say, not so just similar from a coding agent, hey, is this what you want? You know, because I could go all the way down and really tell you every single thing that there is to know about this, but it's also important that it's still valuable for for the user. And that's also actually I do appreciate the the turn count kind of budgeting and and limits, made that actually pretty easy to to implement as well. Yeah. And I think, another question that's popped in that is very interesting about, like, your business model. I love if Alex could, like, quickly just talk to this one. But Brennan says, quick one, when you said it reports the finding, where does it get reported? Yeah. It's probably one of the most important parts of Outtake. So we've spent a lot of time talking about discovery. Ultimately, what matters is remediation in cybersecurity. The specific thing that we're pointing out here when we point at something like an impersonated website, for example, is you gotta go get that website torn down from the Internet. And now this is intrinsically very difficult because it's a system that you, the organization, don't control, unlike traditional cybersecurity where you're protecting things that you do control. And so what you wanna do here is, basically set up high trust capabilities with these third parties that actually own that infrastructure. So, you know, the website might be hosted on a place like CloudFlare. It might be registered, with a domain registrar like GoDaddy. It, of course, shows up in search engines like Google, and these are all places where that website is effectively hosted or registered or distributed. And so to disrupt that, you actually wanna go, you know, remove that hosting capability or remove that registration, maybe d index it from the search engine. The only way that you can do that, of course, is, working directly with these organizations, which I would tell you is a is a is a very, very long uphill battle, but one that Outtake has succeeded in largely because of the quality of evidence that we walk in with. Right? So we don't come in and say, hey. Here's here's an arbitrary fake website. Rather, we say, here's this adversarial campaign, one that affects broadly your platform, and let's try to sort of remove fraud from your platform as well. And so in many ways, these sort of these website hosts, these social platforms have become really thoughtful partners for Outtake. But that's ultimately who we're reporting to. Yeah. So unique. For everyone who's stayed this long, we're gonna do one more question, that's a little bit more forward facing. And that is as threat actors, the Internet, adversaries kind of evolve and potentially getting access to more powerful models over time. We'll start with you, Jack, and then we'll round it out with Justin just from, like, the anthropic side of of how he's thinking about this. But how do you ensure or or think about, like, how this agent is essentially going to defend against, like, more creative attacks in the future as as our adversaries get a little bit smarter? Yeah. I mean, I think that that comes back to everything that we've talked about so far in terms of even how you get to the point where you have a sophisticated long running agent that's able to pin them down in the first place at this point in time. But yes, we're always constantly thinking about, you know, if we're here, we should always anticipate how are we kind of, you know, in an AGI filled way, how do we make how do we kind of think of what does defense look like when, you know, the attacker has AGI? And so even just to kind of bring that to a simple example, kind of focusing on the recon agent, that's why it's so important that we build out such a good feedback loop for eval evals and kind of automate as much of that learning process because all I have to do is continue to ensure that it keeps on getting exposure to each campaign as they get more and more sophisticated, as they add in more countermeasures, and that our system has a very clear way of learning from every single one so that it is right up there with them if they are trying to kinda take away, you know, basically, what does, you know, Sherlock plus Carpathi's, you know, recursive self improvement loop look like. And that's kind of what we we wanna make sure that the recon agent is able to do. But, yeah, just always just ensure that you can kind of have that self improvement is the only thing that you can do right now. Yeah. And just to round that out, what I would say is that AI is a general purpose technology. In cybersecurity, it's dual use. And so as the models get better, they're empowering both the defenders and the attackers. And if you're a defender, what that means is that it's really important that you're pushing the frontier and making the defensive tools as powerful as possible, which is why I really love that Outtake is doing this. I think this is a clear case of pushing what the models are able to do with a a long running agent harness. And, I think this work is just so important. So excited to see this, really coming into the world. Yeah. And it truly takes a village as well. I think, like, working with you guys and seeing all your team in the New York office and and how hot and fast you guys are working and how you experiment with a lot, is really meaningful to us so that we can keep building and iterating the models and and all of the the platform things in the direction that you guys need. So, yeah, big thanks for all your work, and thank you to everyone for attending. A survey should be sent out out to you. Either it already has or it will be soon. But that'd be great if you guys could fill that out so we can get some feedback. But thank you everyone for attending. Big shout out to the Outtake team, and and thank you as well, Justin, for being here. Thanks, Jake. See you, everyone.