Name: Measuring AI Fluency at Your Organization
Uploaded: 2026-10-13T20:06:05.784Z
Duration: 51 min 4 s
Description: Measuring AI Fluency at Your Organization

Transcript for "Measuring AI Fluency at Your Organization": Alright. Hi, everyone. It's so nice to see all of you today. Welcome. As we get warmed up and we let some people, join us here in the webinar, feel free to share where you're joining us from, in the chat. We'd love to hear from you all. Belgium, Cleveland, LA. Alright. It's a very global group. It's so wonderful to see all of you. Thank you for making time in your day. Alright. We will hope to keep this pretty interactive, so feel free to use the chat for questions or reflections or other ideas as they come up. It's wonderful to see all of you. And if you haven't met me yet, my name is Kristen Swanson, and I lead our Education Labs team here at Anthropic, which means that me and my team have the tremendous responsibility and opportunity to help understand what developing AI fluency looks like at scale. So how do we actually learn to use AI, and what does that workforce transformation look like when it goes really well? So I'm gonna be sharing a little bit about that today. And, again, as I said, I want this to be an interactive conversation, so we will use the chat as our place to share as we we go along. Okay. So I think it's important when we start to talk about organizational transformation transformation to start from the place that every transformation begins with a choice, and this choice is critically important for how the rest of the transformation goes. In some cases, transformation can be viewed as filling gaps or helping people do things, that they're not able to do. And we find that when transformations are framed this way, it can be very hard for people to accept the change. However, there's another choice and one that we found through our research is the is the more effective one, and I would say this is also pretty prevalent in the literature. This idea that if people see the change as really extending what they're capable of, giving them new opportunities to try new tasks, new opportunities to solve new problems, they see this as an opportunity for growth, and people that are in that frame of mind are much more likely to accept the change as it rolls out across your organization. And so here at Anthropic, we would strongly encourage you when you're thinking about AI transformation to ensure that you're framing it as this is a chance to increase innovation, to increase growth across the organization, and to give people the opportunity to try and take on challenges that perhaps they've not seen or tried or taken on before. And, you know, I think one thing that's always sort of stood out for me is a few months ago, we asked 81,000 people what they hoped for from AI. And we found that one of the top things people hope for is, in fact, professional excellence. And I think that's because people really do enjoy coming to work, being good at their jobs, and experiencing that satisfaction of helping others, making impact, and doing good work in the world. And so this is, of course, one of the top things people hope for. But it's also one of the top things people worry about that you can see unreliability and jobs in the economy are number one and two of what they worry about. So just as much as people are hoping for that personal and professional transformation, they're also really worried about what their career could look like in this new phase of AI. I'm I'm curious out there in the chat. Let's just take a moment. What do you hope for? What are some of your hopes with AI and transformation that can can happen? Yeah. Amplification of people's skills. Yeah. The transformation that allows people to grow, not grind. I love that phrase. Support, elderly support, freedom, time back for true connection. We know that those human connections are more important than ever, especially when you're in an... Or, going through an organizational transformation. Empowerment to solve your projects. Also, some efficiency and speed. These are all, tremendous things to hope for, and we'll talk about a lot of them today. So when it came to AI fluency and how we rolled out AI fluency many months ago, sometime in February, we actually took a look that started with behaviors, and you may have seen this. This was our AI fluency index. And we started to study which behaviors were more frequent, which behaviors were less frequent, and how those behaviors developed over time. And we released this in February. That was before Fable came out. We found something really, really interesting. We found that these behaviors were critically important in February, and in... They are no longer critically important in the same way today. Now I wouldn't say that these behaviors are necessarily bad. If you do these behaviors with Claude, you can get to a really high quality output. But I would argue that you no longer need these behaviors to exhibit AI fluency. The reason for that is the model intelligence has become so sophisticated that if you don't engage in one of these behaviors, the model is able to figure it out or prompt you for what it is that the model needs to actually get the answer you need. And so what we started to see as model intelligence increased was that these behaviors would start to go down, but task success wouldn't. And so we found that although these success wouldn't. And so we found that although these were a very effective predictor of task success and AI fluency in February, model intelligence is moving so quickly that these specific behaviors are no longer indicative. And just like trying to learn every single feature that comes out, for all of the suite of AI products is probably a little bit overwhelming, trying to pinpoint the exact behaviors at any point in time is probably also a bit of a fool's errand because the models are improving faster than we can actually pinpoint, develop, and implement the specific behaviors at hand. And so when we started to look at AI fluency through this lens of incredibly fast moving model intelligence, we recognized that we needed to expand our approach to AI fluency and that this was no longer about teaching very specific behaviors, very specific features, or very specific products. And for some of you out there, hopefully, that feels like a bit of a relief because I hear often that people experience what I sort of call Claude FOMO, where they're like, am I using it enough? Am I using it right? And the reality is is that, there isn't necessarily a specific recipe to follow for success. But instead, it's really about the mindsets that you have, the access that you have to different models and tools, and then the skills that you use in terms of being focused on a set of complex tasks and working as a team with the organization around you to get to test success very regularly for those tasks. So let's start with what I would argue is the most important part of the fluency ecosystem, which is mindsets. We went through hundreds of literature reviews on the use of AI across different products and different models and different features. And these were the seven mindsets that kept coming up again and again when people were really experiencing high levels of success or satisfaction with AI. And you'll notice that none of these are specific behaviors to practice, but instead, a disposition towards critical thinking, a disposition towards experimentation, and a disposition towards trying new things all the time. And the reason this is so important is because just like mindset three says, today's AI is the worst AI you'll ever use. So you have to constantly revisit the new models and the new forms of intelligence to understand, maybe I couldn't do this before, but can I do this now? So let's take a pause here. Look at these mindsets. Are there any mindsets that you feel are most important to your current personal practice with AI? You can either just jot the number in the chat or say sort of what it is that is jumping out to you as a reflection when you look at these mindsets. Yeah. Blake, context. Julian, number five, collaborator, not an oracle. Yeah. Two and seven. Yes. Yes. And so you can see here as they start to come in, it's not necessarily all of these mindsets all of the time. Different parts of your transformation will help you focus on different aspects of these mindsets. I would recommend, you know, these mindsets are something that you can take back and share with your team to start a conversation about what you want AI transformation to look like on your team. You know, I've heard from many different organizations that are rolling out AI. I've heard a lot about seven. This idea that, hey. It's still my output, and how do we communicate who owns that output and how it was created. I've heard a lot about four. How do we actually do discernment? How do we verify in proportion to the stakes across our team? What does good really look like? And so setting a few mindsets as priorities for your team given where you are in your transformation can be a really effective way to encourage people to exact button to press or feature to use. It's really about channeling these critical thinking and experimentation focused mindsets that help people try and try again. Because there's always going to be a new behavior to try or practice or learn, and this helps people accept that, understand that, and move forward with the change. In addition to mindsets, we also look at task success. And you may have noticed when we looked at the behaviors previously... I'll flip back here for a second. When you look at these behaviors, these behaviors are input behaviors. They are things that people do with the AI. And what we recognized was we really need to take a mindset and value forward approach and focus on, did you get to success on the task you showed up to AI with? And we find that this can vary broadly across people. It can vary broadly across role types, and it can, of course, vary broadly across tasks. And one of the things I've observed is many people, when they're using AI on their teams, they're sort of all trying to get to task success on their own. And what I'd encourage is think about this journey to task success as a team sport. If you all agree that the most important thing is that you are able to produce, let's call it, like, a document analysis that is specific to your business, you probably have a specific format for how to do that. You probably have a specific set of data that you use when you look at that. And as a team, if you start to find a way of working that you all agree on that guess... Gets you to task success on that task, that's a much more effective way of using AI as a group and really understanding workflow transformation than perhaps, just simply, you know, saying, okay. Go try it with Claude, and send everybody out on their own respective journeys. And so think about for your team the most important tasks that you want to do with AI, and then think about how you as a team can work together to figure out how to get AI supporting you really effectively in the production of that task. I'd also say that as you start to become more practiced in using things like skills or projects or file uploads or connectors, you start to get to task success a lot more quickly. If Claude is able to see all of the context and information it needs to get a job done, it's much more likely that job is going to be done how you'd like it to be done. And, again, importantly, this is something that works best when you and your team are focused on making it happen versus every single person in our silo trying to figure it out on their own. And last but not least, when it comes to the AI fluency ecosystem, we need to think about what we do in fact have access to. You probably noticed the AI fluency ecosystem is a product and model agnostic framework. It should work across all of the AI tools that you use in your stack, not just Anthropic. But the access that people have is a critical part of getting to success and getting to organizational transformation. People need to have access to frontier models to understand what frontier models are capable of. People need to have access to different surfaces that are catered to their output to get those outputs done. And so as you start to think about what access people on your team have and how you want to share and spread that access, that becomes an important part of the mindset. Once the access is there, however, it doesn't necessarily mean that people are taking advantage of it. And so you also need to take a look at... If the access is there, what are people actually using? And what you'll find is many people sometimes get, into habitual use of a certain set of features and practices, and they don't necessarily continue to explore. And so being aware of what features might be most useful to different tasks that they're trying to do and helping expose people to that and share some of those examples can be a really effective way to foster fluency, not just at the individual level, but at the team level. So I'm gonna take a moment there, and I'm gonna pause around this AI fluency ecosystem, this idea that honing mindsets that support AI fluency, choosing a few tasks, and helping the team together get to success on those tasks by building connectors and projects and prompts and skills, and then access, making sure that people have access to the tools that they need, that these are the three levers you as a leader have to encourage AI fluency across your team and across your organization. Before we dive into the the next two parts, do we have any questions that I can answer in the chat? Okay. We don't have any questions yet. But if questions pop in, we will... We're gonna... We will get to them later. So don't feel like you missed your only chance. We'll have more opportunities for questions. Alright. Alright. So one of the biggest questions I get at Anthropic is how do we teach AI fluency at Anthropic? What is our practice for taking new employees in our organization and helping them become AI fluent? We actually have a three part process for teaching AI fluency, and all of the materials are available, at Claude Academy. So if you haven't been to Claude Academy, you just head to academy.claude.com. I will pop that into the chat. And, essentially, when you get to Claude Academy, under explore, you'll see a section called learn about AI, and that will have three components of it. And those three components are the three components that we use to teach AI fluency to all of our employees here at Anthropic. The first part is how AI works. And you might think, oh, well, maybe, you know, your team, isn't very technical or maybe they don't necessarily work with code. That's not necessarily what we mean here. When we talk about how AI works, we talk about at the highest level, what are the properties of AI that help it be successful on some tasks and unsuccessful or very limited on others. You know, there was a survey that was done a few months ago, and they asked people in The United States, how AI gets its answers, and most people actually thought it looked the answers up in a database. That's not correct. And if that is how people think AI works, they will probably use it differently than if they understand how it does, in fact, really work. So when it comes to how AI works, we use a a framework, called the capabilities and limitations framework. So instead of thinking of AI as a black box that's either good or bad at things, we see that it has four properties, next token prediction, knowledge, working memory, and steerability. And these properties have capability zones and limitation zones. And we take the time with our employee base to go through each of these areas and to help them understand what types of tasks fall in the capability zone, what types of tasks fall in the limitation zone, and then product features that can help them take something that might be a limitation and push it into the capability zone. When it comes to teaching this, we do keep this at a relatively high level so that every employee who joins, no matter what their job might be, can really have a deep understanding of the properties of AI and how it works. This takes us about a half an hour to forty five minutes to go over, and many people say that it's a really effective way to help them understand what features they might want to use when, what models they might want to use when, and what tasks they need to be a little bit more mindful of when they take on those tasks with AI. If you are interested in teaching this to your organization, if you head to Claude, Claude Academy, which I linked there, we offer a full course on these four properties and how AI works that explain all of this in detail. And what I would... And the course is free. It's also product and model agnostic. So if you are using many different forms of AI in your organization, you can still provide this course because this will explain how all AI models that are generative work, not just Anthropic products. So, again, it's a really nice conceptual piece that can help people before they start diving into the specifics of how they actually start to use AI. So with this broad understanding and base of how AI works, we move on to the second phase. And this is what we call human AI collaboration. You've probably heard of the four d framework if you're here. I know that is a... That has been around the longest at Anthropic in terms of AI fluency. And the four d framework is really the thing that details what safe, efficient, effective, ethical, and safe AI fluency that one on one human AI collaboration looks like. And when we look at this and you look at the four d's, this is something that, again, similar to the how AI works content, we also offer a variety of courses on the 4Ds of AI fluency. But I would encourage you that the best way to think about these 4Ds are in two loops. There's the inner loop, which focuses on description and discernment, and that is when hands are on the keyboard. How do you actually use AI when you're talking with Claude, when you're adding files, when you are describing what it is that you need, and then importantly, when you evaluate what's coming back. And this loop of providing a prompt, evaluating what comes back, refining that, evaluating what comes back is probably, if you look on the Internet around how to use AI, probably about 95% of the content that's out there. This is critically, critically important, but we found through our research that people tend to develop efficacy in this loop very organically because AI is very different than many of the computer tools we've used at work before, it talks back to you. Because AI is constantly giving you feedback and constantly replying to you and constantly talking back to you, we actually find that most people start to improve their description and discernment or that inner loop on their own just from using and practicing Claude on task they do at work. So it's important to set people off on the right foot, but also know that over time, people will start to improve here. And the key area to focus on, yes, Julian, is discernment. Discernment is the hardest part of this inner loop. And the reason that it's very hard is... And we've done a study on this. We released all of the results about a month ago, is is twofold. One is that what I call the discernment tax. Discerning something actually takes a lot more mental calories than generating something. It's higher in Bloom's taxonomy. Evaluation is a very cognitively demanding task. And so every time you're asked to discern, it is harder than actually creating the output. And a lot of times with this inner loop and using AI, there can be even more discernment in your role than there ever was before. And so that is one reason why it's challenging. The other reason, and this came from a study of hundreds of Claude Academy learners, is because people are often using AI to help them on tasks in which they are not an expert. And so when they receive the inputs back, what they recognize is they don't know what good looks like. They're not sure how to check Claude's work. And this discernment barrier is something that... It's really, really important for you as a leader to get ahead of in your organization. Be constantly thinking about what are the checks that would tell me if this thing that I've generated is good, will work for this audience, is something that should be used if it passes the taste bar in your organization. All of those kinds of checks, many of them are inherently built into coding. Right? And so that's why coding has been very effective when it comes to AI. All of these checks already often exist. We need to think about bringing that kind of practice to other tasks. So I'll give you an example. I write, concept notes. And when I... Before I ever started using Claude to help me write concept notes, I took some of my best concept notes. I gave them to Claude, and I said, what makes these good? Like, what are the qualities of these patterns that you see that have made them good? And I actually had Claude create a rubric. And now every time I start writing a product concept note, that rubric, it gets checked against that rubric, and it doesn't pass the output until it's checked green on all of those items that were generated from my best examples. So be constantly thinking when you get to this description discernment loop of ways that you can help your organization cocreate exemplars and rubrics so that discern... The discernment tax goes down a little bit. I love this, from Farha. I asked my learners to start on a task they know how to do. That is a highly recommended strategy. So if you... Let's say that you do a data analysis every month. Before you ask Claude to do that data analysis for you, ask Claude to do the data analysis you already did, see where Claude got it right, see where Claude missed. And then when you start to understand those patterns, you can actually make sure that you've tuned Claude so that Claude will get it right the next time that you provide Claude with a similar task. So, great comments out there in the chat. And, again, that's really the heart of this inner loop. In addition to the inner loop, there's an equally important outer loop. This outer loop is sometimes missing from some of the conversation that you'll see out on the Internet or out in the world. But I would argue it's just as important, if not more important than the inner loop when it comes to using AI. And this is that delegation diligence loop, and it answers two really important questions. One, should I be giving this task to AI in the first place? And two, if I have decided to give this task to AI, how do I disclose ethically the role that AI played in the creation of this particular output? So for example, with delegation, I often hear from folks that they they don't always feel empowered to say, I don't think we should use AI for x, or I think we should use it for y and not, you know, b. I would encourage you to have a really good conversation with your team about the tasks that you want everyone to take on with AI and the tasks that you think are best left for humans. For example, on my team, we actually create, a number of animated explainers with Anthropic, that can help in some of the educational content that we provide. However, because we feel it's really important for that to be a human voice and a human way of explaining the concept for other humans, we actually still do quite a bit of handwriting on all of the scripts, and then we actually all self narrate all of the videos ourselves. And that was a decision we made as a team. Now we may change that decision over time. We may choose to delegate things in different ways down the line, but that was the the delegation choice we made together as a team to think about how we wanted to show up in our work and what decisions and pieces of craft we felt were most important to keep to ourselves. And and when it comes to diligence, we're pretty up front. So you'll notice many of the courses in Claude Academy even include a diligence statement that state how AI was used in the production of the course. Internally, if we write a doc, we say how Anthropic was used in the development of that doc. We've all gotten docs most likely that were created by Anthropic or some other AI model. And that wasn't disclosed, but we all sort of knew it, and we weren't really sure what to make of it. Did this person edit this? Did this person think it through? Is this really this person's ideas? And when you add that diligence statement that says, hey. I use Claude to, you know, refine some of my ideas and format this, or I narrated all of my ideas to Claude and Claude did web research for me, or whatever the case might be. It really gives your reader or your consumer a map for how to interpret the output that you're putting in front of someone. And the reason that's important is because Claude makes different mistakes than humans do. Humans will often make mistakes where it comes to... If you're looking at a very broad set of information, they'll miss something. It's very common. Claude never misses anything. Claude is very thorough, but Claude doesn't always get what is most important right. And so when I see a really good diligence statement from someone, that helps me understand what to look for, what to check for, and what types of mistakes I should be most tuned into. So, again, this delegation diligence loop is equally important to what happens, when you're having that conversation with Claude even though this loop happens outside of the conversation with Claude. And, again, I would encourage you to have conversations and set norms at the organizational level about this because it is something that when it's present, it really helps people gain the mindsets they need to develop that AI fluency and try using Claude for new and different more complex tasks. So those are the first two parts of how we teach people to use AI. One, how it works. Two, how to collaborate one on one with AI. And this third part is perhaps the newest part of our AI fluency curriculum. But I would argue potentially the most impactful. Because when you think about how your work with AI looks today, it probably looks something like this. You open up Claude or whatever tool you're using, and it is a one on one conversation. It's almost like you're putting on an AI super suit, at work that is helping you get all of your tasks done. And while that can be really, really helpful and it can increase efficiency and speed in completing tasks, it's not really how most humans spend their days at work. Most of our time spent at work isn't one on one at all. Instead, it's multiplayer, and we are constantly working with other people, to get our work done and to achieve bigger things. In the future, when we're on those teams of humans, it won't just be humans. It will be humans and agents. So the third way that we teach people about AI at Anthropic is to really help them understand that one of the most effective ways to use AI is on a human agent team. And so when you think about this, you can think about the black dots being humans and the orange dots being agents and many humans being able to interact with many agents to get the job done. And this much more closely reflects some of the practices we've seen be most effective over the past fifty years of collaborative teaming, but supercharges them with the capability of AI. And so there are some technical pieces that have to be in place to take an agent that looks like this and allow it to function like this. And those three things are really, really important infrastructure pieces that you as a leader should be aware of as you start to build out human agent teams. The first is memory. So today, if you have a conversation with your Claude and someone else on your team has a conversation with their Claude, those two those two conversations are siloed. They don't necessarily know about each other. The key to a multiplayer agent is that it has this memory that it can collect across conversations with multiple people. So I work on a team, a human agent team, and that human agent team is the team that actually builds Claude Academy. And if Miles, who's one of my engineers, has a conversation with our design agent, and then I go and I have a conversation with that design agent, I'm getting all of the benefit and all of the context from Miles' conversation when I'm having mine. So if Miles made a decision about a specific way that we should handle displays or links, whatever it might be, I benefit from that, and the agent already knows about that and is able to help me produce work that's in line with the team's norms and the team's goals. The second one is that has its own credentials. You've probably heard or you probably know that when you log in to Claude and you add connectors to Claude today, that means that Claude knows what you know and can see what you can see. And that's very effective for a personal assistant agent. But when you start thinking about agents on a human agent team, you want those agents to actually have the right amount of agency and the right amount of guardrails to do their job without too much access. So for example, our design agent is able to make front end changes, but not back end changes. And when that agent does make those changes, sometimes at our request, sometimes based on other signals, it leaves a very clear trail of the changes that were made and that the fact that they were made by the agent. So that way we have forensics. If we ever need to go back and understand why something happened or how something happened, so that we can make good decisions going forward and that we don't have to worry that our design agent, is is going to make decisions about parts of the code that they are not an expert in. Is it also possible to keep that context for yourself in the miles example, or is it automatically shared with the team? That's a really good question, Dorian. I would say that when you're using that human agent, team, transparency is really, really important. And so things that you... Conversations that you have in the shared team space... So for us, it's a Slack channel. It could be a team's channel or an email thread. Anything that happens there is considered fair game. But if you do DM the agent, right, like, you're in that one on one mode again, then that would be preserved from the agent. So there's a there's like a a place where that agent is, on a human agent team specifically crafted, and then you can also have a separate place that is just one on one between you and the agent. Good question here from Zoe as well. What needs to be done to manage the memory of an agent, especially as people are exploring agentic capabilities? So I think there's, a lot of different ways to do this. There's a lot of different models to do this, and different products have different ways of handling it. I have typically seen two patterns. The first pattern is the agent has a skill that allows it to consolidate its own memories itself. So, like, overnight, it will look at everything it learned from all of those humans talking to it, and it will, like, compact all of the memories and only save the most important bits so that the memory is there. And then humans, if humans want to tell it to not remember something, like, oh, please forget this thing. We we... That's not what we wanna do anymore. Agents, humans can ask agents to forget things. That is one model. The other model I've seen is that only certain people have the ability to persist information to the agent's memory. Now this gives some people more control, but it is a little bit more brittle. And what we find is that when people use this ladder model where only some people can persist information to the agent's memory, the agents tend to not be able to have a fresh enough memory because humans forget to update it. So I would recommend the former to start, and teaching people the, the practice of, like, telling agents to remember and forget things as they are useful and not useful to you all. And then the last piece here is shared context. Right now, you probably do many different things. You know, you have team meetings, you have hallway conversations, you have one on ones, and so on that are not being shared back to your agents today. And that means that your agents probably don't have the tools or the information they need to do a really good job. And so starting to set the right practices around recording all of your team meetings and dumping the transcription into a channel where the agent can see it, or getting used to making sure that information that might be in a DM actually goes to the team channel so the agent is aware of that, Or setting up connections to other tools like Jira or Linear, wherever you might be tracking projects, Asana, so that you can, in fact, make sure the agent knows all of that information and context when project tracking exists. And that is the thing that I would say the biggest trap when it comes to multi agent, human agent teams is that information gets siloed away from the agent, and the agent makes some bad decisions because it doesn't have all of the information that you do. Julian asked a good question. Is there a preferred method for building the shared context or memory? I find that, it's a good question and kind of one that, you know, we see here when it comes to best practices, and that is you want to extend trust to an agent over time. You shouldn't assume that the the moment an agent is created, it's going to have all of the information it needs to do your job perfectly. Start with smaller tasks. And as that agent starts to build up context and build up memory, you can continue to make the task bigger and bigger and longer running and longer running until you're able to give an agent a very long running task, and it's able to complete it on its own. Let's see. Is there an approach to manage and review the agent's shared instructions and memory so they don't drift based off of an individual user's confusion or frustration? Great question, Peter. One of the things that, you know, there's many different approaches to this, but one of the things we found that works particularly well on our team is, we actually... Once a week, we have the agent give us a report on what the agent learned that week. So we say, like, tell us what you learned, tell us what you think you did really well, and tell us what you think you you didn't do well. And for five minutes in our team meeting, we actually pull it up, we look at it, and we talk about it in the meeting. And then that that transcription of how correct we think the agent is or any adjustments we wanna make gets put into the transcription, dumped into the channel, and then we refine, refine, refine, refine. So in five minutes a week, we can, as a team, really align on making sure that agent is being as effective for us as possible. Let's see. Are we talking about Claude agents? Yes. So Claude agents can take many different forms. You could build your own using Claude managed agents. You could also use Tag, which is a similar product that allows you to have that multiplayer experience. There are also other products that are not Anthropic products that allow for multiplayer agents or multiple people to interact with the same agent to get that collaborative output. And one of the things I think that's really interesting is that when you move out of this only having a one on one conversation with Claude and you move into this multiplayer mode, you start to see that the things that made a really good team work well are the same things that make a really good human agent team work well. You've gotta write everything down. People need to have really clear roles. Agents need to have really clear roles. There's gotta be a north star. And then, like I said, start with smaller tasks, build up the context and the memory over time, and that is gonna lead you to an agent that's capable of longer running tasks as well. One of the things that's a little bit interesting is as you start to think about what it means to be on a human agent team, it also means that, especially if you're a manager, all of your ICs are now sort of managers too. We've seen that managers are actually very, very effective at working on these human agent teams because they're already really good at delegating. They're already really good at giving regular feedback. They're already really good about thinking about articulating really clear north stars and helping orient a group of people towards it. Those types of skills that we've historically reserved for leaders and managers are now gonna become critical for everyone. And so what I would encourage you to think about is, especially if you are a manager, what are the things you do as a manager that you may want to start explicitly teaching to your ICs so they can be more effective on these human agent teams where they will also be providing management to some of these these agents. And so that kind of brings us to a close on the three ways that we teach AI fluency at Anthropic. Chapter one, how AI works. Chapter two, that single player human AI collaboration with the inner and outer loop. And then chapter three, this idea of moving from single player AI to multiplayer AI where many humans and many agents work together on a complicated outcome that serves your business or your team. And all of these, you might be like, well, what what do I do now? All of these are available to you if you go to Claude Academy. So, this is Claude Academy. And as I said, under the learn about AI area, you can see how AI works. You can see the collaborating with AI section. That was the second part. And then building human agent teams. And if you click on one of these, I'll start with this one. You can see research and courses on how these work. You can see what different organizations like yours are doing when it comes to multiplayer AI. And then you can get help if you're like, but how do I set this up? What does this look like tactically? A lot of those materials are there for you as well. Everything, is pretty product and model agnostic. It should work for most products and models that you want to use. This is really based off of our research of how humans interact with AI. All of this is free, and much of it is licensed under a Creative Commons license. So if you want to take some of these materials, for example, on the four d's, and you wanna personalize them or remix them for your organization as long as it's for noncommercial purposes with attribution, those are things we love to see and and strongly encourage. The other thing I would say is if you are logged in, and you're logged in with your Claude account, you also are able to take a look at what your organization has done if you're a Claude admin. So if you want everyone to take building human agent teams course and you wanna see if everyone's done it, that is also something that's available to you. And individuals can actually track their learning, can earn badges that you can share on LinkedIn across all of these topics as you dig in, and become more and more capable, in the different areas of AI fluency. One thing, that is just a fun thing that I will plug for you all as well is, we are rolling out new ways to learn with Claude. So if I open, for example, this lesson on syncopancy and what syncopancy can do, you can actually have an interview with Claude where Claude will interview you, and you can just talk to Claude. And Claude will tell you how deep your understanding is, other opportunities that you have to deepen your understanding further, and other resources that are related to where you're at in your learning. So definitely check it out. It's a great free resource that is intended to help you feel really confident when it comes to learning about AI fluency and building your AI fluency. We have about five minutes left, and I will just kind of say these are... And, again, we'll share all of this deck with you, but these are all of the resources that I talked through today that you can take a look at, and, again, use for free, logged in or logged out, at Claude Academy. And I am happy to use the last four minutes, for questions. And it looks like Wesley's got one in here to help get us started. So it says usage is measured, but how is adoption measured for AI? Great question. When we think about AI adoption, we really think about this in terms of the ecosystem and in terms of value. So, AI usage doesn't mean value for your business. It could, but doesn't necessarily. And so that's really why this ecosystem is so important. One is what are the mindsets people bring to the work and how are those changing over time? Two is access as you said. But the third piece is the most important piece when it comes to value, which is task success. So we are able to look at people's tasks that they do with AI and determine if they were successful and how complex those tasks are. If you are a Claude for enterprise user, you can get this information through smart reports. And the key here is understanding how successful tasks successful tasks are and if those tasks are the right tasks for your organization to be aided by AI. And that is strongly how I would measure linking value is picking a certain set or family of tasks and getting the organization to task success as quickly and efficiently as possible to make sure that you're getting the most value you can out of the products and services that you're using. And, Kristen, that's kinda related to your question question as well in terms of measuring impact as adoption grows. Because, again, if access goes up and usage goes up, but mindsets don't change and task success is not going up, you're not necessarily getting more value. And so I would strongly encourage you to think about this AI fluency ecosystem as a mechanism for measuring value over time. We do have and can share, measurement tools that you can use to measure each of these areas, as well as... Again, those are all also available in the smart reports. So, keep that in mind as you start to think about setting the right kinds of goals and the right healthy incentives for transformation. Is there a set of metrics you look at to understand high ROI use cases? Yeah. So, again, this really comes back to this ecosystem that you're seeing here. One is, like, what are people's perceptual mindsets? And two is, what are the tasks tasks that people are doing, and are those tasks successful? And the higher the task success rate on those high complexity tasks, the more value you're going to be getting out of your deployment or your usage. And, again, those are available in the smart reports. It's based off of classifiers that we've published in our economic index and gives you a very value forward way of looking at what you're getting out of AI versus just token usage, which is often not indicative of value. I see these questions about metrics. Again, these, each of these have classifiers that you can use, that are published, and definitely check those out, so that you can start to think about how to measure the value you're getting as organizational transformation happens. So, those are a couple questions there. It looks like there's a lot more here about the questions and the tools. Those tools are available to you. You can find the description here, so I'll show you where it is located. Here, you can see this building AI fluent organizations, walks you through all of these in particular and how you can measure them. If you want Anthropic to measure them for you, that's something you can talk with your CSM about or turn on smart reports to get those specific metrics for you. With that, we are at time. I just wanna say, thank you so much for spending some time today. And I see your question here about ROI tied to business outcomes. Yes. And, that's where you choose the right tasks in the task success area. You know what those are for your business, and those can be very flexible. But, with that, thank you so much for spending your time and offering such great insights and questions. This will be recorded and shared along with the slides, and more information, of course, is available on Claude Academy. It's great seeing you all today and, look forward to continue learning with you and collaborating with you. Thanks, y'all.