Name: Cisco Workshop: Building and Evaluating Deep Agents
Uploaded: 2026-09-02T17:04:31.240Z
Duration: 1 h 25 min 12 s
Description: Cisco Workshop: Building and Evaluating Deep Agents

Transcript for "Cisco Workshop: Building and Evaluating Deep Agents": Hey, folks. Welcome to this session. Thanks for joining. I'm Wei. I come from Cisco first for the last two and a half years. So just a few housekeeping items. We are available on Webex. So I'm gonna put my email in the chat. If you have any questions after the session, feel free to Webex me directly. In the doc section, we also have the sign up link for LangSmith. This is for LangSmith to run on. So if you have any questions, just let us know. But with that, I'm gonna introduce you to Andrew, who will guide you through today's session. Hi, everybody. My name is Andrew Salden. I'm a deployed engineer here, with Langchain, working with Wei, at Cisco. I'm also available on the Cisco Webex, so feel free to reach out to me anytime. Thanks for joining us today. Today, we're going to spend, I think about ninety minutes, walking through DeepAgents. And DeepAgents is an exciting technology for building powerful agents. So we're going to start off with a little bit of introduction, and then we're gonna go through, some code notebooks to see kinda DeepAgents in action and how you'll build them. If there are any questions, there is a a q and a in the chat. So feel free to, post your questions in there. And Angeline, who's hosting, will see them. And kindly interrupt me, and we can always, take a moment to discuss your questions. I'm happy to pause the content and do some more interactive if that's interesting to you. Great. So let's go ahead and get started, learning about what deep agents are, why they're interesting. So today, we're gonna talk about, the anatomy of a deep agent, what they are, how they work, how to build them. And as we do that, we're gonna work through a Jupyter notebook. And, again, as we go, if there are any questions, please put them in chat or even just unmute yourself and speak up. Love to love to hear your questions as we go. Great. So let's get started talking about what deep agents are and why we built them. As you all know, a simple agent an agent basically, you can think of it as a tool calling loop with an LLM. At its very fundamental level, that's all an agent is, calling an LLM in a loop with tools. And this works good for some simple task, but we really need more power as we build real agents that do real work, in the real world. And so that's why we built DeepAgents. DeepAgents is an agent harness. It's designed to help you very quickly get up and running with some very powerful agents, and it provides a couple of things out of the box. It provides the ability to plan. It can take complex, even ambiguous natural language queries from a user and decompose them into multiple steps, break them down into a plan of action. Context management. Context management is extremely important for talking to your LLM. Context is that precious resource that you wanna make sure that you manage intelligently and appropriately. And deep agents gives you a lot of tools for doing that transparently. Another feature of deep agents is the ability to to delegate. We'll see shortly when we start building some deep agents that deep agents are designed with a certain design pattern in mind, and that design pattern is essentially a supervisor worker pattern where you'll have a a main supervisor agent, which can delegate to one or many sub agents, sub agents being specialized, workers. There's also great, controls in the deep agent's harness for memory, and we'll spend some time today looking at how to architect a a memory architecture, for your deep agent. Then, obviously, there's also control there. So human in the loop control for human approval for sensitive actions. There are a couple things you do get out of the box with any deep agent right up front. I managed I I mentioned planning, the ability to decompose complex tasks and make a plan of action. Another very powerful thing that we'll spend some time looking at today is file system support. We all know that, the Frontier models have been trained, fine tuned to use file systems as coding has exploded as a primary use case. And we leverage that in deep agents to use the file system abstraction as a place for deep agents to store, intermediate results and final results and to communicate. It's a channel of communication for sub agents to communicate back to the supervisor agent. We'll also look at memory and skills. There's great support for memory and progressive disclosure of skills as well. And finally, I just wanna call out here that everything we'll look at today is model agnostic. Doesn't really matter what models you use. Even OpenAWIT models are supported, and that's another powerful feature of the DeepAgents framework. So let's just take a moment to situate to situate deep agents within the larger langchain open source ecosystem. At the bottom level of the open source ecosystem from langchain is the langue graph platform. Langue graph is a runtime for your agents, providing durable execution, streaming, persistence, and human in the loop. On top of that foundational layer, we have the langchain package. This is a great framework for creating simple React agents. It gives you support for models, tools, basic agent loops, and middleware. Deep agent sits on top of lang chain and lang graph. In fact, deep agents is essentially a super powered lane chain agent, with a lot of extra things built in to a full featured agent harness. And at the top of that, we use the deep agents harness to build your agent, your custom bespoke logic for your business case. Parallel to that, we also use Langht Smith as our tracing, evals, and deployment and monitoring platform. With that said, I'd like to switch gears now and start looking at some of the deep agents code. Gonna switch over to my Python notebook. Great. So from here on, we're gonna be mostly focused on looking at some actual DeepAgents code and seeing how DeepAgents really works under the hood and getting comfortable with building some DeepAgents. So there's a lot there's a lot of content in this Jupyter notebook. We'll be sharing out the link with you afterward so that you can download the notebook and work through it yourself. There is a lot of content here, and we're not gonna get through it all today. But what we will do today is go over the basics and some of the extended features, so that you'll feel comfortable at the end of this building your own deep agents. Great. So let's get started looking at deep agents. A deep agent at its core is just a regular agent with a small collection of essential tools, a small collection of prebuilt middleware, and a few superpowers, the ability to read from and write to file systems, and the ability to spawn synchronous and asynchronous sub agents. Additionally, when you want the extra power, it has an optional to do list middleware. The to do list middleware is great when you want to take in complex, ambiguous natural language queries and decompose them into a plan of action before you start executing. This is directly analogous to plan mode, for example, in cloud code, where before cloud starts coding anything, it will do a deep analysis and planning of what it actually has to do. Let's get started looking at some code examples. You can see here, it's very easy to create a deep agent. In fact, in just a few lines of code, I could give it in just a few lines of code, I could create a simple deep agent. I give it a simple system prompt. You are an expert research assistant, and I pass in the model here. And with just a few lines of code, I have a deep agent. Invoking the agent is as simple as calling the invoke method on the agent we just created, passing it in a a Python dict of messages. And by convention, when we pass in those messages, we typically tell the LLM who's talking, and we label the content. And in this case, I've simply asked a very basic deep agent, what tools do you have? So we'll see this pattern over and over again today where we create a Deep Agent and we call agent dot invoke, passing it a Python dict of just messages. Having done that, we can easily see the basic bare bones Steep agent has a couple of built in tools here. Mostly these tools are around working with the file system. So we can LS directories, read files, write files, edit them, and do some simple searching like grep and globbing. We also see there's a task tool. This is how we hand off to dedicated specialized sub agents. So we'll be seeing a lot of sub agents today, and the task tool is how we delegate to them. I did man I did mention the planning tool, the to do list middleware. So planning is opt in, and it can be useful when you need it. However, it does add some token overhead, so it's been removed from the default middleware stack, very recently. But to add planning back in, we simply add a new middleware. I'll pause here to note that one of the great things about the Deep Agent's harness is that it's very extensible. There are many extension points, and we can always add in prebuilt middleware available from the main chain open source ecosystem, or we can create our own custom middleware. And we can see that we can extend a deep agent very easily by simply passing in a middleware array when we create the deep agent. Having added the the prebuilt to do list middleware, I can then I can then ask the the deep agent, to make a plan. So we created a new agent with the to do list middleware, passing it in a new Python dict. I can say, hey. I wanna make a bay I want to bake a cake, make a plan. And before the deep agent ever delegates action or ever actually starts executing anything, it will decompose the requirements and create a to do list as we can see here. K. Okay. Let's get on with some other basics before we get into the nitty gritty of it. Our agents wouldn't be very useful if they couldn't interact with their environments, and the standard way of doing this is by giving our agents access to tools. Tools in the lane chain ecosystem are just regular Python, functions annotated with the app tool annotation. In this case, I've created an Internet search tool, and I've given it, the Tavly search client, which is just a very handy, Internet search client. Having defined my tool and then go into my regular create deep agent, and I pass it a tools array. Now in the real world, you would have presumably many more tools, but this is just a teaching case here. So So we're just gonna show, that we're just passing it out a tool here. And, again, once we've defined that agent, we simply call agent dot it invoke, and we give it a query that it can't understand from its own memory and has to, call out to its tool for. Having done that, we can see the agent calling its search tool that we just defined with the query, and we could see the entire result back here. Now this is a rather large result set, and I'll take this as an opportunity to mention one of the really cool superpowers that DeepAgents gives you. It may often be the case that when your agent calls a tool, it gets back a rather large result. It's generally bad practice to naively pack all of the results back into the context window of the large language model. We know that context windows are finite, and we know that the power of the of the model to reason can degrade when the context window fills up. One of the nice things that deep agents will do for you automatically is when you call a tool, it will detect if the result of that tool is very large. And if the result of the tool is very large, and if putting the entire result set into the context window could degrade the cognitive performance of the model. The deep agent's harness will automatically intercept that tool result. It will store the entire result to the file system, and it will simply return a pointer to the file to the model along with a couple of lines representative of the of the result to the model. The model can then opportunistically decide if it wants to read all or just part of the file using its standard file system tools. So this is one of the nice ways that DeepAgents gives you context management, out of the box. Now that we've seen some basic use of deep agents, we're gonna talk now about why why deep agents are designed the way they are and really what makes them powerful. The entire deep agents framework is really predicated on a certain design pattern, and that design pattern is that you have a main supervisor agent delegated to one or more specialized sub agents. What's great about this is that each sub agent has its own context window, its own tools, its own system prompt, its own model, its own skills, and its own middleware. And the main agent simply delegates via its task tool, and it only sees the results of its delegated sub agents. So this is a great conceptual model here. The way it works is a user will talk to the agent using natural language, typically presenting, a fairly complex, query that needs to be decomposed and delegated. The main agent, which we often call the supervisor agent or the orchestrator or the coordinator, is the one that directly talks to the human user. It understands the human user's tasks, and it breaks those tasks into definable, actionable subtasks. Once it's defined as subtasks, if it has appropriate sub agents, it will delegate to those sub agents. In this example, we're seeing that we have perhaps multiple sub agents, each specialized for one area. Maybe we have a coding sub agent. Maybe we have a research sub agent. The idea is that you plug in your own specialized sub agents. Those sub agents work in their own context windows with their own tools and their own skills, writing to their own file system. And when they're done, they can write results back and share those results with the supervisor agent, which can then synthesize all the the research results and present a final answer back to the human user. So this is the core pattern that DeepAgents enables. I'm gonna pause there. I see a lot of activity in the chat. And, Julian, anything I need to address? No? Okay. Looks good. So now that we understand the main design pattern, let's see how we can define some sub agents. Oh, Typically, Andrew. we simply yeah, Angeline. There are about three questions. Maybe we can kind of jump into that one in the meantime as. you pause. I could pause. I don't see that. I only see a chat. I'll go ahead and share it to the screen right now. Okay. Alright. We have one question for from a theory. Okay. This, is Andrew. Amatri. How how does does the plan, perfect. oh, I can read it now. Thanks. I see it. How does the query plan to do or aligned? By virtue of adding these middleware, should we expect it to happen naturally? Yes, Maitre. You should expect it to happen naturally. There's the mechanism here is very similar to what you'll experience if you use quad code and put it into plan mode, for example. The the guidance that I often give is that whereas sub agents can have different models from the supervisor agent, the supervisor agent's model should be a reasoning model that's able to understand and decompose, complex nontrivial tasks. If you give it a sufficiently powerful reasoning model, the harness will provide it with the to do list tool, the planning tool. And then what you will see is very naturally with, almost no work from you, it will very naturally start planning itself, and there's nothing you you really need to do. Do you wanna put the next question up? Why should I use. DeepAgent versus Hermes or OpenClaw? I think that would be a really big digression. And in the interest of keeping this on track, I'd rather not go there. But, Tensile, what I'd say is that Wei and I are on the Cisco Webex. Please reach out to us, and we'd be happy to have that conversation with you. But I do just wanna keep this presentation on track. Was there another? Okay. Yeah. The next question is on the screen. context rot handled in long conversations or tasks? This is a great question, Anurag. So the DeepHS harness, is designed to manage the context window intelligently. I spoke a moment ago about how we will offload large tool results into the file system rather than pollute the main context window of the LLN. But that's not enough. Another way that deep agents will handle context rot in long conversations and I do wanna emphasize, the deep agents harness is specifically designed to handle long running, long horizon complex tasks that can run over minutes, hours, or days. The way we handle context raw is I mentioned at the beginning, DeepAgents comes out of the box with a stack of middleware. One of those middlewares is the context summarization middleware. When we load models, we know about the context window size of every model we deal with. And the context summarization middleware will automatically kick in and summarize a long running conversation whenever the the context window limit approaches, I think, 85 or 90%. This is directly analogous to how you'll see a coding agent kick in during a long session. Where your context window hits 85 or 90%, it will kick in. It will automatically summarize the conversation. Same exact feature happening here in the deep agent's harness. Great. Great questions. I'm gonna keep going. Otherwise, we won't get through everything. But I'll I'll circle back in maybe another ten or fifteen minutes for more questions. Let's take a look at defining sub subagents. We typically define them as Python dicks. You can see here I'm defining a research sub agent, and it's just a dict. It should have a name, a description, and, of course, it should have a system prompt. We can pass the sub agent a different model from the supervisor, and we can give it its own tools list. Once I've done that, I simply call my create deep agent, same way I've been doing previously, but now I pass it in my sub agents array. And notice that it is an array. I can have many sub agents, and that is the normal case. We can also see that sub agents can have their own planning middleware. So if I wanted my sub agents to have deep deep, deep ability to do detailed planning prior to execution, I could also give them the to do list middleware. There is another way to to define a sub agent, and that's through the compiled approach. Functionally, it's equivalent, but we'll call the create agent, giving it all the same parameters as before. The only time you would really use a compiled approach is if you wanted to invoke an asynchronous sub agent. We probably won't have time today to go over asynchronous sub agents, but just know that the ability to invoke a sub agent asynchronously and then pull for its results later is is a new advanced feature in the DB agents harness. And if you do wanna go with that approach, compiled subagents are the way to go. I'll call out that I would encourage you to download this notebook as there's an entire section here on running asynchronous sub agents, and it can be a really powerful technique. Let's move on and talk a little bit about file systems. Since file systems are so incredibly useful and important to the operation of agents, the deep agents harness comes with a couple of different ways to persist memory and work with file system state. By default, we have the the by default, all deep agents come with the default file system, which is a virtual in memory thread scoped file system. You can just get it out of the box, and you don't have to say it or do anything. We also have several different file system implementations. You can use a local file system implementation. You'll notice here that as I in these examples, I'm now passing in a back end parameter. We're gonna look at that in more detail. We can see here that one of the the files to system implementations is the file system back end, and this will give your deep agent access to your local machine's file system. However, more typically, what we'll see is we use the durable store file system back end. The durable store is very useful. We'll see some examples of that in a bit. This is a typical way to give your agent long running persistent durable storage, and it's typically backed by a database like Postgres DB or Mongo DB, or really anything. There are a couple other back end types. I won't go into all of them today, but I do wanna call out the sandbox back end. This is great for giving your deep agent access to a sandbox. Increasingly, agents need to develop their own code and execute their own code, and they need a safe environment to do that. Sandboxes are a first class citizen in the deep agent's harness. Let's take a look at the default back end. As I mentioned, it's an in memory virtual file system, and it's at it's scoped to a thread ID. In the lane chain ecosystem, a thread is our vocabulary for, a multi turn back and forth conversation between a human user and an AI. So when you go to a chatbot and you have a multi turn conversation, that's linked together through a thread ID. Let's take a quick look at how the mechanics of that work in deep agents. I'm gonna create my deep agent as before. And this but this time, I'm going to give it a checkpoint a checkpointer. That's good to checkpoint the state of our conversation at regular intervals so that we don't lose anything. Having given it a check pointer, the next thing I wanna do is pass it in a config dict. This is a very simple structure with a configurable key and a thread ID key. The value can be anything, but this is really a session identifier. So adding this will allow us to maintain memory through multi turn conversations. As before, I simply call agent dot invoke, passing it a message as dict. And I'm saying, hey. Tell me a joke about computer scientists and write it to the file system. Again, it's a virtual in memory file system. But in order for this to work, I need to pass in that config dict. When I do that, the agent happily creates a joke about computer scientists and uses its write file tool to write it at the path joke MD. Now once that's done, we can come back in a subsequent turn of the conversation and read that joke back out of the virtual file system. And, again, the way we do that is by passing in that same config to dict, which has that same thread ID. If I pass in that same thread ID, I'm able to read the context of joke MD back out. And I can see here, here's the joke. Now if I give it a different config with a totally different thread ID and I try to read that back out again, the model can't find it. So we call this thread scoped memory, and it's very useful for multi turn conversations. It's the standard way of handling ephemeral state through a multi turn conversation. We'll spend some time looking at a memory architecture later, but this will also be a part of that. Let's briefly look at the durable back ends, though, the store back end. When you're building an agent, you typically want a couple of different types of memory. That ephemeral memory that we just saw is super important, but you also want durable memory. The DeepHence harness provides us through store back ends. This is long term durable memory. It's backed by a database, and it's persistent across threads, and it's good for storing longer term memories. We're gonna import the store back end. And in this case, because it's an example, we'll use the in memory store, but we'll see another store in just a moment. As before, we're gonna create our deep agent, and this time we're gonna pack it, pass in two new parameters, a back end implementation, that's our store back end, And, a store variable, which is the actual implementation, in this case, the in memory store. We'll see here that memory is actually namespaced, and that's important. We'll look at that in a little little bit later. Let's take a look here. We're gonna create we're gonna invoke our agent again with a config dict and a new thread ID. We'll call it session a. We're gonna write this memory in this one thread session a. We'll call agent dot invoke, passing in that new config dict. Remember that my name is Andrew by writing it to profile MD. Do that, and no surprises, it uses the right file tool, and it writes it to the file system path. Now because this is durable storage and it's not scoped to a thread, I can pass in an entirely different thread ID, and my agent will be able to read that back out to me. We see that when we do that, it's able to read profile MD, and it finds my name correctly. So the key takeaway here is that building a deep agent, there are a lot of primitive tools here for building a very powerful memory architecture. Some memories are ephemeral, and they can be scoped to threads. Other memories, you want to persist across threads, and that's where we use the durable store back end for. You can see here that out of the box, the lane chain ecosystem provides several different implementations of the durable store backends. We've been using the in memory store for this notebook, but we have a Postgres store, a Redis store, a MongoDB store, and it's also very easy to subclass your own. Here's an example of what it looks like to use the MongoDB store. Great. So that about does it for the basics, the deep agent basics. I'm gonna pause here to take a few questions, and then we're gonna pivot into something more advanced. Angeline, do you wanna propose a couple questions to answer? Okay. Great question. So is there a requirement to use advanced models in deep agents and lesser advanced ones in sub agents? No. No requirement whatsoever. You're free to use any model you want for the supervisor agent and any model you want for the sub agents. It's entirely up to you. The correct agent is entirely your choice, and only you will know because only you know your problem. Next one. Okay. If I choose to deploy a deep agent in the Linksmith platform, what are my options for back end and file system? Can I store info that will be shared across spreads? Great question. In in Langsmith so for those of you who don't know, the Langsmith platform is an observability and EVALS platform. However, it is also an agent runtime. You can deploy your agents onto the LYNXmith platform, and they will run-in a durable, solid, scalable agent runtime. When you do that, by default, your agent will get a durable back end store. We happen to use the Postgres store for all of our agents. So you will definitely get a Postgres store, and that will implement a a file system so your agent can read and write a file system that will be backed by Postgres. And, yes, just as we saw Ittai just a moment ago, when you use the store back end, it will persist across threads. I'll also note a couple other things, though, Itay. One great reason for deploying onto Lanesmith is that your deep agent will have access to a sandbox environment, and that sandbox environment can also be mounted as a file system into your agent. Why don't we take one more question? Question is, can we use self hosted models inside of the deep agent? Absolutely. One of the great things about the lane chain ecosystem is it's completely model agnostic. It's very easy to direct that to a self hosted model or to an open source model hosted by a third party provider like Fireworks. Absolutely can do that. In fact, there are teams at Cisco using, their own developed models. K. Great. Let's take a minute to look at some more advanced memory architectures since this is really core to building important, important agents, we'll do that, and then, we'll have time for maybe one one more notebook. But the reason I wanna spend time doing a deep dive into memory architecture is because it's incredibly important for your agents to, understand user preferences and be able to store them efficiently. So we just looked very briefly at a couple of back end implementations. Now we're gonna look in more detail at them. We wanted what deep end users will do is it will allow us to compose, a a multi tier memory architecture. We're gonna build a memory architecture that has, ephemeral memories and long term memories. It's important to know that there are different types of memory in a deep agent, and deep agents allows you to compose them up together. So if we look conceptually, what we'll see is, every agent will have a system prompt. You set it once at compile time, and it's always there. But agents will also have other kinds of memories. They will have skills, and they will have other memories that you define. Skills for anyone who's not familiar, are simply recipes for doing something repeated over time. When you use skills with deep agents, the index of those skills, not the content, but just the index, the name, and description, will be loaded into the memory of the deep agent, right after the system prompt. So the bubble on the left is actually showing how memory is laid out in the deep agent. After the index of skills is where we get into our own custom bespoke memory architecture. We're gonna work through an example here where we're gonna build up a multi tier architecture. We're gonna have user preferences stored in slash memories slash preferences. Those will be user scoped memories. We're also gonna see how to set organizational wide compliance policies, and we're gonna set a read only policy on those so a rogue agent can't maliciously overwrite a company's policy. Those are gonna get packed in here to agent memory. All of that together will be passed into the model for the model calls, and the model will be able to read and write, those those memory files as well. So let's see how so let's take this and see see it in action. We're going to store some user context for the purposes of this notebook in a small Python data class called memory context. And for the purposes of these examples, we're gonna store user IDs and organizational IDs. We're also gonna set up two fictional users, Alice and Bob, and they're gonna have their own context. User ID for Alice will be user Alice, and her org will be Acme. And Bob will be user Bob with his org. Same org. I mentioned briefly in passing that we have this notion of namespaces. It's very important that one user's memories don't get intermingled with another user's memories. It's also very important that a malicious user can't read or write another user's memories. The name space the name spacing is how we're gonna achieve this. So we talked about everything is based around a virtual file system. And we're gonna build up an a virtual file system based on, slash memories, preferences, and slash policies compliance. So those are the the file system routes, and they're gonna tell us where a file lives. But the namespace is gonna tell us whose file it is. We'll see in just a minute that we're gonna pass in these contexts when we set this up. And I just wanna make the point that this this context object is just a built in class in langchain called a lang graph dot run time dot run time. And the runtime object allows us just to pack in user context. So, if you see that that this class reference, this r t, that's what that is. It's simply a runtime class that we can add our own context in, when we invoke the agent. If this this will all come together in a few moments if it's if it's not clear yet, so just bear with me a little bit. So we're gonna build we're gonna use something called the composite back end. The composite back end is gonna allow us to create a custom memory architecture with some ephemeral memories that are thread scoped and some durable memories that are persistent, and it's gonna allow us to set a org wide compliance policy, which is read only. And because we're gonna be passing in namespaces keyed on user IDs, the memories are also gonna be scoped to individual users, and they're not going to be overridable by malicious users. I know that was a mouthful, but just bear with me. It should become clear. So let's see how we can set this up. For the purposes of demonstration, I'm going to seed our agent with a couple of memory files in the form of MD files. Now for a real agent, you would typically store these MD files as actual text files in your source repo. Or if you're using the Langhitz platform, you could also store them in the Langhitz context hub. But for the purposes of demonstration, I'm going to simply define them in code here. So we're gonna set up our backing durable store. In this case, it's the in memory store. But in production, we would swap this out for a Postgres store or a MongoDB store or a Redis store. Having set that up, I'm now gonna feed my store with some memories. Every memory is namespaced. In this case, our name space is gonna be user ID and then the semantic area of memory, that we're gonna store it in. So I'm gonna store Alice's preferences. She likes concise bullet points and Python. Then I'm gonna actually give Bob a totally different user, his own preferences. Know that they're both stored at the same logical path, preferences dot m d, but we'll see that Bob is actually not able to read Alice's memory. Next, we're gonna set up some organizational wide policies. The namespace for this will be org ID and policies, which is just, the subject matter area. And we'll store those under compliance MD. And these are just company wide policies. Finally, we're gonna give Alice her own unique set of skills. And we'll store those under even a third area here and give her her own skill. Having set that all up, we can then actually inspect the durable file system store, and we can see that Alice's preferences are very different. We can see Alice's preferences here and her skills files here. So this is us just looking at what we set up. Now let's take a look at how we wire up the agent here to make use of all these different memory types. We're gonna use something called a composite back end. Previously, we looked at a store back end. We looked at a thread scoped memory store. Now we're gonna use the composite back end. And I'm just gonna skip ahead a little bit so we can see the actual code so that makes sense here. We're gonna define our agent's back end as a composite back end. And when we create the deep agent, we're gonna give it that back end here. So let's talk about what we're doing here. We're creating a couple of different routes, different places in the virtual file system to store the different memory types. Starting at the beginning with the state back end, this is the default that you always get. It's that ephemeral thread scoped memory that we looked at. It keeps files in the graph state. They're thread scoped, but they're not durable, and they're not gonna be persisted to the back end. So any memory that our agent stores or reads that doesn't fall under one of these paths will by default go into the state back end that thread skipped ephemeral storage. Now this is how we set up our memory hierarchy. We're gonna have a slash memories. This is where we're gonna store those user memories. And note that they're scoped on our user ID. We're referencing that lane graph runtime object, the context user ID, and the memories. So this two altogether is the scope for memories, and we can see here it's scoped to user. Notice here that the platform is injecting the user ID at runtime. This is how the platform prevents malicious users from reading or writing other users' memories. We also have a policies file system area. Again, same setup with a different namespace tuple. And then we're also gonna give some of our users their own skills scoped by user ID and under the skills directory namespace. We're also gonna give the agent, as part of its system prompt, a little bit of a instruction here so that it knows about its memories. I wanna also call out something we'll see here that we're gonna instruct the model that when it identifies a preference, when a user says something like, I prefer x over y or I always do this over that, the model will understand that, and it's good to opportunistically write new memories into the user scoped preferences. So we have customized the system prompt here to know about our memories and also to know about our organizational compliance. With that, we're ready to create our deep agent. As before, we give it a model and its system prompt. Now that we're using, memories, we also wanna tell it about we just wanna pass in the memories the location to the memories in the memory array, and we wanna pass in the early location to the skills and the skills array. The reason these are two different arguments is simple. As we saw when we looked at the memory hierarchy image before, if I scroll back up to the top, those memories in green, these are always gonna be read verbatim, byte for byte, right into the the context window. However, skills are not loaded entirely. Rather, they're loaded on demand. Only the name and description of the skill is loaded every time into the system message. So that's why these two memory types, when we create the deep agent, are two different arguments. The technical term for skills loading is called progressive disclosure. The DeepAgents furnace manages the progressive on demand disclosure of the entirety of the skills only when they're needed. The reason for that is that skills can be very large, and we don't necessarily always want them in the context window if we're not going to use them. By passing them in as a separate parameter, we're telling the deep agent, Harness, to use that on demand progressive disclosure loading of skills only when needed. Great. Given that, we can pass in that composite back end that we just defined. And notice also that we're gonna enforce some policy permissions here. I said earlier that the company's compliance policies should be read only. This is how we set that policy up. We set a file system permission, and we say this is the path and that you should deny rights to it. Great. That having been set up, we're now ready to start using the deep agent. Let's take a little bit of a look at how memory isolation works and how those namespaces enforce separation. So I said that we're putting memories preferences MD in the system prompt before every turn. So when the agent starts a conversation, it already knows the user's preferences. Same compiled agent, same path, but note that they get different runtime context. That's that lang graph runtime object. And the name spacing function that we saw resolves to a different store namespace every time. So this is how the framework prevents, malicious users reading each other's memory. It's not a prompting technique. You can't jailbreak it through prompt injection. It's purely enforced through the framework. Let's take a look at that in action. If I ask the agent what are my stated preferences, give them back to me, and then I run through my two different users, Alice and Bob, I can invoke my agent giving it that question and in turn giving each user context. What will happen is the agent will answer back with Alice's preferences. She likes concise bullet points and prefers Python versus Bob's preferences. This is not handled through prompting. This is handled through the framework enforcing isolation. Couple other things we should look at though because you could ask, well, what about if we tried to hack the file system pass? Again, this is enforced at the file system or rather at the framework level, at the harness level. So there's two different ways we enforce this. We talked about namespace scoping above. Let's take a look at path validation. This is another way that we prevent, memories from leaking. So let's see if we can jailbreak out of our paths using some common techniques. This should always be valid. Hopefully, this one will be invalid. Trying to jailbreak out of your path through the utility should be invalid, and this well, we'll see what happens here. If we run through these and we try to invoke, invoke them through the agent, what we'll see is valid paths are allowed, but we're unable to jailbreak out using, dot dot pathing. We're also unable to reference a user's home directory using the tilde shortcut, but we are able to use the tilde as just a a character literal in the middle of the path. Great. That was a lot about memory architecture. Architecture. I think we covered most of it. In the interest of time, I'm gonna leave the the compliance one for, for you to look at offline because there are some other topics that I do wanna make sure we go over. We have about thirty minutes left, and I wanna take another break for questions. So I'm just gonna pause here, and ask, Angeline, do you wanna maybe bring up a few more questions? Yeah. Absolutely. Give me one moment. Kinda share this question from Danielle. Okay. Hi, Danielle. Question is, how is the to do list derived, updated, and controlled across parent and subagent boundaries of deep agents? Okay. That's a great question. So the question is basically about how does the to do list work with sub agents, and how do they update each other? So what will happen conceptually is the user gives the supervisor agent, a complex a complex task description. The supervisor agent will break apart the complex task description into multiple components. It will then if I go back here to the basics notebook just go to the top here where we looked at the where we looked at this to do list for planning on how to bake a cake. You'll notice that in this to do list, every task is marked pending. What happens next is the supervisor agent will start iterating over all these tasks that are in the pending state. And if it knows that it has a subagent which can implement the task, it will delegate it to the subagent. It will flip the state from pending to in progress, and it will simply wait for the subagent to take that task. The subagent then will take whatever time it needs to do the task, and it will report back its results to the supervisor agent. The supervisor agent will then look at their the messaging from the subagent, and it will determine was the task completed or not. If it feels more work is is required, it will continue, delegating to the subagent until the task is complete. And as it goes, it will mark these from pending to in progress to complete, and it will go through them one by one. Now note that there are ways of doing asynchronous sub agents so we can execute multiple sub agent tasks simultaneously in parallel and have them all work on subproblems. But, generally, the delegating works, because reasoning models are good enough now to know, how to delegate to, to sub agents. I hope that answers your question. Again, Wei and I are available on the Cisco Webex. Happy to continue these chats, later. Great. Next question. How does agent memory relate to open knowledge format? I have to confess I know nothing about open knowledge format. It sounds super interesting. Matt, please reach out to me on the Cisco Webex, and let's continue the conversation there. I'd love to learn more about it. Next one, Angeline. Is the context window summarization configurable to be less than 85%? Yeah. I believe so, Remo. One of the things we haven't touched on, because we want the deep agent's harness to be configurable and model agnostic, is that we have this concept of a model harness. And the idea here is that a rather sorry. We have the concept of a model profile. The idea is that, not every model is trained the same way, and they may have slightly different tool calling conventions. For example, the OpenAI tool for editing a file is like patch file, whereas the anthropic tool edit is, like, called edit file. And the open weight models are even more diverse in how they've been trained. And so what DeepInches will allow you to do is when you provide a model, you can provide what's called a model harness. And the model rather sorry. I keep saying harness. A model profile. When you provide a profile, it will inform the deep agent's harness on some specific ways to tune its behavior for the model. Maybe one more question, Angeline. The question is, when is context summarization triggered in a multi turn use case? Is there a real context overflow error style summary? So context summarization in a multi turn case is no different from a single turn case. It's gonna be triggered when we hit a certain watermark, high limit there. Not sure I get the question about error style summaries and retry false facts. There this is generally not an error condition. This is simply a context management condition. So that's not really something that will be retried. Yeah. Sorry, Himanshi. I'm not really getting getting a question. Please reach out to me on Webex, and we'll try to dig into it more. Okay. Great. I wanna spend some time now and the time we have left talking about some more advanced features. And I'd like to go over code interpreters and programmatic tool calling. So there there's a spectrum here with the kind of power we wanna give to an agent. We know that agents are typically used to write code and execute that code. On the one hand of the extreme, you can have the agent write and execute code directly on the local machine, And it's very dangerous, but you can do it. On the other end of the extreme is we give every agent its own dedicated sandbox, and the sandbox is a safe environment to develop and execute code and maybe make external network calls. But there's actually a happy middle ground that's much more lighter weight and can be very useful in many cases. And so DeepAgents now comes with a built in JavaScript code interpreter, which allows you to use a technique called programmatic tool calling. So the interpreter gives the deep agent an in memory JavaScript runtime inside the agent loop. And what can happen now is that the agent can write code, and the runtime will run it, and only the final results get returned to the model. We tell the runtime, the built in JavaScript runtime, that it has, a list of tools that it's able to call from within the runtime. And this could be super helpful in speeding up repetitive tasks and reducing token spend. Let's let's take a look here. We're gonna take a look at this and see how it could be beneficial. So to set the stage, imagine that you have a repetitive task that you have to loop over. In this case, I wanna do research about coffee. I'm a coffee addict, so I wanna know everything that there is to know about coffee. So I have several themes about coffee that I wanna research, and I have a web research tool that I use to call out to the Internet to research my coffee topics. Pretty standard stuff. Now the naive and typical way of doing this is with direct tool calling. We're gonna baseline this by calling the LLM. The typical way is that you have one tool call per theme. You tell the LLM you wanna research it. You do the tool call. You wait for the result. You get that back, and then you you proceed on to the next one. So in this list of coffee themes, I have 12 different themes that I wanna research. So let's look at doing it the naive way. We set up a simple system prompt. You are a market research analyst for a coffee company. Research every theme the user gives you. We'll create our deep agent to pass you the model and our web search tool and our system prompt. We're then gonna simply invoke that model and see what it does. As we do that, it runs, and we can see that it faithfully does the 12 web search calls and has a whole bunch of input tokens, some output tokens, about 59,000 tokens, almost 60,000 tokens, and it took about thirty four and a half seconds. Okay. So this is the typical approach. The question is, can we do better? And the answer is yes. We can do much better with the built in lightweight code interpreter and programmatic tool calling. So what's gonna happen now is that we're gonna actually inject a new middleware into our deep agent. The code and oh, sorry for that. We're gonna inject the code interpreter middleware, and we're gonna tell the code interpreter middleware which tools it's allowed to call. We're gonna give it access to the web search tool. And once it does that, it's gonna write some JavaScript for us. Let's scroll down and take a look at this, and then we'll see how much more performant it is. We're gonna create our deep agent similar as before. This time, we're gonna give it a custom middleware, the code interpreter middleware. We have to tell the code interpreter middleware which tools it's allowed to use. So we're gonna specifically give it a PTC array. That's the programmatic tool calling, an array of which tools it's allowed to use. It also takes a few other parameters which you can tune about timeouts and such. What happened when we gave it this code interpreter middleware was our deep agent now gets a new built in tool called the eval tool. So the eval tool is now a tool built in to the supervisor agent, and he can use that to generate JavaScript code and invoke JavaScript code with its built in JavaScript runtime. We're gonna amend the system prompt. It's still a coffee researcher, but we're gonna tell it about the eval tool and that it has a web search tool only inside the interpreter. Notice what's missing here. I did not put the web search tool into the tools array of the deep agent. I don't want that to do that. I don't want my deep agent to call web search. I'm only putting it inside the tools array for the code interpreter middleware. I'm then telling it, hey. Research all my themes efficiently. Write one of our script that runs every search in parallel. With just a little bit of that prompting, let's see what happens. I run this and, sorry. I had to probably rerun this notebook. That's okay. This part is not important. We can see what the x what the agent wrote. Showing the eval code, this is the JavaScript that the agent wrote. It and it's applying a a pretty standard MapReduce paradigm here. So this hints at something. Programmatic tool calling is super efficient when we have a lot of very similar tasks, which we can fan out in parallel in a MapReduce style computation. We're mapping out all the themes, and we're calling the tools web search in them. And when we do that, what we can see is quite a significant reduction in token spend and clock time. This table shows it pretty well. The direct tool calling approach had about 56,000 input tokens and about 3,000 output tokens. The programmatic tool calling approach was went from 56,000 to 27,900 29,000 to 2,400. So we got a 53% reduction in the token consumption, and we accomplished the same exact task. We also noticed that we had a 13% reduction in execution time. Now in my testing, I've seen execution time reductions up to 40% and close to 50%. It will vary depending on the task you give it, but these are pretty impressive results. So if you have tasks which are highly parallelizable, I strongly suggest you consider using a code interpreter to map reduce them out in parallel. It will save you a lot of money, and it can save your users a lot of time. Okay. I do wanna spend some more time on one other topic today, but let me pause there and take some questions about the code interpreters and programmatic tool calling. Angeline, do you wanna bring some up? Yep. One moment. We have one from Ido. K. Ito. How do you ensure the code generated by the agent doesn't consume too many resources, for example, CPU, thereby degrading other flows running on the same machine? Great question. Let's take a look here at the code interpreter middleware. I believe that there are some controls here. We see a memory limit here, for example. We also see a time out parameter. So there are controls here for bounding what it can do. Yeah. It's a great question. So time out, max result characters. So this is gonna also limit the number of tokens that I can feed back into the model so that I can't, span the model with an outrageous number of tokens. And we did see that it has some memory limits as well. CPU limit, I didn't see there, probably because, if you really need CPU control, you probably need an actual sandbox. But great question. Yeah. Next question. Okay. Mohammed asks, can we configure deep agents to write files and offload context directly to an object store like AWS s three instead of using the local file system? Yes, Mohammed. We can. So two approaches there. We saw in the if we come back here to the deep agent's basics, remember this table here where I showed you some of the prebuilt back end stores, Postgres, Redis, Mongo. Note here that you can also subclass base store for your own durable back end file system implementation. So that's path one. But path two is actually to mount an s three bucket as a file system. And that's actually a really good segue into the the notebook on sandboxes. So why don't we spend the last little bit of time talking about sandboxes? And I'll give you a preview. I'll just scroll to the end. The preview is that we can we have something called sandbox mounts. And sandbox mounts allow us to mount cloud object store and GitHub repos into our sandbox as file systems. So let's take a look at sandboxes. Let's go up to the top here. So let's understand the motivation here. Sandboxes are gonna allow your agent to execute arbitrary code, access files, and make outbound network requests without compromising the security of your environment. They're super useful for things like coding agents and data analysis agents. When you configure your deep agent with a sandbox, it gets all the standard tools, plus it gets a new tool called execute. And this is what's gonna allow the deep agent to run arbitrary shell commands inside the sandbox. Deep agent supports several different sandbox providers out of the box. LangSmith, of course, the platform LangSmith has its own sandboxes, but we also support other sandbox providers. I've also seen this used with Kubernetes sandboxes. Let's take a look at how we set this up. We're just gonna create a couple of extra imports. And to motivate the example, we're gonna build a data analysis agent. I use this all the time, to be honest, because I get messy CSVs for my customers that I need to run through analysis. And, passing data files into an LLM just generally doesn't work great at scale. So let's see how we can build a data analysis agent. A common case that occurs to me often is I get CSVs from my customers, and the CSVs are often messy. To continue with our coffee theme because I I am a coffee addict, imagine that we have the CSV of, coffee sales and prices, and we'll see that it's a little bit messy. Sometimes there's a leading dollar sign. Sometimes there isn't. Sometimes there are missing values. Typical real world messy data. We could, of course, pass the entire CSV into the model and have it analyze it for us, but much more efficient to actually write code to analyze it. So we're gonna create a sandbox here. Nice thing about sandbox is is you can bring your own Docker image. In this case, I'm gonna be using the Python three two slim, but you can pass in any Docker image at all. To set it up, we're just gonna create a sandbox client, and then we're gonna call client dot create sandbox snapshot, giving it the name, the image, and initial file system capacity. Once that's done, we use that client to create a new actual instance of the Lanesmith sandbox. And once we have that, we can just start executing commands directly into the sandbox. So I can call make dir. I can do pip install, and, of course, I can upload and download files. So in this case, I'm gonna upload that CSV, encoding as UTF eight, and then I can just head the file to see that, in fact, it was uploaded. The next thing is we need to tell the deep agent that it has a sandbox. To do this, pretty standard, create our deep agent. Here in the back end, we're gonna use the sandbox back end. We also wanna tell the deep agent a little bit about our working directory is here and where it has an online and where it has the CSV of orders. We also say you're an analyst, and you're gonna clean the data. You're gonna work with it, and you're gonna write Python scripts, and you're gonna save your images to work in PNG format. Once we set that up, it's pretty easy to get started using the sandbox. Let's give it a let's call agent and invoke with a new request. We're gonna say, hey. My CSV data is here, and it's messy, and I need you to analyze it according to top products and monthly revenue. Finally, I wanna tell it to clean the data, compute some numbers, and save a chart. Let's see what happens when we invoke the deep agent. Here's my human prompt. The AI response. I'll clean the data. And what does it do? It gets started with its read file tool. Pretty typical. We've seen this many times already today on the file path for our CSV. It starts by inspecting the data, reads the entire thing, and then what does it do? It gets to work writing code, uses its write file tool to write a new Python script, starts importing pandas, matplotlib, and numpy, reads the CSV into a data frame, and starts its data cleaning and data analysis. It writes a lot of, ugly matplotlib code, which is great because I hate writing matplotlib. And at the end of it, it gives our our final output. Finally, it's gonna call that execute tool we saw. Execute is how the deep agent executes arbitrary shell commands inside of the sandbox. It CDs into our work environment and calls Python on the file. And great. We get the results we wanted. Right here, our data is cleaned and nicely analyzed for us. It's also saved the output of an image to chart.png. Now the sandbox always has download files and upload files tools. So we can just call download files to get that chart, and we can see nicely the matplotlib charts that our Python script created for us analyzing our CSV. Now skipping back ahead to the the original question which motivated this, which was, can we use s three? Yeah. Let's take a look at that in more detail. In the real world, if we were gonna do data analysis, chances are we have a lot of it, and it's probably living on something like s three. So rather than uploading files into the sandbox and downloading files from the sandbox, that's probably not practical. What we can do is we can simply directly mount a sandbox. We can directly mount an object store into the sandbox at a file system location. In this case, we'll use s three. We'll set it up in the standard way by using the standard, Bo to client, passing in all the standard parameters, region, access key, and secret key. For the purposes of this demo, I'm gonna use the put object API to put my sales CSV. But in the real world, you would already have your terabytes and petabytes of of data in s three, so you wouldn't need to do this. Then we're gonna configure the mount point for the deep agent, and we do that with a mount config. The mount config takes that auth object that we just set up, and then it sets up one or several mounts. Here, we're gonna set up a data input mount, and we're gonna say this is a read only input, so we can't write to that location, give it all the s three parameters that we need. We'll set up a secondary mount for data output, and this will be our read write place for our the results of the analysis. Having done all that, the only thing left is to mount s three bucket into the sandbox at at the file system. And so, again, we create that sandbox client. We instantiate it. And and we're able to now, directly execute on the location of the input mount and, read directly from there. And, of course, we can just output, right back to the output output, to the, output location. Okay. Why don't I pause there, for final round of questions? Alright. One here. How do we ensure the validity of the code generated does not impact efficacy by optimization? Well, Ultimately, you're you're responsible for the artifacts that you generate. They're they're I mean, you can always inspect the code that's generated. I I see more and more these days that, not every code that's generated by an LLM is reviewed by a human reviewer, but, ultimately, this speaks to your review process and your software development life cycle process. There there is no guarantee that it's gonna always generate the most efficient code, but I think the whole world is moving towards, you know, LLM based cogeneration. So, I think they've gotten very good at generating very good code. But, ultimately, it's up to you to determine what what constitutes good and what constitutes, efficient. Alright. The other questions relates back oh, actually, we have a new one here. Let me just go ahead and share this one. Tariq, do you have any references or case studies for large complex enterprises deploying PTC? No. I don't. I don't have any, any references for that. Be interesting to know, but I'm not aware of any. Is the PTC middleware ready for production? It was alpha until very recently. I'm not sure if they've GA'd it, but, yes, I in my testing, it seems, it it does seem ready for that. If it hasn't been toggled to GA yet, it should be pretty soon. How should we evaluate deep agents, and which metrics are most meaningful? Well, evaluation is is a whole separate topic. There is a notebook on that here in this, in this repo, which I will be committing later today. But there is no one size fits all metric. I can't tell you which metrics are most meaningful because what you want to measure is really a business outcome, and that will be different. Maybe there's some high level metric like hallucination, and maybe that's fairly universal, but, really, the evaluations are gonna be particular to your use case. So I can't say that which metrics are most meaningful to you. What I can say is that Lang Smith gives you a platform to structure evaluations in in a structured and methodical way. And you should have, a dual prong approach, really. One, you should have online evaluations, which are evaluations which run on a sample of your live traffic and check for online behavior of the agent. Secondarily, you should have offline evaluations. In LangSmith, we call these offline experiments. And the image here kinda sets up what it is, really. You want to curate some datasets with golden examples of what good looks like, and then run your agent through those examples and observing the outputs, running them through multiple evaluators. But this is a whole huge topic actually for an entire session, and I don't think that we can do, justice to the topic of evaluations in the three minutes we have left. So I would say if there's interest in doing a session on EVAOS, let's work with, let's work with the Cisco team, the leadership there, to to set that up. I'd be happy to do a session on that. Well, thank you. You've been a great audience. Really appreciate your time. Alright. Thank you all so much. Just a reminder that we will be sharing all of the recording, the notebooks, GitHub repo, and additional resources should be coming your way very shortly. Thank you all so much for joining us. Thanks, everyone.