Name: LangSmith 101
Uploaded: 2026-03-25T14:31:49.165Z
Duration: 59 min 40 s
Description: LangSmith 101

Transcript for "LangSmith 101": Welcome, everyone. We're going to get started in just a minute. We'll wait for a couple more minutes for people to filter in, and then we'll start the session. Alright. We're going to start the session. Thank you everyone for coming. My name is Robert. I'm a deployed engineer at LangChain. And today, we'll be going over how to debug, evaluate, and ship reliable agents. So like, the chat has mentioned, this session is being recorded, so you'll be able to revisit it afterwards or share it with your team. With that said, let's get started. Today, we're going to cover how to debug, evaluate, and ship agents, and we'll do it in the context of how you can accomplish it using Langsmith. So right now, every company is implementing AI. I'm sure this isn't really a surprise to any of you on this call. But especially in recent months, it's become more of a when, not an if. And so AI is being used in a lot of different ways. It's being used to generate more revenue per employee. It's being used to improve product experiences and to increase the efficiency of internal teams. And so I'm sure a lot of you on this call today are in the process of building with AI. So I want to start with a quick poll to see where everyone is in their customer journey. How far along are you in working with AI? Let us know, and that can help us shape the session. Cool. Looks like we're seeing some results come in. It looks like there's a lot of people that are both piloting AI initiatives as well as releasing agents in production. That's something that we love to see. I think six months ago, we would have seen a lot fewer people respond that they already have agents in production. So we're definitely seeing this shift, right now where a lot of AI products are increasing in maturity. Cool. So with that said, let's continue on to talk a little bit about how to effectively observe those agents once you have them in production or even while you're still working in a pilot. So one thing that we've learned at LangChain Building with AI is that it's very different from working with traditional software. When you work with traditional software, you can be pretty confident that everything you do is deterministic. You can look at the stack traces. You can look at the code, and you can understand the execution and why everything happened the way that it did. This all changes with LLMs. LLMs are fundamentally nondeterministic. And even from the very first LLM implementations, there would be nondeterminism and difficulty predicting what exactly would be output when you leveraged AI in your application. So the earliest implementations of LLMs really tried to constrain what the LLM could do. The most common architecture that was pretty prevalent was the Rag architecture, where you would have a retrieval augmented pipeline where the LLM would only be responsible for one small generative step. However, now we're seeing a lot more agentic use cases where LLMs are in charge of a lot of the application flow. LLMs are making decisions. They're calling tools, and they're making real changes when it comes to external services. And one consequence of this is that your app logic no longer really lives in your code. When you have an LLM that's making decisions, that doesn't show up in a stack trace, and that doesn't show up when you look at the implementation. You look at your code, and you'll see a call to an LLM. And then all of the rationale and the logic is offloaded to the model's reasoning. And so this makes it a lot more difficult to reason about why the application is doing what it is and why the agent is making the decisions that it does. And so the app logic for your application no longer lives in your code. Instead, it lives in your traces. And so the traces are really the core of what drives an agentic application. And LangChain is designed to turn those traces or logs into systematic improvement. So the flywheel that we see a lot of customers implement is that they will observe their agents in action. They'll see what decisions their agent is making and why, and then they'll build evaluations on that data to understand where their application is lacking and where they need to improve benchmark performance. And then they'll make changes to their application and deploy those back into production to then complete the flywheel and generate more observability data. And so this core flywheel, observe, evaluate, and deploy, is a pretty critical flow for a lot of these successful AI companies that we've seen. And so in today's session, we're going to talk about how you can use LangChain to build this flow into your own applications and to build it into your own workflows as you're developing agents. With that said, let's jump over to a demo of LangChain and the product itself. So let me switch my screen share. So what we have open is home page and its main interface. So in today's session, we're going to be using LangChain to show you how you can observe, evaluate, and deploy your agents. And we're going to break down each of the sections that is here on the left hand navigation side. So there's a lot going on here on the home page. Don't worry. We're going to be covering everything in detail. And we're going to navigate down through three main sections of topics. So the first section is going to cover tracing and monitoring, and it's going to be about observability, how to understand what your agent is doing and why it's doing it. The second section is going to be about evaluations, so that'll be data sets and experiments and annotation cues. This section will be about benchmarking your agent's performance and then using those insights to improve your agent. And finally, the last section is going to be around prompting. Prompting, as we understand it, is more of an art than a science. And so we've built several tools to help make that process of iterating on your prompt easier and quicker for your agents. So three main sections, gonna be observability, evaluation, and prompt engineering. So we're gonna start off with observability. And one of the ways that I like to demo how Langsmith works is to show how it works with a real application. So I'm going to switch screen shares again real quick to show you a chatbot that we have in production called Chat Linkchain. Chat Linkchain allows you to talk to Linkchain's documentation so it can answer questions about our open source and about Langsmith if you ever want to get a quick answer without having to search through the docs yourself. So let me switch screens real quick. So this is Chat Linkchain. And as you can see, it's a pretty simple rag interface where I can ask it a question like, how do I self host Blanksmith? And what Chat Linkchain will do is it will search through the docs. You can see it's taking some of those actions here in the chat window. And then once it's done gathering the information that it needs, it's going to generate a long form answer to tell me how I can self host Langsmith. Under the hood, we have integrated chat link chain with Langsmith itself so that everything that happens in this execution process is traced, and we'll be able to take a look at it in Langsmith itself. So we have our final answer generated here. Let's switch back to our workspace to see what happened under the hood. Cool. So we're back in Langsmith proper, and you can see that we have a couple of tracing projects that have been configured for ChatLinkedIn. So I'm going to click into this main ChatLinkedIn tracing project. And inside this project, you'll see that it contains traces or logs associated with every question that is being asked to chat LangChain. And I'm not the only one asking questions. This is an agent that we have in production, so you can see that the most recent messages are from someone else. My question is down here. How do I self host Blanksmith? And so in a given trace, we can click in to see exactly what happened when I asked this question. So on the main right hand side, you'll be able to see this input from me, how do I self host Langsmith. And in the output, we can see all of the actions that Chat LangChain took to generate that long form answer. So if I scroll all the way down to the bottom, by default, we capture the whole conversation history. But you can see the exact long form answer that it output in the main UI represented represented here as an AI response where it gives me instructions on how to self host, and it also gives me some sample YAML files. Now the most important part of a trace is not here what's on the right, but is this left hand sidebar. So the most important value of tracing is that it shows you exactly what happened under the hood when your application executed. You can see that as part of chat LangChain, we have a guardrail step as well as several tool calls to search the documentation. And under each step, we can see what Chat Link Train did to get its results. So this guardrail step is powered by an LLM step. And if we click into the LLM step, we'll be able to see the exact system prompt as well as the output associated with this guardrail LM. So we're using x AI to make this LM call. And in this case, the guardrail decision was that our question was not malicious. It wasn't prompt injection, and we're allowed to continue on with our execution. The same thing applies with everything that happens in our tool calls. So our AI model decides to search our documentation to help answer the question of how do I self host LangChain, And it is able to show up here in the trace where we can then take a look at every single input, output, and result of each tool call to understand whether or not LangChain is getting good results. And so the core way that you can use a trace is that you can take a look at the final answer, and you can make a judgment on whether or not LangChain did a good job. If you feel like LangChain did not give you the content that you needed, you can then step back through the trace and understand at every single step what happened and where LangChain went wrong. Now one thing that you might think when looking at a trace like this is that this trace itself is pretty long. And so it might be a pain to click through each and every one of these steps even when you have this nice trace view to show you what happened under the hood. So this is something that we've realized as we've built more long running, more complex agents here at LangChain. And so as part of the tracing view, we also have a Copilot that lives here in platform that can help you understand what happened in the trace. So if I go up to this top level trace, you can see that what the Copilot called Poly sees changes correspondingly. And so I can ask Poly a question to say maybe what tool calls happened in this trace. And Poly is able to take a look at the full context of the trace and understand which tool calls were occurred and gives me a quick summary, as you can see on the side, which tool calls happened and how many times they happened. So Poly is a way that you can quickly understand what happened in the trace at a high level without having to click through every step yourself. And this can be a really nice way, especially with longer traces, to understand where you should focus on as an engineer or as nontechnical SME taking a look and trying to understand why your agent failed where it did. So in this trace view, you're able to understand, at an individual level, exactly what happened when your agent ran. But most agents today are multi turn. Whether that's a chatbot or an ambient agent, usually to accomplish long and complex tasks, the agent will have to take multiple turns to complete different steps and evaluate its results. And so another view that you can look at within Langsmith is the thread view. So the thread view is a useful way to understand what happened across an entire session instead of just across one invocation of your agent. And so if we look at the thread view for LangChain, we'll be able to see entire conversations with LangChain and not just one question and response. So clicking into one of these sample threads, you can see that there's multiple traces and turns under the hood. And so in the turn view, we can see each human question as well as each AI output. And then we can scroll down to a later turn to see the follow-up question that the human had as well as additional outputs from our agent. So this is a nice way to see exactly what's happening across an entire session rather than an individual trace. But if you wanted to use the thread view to dig down into each individual trace that comprise the conversation, you're still able to do so by switching over to the trace view itself. And so you can see that we have all of these same steps. We have the guardrail. We have the model. And we have the tool calls here on the left hand side when we switch into the trace view. And so this allows Langsmith to surface the different execution paths that your application will take. And it allows you, as the developer, to understand what's happening in your agent and how to improve it. But one important thing is that a lot of what I'm showing you with the trace view is at a very granular level. You can look at individual traces. You can look at individual threads, but that doesn't scale when you have your application in production and you have large amounts of traffic. It's not really reasonable for me to scroll through every single trace, every single question that users are asking Chat Linkchain. I want to be able to understand at a high level how well Chat Linkchain is doing as well. And so one really useful tool that we have in Linksmith that can help you accomplish this is Insights. So Insights is the furthest tab here at the top on the right. And what Insights does is that it will show you the patterns, trends, and gaps associated with your agent. How it works under the hood is that Insights will take a topic that you specify, and it will run a custom clustering algorithm to show you different trends related to that topic. So for example, one trend that you might be interested in is usage patterns. So as the developers of Chat Linkchain, we want to know how exactly are users using our product, what types of questions are they asking Chat Linkchain, and are they getting good results from our agent. And so when we click into one of these insights jobs, we'll be able to see at a high level all the different categories of how users are using Chat Links chain. So one interesting thing here is that on February 25, you can see that the most popular topic that people were asking about was core integrations with our open source. But this was around the time that we did a big push on our latest open source release, which is Deep Agents. And so as you look throughout the week that this Insights job was running, you can see that Deep Agents shows up as a small percentage of questions that continues to grow over time. And so you can see deep agents slowly rise up the list in terms of how prevalent it is in terms of user questions. People are using Chat Linkchain more and more to understand how to use this new open source package that we released. And so this is a great way to get some insights into how people are actually using Chat LangChain at a high level and to understand trends. So because we see that Deep Agents is rising in popularity, it's a great idea for us to shore up our docs on Deep Agents and to make sure that users who are coming to Chat LangChain to understand Deep Agents have a really good experience. So more than just these clusters, you can also see a nice summary of exactly what's going on with the trends in text format here below. And if you scroll further down, you can actually see a breakdown of every single cluster and its subcategories here below with percentages and scores on how Chat Link Gen is doing. So some of the attributes that we're tracking include things like helpfulness, incident signals, and dock groundedness. And so under each of these categories oops, sorry under each of these categories, we'll be able to see within the questions about LangChain, the most common subcategory is LangChain concepts, and our helpfulness score is 6.3, which isn't bad. But one thing that might be concerning to us is that RAG is the second most common subcategory, and yet our helpfulness score is only a two point zero. And so this is another signal to us, at a high level, that we should probably shore up our docs around RAG because users aren't getting good answers from Chat LangChain when it comes to RAG related questions. We can even click into this subcategory to see the exact traces that are associated with Rag across this week period and to understand what happened in each individual trace and why Chat Link Chain didn't do a good job. And so when it comes to getting a high level insight over what's happening in your application, Insights is a great way to do it without having to manually crunch a lot of data yourself. And if you wanted to create a new Insights job, you could do so here in the new Insights panel, where you can choose the exact runs, the exact models, and the exact categories you want to choose for your clustering. So the last section that I want to cover within observability is monitoring. Insights is one way that you can get a high level overview of what's happening within your application, but it's not the only way that you can understand how your application is performing. If we click over to the monitoring chart, we'll see that you have a lot of prebuilt dashboards that will be included with every single tracing project you make in Langsmith. So as you can see, when we click over to the monitoring tab, we're here under dashboards, and we can see that we have a chart or a set of dashboards that is already precreated for Chat Linkchain. This is everything that you get out of the box. You don't have to do any configuration. We will track these stats for you. And so some of these things that you get for free with Langsmith is that we will track your trace count, your usage, your latency and error rate, as well as your LLM calls and cost. So you can see exactly how many LLM calls you're making, as well as your total cost and average cost per invocation of your agent. And this is even broken down by output tokens and input tokens so that you can understand whether your costs are coming from very long prompts on the user side or very long outputs from your agent side. Scrolling down, you can also see a breakdown of which tools your agent is using most frequently. This can be useful to identify where errors are coming from as well as which cases or or use cases from your customers are most popular. In this case, Chat Link Chain most frequently will search the docs before searching support articles. And so you can see similar stats on latency and error rate for your tools as well. And there's a similar factor when it comes to run types. But the most interesting stat that we compile for you is going to be at the very bottom, and this is going to be feedback scores. So these scores are going to be metrics that you can customize and configure yourself to track on your application. So for LangChain, we've built several custom metrics that we want to track. We want to understand, did LangChain complete the task that the user assigned them? Was LangChain helpful? Was LangChain answers grounded in the documentation that we provide? And we can even use custom metrics like question category to do a version of Insights Lite, where we can see which topic users are asking about, whether it's deep agents, evaluations, observability, or something else. And so all of these feedback metrics at the bottom are custom built by us, and those are something that you can configure as well, which we'll be covering in evaluation. So these feedback scores let us know over time, how is Chat Link Train doing in production? When it comes to real users using LangChain, is Chat Link Train being helpful? Is it completing the task? And is it giving grounded answers? So this is really the point of where we're going to transition into evaluations. How do we understand how to build these evaluations and these custom metrics so that we can have this high level insight into how our application is performing? To show you guys that, I'll be switching back into the tracing view where we can set up online evaluators. So all of those feedback charts that you just saw in monitoring are driven by online evaluators on your tracing projects. In LangChain, there's two types of evaluators that you can create. Online ones will run over your production traces. They'll run over your live data. And so you can you can configure them on the evaluators pane within any given tracing project. The second type of evaluator is offline evaluators. And offline evaluators are going to run-in development. They're gonna run over some curated data set that you've put together. And so we'll cover that next under data sets and experiments. But first, I wanna show you what it looks like to create a online evaluator on your tracing project project and to see what scores and feedback it gives you for your application. So we can click into this evaluators tab, and we'll be able to see a list of all of the evaluators that we have already configured for Chat LangChain. You might notice that helpfulness, dock groundedness, and which LangChain products were mentioned all were charts that we saw in our monitoring tab. And those charts are drawing from the feedback that these evaluators leave. So you can see that there's a couple of different types of evaluators that we can create. There's LLM as a judge, which runs using an LLM to score the output. There's custom code, which runs over code that you define, so deterministic output. And then there's also thread level evaluators. And this will run over a whole session to understand whether LangChain accomplished the task. And so to see what a evaluator looks like, we can click into one of these, and we can click into the edit view. So when you're making an evaluator, you give it a name, and you can set up a sampling rate for how many traces you want it to run over. So this is mainly for cost purposes. You could set it to 100% to run over all the traces in your project. But if you're operating at scale, it's usually more tenable to run it over a certain percentage. In this case, we use 10% of the traces that we want to sample and score for helpfulness. You can even set up some filters to be more targeted with this so you can run over very specific subsets of traces, like only traces where the latency is higher than ten seconds or only traces where the latency is really short or only traces where it has an error status or anything similar that you want to define. I'm going to leave it as running over all of our traces as just a general catchall for how helpful LangChain is being. And this is an LM as a judge evaluator. So the next part of this evaluator is that you can choose which model you want to use to grade your outputs. In this case, we're using GPT five nano. But if we click into it, you'll be able to choose any provider that you want, whether that's Anthropic, Google, Azure, or even your own internally hosted models as well. We can take any custom OpenAI compatible endpoint and hook it up here so that you can use it as an evaluator. After you've defined the model, then you can define a prompt that tells the LLM how you want to score your output. In this case, we're gonna give a rubric that contains instructions of how we define helpfulness. So we give some attributes of what a perfectly helpful answer looks like as well as what some unhelpful answers look like. And so then we'll take the inputs of our trace, and we'll take the AI's final output. And then we'll pass it as input variables for the LM as a judge to score. And so finally, here at the bottom, can see that we're tracking helpfulness as a score from one to 10, and we're including AI reasoning to understand why each trace got the score that it did. So I'll leave this evaluator unchanged. And we can take a look at another one of the evaluators to see that the general process of creating an evaluator will have a very similar interface. For example, if we click into this thread level evaluator, we can see that the general shape is very similar. So when we have an evaluator that runs over an entire thread, we can still set a sampling rate. We can still set some filters. But we do have this new field here of idle time. So by default, a thread level evaluator will trigger once there's no new input or conversation to be had. And you can set the time that you want to run as the trigger. So in this case, if there's been no activity for an hour, then we're going to refresh the evaluator and rescore the conversation. But, otherwise, the evaluator will look very similar where you can still choose a model to do your scoring. You can still give it instructions. And in this case, you can pass in which parts of the conversation you want the LLM to score. So this thread level evaluator will score all the human AI responses in the session, but you can also choose things like all messages, which is a nice way to be able to filter on showing all of the tool calls, all of the individual steps that the AI takes to generate its answers as well. In this case, we're gonna leave it as human AI pairs to prevent some of those messages from adding noise to our evaluation. But once you've configured any of these evaluators, as you can see here, you'll then be able to see their feedback in the traces themselves as well as populating the monitoring dashboards that we have already showed you. So in any individual trace, if we scroll over to the right, we'll be able to see a feedback column. And since we're sampling traces, not every trace will have feedback on it. But if we scroll down, we can see that this trace has a doc groundedness score, and it has a helpfulness score. So it's been sampled by our evaluators, and we can see how well LangChain was doing to answer this particular question. So this user asked, what role should I assign a key? And LangChain had a helpful answer that didn't pose an incident but was not grounded enough in our docs. So setting up online evaluators, the usefulness of it is that it lets you understand how well your agent is performing at a high level. It lets you understand at a bird's eye view whether or not your agent is being helpful, whether or not your agent is grounded. And, it'll populate those monitoring dashboards for you to be able to track in a nice chart form. But the highest signal still comes from human review. Online evaluators using LM as a judge can scale to levels that a human could never reach. But to have really high quality insights, it's still useful to have a subject matter expert take a look at some traces themselves. And so this is going to tie back into what we were talking about with the lifecycle of agent development. Once you have the traces here populated from your agent and you've set up online evaluators to generate scores on them, you can use those online evaluators to filter down which traces have high signals for your human annotators to review and add to curated development test sets. So the goal here is to take all of the tracing data that Langsmith captures and to leverage it so that you can use those traces to improve your application. The most common flow that we see is to take all this tracing data, to set up online evaluators to see which traces are useful, and then to send those useful traces into curated datasets to run tests over. And so I'm gonna show you what that process looks like in Langsmith. We support it through a feature called automations. And so in automations, you can see some of the ones that we've set up here. You're able to create a rule that applies to any type of trace that you like. So in this case, if I create a new automation, we'll be able to see that I can give it a name. So I can say helpfulness. And we'll be able to select a sampling rate and a filter as well. And this is where it's useful to use a filter on your online evaluator scores. So one of the things that you can filter on for an automation is how well your trace has scored on any evaluator you've configured. So we had the helpfulness evaluator that we took a look at earlier. We can filter down to only traces that have been scored as unhelpful, which we can say as a score of less than three helpfulness. And so only these unhelpful traces where LangChain did poorly do we want to have a human annotator to review to find out exactly what went wrong. And to do so, we can set up this automation to send these unhelpful traces to an annotation queue or to a data set. And so this lets you filter down the amount of data that you need a human to annotate to be something that's more scalable, to be something that's more manageable when it comes to however many annotators or reviewers you have available on your team. And so if we choose to set up this automation to send unhelpful traces to an annotation queue, we could then take a look for a human reviewer and see how they would be able to review this trace and curated it so that you can then apply that trace as part of your experiments. So I'm going to skip this rule because we already have quite a few automation rules set up for LangChain and jump down to annotation queues where we can see some of the reviewing queues that have been populated by our automation rules. In this case, if we take a look at this golden annotation queue, we'll be able to see all of the traces that have been sent for grading by our reviewing team. So in this case, we have a sample trace. Here's the input where the human has a request, and the output is going to be everything that the AI did to return an answer. And here on the side, you can see that our graders have instructions on how they should annotate the queue. They have a score that they're meant to give, so I can give this a score of correctness as being, I think that this answer was correct. And I can also look at some of the notes that other users have left here, and other reviewers have commented. So Vic thinks that this is a great trace, and Andrew also agrees with him. And so I could label this trace as correct. And if I felt like this was a good trace to train on, I could then add it to a dataset from the top bar or using the hot keys shown here. And so the core sort of flow here is that, as a reviewer, you have your instructions, you have your grading input, and you can take a look at all of these traces. You can even edit the traces if you need. And you can score whether or not it's a useful trace to be used as part of your agent's improvement cycle. So if I think this is a good trace to be used as a curated test set, I can add it to a data set, and then I can move on to the next trace to review. And once it populates a dataset, you'll be able to see it shown here under datasets and experiments. So this is going to be the last section of our evaluation section. We covered observability, and we're nearing the end of evaluations. In this case, we'll sort of close that development loop of how you can then use the insights gained from your traces to actually improve your application. So this is going to be where we're covering offline evaluations. Offline evaluations are what you run-in development to improve your agent. And in an offline evaluation, it'll generally be based around a curated test set we call a data set. If we click into one of these data sets, you can see that we call this one a golden data set because it contains examples that we think are really useful to grade Chat Link Train on as a benchmark. So if we click into one of these examples, we can take a look at what that means. Each example consists of an input that we think is important to record performance on. So this might be a frequently asked question to chat link chain, such as, you know, break down these topics and explain them to me, state machines, nodes, edges, and conditional routing. And we also have a reference output. So this serves as the ground truth benchmark we want to compare chat lane chain's performance against. This is what we would consider a very good answer so that we can use it as a comparison when grading Chat Linkchain. In this case, you can see that the answer has a long form text format, and it also includes helpful code snippets to break down each of these different topics a user has asked about. So in the data set, we have lots of different rows that consist of these input reference output pairs, where each input is a frequently asked question, and the reference outputs are the ground truth golden standard for a good answer to that question. And what happens in a offline evaluation is that you'll be running your application, in this case, Chat Linkchain, over these inputs. And Chat Linkchain will generate its own outputs that you can then compare to the ground truth. And the way that you compare it to the ground truth is, again, using evaluators. So if you look at the evaluators here, you'll notice that it's very similar to online evaluators, where you can configure an LM as a judge. You can configure custom code evaluators, and you can do all of the same things that you can do online but in an offline experiment. So you can see when we offline evaluator, you still have the option to choose your LLM model. You still have the option to define the prompt for this LLM as a judge, and you have the same format for how you're going to leave and receive feedback. And once you've configured the evaluators, you can then run experiments. So experiments are going to be what actually runs Chat Linkchain over the dataset and produces those outputs that get scored by the evaluator here. So you have a couple of options. You can run-in the UI using our playground interface or our studio interface, but the most common option is to run-in the SDK using code. And so if we click this option, you'll be given some sample code snippets that you can use to then run your offline evaluation and interpret the results. So you can see we define the dataset here, define some evaluators, and you call an evaluate function where you pass in your application, in this case, LangChain. Once you've run the experiment, your results will show up in this main view. And so you can see we've run several experiments on LangChain, each with a different model. So GPT-five, GPT-five-two, Claude SONNET, and Claude Haiku. And so one of the things that you can do here is that for each experiment, you can choose whether or not to set it as the baseline. So if it's not set as the baseline, you can see that all of the scores that we track, such as accuracy, answer relevance, and code checking, are shown in plain text. But if we do set a baseline, we can see how they all compare against the baseline experiment. So in this case, if GPT-five minutei is our baseline, you can see that Claude Sonnet has a higher accuracy score, a higher relevance score, and a high code checking score as well. And so all of these stats are also shown in this chart so that you can track it over time and compare it to latency and token output as well. So one useful thing is that Poly, our Copilot, also lives here in the experiments view. So I can ask Poly, which model or experiment had the best results? And Poly can also help you filter through some of the signal here to understand which experiment or version of your application performed best on the given data set. And so the core way that you can use offline evaluations and data sets is that every time you make a change to your application, you can rerun it on the data sets that you've already defined. So if you change out the prompts, if you change out the models like we do here, or if you change out the architecture, you can see whether or not that actually improves performance on your data set. And in this case, what Poly is telling us is that Claude SONNET 4.6 had the best results. It had the highest accuracy, answer relevance, and code checking score. So we can also see that looking at the experiments below, where every time we change out one of the models, we can see whether our accuracy, answer relevance, and code scores go up or whether they go down. And that helps us understand which model is the best to choose for us. It also helps you understand which prompt might be the best if you're experimenting over your prompts. And if we wanted to do a deep dive on any particular comparison, we can actually select multiple experiments and enter a special comparison view. So if we're looking at these experiments side by side, you'll be able to see what the input question was, what the ground truth answer was, and what the actual outputs were generated by your application. So in this case, we have two outputs, one that's generated by SONET and one that's generated by GPT 5.2. And we'll be able to compare those outputs side by side as well as how well they scored and why SONET, in this case, scored better for all of the metrics that we're tracking. And so this will show through for every single row of the experiment so you can do a deep dive if you want. So that wraps up the evaluation section of Langsmith. And to recap what we've covered so far, we've covered observability. We've covered evaluation. And we've covered how you can use them as part of a life cycle, where you trace what your application does. You log it into Langsmith. And then you use those traces to benchmark your application, both online performance and offline data sets, so that you can continuously build this flywheel of improving your application, testing it, releasing it into production, and generating more traces. Now the last part that I want to cover is prompt engineering. So this is going to be the prompts in the playground section of LangChain Langsmith. And the part of where this fits in into the life cycle is it helps you speed up the iteration of improving your application. So once you have an evaluation, you might understand, okay. Chat LangChain did bad on these types of questions. Your next step is to adjust your application, Chat LangChain, to make sure that it can answer those questions that it previously missed. And one way that you might wanna do that is by changing the prompts. And so the playground interface in Langsmith is a interface that lets you quickly adjust the prompts and see how it affects your outputs when you run it with Chat LangChain. In this case, you can access the playground just by clicking into this tab. But one way that I really like to use the playground is to enter it from an existing trace. So if I click a trace that already exists in my project, I can go to any LLM call, and I can actually pull this exact LLM call into the playground interface with the same inputs and outputs. So if I click this playground button, you'll see that it's pulled that exact system prompt and output here into the playground interface. And now that it's here, I can change whatever I want about the prompt and see how it affects the output. So I could say, don't want to use Grok. I want to use GPT five four. And maybe I think that this prompt is too long, so I'm going to delete parts of this prompt and then see how it impacts our guardrail decision. So I could click start with my changes and see if everything is still allowed. And you can see that we do still allow this question even with a more lenient guardrail prompt. One nice thing is that Poly also lives here in the playground, and you can ask Poly to help optimize your prompt for you as well. So I could say, please make our prompt a little more concise. And Poly would be able to adjust the prompt here if you are having trouble coming up with your own improvements to the prompt. So the last thing that I'll cover here in the prompting section is that after you've made these changes, you've seen how the change affects the output, you feel like you really like this prompt that you've made, you can also save it into the prompt hub. So any change that you've made to your prompts, in this case, Polly is currently editing this prompt to make it a little bit more concise, if we feel like we like this result, we can then save it into the hub. So I'll save it as a test and then be able to use it in our application. So what this does is that we have now saved this version of our prompt into the PromptHub interface, which gives us a lightweight version control experience so that every time we make a new change, it'll generate a new commit of this particular prompt. And you can see that our prompt is saved as a template that we can use a code snippet at the bottom to pull directly into our code. This is something that I really like doing in development especially because this way, I don't need to rerun my whole application every time I change a prompt. Or if I change a prompt in the playground, I don't need to redeploy my application. I can have my application pull directly from the Prompt Hub and get the latest changes from Langsmith. So I've spent a lot of time talking about Langsmith. We've covered a lot of the features here. Haven't actually covered everything that Langsmith has to offer. There's even more, like deploying your agents in one click or a no code builder called Fleet. But we've covered the core life cycle of how you can observe your agents with LangChain, understand what they're doing under the hood, evaluate your agents to then benchmark them, and then use the prompt engineering tools to improve your agents and redeploy into production to generate more traces and complete that flywheel. So at this point, I'd love to open it up for any questions that you guys might have, so that I can answer anything that might be top of mind for the audience. So I see first question in the tab with evaluators. There could be three types of evaluators: LLM as a judge, thread level evaluators, and code. So this question is asking about the code evaluating agent. How does it work? What code can go in there? And, yeah, I think the easiest way to answer this question is I am going to go back into LangChain and show you what the code evaluator looks like. So let's go back to LangChain. And we can go to chat link chain and create a new evaluator. So the types of evaluators that you can make are LLM as a judge, where you have an LLM score. You have some prebuilts down here. You have thread level evaluators, composite evaluators, and code evaluators. So I'll cover these two, which I didn't go into detail in the presentation. But a code evaluator is just writing your own function to compute the scoring metric for you. So under the hood, every evaluator is just a function. And if you want, you can write your own function to understand how well your application did. So what will happen is that this function, perform eval, will take in the run object. So it'll take in your trace. And you can write your own function that calculates some sort of metric on that trace. Maybe you're interested in if a certain keyword showed up. Or maybe you have a PAI checker so that you have some regex detection that you want to write. All of those are things that are very deterministic and you can define directly in your own code function. So you would be able to write that function here, and you'd be able to access the outputs and the inputs of your trace. So writing that function can be done in Python or JavaScript, and you'd be able to test the function on your traces as well. So in this case, instead of using an LMS judge to score your output, you would be using your own custom written function to do that instead. One thing that's notable with code evaluators is that for security reasons, not every Python library is supported. We support the main standard ones like NumPy or like Regex or similar ones that you can use, but we don't support arbitrary packages to be imported in your code. Cool. With that said, let's go back to see if there's any other questions. Can LangChain be integrated with other agent building tools such as Copilot Studio? Yes. LangChain can be incorporated with any sort of agent that you want to build. LangChain is actually completely framework agnostic, so you don't need to be using our open source. You don't need to be building with a specific framework. We integrate with a lot of the popular frameworks like Google ADK, know, Crew AI, Symantec Kernel. And we also support OpenTelemetry so that no matter how you're building, you'll be able to trace effectively into LangChain. Let's see. Next question. How do traces in LangChain help identify failures in prompts, retrieval, rag, tool usage? And given a failed trace, how would you isolate the root cause? This is a great question. Generally, failures happen in one of two areas. It either happens in an LLM call or a tool. So if I switch back to LangChain, let me stop sharing, I think. Yeah. If I switch back to LangChain, we'll be able to take a look at a sample trace, and I can walk through what it looked like to evaluate this. Looks like that's not a super useful one. So given this trace, what I would do as a user is, first, I would start at the end. I'd start at the final output that the AI generated, and I'd understand, okay. Do I think this is a good answer or not? If I don't think this is a good answer, I'd take a look at what the agent did to generate this answer. So one thing that it does in this case is that it searches the docs. I would inspect these tool calls to understand, are these docs useful? And so one thing that's kind of relevant here is Polly, the Copilot, because this doc has quite a bit of text in it, and it's not formatted in the easiest way to read. So I could ask Polly, hey. Does this doc look right? Is this doc relevant to the question that the user asked? And if not, I would then be able to dig in further deeper into this doc and say, okay. If this doc isn't right or if the content is bad, why was it retrieved by this input query? Was the query wrong? Was the tool implementation itself wrong? Or was the content mostly correct but just worded in a confusing way? And so what I would do is I would step back through each of these actions that the agent took and understand whether each action is helpful or unhelpful. So something that's also useful is that in each LLM call, you can see the system prompt, and you can see the output that the agent is taking. In this case, the agent is trying to search the docs with specific queries. And it even gives some logic in terms of why it's doing the tool searches that it does. So I would also read through these AI reasoning traces to understand, does this feel like this is valid logic? Do I think the AI is making the right decision, or do I think it got confused with the information that it was given? And so, again, in long traces, it's very useful to have Poly to be able to help you identify where to dig deeper within a lengthier trace as well as to help summarize some of the actions that your agent took. But going through this process of searching through the steps will tell you whether or not it was a tool call that was wrong, whether or not that it was an AI like call that had poor reasoning. And it'll essentially let you trace down to the root cause of any given issue, whether that's RAG or an incorrect track trajectory or something else. So one thing that might be relevant is that even just looking at the top level, you can see what order the tools were called in. And you can see in our waterfall view which tools were called in parallel. So you can understand, okay. Maybe the agent is calling the right tools, but it's calling them in the wrong order. And if it's calling it in the wrong order, then there's probably some logic that it has in the AI call itself, the LLM call, that will help help us understand why it's calling these tools in the wrong order. So that's kind of a investigative process that I'll take on any trace that I feel like is a good example of bad output from chat link chain and I want to dive deeper into. Question on what is Langsmith's charging model? So Langsmith is charged based on usage, based on credits. So Langsmith will charge you based on the number of traces that you generate, which means that you only really get billed for what you actually use. And each trace itself can be as long as it needs, and it still only gets billed as one trace. So next question, who created the golden dataset that I showed on screen? So our team created the golden dataset, the engineering team did, with the help of using the automations that I showed within Langsmith. So our process is also to use that feedback filter, so to use the case of where we're getting all of the traces, where our online evaluators have scored as helpful or have scored as unhelpful or have scored as grounded. And then we're using that to populate an annotation queue. And from the annotation queue, our team will then populate those data sets. Next question. How do you know if any good traces have been missed out as you only review a selection of traces? So the core nature of sampling is that you will miss some good traces. But a big part and a big value add of having online evaluators configured is that you can have confidence that for really good traces that offer a lot of signal, you'll be able to review some of them and to draw insights from. Now a big challenge is that there's a lot of insights across the total volume of your application. There's only so much of those insights that you can actually manually review. And so that's part of why having this funnel with both online evaluators to get an automated sense at a high level how your application is doing is a good supplement to having manual human review, which is really designed for more curated targeting testing. And so when it comes to the sampling process, it's sort of inevitable that you'll miss some traces. But that's part of why online evaluators exist in the first place, so that you still have a broad picture of how well your application is doing, even if you're not diving deep into every single trace. Next question. What is the accepted format for a bulk upload of the evaluators if I already have a golden dataset? So we do have an API reference that you can use to upload from the SDK. We can send that those docs after the fact. But, yeah, like, that reference will show you how to bulk upload any evaluator without having to go through the UI so that you can programmatically manage things if you want. Next question. What metrics does Langsmith provide? So Langsmith covers a lot of metrics out of the box. I showed quite a few in those monitoring dashboards. There are things like cost, LLM calls, tool calls, run type, your overall, like, latency and error metrics for all of the things that I just mentioned. But you are also able to create your own custom charts as well as your own custom metrics using online evaluators like I showed. So what this means is that we do give you a lot out of the box, so your basic metrics in terms of, like, usage and latency. But if you have anything specific that you need, you're able to configure your own evaluator and your own metric to then track it in those charts. Cool. Next question: How could you trust an LLM as a judge to evaluate other agents? I love this question. This is something that we've thought a lot about, and it's part of the reason why we have annotation cues in the first place. So I know we're almost at time. I'll share my screen again one more time to answer this question, and then we'll see if we have time for a couple of of last ones to cover. So if I go into Langsmith again, it's a great question on how you can trust an LLM to generate good signal if it's grading another LLM. And so the answer that we have to this is that you should align your LLM to human judgment. So you can have confidence in an LLM if you have confidence that it grades the same way that your human graders would. And there's a way that you can test this using a flow in Langsmith called sorry. A flow in Langsmith called align evaluators. So when you create a new evaluator, one cool option under the LLM as a judge evaluator is to create from labeled data. What this means is that when you create this evaluator, let's just call it correctness, you'll actually be able to sample runs to annotate as a human. So as a human, I could select, let's say, five traces to annotate. And under the hood, this is going to build an annotation queue for me that you can then add more traces to to increase your confidence even further. But once I have this annotation queue, I can grade as a human whether or not I think this output is good or bad, if it's correct or if it's incorrect. And so I could say this output is correct or sorry, incorrect. I could say this output is correct. I'm gonna grade a couple of them just randomly in the interest of time. And once you feel like you've graded enough examples, you can then enter a playground experience where you can test your l m LLM as a judge prompt. And so what happens here is that all of the outputs that I graded, you can see my score here as the human score. So I thought this trace was incorrect. I thought this one was also incorrect, and I thought this trace was correct. But if I have an LLM as a judge prompt, I could say grade grade if this is correct. I could test this prompt and run it against my own scores. So the goal of what you would want to do here is you would want to adjust your LLM as a judge your LLM as a judge's instructions until it grades the same way that you did as a human. And so you can see, in this case, the LLM agreed with me, and that's a good thing. Right? In this case, we're just doing things kind of randomly. But you would use this process to iterate on your LLM as a judge instructions until you had confidence that it's grading the same way that your human graders would. And in that case, you would then feel more confident that you're getting real signal out of the LM as a judge and not just noise. So awesome question. I know we're basically right at time, so feel free to always follow-up with us after the fact. Happy to answer any other questions async with the help of our community. Cool. With that said, thank you everyone again for attending. Thank you for the engagement and for asking lots of really great questions, and we'll see you again in our next session.