Transcript#
This transcript was generated automatically and may contain errors.
Okay, I'm going to talk about effective agents. And what I really mean is I'm going to talk about what it takes for you to build something like an agentic AI system. And again, whatever I mean by that, I mean, there's so much AI in the world, I'm talking about the kind of thing where you have a conversational interface, and some AI agent will do something on your behalf. And then maybe you'll have a follow-up conversation, and you'll continue to direct the AI to do things. Think cloud code, think co-pilot chat, that kind of stuff. If you want to make your own one of them, this is the talk for you.
And the reason I know about this is because I work for Posit, and we've been working on Posit Assistant, which is an AI assistant that's now available in our studio. So we now have the mythical fifth pane on the left-hand side. You can talk to Posit Assistant. It can run our code for you. The assistant can see your plots, and you can do the kind of things that you can do with something like cloud code, but in our studio, and it's really designed to work well with the kind of workflows that you will be doing in something like our studio. I'm talking exploratory data analysis, creating reports, creating dashboards. We've really thought very carefully about how to create an AI assistant to help with that kind of work.
And usually what that actually means is that a lot of the time we thought, when should the agent bring back a human in the loop? That's kind of what we've been thinking about. So not async agents, which we've talked about today, not one shot back and forth, but a conversational AI agent. Posit Assistant is also available in Positron. Positron actually had an older agent, Positron Assistant, for a little while now. If you haven't used Positron's AI stuff in a while, go back, try it again, because we've rebuilt the entire thing from scratch. And that's kind of what this talk is. We rebuilt the thing from scratch, and I'm going to tell you everything we learned in the process of doing that.
How to build your own AI agent
So you want to build your own AI agent. How does one do so? You could do this. You could use an agent SDK. Now I'm assuming for the rest of the talk that you're not doing something like self-hosted models. You're using frontier models on a provided service. You have some kind of platform already set up with guardrails or whatever else you need. You have access to a good model. That might be eventually hitting open AI. It might eventually be hitting Anthropic. But that's kind of the agents that I'm talking, the LLMs that I'm talking about.
Now, if you decide to use these tools, there are some good parts and bad parts. The good part is that's very easy. You get an AI agent ready to use. It's got tools built in. It knows how to do things. It's easy to set up. It's easy to integrate into other things if you're an engineer. But the downside is you're now locked into Microsoft or Anthropic, depending on which SDK you use. In the end, these kind of tools, they're not that popular. You can use them, but you're kind of locked in. They're also hard to customize. Obviously, this is important for Positive Assistant because we want a very customized data science experience. So we can't use an off-the-shelf SDK where we can't modify the tools or the tool loop.
So instead, we've decided to build our own. And there are a few ways you can do this if you want to do this yourself. If you're an R user, you can use Elmer. If you're a Python user, you can use Chatless. We built Positive Assistant in a programming language called TypeScript. It's similar to JavaScript. And we use the Vercel AI SDK. And what these do is they let you talk to model providers in a standardized way. So you can talk to Anthropic or OpenAI or Google, and you can use the same primitives as you would over all the various different providers. And it means that you don't need to know the specifics of how to talk to each provider.
Fundamentals of talking to an LLM
Now, despite that, I'm going to take some time to talk about how you would do that. I want to talk about some fundamentals of how you actually talk to an LLM via a Frontier provided service. And the reason I want to do that is because everything you need to know about creating an AI agent that works autonomously all comes down to these fundamentals in the end. It's not magic. It's repeated loops of talking to an LLM provider.
For a while, we're going to get a little technical, and I'm going to talk about what happens when you type hello world into a box and you hit the send button, what does it look like? So this is JSON way of formatting data. And when you talk to one of these AI models, you're sending formatted data over the network, over the internet to their provider service, potentially via some kind of guardrails or proxy service and stuff. But at the end of the day, it's going to hit Anthropic or OpenAI or Google. And here I'm using what you would send if you were talking to Anthropic. I'm skipping some bits and pieces, simplifying, but it essentially looks like this. You choose a model, you set a system prompt that controls the behavior of your agent, and you send a user message, what the user said.
Now, when you do that, you will get a response back and the response will look like this. Again, I don't want you to kind of worry about all of this stuff. What I want you to pay attention to is this content block. You'll get back some content from the assistant that looks like that. Now this could be streamed in. Again, there's complications, there's IDs, there's sort of stop reasons and usage information. Fundamentally, you're getting a response from the assistant. And then that's kind of the rest of what we're doing is just back and forth, requests and responses, requests and responses with additional bits tacked on.
Now, if in the old days you use web APIs before the advent of LLMs, you might think, okay, this is a conversation. I want to continue that conversation. I want to ask another question. How do I do that? And you might think, oh, well, maybe I just pass this ID and I pass another user message. And it turns out that's not the case. The fundamental thing you need to remember when working with AIs at this level is that you're sending the conversation over and over again. The entire history of that conversation gets sent every time the assistant makes a reply. And that's kind of weird. It's a bit different to how a lot of web services used to work, where you'd have something like a cookie or a session to track things over time. No, you send the conversation every time.
The entire history of that conversation gets sent every time the assistant makes a reply.
And then two things should immediately flag to you when I tell you that. One, if you have a long running agent or a long conversation, surely this gets very long and you have to send a lot of data. Yes. And the other thing that should shout out to you is that when you do that, after a while, you are paying for all of these tokens again and again and again. And every time that someone sends another message, you get another message at the bottom of the list. It's an append-only data structure if you're having a single linear conversation. And all that's going to happen is right at the end, you're just going to change this part. And everything else is going to stay the same. Well, you pay for it all anyway. You pay for it again and again. That's going to become more important later when I talk about caching.
The agentic loop and tools
When you're using an AI agent, it has a way to interact with the world. We call them tools. A tool description looks like this. You have a name, you've got some text that describes it to the model, and then a way to tell the LLM how to invoke this tool. And again, if you've used something like Elmer or ChatList before, any of these tools that exist for languages, you know all this. You just do it in a different way, a much nicer format. So a full request with tool loops, tool use block here. The model is opening a file. That response comes back to the model, back and forth. Kind of looks like this.
Now, I'm not going to make the rest of the talk full of JSON blocks because I can't make you look at that any longer. I'm going to use a schematic view. And what I want you to just get an idea for is what we call the agentic loop. This is the lifeblood of an AI agent. You have some kind of setup. You have a user turn. And then each user turn is prompted by someone has sat, typed a question in a box and pressed enter. That's a turn. Now, because inside a single turn, an assistant might use some tools, every time the assistant uses a tool, barring things like parallel tool calls and various subtleties, but generally every time a tool is used, that's a single back and forth because a request has come back to say, please use this tool. And the value of the response has gone back to the model. So back and forth and back and forth. And that's the loop. Read the assistant message. We check for tool calls. We run the tool calls. We send them back. We check the assistant message. We check if there's tool calls. We run the tool calls. We send it back. And we do it over and over again. That's the agentic loop. And then eventually the assistant will stop and it'll say, okay, I understand. Here's your answer. And then we wait for the next turn. And that's how things like Posit Assistant work, that's how Claude Code works and Copilot Chat.
Lessons from building Posit Assistant
So now that we're all experts on fundamental messaging APIs, I'm now going to talk about some of the learnings we had building Posit Assistant so that I can share it with you. So the first thing I'll say is that Claude is a liar. He will bare-faced lie to you again and again. And, you know, we've talked about hallucination. Hallucination, actually, you know, it's getting better. It still happens. It's not as bad as it was. But this is kind of a more subtle problem that still exists.
We have a system in Posit Assistant so that when the model runs a piece of R code that produces a plot, that plot is sent back to the model. So the model can view that plot using a system called Vision. And most frontier LLMs can handle images these days. But on several occasions during development, we've broken this feature and we didn't realize that we'd broken the feature. And the reason why was because the model is just so, just convincing in its responses. It was like, surely it's not broken. I can see.
And I'm going to show you an example of this. I've hooked an agent up into the slide. So here I've got an example asking the agent to plot something, a horsepower mass per gallon, famous empty cars data set. And I'm going to ask it to describe what it sees. So I've sent that off. Okay, so it's trying to run some tools. It's running a piece of R code and that R code's executed and some result images sent back. The assistant said the plot shows a clear negative relationship between horsepower and fuel efficiency. Okay, brilliant. That's right. That's doing its job, except watch this.
Yeah, what I did is I made it so that literally the word image is returned, not the image data. And it took me to ask, ask the model, you know, did you really see that plot? No, it kind of just used what it expected to see because the empty cars data set is such a well-known data set. It knew what the plot would look like. And that's what it described. Now I knew this was going to happen. Your users might not know that's going to happen. How would they know? There could be subtle biases introduced into the data because this kind of thing happened. So these are something to be aware of that. Yeah, Claude will lie to you.
And in fact, we at Posit, we've been trying to understand how this works. We've done some deep dives. And what we've really found is that LLMs really do see what they expect to see. And this is a really interesting article. If you're interested in that, Simon and Sarah, they take famous data sets and flip them. So this is empty cars. But here we've got as horsepower goes up, the miles per gallon goes up. And that's not right. What they've done is they flipped the plot. Well, they flipped the data and then plotted it. And they send that to the LLM and they ask, can you describe this plot to me? And overwhelmingly at the time, what were frontier models will get this wrong. They will describe that image wrong. They will say always horsepower goes up, miles per gallon goes down because that's what they expected to see. And you can see here that most models failed at this task. So yeah, do keep that in mind.
What we've really found is that LLMs really do see what they expect to see.
Always, always, always, if you're building AI agents, check the raw messages, check the network. Don't just ask the LLM why it did something. Like, oh, why did you do this? Why did you do that? What were you thinking when you returned this? LLMs are really bad at introspection. They are not good at analyzing their own conversation history. It's actually possible to sort of forge that history, because you're sending the conversation history every time. You can forge this and make the assistant say whatever you want. And then if you ask it, why did you say that? It will say something that sounds convincing about why it happened. So you gotta be really careful. Don't ask the model why it did something. Check the raw data underneath and really see if you can figure out what's happened that's returned some weird result.
Top tip, if you can, use a man-in-the-middle proxy. If you don't have access to the raw data, for whatever reason, and you're doing this on a machine you control with admin access, then do try out these tools. ProxyMan's a great one. There's CharlesProxy and some open source stuff. They'll let you view the raw traffic. Even if the debugging information in your app isn't good enough, you can use these tools to do so. So what I'll do is I'll really quickly explain sort of how these proxies work. So I was to sort of pick two people in the audience and say, oh, whisper in that person's ear a three-digit number. And they did that. None of us would know what that three-digit number is. We just wouldn't know because it's sort of this private communication, right? That's like the internet. That's how most internet communication happens. What I'm going to do is I'm going to go over here. It's 165, yeah. That's what a man-in-the-middle proxy does. It watches your traffic and it shares it so that you can see exactly what's going on. So you now will know the three-digit number. And very similar to that, if you set up a man-in-the-middle proxy on your machine, you can view all of the raw traffic on, you know, any tool you have installed on that machine. Like I said, you need admin access. A bit of a dangerous thing to do if you're not being careful because you can leak things. But, you know, it's a really, really, really useful tool for fundamentally seeing what is being sent to an AI model.
Token caching
Okay, many of the frontier providers, not everyone, but most, you know, Anthropic and OpenAI certainly, will provide deep discounts if you have cached LLM tokens. What I mean by that is that if you send the same thing twice within some predetermined time period, you'll pay a lot less for those tokens the second time. OpenAI have a way to do this automatically. Anthropic have a manual cache control markers. You can do it automatically with Anthropic these days, but with PositiveSystem, we like the manual cache control markers because it lets us do clever things with the MFR context and things like that. But yeah, these discounts can be very, very deep, like 90% cheaper by using caching.
The way this works is that when you send a user request, like here, you can attach headers. This is a cache control header attached to that user message. And, you know, this could return some kind of like 40, you know, tool calls or whatever. This could be a very long message. But next time the user sends a message following up from that, everything in blue, the model has seen before. All of this, all of this has seen before, all of this has seen before. And you only pay full price kind of for the latest, which if you imagine a very long conversation with lots and lots of tool calls and lots and lots of turns, the ratio of the new stuff is very small as compared to everything that the model has seen before. So you can save a lot of money for you or your company by taking advantage of these caching discounts.
With PositiveSystem, we try and achieve, you know, greater than 80% cache rate hits. There is a way to view actually what your cache hit rate is in PositiveSystem. We can make it up to 90%, but there are some gotchas. So if you are thinking of using caching, some top tips. Try not to change the system prompt. We used to do a thing where the system prompt sent to the model would change depending on what you're doing. It was a sort of template system. You know, so if you're using R and then you flip to a Python console, the prompt would change to indicate that.
Unfortunately, you'll find that changing the system prompt completely invalidates the caching. It's changing the very start of the message sent to the LLM. So you have to pay full price for everything again. So as much as possible, try not to change the history if you can.
Also, if you are doing that and you're not trying to change the history, sometimes you can use these XML style tags in your user message that's kind of injected to change the behavior of the model. Cloud Code does this, or at least it used to the last time I looked. You can change the behavior of the model by putting instructions inside XML tags that look kind of like this. And the LLM will listen to those. It's not as strong as true system prompting, but it is enough to change the behavior of the system without changing your entire history by changing the system prompt.
That's true of tools too, by the way. We used to make various tools available or not available depending on the context in Positron. But again, if you change which tools are available, that completely invalidates the cache and you're paying 100% for, you know, 200,000 tokens, 250,000 tokens on the next turn. So you gotta be careful.
And keep an eye on how long you expect your user to reply to you. So if you're writing an AI agent and you're expecting a very asynchronous conversation, a back and forth within five minutes, you can probably get away with the five minutes. If you're expecting a user to come back after, what, 20 minutes, 40 minutes, you might wanna think about using a longer caching header. And what's available, it's gonna depend on your provider.
Managing tool outputs
Okay, so yeah, so tools are the main way for LLMs to interact with the outside world. They're kind of what makes the AI agent able to do some work on your behalf. Another top tip for you, manage your tool outputs.
If you have things like a bash tool or a tool to run R code, don't just naively put the results of that tool into your conversation history. Here's an example where this is sort of, it's not really data related, but some kind of like sysadmin related thing. Someone might print some log files and if they're not careful, you might accidentally get gigabytes of log files in your conversation history. And that's gonna break things.
The model's not gonna be able to use over a million characters of code. They're not very good at that. They'd be able to pick out individual things, but they wouldn't be able to aggregate, for example. They're not good at that. But not only that, but the next time the user says something, the whole conversation history is gonna break because you've blown up, you filled the context with this log file and there's only a finite amount of space that you can use.
So here's a good trick. If the output from a tool happens to do something like that, don't give it straight to the model, write it to disk in some temporary location and give the path to the model instead. Because these models, they're trained to use tools like search tools, read tools, you know, grepping. They're trained to use these tools where they can take very long files. Now this could be millions of lines of code of log files and they're trained to be able to extract, search for and extract partial parts of that without putting the whole thing in your conversation history. So there's something like this. Let the model know that it's a truncated output, you know, some end lines, write to disk somewhere and let it know what it should expect. That's really useful.
Using sub-agents to preserve context
If you find that you're using a context a lot with your AI agentic workflows, what you can do is use sub-agents. Sub-agents are useful if you're doing some kind of work and you don't want to use all of that context history. You want to use that all up with your agent. You want to kind of just send off a sub-agent to do that work for you.
This is useful because what you can do is as a tool call, you can create a sub-agent. You can set up a tool that can create a sub-agent and what that will do is your sub-agent will have some kind of system prompt that might be different from the main agentic loop and the user message, rather than written by the user, will be written by the model in this tool call. And then this can rip, right? This could be, you know, 200,000, 300,000 tokens, pool calls over and over again, 40 tool calls, 50 tool calls, some kind of long running agentic workflow could happen all inside this sub-agent.
But that final message from the assistant when the work has been done can then be taken and then injected back into the original tool loop as a result from this tool. So what you've done is you've compressed all of this work that happened down into a single message of just a few tokens and you've put it inside your main agentic loop and then the assistant can continue and it means that you've used up, you've done a lot of work agentically without using up your context history and it means that the user can continue their conversation, you know, go in a different direction and do some other work. The game here is to basically try and last as long as possible before having to compact. That's kind of the game.
Feedback loops and grounding the model
So earlier, Colin talked a lot about feedback loops and as the future of AI. And I agree strongly, tools and feedback loops ground the model in reality. Without it, you know, after a single hallucination, the model just spins off and you've kind of lost it and it does, you know, it goes off the rails. Feedback loops bring it back into the ground.
A lot of things that are becoming popular in terms of workflows with AI agents, things like test-driven development, again, this is something Colin talked about, planning and implementation phases, using languages with good compilers, Rust and TypeScript with good error messages. All of the reasons that these are becoming popular in the software engineering world is because it grounds the model in reality and it stops it spinning off in one direction and getting lost.
The same thing happens in kind of, not necessarily software development, but other orchestration and AI agent flows. What we've gone for in particular with Posit Assistant is when we know that the user's doing some kind of data exploration, we kind of trust the model less and bring the human into the loop more. And that is, you know, it's not as deterministic as this, but it's still a way of grounding the model.
So if you are writing AI agents, what I want you to think about is in your workflow or your pipeline or whatever you're building your AI agent to do, what parts of that workflow can be independently automated deterministically? What can be verified or tested? Think about that, turn it into tools, because then the model can use those tools to stop it spinning off. It keeps it grounded, keeps it on track. Really think about that the best you can. For everything else, if there are things that you know must be true in your workflow, but you can't test for whatever reason, think perhaps there's a time to bring a human into the loop. Software engineering is one of these things where these tools have become very popular because they can run asynchronously and agentically, and they can run, you know, for a very long time, hours in the background. Maybe that's not the case with your particular work. Maybe you need to bring a human into the loop more.
Keeping agents simple and using known protocols
Okay, I will really quickly say that it's tempting with these tools when you're building your own AI agents to sort of add in as much customization as possible. Oh, maybe the user could add this or that. Maybe we should have some kind of graph-based agentic workflow. Maybe we should have very complex sub-agent definitions, things like that, custom instructions that the user can supply. What I will say is that that stuff's really nice, but do, if you can, instead stick with known protocols and standards that already exist. We learned that these things are much easier for the models to pass, largely because they're trained to do these things.
Things like agents.md or copilot.md or whatever, things like skills that just mark down text. Models are good at reading text. Model context protocol, the kind of thing Neil talked about. Models are trained to be able to use the tools from here. Users know how to use those things. They can bring them in from competing products, and you can onboard them onto your AI agent, and models already know how to use these things. So try, if you can, if you're thinking of customizing the agentic loop to allow user customization for workflows, try and do it by introducing a skill, for example, which will allow you to do things like slash command.
Probabilistic debugging and agent guardrails
Okay. Agents are trained to be friendly assistants. We already saw that earlier, where Claude just lied to me to be as a friendly, useful person as possible. And because of that, they'll work to the end and back to try and do the job that you ask for, including working around your guardrails.
Here's an example, fairly recently, I think I saw this like maybe yesterday or the day before, where someone was working in an environment that did not have sudo access. They did not have access to admin settings on their machine. And yet the model saw that, it saw that there was no sudo, and it just went ahead and did it anyway by this sort of backdoor that exists if you happen to have Docker access on your machine. And this is kind of known. It's kind of this not very well-spoken about thing that if you have a Docker access on your machine, it's the same as having admin access.
The model, you'd think, surely the model knows that it shouldn't do that, that that's kind of wrong, right? But no, they're being helpful. They want to write a file with admin permissions, so it's going to find a way to do it.
They want to write a file with admin permissions, so it's going to find a way to do it.
We find this with Posit Assistant, if you're trying to choose guardrails, like, oh, we'll temporarily turn off access to the R console, so it can't run code or whatever, right? Well, it's got access to Bash, so it'll just start a new R session in Bash and do it that way instead. It doesn't realize that it shouldn't really do that.
And that takes me to this sort of idea of probabilistic debugging. So how do we train the models out of those kinds of behaviors? It's hard to do. There is the idea of evals, evaluations, so we can measure the behavior of a model by putting different scenarios in front of the model and seeing what it does. And then we can use a bigger model to judge whether that model did the right thing or not, whether it passed or failed some behavioral test. They're really useful, evals, for debugging. We use evals to debug exactly how Posit Assistant behaves. But I will say this can get very expensive, especially when you're using a larger model to judge the behavior of a smaller model. Those costs can add up, so there's always vibes. Just try it, see how it goes. Maybe it's fine, I don't know.
I'm joking, but honestly, like dogfooding, really trying your own tools, sit down with your agent after you've set up the agentic loop and really try it in earnest and try and break it. So you see what happens when you prompt inject it or whatever. You kind of want to get a feel for how that model behaves because that's what your users are going to see.
Lessons learned and final takeaways
Okay, I thought the idea of this slide was kind of like, what would we do differently if we were to start again? But to be honest, that was this entire talk because we kind of did start again with Posit Assistant. We switched from a Microsoft copilot-based infrastructure to a more agentic workflow. So everything I've told you today already is kind of what we would do if we were starting again.
But I will really quickly add, these are kind of my takeaways, but I will say this one's important. Try to avoid adding complexity to your agents. Try and keep things as simple as possible because you can do things like prompt your agent to a mile away and back. You can do things like set up very complicated workflows. But what you will find is this time next year, the models are five times as big. They're five times as clever. And all of that work that you spent building those very long prompts, building those workflows are no longer needed. This is called the bitter lesson and it's been around a long time before LLMs. It's a very old AI idea that don't work too hard because the models are going to get better anyway. So yeah, keep it simple. Thanks very much.
This is called the bitter lesson and it's been around a long time before LLMs. It's a very old AI idea that don't work too hard because the models are going to get better anyway.
