The Model is Your Other User
I’ve just made a substantial update to go-llm, a project I started to learn how to work with LLMs and the ecosystem around them.
About a year ago, that meant chat completions, tool calling and embeddings. I got as far as what is now called the “agent loop”: passing messages back and forth with the model until either there are no more tools to call, or the context is full and nothing more can be done.
Now we’re in the age of the “assistant”: these loops are started by something other than a direct call, such as a scheduled task or an external event. You ask your assistant to do something, and it acts on your behalf, seemingly intelligently. It’s convincing, and useful to a lot of people. Meta’s Muse, the OpenClaw phenomenon, Instinct and Anthropic all have products in this space, so I updated go-llm to do the same, albeit without their range of tools or their resources.
I learnt a few lessons along the way.
The main lesson from my first round was that I needed to limit what goes into the model’s context window. It can hold a wealth of information: your previous messages, instructions on how to respond, and the tools the model can call. There are several ways to keep it in check. The crudest is to trim your history until it fits the context window (measured in input tokens), leaving a little headroom for the reply.
That’s why long-term memory matters: a tool the model can use to retrieve important facts later. With a reasonably capable model, memory changes every conversation you have. For a few extra tokens to describe the tool, your assistant behaves quite differently, based on long term information stored about you.
Mine has an overlong prompt, suggested by a wise model. I’m sure some experimentation would trim it down, but most providers now cache prompts (Anthropic, OpenAI, Google Gemini), so unchanging text in the system prompt is very cheap to send…so maybe, it’s time to unlearn the idea that shorter prompts are always better ;-)
This is a step-change in software development too. Any good product manager knows that your audience’s feedback should guide development, even to the point of pivoting the product entirely. But when you build an assistant, you have two audiences: the human, and the model. The model needs guidance too. That means not just completing the next word and sentence, but using the tools you give it efficiently, and knowing what to do when something doesn’t work the first time.
I found it incredibly useful to ask the models themselves how easy a task was to do. Their answers usually led to changes in how I designed and implemented tools and messages. For example, I now annotate each message with where it came from and who sent it, because the model tailors its responses to who is asking. It’s a change that the model itself suggested, if I may anthropomorphise for a moment.
A modern “conversation” with a model isn’t just with the “user”. The request comes from many sources. It also includes tasks, emails, calendar reminders, news articles and more. Giving the model that context is another step-change, beyond memory: in how intelligent these systems appear, in how efficiently they respond, and ultimately in lowering the cost of doing the same task.
