|
[Webinar] How to stop babysitting your agents (Sponsored)
 |
Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in the correction loops.
More MCPs, rules, and bigger context windows give agents access to information, but not understanding. The teams pulling ahead have a context layer to give agents exactly what they need for the task at hand.
Join us for a FREE webinar on Sep 23 to see:
Where teams get stuck on the AI maturity curve and why common fixes fall short
How a context layer solves for quality, efficiency, and cost
Live demo: the same coding task with and without a context layer
If you want to maximize the value you get from AI agents, this one is worth your time.
When an application adopts a large language model (LLM), they generally choose the most capable model possible. This means that every single request is sent to that expensive model.
While this approach is easier to implement, it can become quite expensive in the long run. For example, a request such as “classify this support ticket as billing, technical, or account-related” doesn’t require the same level of reasoning as “investigate why these financial records don’t match properly and explain the likely cause.”
With smart model routing, we can solve this problem. In such a routing approach, we choose a specific model for each request. In other words, simple work is sent to a small model that might be less expensive, and difficult work is routed to a more capable model. If most requests are simple, this approach can reduce the total cost in a big way, sometimes by even around 10 times. Also, the quality of the response doesn’t go down noticeably.
However, cost reduction isn’t a given. It also depends on the types of requests the application receives, the price difference between models, and how well the routing system performs. In this article, we are going to look at various aspects. Here’s what we will cover:
Why do LLM applications become expensive?
What is model routing?
How can model routing provide cost savings?
How to judge a request before answering?
Using a small model as a router
Cascading: Trying the cheaper model first
Semantic routing
Learned routing
Common ways routing can go wrong
 |
Why LLM Applications Become Expensive
The total cost of using an LLM API usually depends on the number of tokens processed.
To be clear, a token is a small unit of text. A short word might be one token. But a longer word can be split into multiple tokens.
There are usually two important token counts:
Input tokens include the user’s message, system instructions, conversation history, and any documents supplied to the model.
Output tokens are the tokens generated within the response.
Depending on the LLM provider, input and output tokens can have different price points. Larger and more capable models generally cost more because they require more computing resources. They may also spend additional computation for reasoning. This extra capability is very important for solving complex problems. But this capability is wasted when the task is simple.
For example, imagine a customer-support application that has to process a million requests per month. Within those requests, some users may ask for refund policies. Others may want an address extracted from an email. Some might have complicated account problems that need careful analysis. If each request goes to the most powerful model, the company has to pay a premium price even for work that is quite simple for this capable model. You could think of this as hiring a senior software architect to rename files, sort support tickets, and format dates. Sure, the architect can technically do those things. But it would be a waste of the architect’s capability and a case of poor resource management.