Model Routing Is an Optimisation, Not an Architecture

Reading time: about 9 minutes.

Six weeks into a new AI project, someone draws a box on the whiteboard labelled "router." The idea is to send easy requests to a cheap model and hard ones to an expensive model. Everyone nods. It sounds responsible. It gets built into the architecture diagram before a single user has typed anything.

That box is the mistake. Not the idea, the timing.

TL;DR

Routing is a cost optimisation. Like every optimisation, it needs measurement first. Build it when you have production data showing which requests a cheaper model handles correctly, not when you are still designing the system.

• The router is not the hard part. The logic is about fifty lines. Proving which requests are safe to send to a cheap model takes months, and that is the actual project.
• You cannot route without data you do not have yet. On day one you have no traffic, no labels, and no idea which requests are hard. Every routing rule you write then is a guess.
• Raw model confidence is not a routing signal. Models sound most certain when they are wrong. Calibrate the score against real error rates first.
• Cascading is not free. When a request escalates, you pay for both models and both delays. Escalate too often and you have spent more than doing nothing.
• The price gap has shrunk. Cheap and expensive tiers are now about 5x apart, not 20x. Caching and batching are usually the better first move, and neither can give a user a wrong answer.

Routing sits at the end of the sequence, not the start. Everything to its left has to exist before the router can be anything better than a guess.

What This Article Answers

• When should you add routing, and when is it too early?
• Is routing the same thing as cascading? (No, and the difference costs money.)
• How many models should you route between?
• Can you use the model's own confidence score to decide? (Not directly.)
• What does routing actually save, now that model prices have converged?
• Where is the break-even point, and when does a cascade start losing money?
• What should you do instead, if it is too early to route?

How the Router Ends Up in the Diagram

The conversation follows the same path almost every time.

A team starts building an agent. Someone runs the numbers on the frontier model and gets a figure that looks alarming at scale. Somebody else points out that most requests look simple. A third person mentions an article about a company cutting costs 85% with routing. By the end of the meeting there is a router in the design, and it feels like the responsible choice. Nobody argued for wasting money.

Here is what nobody says out loud in that meeting: the team has no traffic yet. They do not know what the requests look like. They do not know which ones are hard. They have never seen the cheap model fail at anything, because they have not run it against anything real.

So the routing rules get written from imagination. Short requests go to the cheap model. Anything with the word "analyse" goes to the expensive one. It sounds sensible and it is guesswork with a straight face.

Six months later the system is live. Costs are lower, so the router looks like a success. Meanwhile a slice of users is quietly getting worse answers, nobody has measured it, and nobody will, because measuring it means admitting you cannot currently tell.

That is the failure this article is about. Not routing. Routing built before there was anything to route on.

A Router Is a Prediction, and Predictions Need Evidence

Strip away the engineering and a router makes one claim: this request will be handled well enough by the cheaper model. That is a prediction about an answer nobody has looked at yet, and it is only as good as the evidence behind it.

Most routers ship with almost none. Someone sampled thirty requests, skimmed the cheap model's answers, thought they looked fine, and wrote rules around that impression. The requests that break you are the strange ones, and there are never thirty of those in a sample of thirty.

Real evidence means four things: a labelled set of real requests with known-correct answers, accuracy measured per request type at each tier, a confidence signal calibrated against actual error rates, and a number for what a wrong answer costs in each type. Every one of those needs production traffic. That is the whole argument for why routing comes late.

A 2026 survey by Moslem and Kelleher maps the entire design space of routing methods: when the decision gets made, what feeds it, how it is computed. What the map does not contain is any method for knowing whether a routing decision was right. That gets handed to evaluation, which is where these projects stall.

Routing and Cascading Are Not the Same Thing

These two words get swapped around, and the difference shows up on your bill.

Routing decides before any work happens. The request comes in, something classifies it, one model handles it. One model, one charge. If the classifier is wrong, the user gets a poor answer at a cheap price and you usually never hear about it.

Cascading decides after a first attempt. The cheap model answers, something checks that answer, and a weak one gets sent up to the expensive model. This catches mistakes the router would have missed. It also means that when escalation happens, you paid for both models and the user waited for both.

That is the part teams get wrong in the spreadsheet. An escalated request is not "an expensive request." It is an expensive request plus a wasted cheap request plus double the waiting.

Run the numbers. Say the cheap model costs 1 unit and the expensive one 5. Escalate 20% of traffic and you pay 0.8(1) + 0.2(6) = 2.0 against a baseline of 5. Good deal. Escalate 40% and it is 3.0, still worth it. Escalate 70% and you are at 4.5, having saved almost nothing and made everything slower.

And there is no standard threshold to copy. Bouchard's 2026 analysis works through the maths and deliberately refuses to name a universal number, because the right one depends entirely on your task's own accuracy and cost trade-off. Anyone who hands you a default threshold is selling you something.

# The shape of a cascade. Note what is missing:
# no default threshold, because there isn't a portable one.

answer = cheap_model(request)
risk = calibrated_error_probability(answer)   # NOT the raw confidence score

if risk < threshold_measured_on_your_own_data:
    return answer                    # about 1 unit

# escalating costs BOTH calls, and the user waits twice
return strong_model(request)         # about 6 units, and 2x the wait

The Model Sounds Most Sure When It Is Wrong

There is one shortcut nearly everyone reaches for, and it quietly breaks things: using the model's own confidence score to decide when to escalate.

It sounds obvious. The model reports how certain it is, so escalate the uncertain ones. But these scores do not mean what they appear to mean. Models are trained in ways that reward sounding confident, and the training that makes them helpful and agreeable makes it worse. A fluent wrong answer and a fluent right answer come out of the same machinery and look equally sure.

Route on that number and you have built something backwards: it escalates the requests the model felt uneasy about, and sends the confident fabrications straight to your users.

The fix is calibration. You map the model's score onto how often it is actually wrong, using labelled examples from your own traffic. Kotte's UCCI work from May 2026 shows it working: take the small model's uncertainty, calibrate it against real error rates, then pick the threshold by minimising cost on a validation set. On a workload of 75,000 labelled queries, calibration error fell from 0.12 to 0.03 and cost dropped 31% with no loss of accuracy. As a rough guide, above 0.10 the score is not trustworthy for routing, and above 0.15 a fixed rule would serve you better.

The middle step is the one teams hope to skip, and it needs labelled data from your own traffic. There is no way around that.

Underneath every cascade sits a second assumption almost nobody checks: that the expensive model can handle what the cheap one could not. UCCI names this as a limit of its own theory, and it breaks in production, because difficulty is mostly a property of the request rather than of the model. A vague question is vague for every model. If the answer is not in the retrieved documents, no model can find it, which Part 6 covered as a retrieval problem wearing a model-quality mask.

So the cascade sends its hardest requests upstairs, pays double, and recovers less quality than the arithmetic assumed. Measure that recovery rather than assuming it. The question is not "how good is the expensive model," it is "how good is it on the exact requests the cheap one failed."

The Price Gap Closed While Everyone Was Quoting Old Numbers

Most routing advice was written when cheap and frontier models sat far apart on price. Check today's numbers before planning around that. Haiku 4.5 costs $1 per million input tokens and $5 per million output. Opus 5 costs $5 and $25. That is 5x, not 20x, with Sonnet 5 between them at $2 and $10 on introductory pricing through the end of August 2026.

Now compare the alternatives. Prompt caching costs 1.25x normal input price to store something and 0.1x to read it back, so it pays for itself after a single reuse. The Batch API takes 50% off and stacks with caching. An agent resends the same large system prompt and tool list on every turn, so caching often saves more than a router would, with none of the risk. Caching cannot give someone a wrong answer. A router can.

The headline results still look impressive: RouteLLM reported 85% cost savings at 95% of quality, MixLLM 97% of GPT-4's quality for 24% of the cost. Read them carefully. They route over benchmark datasets, where difficulty is tidy and the right answer is already known. Your traffic is not a benchmark, and the same survey admits these methods generalise poorly to new domains.

Nobody Gets Promoted for the Request That Should Not Have Been Downgraded

Routing savings are easy to see and easy to claim. A chart goes down, someone says a number in a review, it gets attributed to a person. Routing failures are invisible: slightly worse answers spread across users who never complain loudly enough to trigger anything. Part 6 called this the beautifully formatted wrong answer. Here it arrives with a budget line defending it.

So thresholds drift one way. Every quarter somebody asks whether more traffic can move to the cheap tier. Nobody asks whether last quarter's move was a mistake, because answering needs an evaluation nobody built.

The counter is procedural. A routing threshold is a production setting with a quality claim attached, so treat it like a prompt: versioned, reviewed, and gated on a measurement that checks quality per request type, not overall. An average is exactly where a routing regression hides, because the cheap model handles the easy majority fine and the overall number barely moves.

What To Do Instead, If It Is Too Early

If you do not have the data yet, you are not stuck. You are just not routing yet.

Turn on caching first. It is the largest saving available with zero quality risk, and for most agents it beats what routing would have delivered.

Use batching wherever latency does not matter. Half price, no downside for work nobody is waiting on.

Start collecting labels now. Sample real requests, record which model answered, and get correct answers written down. This is the asset routing depends on, and it takes months to build, so start it in week one even though you will not use it until much later.

Pick one request type, not all of them. When you do route, start with the single most obviously repetitive category, measure it properly, and expand only after that one is proven.

Check whether it needs a model at all. Part 3's shape ladder had six rungs. Routing between the top two only matters if you already confirmed the bottom four cannot do the job.

Cheap Is a Property of the Answer, Not the Model

A cheap model that returns a wrong answer is not cheap. It is rework with a small invoice attached, and in anything customer-facing it is a support ticket, a lost sale, or a trust problem that costs far more than the expensive call you avoided.

Teams that get this right do not start with the router. They run the system, collect real requests, measure what each tier gets right, and only then write the fifty lines. The router comes last because it was never the hard part.

The bill is not the difficult number. The difficult number is what the cheap tier gets wrong, and how long it takes you to find out.

Coming Up in This Series

Next up: Prompt management as code: registries, versioning, and rollout. Routing thresholds, tool descriptions, and system prompts share an awkward property: they change how the system behaves, but they skip the pipeline built to catch changes in behaviour. A prompt gets edited in an afternoon with no test, no review, and no way back, and then everyone spends a week arguing about whether quality got worse. Prompts are code that decided to skip code review, and the fix is simpler than it sounds.

---

If this resonated and you're building production AI systems, follow along. The series covers the 21 things I think senior AI engineers and architects need to reason about: RAG pipelines, tool design, security, evaluation, cost, and the operational patterns that separate demos from systems you can actually run.