What Is Model Routing (and When Should You Downgrade)?

Quick answer
Model routing chooses a model per request based on task type, difficulty, latency target, or cost budget, instead of sending everything to the largest model.
The saving comes from the price gap between models, so routing pays off when a meaningful share of requests can be handled by a cheaper one without quality loss.
A router needs a quality check per route: an evaluation set, a fallback to the stronger model, and a way to detect when the cheap route starts failing.
Static rules — task category, input length, tenant tier — are easier to reason about and debug than a learned classifier, and usually capture most of the benefit.
Never route silently on a user-facing task where a wrong answer is expensive; make the downgrade observable and reversible.

What is model routing?

Model routing is a layer between your application and one or more model providers that selects a model for each request. The selection can be a static rule (summarise this support ticket with the small model), a classifier that estimates difficulty, or a policy based on tenant tier, latency budget, or remaining spend.

The purpose is to stop paying premium prices for work a cheaper model does well. Most applications contain a mix: classification, extraction, and short rewrites are often fine on a small model, while multi-step reasoning and code generation usually need a stronger one. Routing makes that distinction explicit instead of averaging it.

What signals can a router use?

Task type is the most reliable signal because you already know it from the call site. Input length and required output format are also strong: long-context summarisation and strict JSON schemas have different failure modes. Tenant tier and user plan can justify different quality levels if the product promises them.

A learned difficulty classifier is the most ambitious option and the hardest to trust. It needs labelled examples of requests that the cheap model failed, which are expensive to collect and drift as models change. Start with rules, measure the quality gap, and add a classifier only if the rules leave obvious value on the table.

# Rule-based routing with an explicit fallback.
def choose_model(task: str, tokens: int) -> str:
    if task in {"classify", "extract", "rewrite"} and tokens < 4000:
        return "small"       # cheap, evaluated on these tasks
    if task == "codegen":
        return "large"
    return "large"            # default to quality, not to savings

# The caller must still validate the output and escalate on failure.
result = call(choose_model(task, n), prompt)
if not passes_schema(result):
    result = call("large", prompt)

How do you know a downgrade is safe?

Build a small evaluation set per route before switching. It should contain representative inputs and the failure cases you care about, with an expected output or a rubric. Run both models on the same set and compare, then keep the set as a regression test for future model versions.

In production, add a cheap validator: schema validation, a confidence signal, a disagreement check, or a sample of human review. When validation fails, escalate to the stronger model and record the event. The escalation rate is the metric that tells you whether the cheaper route is actually working.

What can go wrong?

The most common failure is a routing rule that looks sensible but sends a task the small model handles badly, with no validator to catch it. The second is silent quality drift: the provider updates a model, the cheap route degrades, and nobody notices because the escalation path was never wired.

The third is cost inversion. A router that retries failed cheap calls on the large model can cost more than sending everything to the large model once, especially when the retry includes the full prompt again. Count the total cost per successful task, not the cost per first attempt.

Where this bites vibecoders
Generated code calls one model everywhere because the SDK example does. That is a reasonable start, but the bill is dominated by the easy calls — classification, formatting, short answers — that a smaller model handles. Add routing at the call sites where the task is known, and measure the escalation rate before widening it.

Where AI coding assistants get this wrong

Routing by a difficulty guess with no evaluation set.
Downgrading a task that silently produces plausible but wrong output.
Forgetting to validate the cheap model’s output before returning it.
Counting the cost of the first attempt instead of the total per successful task.
Hard-coding a model name in twenty places so routing cannot be changed.
Assuming a model’s price and quality stay constant across versions.

Checklist

List the distinct tasks your app sends to a model.
Measure cost and quality per task on the current model.
Pick the tasks with the largest easy-call share.
Evaluate a cheaper model on a representative set.
Add a validator and an escalation path.
Monitor escalation rate and cost per successful task.

FAQ

Is model routing the same as a model gateway?

Routing is one function of a gateway. A gateway also handles authentication, rate limits, logging, caching, and provider abstraction. You can implement routing in application code without a gateway; a gateway centralises it across services and languages.

How much cheaper is a small model?

Prices differ substantially by provider and model, and they change. Do not rely on a remembered ratio; read the current price sheet and compute the cost for your token mix. The saving also depends on the share of traffic you can route, which is usually smaller than expected.

Should routing be per request or per conversation?

Per request is more flexible but can produce inconsistent tone and quality within a conversation. Per conversation is simpler and more predictable while still allowing cheap handling of simple sessions. Choose based on whether users would notice a mid-conversation change in style.

What is the safest first route to downgrade?

A task with a deterministic check: classification with a fixed label set, extraction validated by a schema, or a rewrite checked by length and banned words. The validator gives you an objective signal, so a downgrade is a measured decision rather than a hope.