Pattern Matrix/White Paper/R2

R2 · Complexity-Based Routing

The problem it solves

Using the most expensive, most capable model as the default is the most common form of waste in 2026 agent industry practice. The price gap between GPT-4o and GPT-4o-mini is 16x, and the gap between Opus and Haiku is close to the same order. If an agent routes every query to the most expensive model while more than half of those queries are in fact template-fill-in level simple tasks, that share of the bill is money spent for nothing.

Complexity-based routing replaces "use the expensive model for everything" with "split by need." Its premise is that real traffic has many easy cases and few hard ones—simple queries make up the bulk, complex queries are the minority. Route simple queries to a cheap model and reserve the expensive model for the complex queries that genuinely need it, and the bill is immediately cut in half with essentially no quality loss. Industry data shows that teams who route correctly commonly achieve a 47% to 80% cost reduction. It is complementary to parallel exploration (R3): routing handles everyday savings, while parallelism buys quality for critical decisions, and both can coexist in the same agent.

Why the coordinate is "Reasoning × Route"

• Vertical axis · Reasoning: A routing decision is at heart "should this question use cheap fast thinking or expensive slow thinking." What it selects is a reasoning policy. It belongs to the reasoning-strategy layer, not to splitting a task across multiple agents.
• Horizontal axis · Route: Based on the complexity signal of the input, it dispatches different queries to different reasoning depths (model tier + effort tier). This is a natural branch-selection structure: a query comes in and one path is chosen.

Core mechanism

A single complexity-based routing pass consists of three stages:

1. Extract signals + classify: Extract complexity signals from the query (length, keywords, domain, historical success rate) and hand them to a classifier that decides which tier to take. The classifier itself must use a cheap model or rules, keeping the decision cost within 5% of the total—you cannot call an expensive model upfront just to make the routing decision.
2. Tiered execution: A common setup is three tiers (Cheap / Medium / Expensive), all sharing the same call interface and only switching the model name. Routing selects not only the model but also the model's effort tier, combining the two dimensions.
3. Escalation fallback: After the cheap tier answers, check whether the result is trustworthy (confidence, schema, length); if it falls short, escalate and retry. The escalation chain must have a hard ceiling (usually 3 tiers); if it still fails at the top, raise an error or hand off to a human—every escalation chain needs a circuit breaker.

In 2026 the routing industry split into three lines, which determine how a team chooses:

Serious production agents generally take the explicit-harness line, because cost control is at the core of product economics, and handing that power to a model vendor or a third party is giving up control. PoCs and small projects can use the other two.

Production scenarios it fits

• Many easy cases, few hard ones, with cost sensitivity but no room for accuracy to drop: internal BI self-service queries, customer-support Q&A, document processing—most queries are simple, a few are complex, and routing spends money exactly where it matters.
• Multi-vendor mixed scenarios: a team using Claude / DeepSeek / its own fine-tuned model at the same time cannot hand routing to a single vendor and must do it at the engineering layer.
• Specialized agents that split by action type: code changes use the main model, git operations use a weak model—routing by action rather than by query complexity is a variant of the routing idea.

Where it goes wrong

• Classifier built on pure rules: keyword matching generalizes poorly to unseen query forms and misroutes complex queries into the cheap tier. In production you should use a cheap model for classification—it generalizes better and still costs within 5% of the total.
• Acceptability check looks only at length: checking only whether the output is long enough lets through results that violate the schema, exceed numeric bounds, or cite the wrong source data. Do schema-aware validation.
• Fallback ends up more expensive than going straight to the expensive tier: running the cheap tier first, finding it inadequate, then escalating to the expensive tier costs more in total than going expensive from the start. As soon as the fallback rate rises, check whether the classifier is misfiring.
• High-risk queries also go to the cheap tier: queries involving finance, privacy, or compliance cannot be skimped on even when they look simple, because the cost of an error is 100x more than the tokens. Force them through the most reliable tier.
• A single-model team forcing routing in: when only one model is used, or the task is always of the same kind, or the cheap model is already good enough, routing is pointless.

Key metrics

• Per-query cost distribution (healthy zone is long-tailed): cheap tier 60-80%, medium tier 15-30%, expensive tier <10%. If the expensive tier exceeds 30%, either the business really is complex or the classifier is misrouting too many simple queries to the expensive tier.
• Routing accuracy (healthy zone >90%): the share handled by the cheap tier without escalation. If only 60% avoid escalation, the classifier is pushing too many complex queries into the cheap tier.
• Fallback rate (healthy zone <10%): the share of cheap-tier results that trigger escalation. This is the leading signal for the other metrics—when it rises, the cost distribution shifts and accuracy drops immediately, so bring it down first.
• Average decision time (healthy zone <100ms): how long the classifier takes to make its own decision. Exceeding this value means the routing judgment is too heavy and the agent waits before it does any work.

Minimal implementation skeleton

Query comes in → extract complexity signals (length / keywords / domain / history)
            classifier (cheap model or rules) → choose tier + confidence
            High-risk query (finance / privacy / compliance) → force most expensive tier, skip the split
            Execute:
                cheap tier runs → result acceptable? → return
                          not acceptable → escalate (schema validation + cost estimate)
                escalation chain has a hard ceiling (e.g. 3 tiers); still failing at the top → raise error / hand off to human
            Emit a trace on every routing decision (query summary / tier / signals / confidence / actual cost / whether escalated)
            

Four points for getting it into production: use a cheap model instead of pure rules for the classifier; make the acceptability check schema-aware; add a cost ceiling to the fallback chain to keep extreme cases from blowing up; periodically promote query types that repeatedly escalate to default directly to the high tier, saving the repeated escalation overhead.

Enterprise landing example

A mid-sized SaaS company's internal data-analysis agent let the product, growth, and finance teams query BI data in natural language. The first version defaulted to the most expensive model for everything ("the strongest, so nothing goes wrong"), and three months after launch finance received a monthly bill of 480,000. The team broke down the traffic: 41% of queries were SQL template fill-ins ("number of users who registered last week"), 22% added a grouping ("retention curve grouped by region"), genuinely complex queries needing multi-step attribution were only 19%, and those needing causal modeling were only 4%. Feeding template-fill-in queries to the most expensive model is paying 15x for nothing.

The rewrite made six key decisions: three tiers plus a fallback tier (cheap tier runs template fill-ins, medium tier runs groupings, expensive tier runs attribution, top tier runs causal modeling); a dual-track classifier of rules plus a cheap model (80% goes through rules, 20% through the cheap model, holding routing cost under 200 per month); estimate total cost before escalating to avoid escalation that ends up more expensive; force the three high-risk categories—finance, privacy, compliance—through the most expensive tier; full traces on every routing decision and a weekly health report; every 30 days, promote query types that repeatedly escalate to default directly to the high tier. Three weeks later the bill dropped from 480,000 to 120,000, the error rate held within 0.5%, and p99 latency also fell.

Relationship to other patterns

• Parallel exploration (R3): complementary. Routing picks one tier at a single point in time to save money; parallelism opens N lines at once to buy quality. Within the same agent, everyday work goes through routing and critical decisions start parallelism.
• Chain of thought (R1): the tier that routing selects already contains the CoT effort tier. Routing is the version of CoT effort control raised from a single call to the task level.
• Dual-mode architecture (R5): dual-mode is the architectural extreme of routing—ordinary routing is "pick one tier and run," dual-mode is "run two tiers at once," pushing "tier selection" all the way to "splitting the agent."
• Failure journal / guard-type patterns: forcing high-risk queries through the most reliable tier shares the same protective reasoning as "some boundaries cannot be skimped on."

What this pattern teaches

The essence of complexity-based routing is not "pick a model by looking at the query," but turning the three variables of token, latency, and quality into a Pareto frontier and choosing a specific point on the curve according to the business SLA—this is the work of product economics, not of model parameter tuning.

This page is part of the ADPS pattern white papers. Back to the pattern matrix, or see the runnable code catalog.