Pattern Matrix/White Paper/R2

Problem

Using the most capable and expensive model for every request allocates the same resources to template filling and complex analysis. Prices and model capabilities change, so routing should compare current candidates on a local task set rather than rely on a fixed price-ratio table.

Complexity-based routing assigns model resources from task difficulty and risk. Simple requests use a lower-cost tier; difficult or high-risk requests use a more capable tier. Savings depend on the traffic distribution and model mix and should be calculated from replayed production traces. The pattern can coexist with parallel exploration: routing allocates routine requests, while parallel branches add verification for selected decisions.

Classification: Reasoning × Route

• Vertical axis · Reasoning: A routing decision is at heart "should this question use cheap fast thinking or expensive slow thinking." What it selects is a reasoning policy. It belongs to the reasoning-strategy layer, not to splitting a task across multiple agents.
• Horizontal axis · Route: Based on the complexity signal of the input, it dispatches different queries to different reasoning depths (model tier + effort tier). This is a natural branch-selection structure: a query comes in and one path is chosen.

Solution and mechanics

A single complexity-based routing pass consists of three stages:

1. Extract signals + classify: Derive complexity signals from the query, domain, and historical outcomes, then use rules or a lower-cost model to choose a tier. Include the classifier's own cost and latency in the evaluation.
2. Tiered execution: Define capability and cost tiers from local evaluation results while keeping a shared call interface. Routing may select both a model and its effort setting.
3. Escalation fallback: After a lower-cost tier answers, check its schema, evidence, and business constraints. If it falls short, escalate and retry. The chain needs a configured ceiling; failure at the top should raise an error or hand off to a human.

Production systems commonly use three routing approaches:

Serious production agents generally take the explicit-harness line, because cost control is at the core of product economics, and handing that power to a model vendor or a third party is giving up control. PoCs and small projects can use the other two.

Applicability

• Many easy cases, few hard ones, with cost sensitivity but no room for accuracy to drop: internal BI self-service queries, customer-support Q&A, document processing—most queries are simple, a few are complex, and routing spends money exactly where it matters.
• Multi-vendor mixed scenarios: a team using Claude / DeepSeek / its own fine-tuned model at the same time cannot hand routing to a single vendor and must do it at the engineering layer.
• Specialized agents that split by action type: code changes use the main model, git operations use a weak model—routing by action rather than by query complexity is a variant of the routing idea.

Known failure modes

• Classifier built only on fixed rules: Keyword matching may misroute unseen query forms. Add uncertainty-based escalation or a lower-cost model, and compare the cost of false downgrades on a replay set.
• Acceptability check looks only at length: checking only whether the output is long enough lets through results that violate the schema, exceed numeric bounds, or cite the wrong source data. Do schema-aware validation.
• Fallback ends up more expensive than going straight to the expensive tier: running the cheap tier first, finding it inadequate, then escalating to the expensive tier costs more in total than going expensive from the start. As soon as the fallback rate rises, check whether the classifier is misfiring.
• High-risk queries also go to the low tier: Finance, privacy, or compliance queries may need a reliable tier even when syntactically simple. Encode risk overrides outside the model.
• A single-model team forcing routing in: when only one model is used, or the task is always of the same kind, or the cheap model is already good enough, routing is pointless.

Verification and metrics

• Per-query cost distribution: Break cost down by routing tier and task class. When the high-capability tier changes materially, distinguish harder traffic from classifier error.
• Routing accuracy: Use labels, human review, or a stronger reviewer to judge whether the selected tier met the task requirement. Track unsafe downgrades separately.
• Fallback rate: The share of lower-tier results that escalate. Changes call for inspection of thresholds, model capability, and input distribution.
• Routing decision time: Measure the classifier's contribution to total latency. If it becomes material, use rule prefilters, caching, or asynchronous features.

Reference implementation

Query comes in → extract complexity signals (length / keywords / domain / history)
            classifier (cheap model or rules) → choose tier + confidence
            High-risk query (finance / privacy / compliance) → force most expensive tier, skip the split
            Execute:
                cheap tier runs → result acceptable? → return
                          not acceptable → escalate (schema validation + cost estimate)
                escalation chain reaches its configured ceiling; still failing at the top → raise error / hand off to human
            Emit a trace on every routing decision (query summary / tier / signals / confidence / actual cost / whether escalated)
            

Four points for getting it into production: use a cheap model instead of pure rules for the classifier; make the acceptability check schema-aware; add a cost ceiling to the fallback chain to keep extreme cases from blowing up; periodically promote query types that repeatedly escalate to default directly to the high tier, saving the repeated escalation overhead.

Illustrative scenario

Consider an internal BI agent serving product, growth, and finance teams. Trace review shows that some requests fill SQL templates or add grouping, while others require multi-step attribution or causal analysis. The first version uses the strongest model for all of them and cannot explain whether resources are being spent on difficult work.

The revised system assigns tiers to template queries, aggregation, attribution, and high-complexity analysis; combines a rule prefilter with a lower-cost classifier; estimates total cost before escalation; forces finance, privacy, and compliance requests through risk-based overrides; and retains every routing trace for replay. Cost and quality changes are calculated from those traces rather than assumed from an industry percentage.

Related patterns

• Parallel exploration (R3): complementary. Routing picks one tier at a single point in time to save money; parallelism opens N lines at once to buy quality. Within the same agent, everyday work goes through routing and critical decisions start parallelism.
• Chain of thought (R1): the tier that routing selects already contains the CoT effort tier. Routing is the version of CoT effort control raised from a single call to the task level.
• Dual-mode architecture (R5): dual-mode is the architectural extreme of routing—ordinary routing is "pick one tier and run," dual-mode is "run two tiers at once," pushing "tier selection" all the way to "splitting the agent."
• Failure journal / guard-type patterns: forcing high-risk queries through the most reliable tier shares the same protective reasoning as "some boundaries cannot be skimped on."

Design conclusion

The essence of complexity-based routing is not "pick a model by looking at the query," but turning the three variables of token, latency, and quality into a Pareto frontier and choosing a specific point on the curve according to the business SLA—this is the work of product economics, not of model parameter tuning.