Most AI applications start with one model. The choice is easy because there is nothing to route. As the application grows, teams add a reasoning model, a cheaper general model, an embedding model and perhaps a vision model. The original default remains in place, so expensive requests keep flowing to it even when a smaller model would work.
LLM model routing changes that decision from an application constant into a per-request policy. A router estimates which candidate models can handle a prompt, applies the workload's quality and latency requirements, then selects an eligible model based on cost or another operating objective.
The goal is not to send every request to a small model. It is to stop paying for capability that a particular request does not need.
What an LLM router decides
A useful router answers a narrow question: which model should serve this request?
That decision may use prompt difficulty, task type, context length, modality, model availability, observed latency and price. Production policies can also exclude models that do not meet residency, license or security requirements.
Routing is different from load balancing. A load balancer chooses among replicas of the same service. A model router chooses among models with different capabilities and costs. Most production systems need both.
Why static routing rules stop working
Rules such as "use the largest model for complex prompts" are useful during a prototype. They become harder to defend once traffic changes.
First, labels such as "complex" rarely match actual model performance. A smaller model may handle most extraction, classification and structured-output requests accurately. Second, new model releases change the cost-quality frontier. Third, a rule written for normal traffic does not respond to a slow or saturated model.
Static rules still have a place. They are appropriate for hard constraints, including data residency, prohibited model families and minimum context length. The problem is using them as a substitute for measured capability.
How xrouter-llm works
xrouter-llm is a prompt-aware routing-decision service. It predicts the probability that each candidate model can complete a prompt, then chooses the cheapest candidate that clears a configured threshold. It returns the routing decision rather than calling the selected model itself.
The current router separates the problem into two signals:
- A capability profile derived from published model benchmarks.
- A prompt-difficulty estimate trained on observed completion data.
The routing policy combines those signals with model costs. Teams can extend the registry with their own models and retrain the difficulty estimator on representative traffic.
On the repository's tested dataset, the current release reports 53.2% lower realized cost and a 1.9 percentage-point improvement in completion. These figures describe that dataset and configuration. They are not a guarantee for a different workload.
The repository is source-available under the Xagent Source License. It is not licensed under an OSI-approved open-source license. Internal and single-tenant use is permitted by the repository's summary, while hosted and competing-service uses are restricted. Review the license before deployment.
What to measure before enabling routing
Start with an offline replay. Sample real prompts, remove sensitive data as required, and run each prompt against the models you want to compare. Record task success, latency, input and output tokens, failure type and cost.
Then define a quality bar for each task family. Exact match may work for classification. Structured extraction needs schema validity and field-level accuracy. Open-ended generation often needs a rubric or human review. One metric will not cover every workload.
Evaluate the router on four outcomes:
- Quality loss or gain relative to the current default model.
- Realized cost per successful request, not cost per raw request.
- P50, P95 and P99 latency, including retries and fallbacks.
- Routing stability when models, prices or traffic mix change.
A router that saves money but increases retries may be more expensive in practice. A router that performs well on coding benchmarks may still misclassify customer-support prompts if it has never seen similar traffic.
A safe production rollout
Begin with shadow mode. Let the existing model serve users while the router records the model it would have selected. Compare those decisions with evaluation results.
Next, send a small percentage of low-risk traffic through the router. Keep the current model as a fallback and attach the routing decision to every request trace. Increase traffic only after quality, latency and cost remain within the agreed thresholds.
Version the routing model, registry and policy together. If a result changes, operators should be able to identify which version made the decision and roll it back.
Where routing fits in the inference stack
Model routing works best above a unified serving layer. The application sends one OpenAI-compatible request. The router selects a model, and the inference platform schedules that model on an eligible GPU pool. Observability then connects the application request, routing decision, model version, hardware and cost in one trace — the same operational picture we described in from model selection to fleet management.
Xinference provides the serving and control layer for this pattern across managed cloud, private cloud and on-prem infrastructure, with the platform handling scheduling and request-level observability. The application keeps one API while the models and hardware behind it change.
Test routing on your own traffic before trusting a headline benchmark. The best router is the one that can prove which requests became cheaper, which stayed on the stronger model and why.