A model can return a convincing answer on a laptop and still fail under production traffic. The gap is usually outside the model: unclear service objectives, untested concurrency, weak failover, slow rollouts and missing request traces.
Use this checklist before an inference endpoint becomes a dependency for customers or internal teams. If you are still standing up that endpoint, start with deploying your first model in 10 minutes. The target values will differ by workload. The important part is to define them, test them and assign an owner when they are missed.
1. Define latency objectives by percentile
Set separate objectives for time to first token, time per output token and end-to-end latency. Report P50, P95 and P99 rather than a single average.
Measure at the user-facing boundary. Queue time, retrieval, routing, tool calls and retries all affect the experience even if model execution is fast.
For each endpoint, document:
- The maximum acceptable time to first token.
- The maximum P95 and P99 completion latency.
- The expected input and output length.
- The concurrency and hardware used for the test.
A target without its test conditions is hard to reproduce and easy to misread.
2. Find the throughput ceiling
Load test the service until it saturates. Increase concurrency gradually and record completed requests, tokens per second, queue depth, errors, memory use and latency percentiles.
The objective is not to produce the largest number. It is to identify the point where added load causes tail latency or errors to rise sharply. Set operating limits below that point and keep headroom for failures and traffic bursts.
Use a representative mix of prompt lengths. A test made only of short prompts can hide prefill and memory pressure that appear with real documents.
3. Test overload behavior
A production service needs a predictable answer when demand exceeds capacity. Options include bounded queues, admission control, rate limits, autoscaling, a smaller fallback model or a managed API fallback.
Rejecting a request quickly may be safer than letting it wait until an upstream timeout. Confirm that clients receive a documented status code and retry guidance. Add limits to prevent several retrying services from creating a larger traffic spike.
4. Prove the failover path
Stop a worker during a load test. Make a model unhealthy. Remove a GPU from the pool. Confirm that the control plane detects the failure, stops new traffic to the affected replica and restores capacity as designed.
Define which failures allow a fallback model. A smaller model may be acceptable for classification but not for a regulated decision. If quality changes during failover, the application and logs should identify that change.
Document the recovery time objective and the person or team that owns an extended outage.
5. Measure GPU efficiency in context
GPU utilization alone is not a success metric. High utilization with poor latency is not useful, and low utilization may be reasonable for a strict interactive SLA.
Track GPU compute, memory, KV-cache use, batch size, queue depth and tokens per second together. Compare them with the workload's latency and reliability objectives.
Continuous batching, quantization, model co-location and cache reuse can improve efficiency. Test each change with the same request set. Some optimizations help throughput while increasing tail latency or reducing output quality.
6. Rehearse a model update and rollback
Deploy a new model revision before launch, even if the first version has not changed. Validate the complete path: weight approval, compatibility checks, warm-up, health checks, traffic shift, evaluation and rollback.
Pin model and runtime versions. A mutable model name makes an incident difficult to reproduce. Record the model digest, serving engine, quantization, tokenizer, configuration and hardware for every deployment.
Use canary or blue-green rollout when the service is important enough that a bad update cannot affect all traffic at once.
7. Build request-level observability
Uptime tells you whether the endpoint responds. It does not tell you whether the system is useful.
Each request trace should connect:
- Request and tenant identifiers.
- User or service identity, subject to privacy policy.
- Routing decision and policy version.
- Model, tokenizer and runtime versions.
- GPU pool or deployment target.
- Input and output token counts.
- Queue time, time to first token and completion latency.
- Tool or retrieval dependencies when applicable.
- Error, retry and fallback events.
- Estimated or realized cost.
Do not log sensitive prompt content by default. Use configurable retention, redaction and access controls. In some environments, metadata and hashes are more appropriate than raw inputs and outputs.
8. Verify security and governance controls
Test authentication, authorization, tenant isolation, quotas, secrets management and audit export. Confirm that a user cannot call an unapproved model or retrieve another tenant's data through the inference service.
Review how model files enter the environment. Scan artifacts, verify checksums and restrict who can register or promote a model. Treat model weights and custom code as software supply-chain inputs.
For private deployments, test the promised network boundary. A service described as offline should continue to work without external network access and should not attempt to send telemetry outside the environment.
Record the launch decision
For each check, record the target, observed result, environment, date and owner. Mark exceptions explicitly. A launch review should show what passed, what remains risky and who accepted the risk.
Xinference provides a unified API, multi-engine runtime, distributed inference, scheduling and request-level observability for private and managed deployments. The platform removes repeated infrastructure work, but the launch criteria still belong to the workload owner. As the number of endpoints grows, these checks become the operating baseline for managing a fleet of models rather than a one-time launch gate.
A good production review does not ask whether the model works. It asks how the system behaves when traffic, hardware, models and dependencies stop behaving as expected.