The Platform

The stack, tuned to the millisecond.

Runtime, infrastructure, and tooling, working as one system on the GPUs you already have.

Get started Talk to an engineer

Trusted in production.

Every layer, tuned for throughput.

Paged-KV caching, quantization, and batching, built into the runtime.

Paged-KV cache

Reuses attention memory across requests, so one GPU serves more tokens per second.

Multi-engine runtime

vLLM, SGLang, TensorRT-LLM and MLX behind one API, picked automatically per model.

Quantization

FP8 and FP4 out of the box, cutting memory footprint without a manual retune.

Continuous batching

New requests join a running batch mid-flight, keeping GPUs saturated request after request.

Auto-scaling

Replicas track real traffic, scaling out ahead of a spike and back down after it.

Observability

Time-to-first-token and time-per-output-token on every request, no extra setup.

Intelligent routing

Every request lands on the fastest healthy replica, load balanced automatically.

Cold-start optimization

New replicas start serving in seconds, not minutes, even for large models.

Any cloud. Bare metal. Your account.

Pool GPUs across providers, or run entirely inside your own walls.

Pooled GPUs

Pool heterogeneous NVIDIA GPUs (A100, H100, H200, B200 and more) as one.

Multi-cloud execution

Run in your own cloud account (AWS, GCP, Azure or Oracle) or on-prem: the same engine and the same dashboard everywhere.

0major clouds, plus bare metal

Bare metal and private

Install on your own hardware or inside a fully private network, no public endpoint required.

0public endpoints exposed by default

Security & Governance

Audit-ready by default.

Data residency, access control, and audit trails, built in from day one.

Data residency

Deploy in AU or SG: your own cloud or on-prem, data stays in region.

RBAC

Role-based access control down to the API key, enforced org-wide.

Audit logs

Every request and admin action logged, exportable for compliance review.

SSO / SAML

Single sign-on through your existing identity provider.

Private deployment

Fully private install inside your network, disconnected from any public network.

Faster tokens. Same GPUs.

Enterprise throughput, measured on identical hardware, no extra spend.

Up to

0×

faster inference on the same hardware.

No cold starts

replicas warm in seconds

p50 / p95 / p99

latency tracked on every request

Throughput · tokens/sec

Standard engine

1.0×

Xinference Enterprise

2.0×

Illustrative comparison, not a certified benchmark.

More ways to put it to work.

See the rest of the stack, from cost breakdowns to code.

See the cost breakdown.

Compare closed API pricing against self-hosted inference on your own GPUs.

Explore Product

Every modality, one runtime.

LLMs, embeddings, transcription, text-to-speech and image generation, on one engine.

Explore Solutions

Ship your first model today.

One command to deploy, plus SDKs, docs, and a CLI that gets out of the way.

Explore Developers

The Inference Stack

One stack.
Every millisecond.

See the benchmarks, then run it against your own workload.