Xinference 4.0

1. ⚡ Xavier across vLLM, SGLang and MLX

Prefix KV cache sharing used to be vLLM-only. In 4.0 it spans all three engines, so compatible replicas reuse context that has already been computed instead of paying for the same prefill twice.

• ▸All three engines — vLLM, SGLang and MLX replicas share prefix KV cache.
• ▸GPU-first tiered snapshots — Xavier snapshot storage is GPU-first with a tiered KV history, and PD defaults to GPU handoff.
• ▸Native NIXL — vLLM and SGLang PD deployments get a native NIXL transfer backend.
• ▸A full round of tuning — batched KV reads across layers and across scheduled requests, loads overlapped with vLLM decoding, BF16 bits preserved in transfer, persistent small slabs to cut padding, lower eviction and size-scan overhead, cheaper cold-prefill snapshots, and CUDA Graph allowed on V1.
• ▸Safety — KV compatibility is validated before a remote hit is accepted; Qwen3.5 recurrent state supports PD handoff.

2. 🔀 Cross-engine and cross-hardware P/D separation

Prefill and decode no longer have to agree on an engine — or even on a vendor. Pick each role independently and let Xinference’s unified serving interface join them up.

• ▸Across engines — run prefill and decode on different engines, including vLLM ↔ SGLang.
• ▸Across hardware — bidirectional NVIDIA ↔ Apple silicon deployments: prefill on the GPU box and decode on a Mac, or the reverse.
• ▸One interface — engine and hardware choices per role, served through the same API surface.

3. ♻️ Reload engines without reloading model weights

Changing concurrency or context length used to mean unloading the model and pushing tens of gigabytes of weights back across the bus. Opt-in GPU weight caching keeps the weights resident and rebuilds only the engine.

• ▸Weights stay put — vLLM and SGLang retain GPU weights while the inference engine is rebuilt.
• ▸Tune live — adjust supported concurrency, context, memory and CUDA Graph settings.
• ▸No dropped traffic — the change waits for in-flight requests to finish, and the model UID is preserved, so callers never notice.

4. 🕸️ Distributed vLLM V1 inference

Combine tensor and pipeline parallelism across Xinference workers through the built-in xoscar executor — with no separate Ray cluster to deploy or operate. This release completes vLLM V1 pipeline-parallel execution and improves pipeline scheduling and failure handling.

5. 🧩 OpenAI Responses API and cache-hit reporting

A stateless /v1/responses endpoint serves Responses-based clients, and prompt reuse finally becomes measurable.

• ▸Responses API — stateless /v1/responses, including streaming and tool calls.
• ▸Cache-hit tokens — vLLM, SGLang and MLX report reused prompt tokens through usage when the engine supplies the data.
• ▸Extension points — new distribution hooks for API composition and worker admission.

6. 📋 Unified log center, one-command install

Two operational quality-of-life changes that land together.

• ▸Log center — the Web UI shows recent runtime logs, and with Elasticsearch configured it queries historical indexed logs too; runtime and historical views are unified.
• ▸Request ID correlation — one request ID ties API, audit and runtime events together.
• ▸One-command install — install and start Xinference on Linux, macOS and Windows into an isolated Python environment.
• ▸Optional system services — background operation and automatic startup, with rollback when a managed service upgrade fails.