Release Notes · September 2026

Xinference 3.4

Prefill/decode separation and multi-worker replica scaling, resumable cache management, multimodal embedding and rerank, plus Gemma-4, MonkeyOCR and a wave of new models — with Python 3.14 support.

⚡ PD Separation 💾 Resumable Cache 🧬 Multimodal Embedding 🐍 Python 3.14

What's New

Highlights

⚡

Inference architecture

Prefill/decode separation, native vLLM multiprocessing executor routing, and multi-worker replica scaling.

  • • Prefill/decode separation
  • • Native multiprocessing executor
  • • Multi-worker replica scaling
💾

Cache & downloads

Resumable cache management, a download-only flow, launching straight from cached models, and oci:// model URIs.

  • • Resumable cache management
  • • Download-only flow
  • • oci:// URIs via llmman serve
🧬

Multimodal & new models

Multimodal embedding and rerank, alongside Gemma-4, MonkeyOCR, dots ocr, MiniCPM5-2B and Fish Audio.

  • • Multimodal embedding & rerank
  • • Gemma-4 with batching
  • • MonkeyOCR, dots ocr, MiniCPM5-2B
🌏

Platform & experience

Python 3.14 support, a zh-TW locale, TTS streaming playback and reusable launch history.

  • • Python 3.14 (xoscar 0.10.0)
  • • zh-TW Traditional Chinese locale
  • • TTS streaming in the Web UI

Open Source

Community Edition

Installation

pip

pip install 'xinference==3.4.0'

Docker

docker pull xprobe/xinference:latest

✨ New Features

  • ▸ Native multiprocessing executor routing for vLLM
  • ▸ Gemma-4 Transformers backend with batching support
  • ▸ Download-only model flow
  • ▸ dots ocr model support
  • ▸ Engine registration hooks
  • ▸ Resumable cache management
  • ▸ Open cached models for launch
  • ▸ UI: select the sole available model engine
  • ▸ oci:// model URIs via llmman serve
  • ▸ MonkeyOCR model support
  • ▸ Fish Audio S1-mini and S2-Pro support
  • ▸ Multi-worker replica scaling
  • ▸ Multimodal abilities for embedding and rerank
  • ▸ UI: prefer the system download source
  • ▸ Prefill/decode separation
  • ▸ Multilingual TTS language controls
  • ▸ zh-TW (Taiwan Traditional Chinese) locale
  • ▸ MiniCPM5-2B support
  • ▸ WebUI: reuse launch history across models

🛠 Enhancements

  • ▸ Granular system settings permissions
  • ▸ JoyAI image editing model support
  • ▸ TTS streaming playback in the Web UI
  • ▸ Complete SD WebUI image workflow compatibility
  • ▸ Updated frontend language order and product links
  • ▸ Python 3.14 support, requiring xoscar 0.10.0
  • ▸ Fixed the CPU Docker image and added non-publishing manual builds

🐞 Bug Fixes

  • ▸ Require xoscar native find-links support
  • ▸ Align the Jina flash-attn version for embedding
  • ▸ Protect virtual environments in use
  • ▸ Emit streaming tool call metadata once
  • ▸ Restore supported vLLM and SGLang model engines
  • ▸ Apply the download worker environment to snapshots
  • ▸ Gate Hy-MT2 remote code execution
  • ▸ Support legacy cache conversion in Transformers
  • ▸ Isolate batch inference failures in the scheduler
  • ▸ Handle multi-device cache reduction
  • ▸ Report bans lifted, not records dropped, from bulk unban
  • ▸ Correct the F5-TTS logging format
  • ▸ Merge engine capabilities across workers
  • ▸ Restore audio streaming with TorchCodec
  • ▸ Make register_builtin_model idempotent across refreshes
  • ▸ Improve model experience actions and latency feedback
  • ▸ Contain embedding result scrolling
  • ▸ Handle runtime and device options in the flexible Transformers launcher
  • ▸ Launch GGUF rerank models without an explicit engine or quantization
  • ▸ ChatGLM: emit a usage-only final streaming chunk
  • ▸ Transformers: propagate batched generate request IDs, handle batched logits when selecting the next token, and enforce token limits across batched decode steps
  • ▸ Decouple heartbeat and full status reporting
  • ▸ WebUI: restore server-backed launch history
  • ▸ Preserve model GPU memory across transient failures
  • ▸ Refresh dynamic model subpool PID ownership
  • ▸ Avoid an implicit event loop in async client cleanup

📚 Documentation

  • ▸ Update README model highlights for v3.3.0
  • ▸ Preserve ACE-Step1.5 details outside generated docs
  • ▸ Fix the model type card layout
  • ▸ Add v3.3.0 to the release notes index and locale catalogs
  • ▸ Align the pypiserver Jina flash-attn manifest pin
  • ▸ Bump actions/checkout to v7