Xinference v2.8.0 Released

Comprehensive observability, rock-solid stability, and massive vLLM ecosystem enhancements.

🎉

Community Spotlight

Huge congratulations to our community member @m199369309 on being promoted to Maintainer! Thank you for your outstanding and continuous contributions to the project.

✅ Release Highlights

📊 Observability

  • Full Prometheus Metrics covering Cluster / Worker / Model
  • Built-in Grafana Dashboard integrated into Web UI
  • New Prometheus Alert Rules for cluster-level monitoring

🛡 Stability & Auto-Recovery

  • Support for Graceful OOM Handling
  • Automatic recovery for Worker model exceptions
  • TCP Keepalive & Reverse Channel Probe
  • Enhanced heartbeat and logging mechanisms

⚡ vLLM Ecosystem

  • Support for vLLM v0.19.0
  • Modular vLLM Patch Framework
  • Fixed Hybrid KV Cache issues
  • TTFT metrics upgraded to Histogram (streaming & non-streaming)

🌐 Community Edition Updates

📦 Installation

pip install 'xinference==2.8.0'

Docker: Pull the latest image or update via pip inside the container.

🛠 Enhancements

  • Dashboard & monitor modules migrated to monitor/
  • Optimized Grafana panels & PromQL expressions
  • Optimized list_models caching mechanism
  • Atomic writes for .pth files
  • Updated JSON configs (Embedding / LLM / GLM-4.6)
  • ARM64 Docker builds bypass libc6 upgrade issues

🐞 Bug Fixes

  • Fixed Mixed Content issue for Web UI under HTTPS Proxy
  • Fixed transformers version conflict for embedding/rerank
  • Fixed Qwen3.5 tool call parsing & completions TypeError
  • Fixed jina-embeddings v3/v4 task parameter handling
  • Fixed enable-thinking default behavior
  • Fixed Grafana Dashboard layout issues

🏢 Enterprise Edition Updates v1.0.4

🗂

Added API for model instance parameter history and deletion (#431)

🛢

Support for KingBaseES and DM (Dameng) databases (#392)