Developers

Ship OpenAI-compatible inference in one line.

vLLM, SGLang, TensorRT-LLM, Transformers or MLX underneath: one unified API across every engine, running on your own infrastructure.

Get started Talk to an engineer

One-line migration Model catalog Serving engines API & keys Python SDK Docs ↗

One-line migration

Deploy. Swap one line. You are live.

Already calling the OpenAI API? Your code does not change. Only the base URL does.

Deploy a model in one command

$ xinference launch \
    --model llama-3.3-70b \
    --gpu H100 --replicas 4

✓ Model live at :9997/v1

Point your OpenAI client at it

- base_url="https://api.openai.com/v1"
+ base_url="https://your-xinference/v1"

client = OpenAI(base_url=base_url, api_key=XINFERENCE_API_KEY)

Zero code changes. Smoke test it.

$ curl -s https://your-xinference/v1/chat/completions \
    -H "Authorization: Bearer $XINFERENCE_API_KEY" \
    -d '{"model":"llama-3.3-70b","messages":[{"role":"user","content":"ship faster"}]}'

HTTP/1.1 200 OK

Model catalog

300+ models. One command to launch any of them.

LLM, embedding, rerank, image, speech and vision models, all OpenAI-compatible.

300+

Models supported

Every modality

LLM, embedding, rerank, image, speech and vision in one runtime

Multimodal

Text, image, audio and video in one API

1 command

xinference launch --model, and it is serving

Browse the model library ↗

Serving engines

Same API. Any engine underneath.

Xinference picks a sensible default. Override the engine or replicas anytime.

High-throughput serving with paged attention. The default for most LLMs.

$ xinference launch --model llama-3.3-70b \
    --engine vllm --quantization awq --replicas 4

API and key governance

Built for teams, not a single laptop.

Every endpoint is OpenAI-compatible. Every key is scoped, budgeted and audited. Full API docs ↗

OpenAI-compatible endpoints

Chat, completions, embeddings, rerank, images and audio: one schema across every model.

API key budgets

Set spend and token budgets per key. Requests stop cleanly at the limit, not mid-stream.

Rate limits and expiry

Per-key rate limits, scoped permissions, and automatic key expiry.

Usage analytics

Requests, tokens and latency broken down by key, model and team.

Team workspaces

Team workspaces with per-key budgets and rate limits.

Audit logs

Every request and admin action logged for compliance review.

POST /v1/chat/completions

Streaming + tool use + JSON mode

POST /v1/embeddings

Dense embeddings, batched

POST /v1/rerank

Cross-encoder rerankers

POST /v1/audio/transcriptions

Whisper-compatible STT

POST /v1/images/generations

FLUX, SDXL, SD3

GET  /v1/models

List every deployed model

Read the docsConcepts CLI reference Hardware guides Deployment and security Recipes

Python SDK

Deploy, get an endpoint, call it. Three lines.

Use the Xinference SDK, or keep the OpenAI SDK you already have. Same API either way.

Python SDK

from xinference import Client

xi = Client("https://your-xinference/v1")

# OpenAI-compatible chat, same schema on every engine
out = xi.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "ship faster"}],
    stream=True,
)
for chunk in out:
    print(chunk.choices[0].delta.content, end="")

Try it in the quickstart guide ↗

Built to scale

What the runtime gives you, in numbers.

Serving engines, one API

Multiple formats

Multiple quantization formats, tunable per model.

1 → N

Replicas per model, no redeploy

One schema

One OpenAI-compatible schema across every endpoint.

Ready?

Deploy your first model
on your own infrastructure.

300+ models, five serving engines, one OpenAI-compatible API.