Deploy Your First Model on Xinference in 10 Minutes

Xinference gives your application one OpenAI-compatible endpoint in front of 300 or more open models, running on your own infrastructure. If your code already speaks the OpenAI API, moving to Xinference is a one-line configuration change, not a rewrite. This walkthrough takes you from a clean machine to a running model you can call over HTTP, in about 10 minutes.

What you need

A machine with Python 3.9 or later. A GPU is recommended for anything beyond small models, but Xinference will also run CPU-only for testing.
Network access to pull model weights the first time you launch a model.
Five to ten minutes, most of which is spent waiting on a model download.

Step 1: install

Install Xinference from PyPI. The base package includes the CLI, the local web UI, and the model launcher.

$ pip install "xinference[all]"

This pulls in the supported serving engines. If you already know which engine you want, for example a leaner install with just the transformers backend, the documentation covers scoped install options.

Step 2: start the Xinference server

Start a local Xinference instance. By default it listens on port 9997 and exposes both a web UI and the OpenAI-compatible API.

$ xinference-local --host 0.0.0.0 --port 9997

Xinference is starting up.
Model registry loaded: 300+ models available.
Listening on 0.0.0.0:9997 ...

For a production deployment you would instead point this at a Xinference cluster running across your own GPU pool, but the local server is the fastest way to see the workflow end to end.

Step 3: launch a model

Pick a model from the built-in catalog and launch it. Xinference handles downloading the weights, selecting a compatible serving engine, and exposing the model under a stable model name.

$ xinference launch \
    --model-name llama-3.1-instruct \
    --size-in-billions 7 \
    --model-format pytorch

Launching model: llama-3.1-instruct (7B, pytorch)
Downloading weights ...
Model ready. UID: llama-3.1-instruct-7b-abc123

You can launch the same command with a different --model-name to try an embedding model, a vision-language model, or a speech model. The launch flow is identical across all 300 or more supported models.

Step 4: call it like you'd call any OpenAI-compatible endpoint

Xinference exposes a standard chat completions endpoint. If your application already uses the OpenAI Python SDK, the only change is the base_url.

from openai import OpenAI

client = OpenAI(
    base_url="http://your-xinference:9997/v1",
    api_key="not-needed",  # or your configured key
)

response = client.chat.completions.create(
    model="llama-3.1-instruct",
    messages=[
        {"role": "user", "content": "Summarize the key risks in this contract clause."}
    ],
)

print(response.choices[0].message.content)

Or call it directly with curl, which is useful for a first sanity check:

$ curl http://your-xinference:9997/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "llama-3.1-instruct",
      "messages": [{"role": "user", "content": "Hello!"}]
    }'

What just happened

You now have a model running on your own hardware, reachable through the same API surface your application already understands. Nothing about your application code needed to know the model changed. That's the point: base_url = "http://your-xinference:9997/v1" is often the entire migration.

A few things worth knowing as you go further:

Swapping models is a launch command, not a redeploy. Run several models side by side and route between them.
Xinference selects quantization and batching automatically based on the GPU class you launch on, so the same launch command behaves sensibly whether you're on a single card or a multi-GPU pool.
The same control plane that runs your first model is the one that scales to a fleet: routing, autoscaling, and observability come from the same install, not a separate product.

Where to go from here

From a single local instance, the natural next steps are pointing Xinference at more than one GPU class to get automatic model-to-hardware placement, wiring in your existing framework of choice (LangChain, LlamaIndex, or your own code all work unchanged against the OpenAI-compatible endpoint), and deciding whether you want Xinference self-hosted in your VPC, on the Xinference standard bundle, or fully on-prem. All paths run the same control plane and the same API.

Deploy your first model on Xinference in 10 minutes.