Xinference puts one OpenAI-compatible endpoint in front of 300 or more open models running on your own infrastructure. If your code already uses the OpenAI API, moving to Xinference is a one-line configuration change. This walkthrough takes you from a clean machine to a running model you can call over HTTP in about 10 minutes.

What you need

  • A machine with Python 3.9 or later. A GPU is recommended for anything beyond small models, but Xinference will also run CPU-only for testing.
  • Network access to pull model weights the first time you launch a model.
  • Five to ten minutes, most of it spent waiting on the model download.

Step 1: install

Install Xinference from PyPI. The base package includes the CLI, the local web UI, and the model launcher.

$ pip install "xinference[all]"

This installs the supported serving engines. If you already know which engine you want, such as a leaner install with only the transformers backend, the documentation covers scoped install options.

Step 2: start the Xinference server

Start a local Xinference instance. By default it listens on port 9997 and exposes both a web UI and the OpenAI-compatible API.

$ xinference-local --host 0.0.0.0 --port 9997

Xinference is starting up.
Model registry loaded: 300+ models available.
Listening on 0.0.0.0:9997 ...

For production, you would use a Xinference cluster across your own GPU pool instead. The local server is the fastest way to see the workflow end to end.

Step 3: launch a model

Pick a model from the built-in catalog and launch it. Xinference handles downloading the weights, selecting a compatible serving engine, and exposing the model under a stable model name.

$ xinference launch \
    --model-name llama-3.1-instruct \
    --size-in-billions 7 \
    --model-format pytorch

Launching model: llama-3.1-instruct (7B, pytorch)
Downloading weights ...
Model ready. UID: llama-3.1-instruct-7b-abc123

You can launch the same command with a different --model-name to try an embedding model, a vision-language model, or a speech model. The launch flow is identical across all 300 or more supported models.

Step 4: call it like you'd call any OpenAI-compatible endpoint

Xinference exposes a standard chat completions endpoint. If your application already uses the OpenAI Python SDK, the only change is the base_url.

from openai import OpenAI

client = OpenAI(
    base_url="http://your-xinference:9997/v1",
    api_key="not-needed",  # or your configured key
)

response = client.chat.completions.create(
    model="llama-3.1-instruct",
    messages=[
        {"role": "user", "content": "Summarize the key risks in this contract clause."}
    ],
)

print(response.choices[0].message.content)

Or call it directly with curl, which is useful for a first sanity check:

$ curl http://your-xinference:9997/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "llama-3.1-instruct",
      "messages": [{"role": "user", "content": "Hello!"}]
    }'

What just happened

You now have a model running on your own hardware, reachable through the same API your application already uses. The application code does not need to know the model changed. Often base_url = "http://your-xinference:9997/v1" is the entire migration.

  • You swap models with a launch command instead of a redeploy. You can run several models side by side and route between them.
  • Xinference selects quantization and batching automatically for the GPU class you launch on, so the same launch command works on a single card or a multi-GPU pool.
  • The control plane that runs your first model also scales to a fleet. Routing, autoscaling, and observability come with the same install.

Where to go from here

From a single local instance, you can point Xinference at more than one GPU class to get automatic model-to-hardware placement. You can connect your existing framework: LangChain, LlamaIndex, and your own code all work unchanged against the OpenAI-compatible endpoint. You also need to decide where Xinference runs: self-hosted in your VPC, on the Xinference standard bundle, or fully on-prem. All of these run the same control plane and the same API.

Read next: Production LLM inference checklist: 8 tests before launch