Self-hosted inference platforms compared: buyer’s guide 2026.

A self-hosted LLM is a language model you run on infrastructure you control, such as your own servers or your cloud account. For one developer on a laptop, Ollama is the quickest start. For one model under heavy traffic, use a serving engine such as vLLM or SGLang. For many models across teams, add a platform layer that runs those engines behind one API.

What is a self-hosted LLM?

A self-hosted LLM runs on hardware or cloud capacity that your organisation operates, instead of being called through a provider's API. You download open-weight model files, load them with an inference engine and expose an endpoint to your applications. The opposite is a managed API, where the provider runs the model and you pay per token.

Self-hosting covers a wide range of setups. It can be a single GPU workstation, a cluster in your data centre, or GPU instances inside your own VPC. What they share is that you choose the model, the engine and where requests are processed.

Why do teams self-host LLM inference?

Teams self-host for control over data, cost shape and model choice. Requests stay inside an environment you run. Cost follows the capacity you provision rather than every token you send. You can deploy custom models, fine-tunes and quantised variants that a provider may not offer.

The trade-off is operational. Self-hosting needs GPU capacity, tuning knowledge, monitoring and someone on call. A hosted API is often the better choice for small or unpredictable traffic. It is also the only route to frontier models sold solely as an API. For the full cost comparison, see LLM inference cost: managed API vs self-hosted vs hybrid. For the provider side, see Xinference vs managed inference APIs.

What hardware do you need to self-host an LLM?

GPU memory is the first constraint. The model weights must sit in GPU memory while it serves. The KV cache, which stores attention state for active requests, needs room on top. A rough sizing rule from the LLM Inference Handbook is:

Memory (GB) = parameters in billions x (bit precision / 8) x (1 + overhead)

Two worked examples using that rule:

Lower precision cuts memory but can affect accuracy, so test quality on your own prompts after quantising. Leave headroom beyond the weights. Longer contexts and more concurrent users grow the KV cache quickly.

Which self-hosted LLM platforms are there?

The options sit at three levels: local runners, serving engines and platforms that manage engines. The sections below cover the most common choice at each level.

Ollama: local models with almost no setup

Ollama is the easiest way to run an open model on a laptop or workstation. You install it, pull a model and start chatting or calling a local API. Its GitHub page lists families such as Kimi, GLM, DeepSeek, gpt-oss, Qwen and Gemma, and it offers an OpenAI-compatible endpoint.

The LLM Inference Handbook describes Ollama as built on llama.cpp and aimed at simple local use. That makes it a strong tool for prototyping and developer onboarding. It is not where most teams run shared production traffic for hundreds of users.

vLLM: high-throughput serving for one model at a time

vLLM is an open-source serving engine its maintainers describe as "high-throughput and memory-efficient" (GitHub). Its memory management comes from the PagedAttention paper. It exposes an OpenAI-compatible HTTP server and, per its installation docs, runs on NVIDIA CUDA, AMD ROCm, Intel XPU and CPU, among others.

vLLM is the usual choice when one or two models face heavy concurrent traffic. You still own deployment, scaling, monitoring and failover. If you serve several models, each typically runs as its own server, and your team connects them behind a gateway. Read more in Xinference vs vLLM.

SGLang and other engines

SGLang is a "high-performance serving framework for large language models and multimodal models" (GitHub). It covers similar ground to vLLM, and the faster of the two depends on your model and traffic. Other engines worth knowing:

• TensorRT-LLM: NVIDIA's library for optimised inference on NVIDIA GPUs.
• llama.cpp: LLM inference in C/C++, portable across a wide range of hardware, including consumer GPUs.
• Hugging Face TGI: its documentation says it is now in maintenance mode. If you run it today, plan a move to another engine.

NVIDIA NIM: packaged containers for NVIDIA GPUs

NVIDIA NIM packages models into "validated containers" with "curated weights" for NVIDIA hardware (NIM overview). In NIM 2.0, the core engine inside each container is vLLM, with one backend per container. NIM suits teams that have standardised on NVIDIA GPUs and want NVIDIA-tested images. Check NVIDIA's current licensing terms for production use. See Xinference vs NVIDIA NIM for a closer look.

Xinference: one API across many models and engines

Xinference is an AI inference platform for enterprise teams. It runs in your own environment: your VPC, your data centre or your servers. It puts one OpenAI-compatible API in front of more than 300 models across language, embedding, reranking, speech, image and multimodal tasks, plus your own fine-tunes.

You choose the serving engine per model when you deploy it: vLLM, SGLang, llama.cpp, Transformers or MLX. Changing engine means redeploying that model. TensorRT-LLM is available in Xinference Enterprise only. The trade-off is one more layer to run. If you serve a single model, a bare engine is simpler. Details are on the Dedicated Inference page.

How do self-hosted LLM platforms compare?

They differ most in scope: a local runner, an engine for one model, a packaged container, or a platform over several engines.

Third-party details from each project's own documentation, checked 6 October 2026.

Can you combine more than one platform?

Yes, and many teams do. A common path starts with Ollama on laptops for early prompt work. A serving engine in staging then tests real load. The production setup depends on how many models you run. A model that works in one usually runs in the others without retraining, although tuning differs.

The handbook makes the same point: no single runtime suits every model or workload. Support for a new model often lands in one engine first. That is the case for keeping the application layer engine-agnostic. If every service calls one OpenAI-compatible endpoint, you can change what sits underneath without touching application code. This matters more as the portfolio grows into a fleet of models rather than a single endpoint.

How do you choose a self-hosted LLM platform?

Start from the workload, not the tool. Answer these in order:

1. How many people will use it? One developer exploring models: Ollama.
2. How many models will you serve? One or two under heavy traffic, with an engineering team to run them: vLLM or SGLang.
3. Are you standardised on NVIDIA GPUs and want vendor-tested images? Look at NVIDIA NIM.
4. Do you need many models of different types behind one API, in your own environment? Look at a platform such as Xinference.
5. Still unsure? Run the same representative workload on two shortlisted options. Compare model support, throughput, latency and the hours your team spends running each.

For side-by-side pages against managed providers, see all our comparisons.