LLM inference means running a trained large language model to turn a prompt into output, one token at a time. The model's weights do not change. It is the opposite of training, which updates those weights. Every chatbot reply, summary or model API call you make is an inference request.
| Training | LLM inference | |
|---|---|---|
| What happens to the weights | They are updated | They are read only |
| When it runs | Before the model is released, and again for each new version | Every time someone sends a prompt |
| Hardware pattern | Large clusters working together on one job | Serving many requests at once, often on fewer GPUs |
| Cost shape | A large cost up front | A running cost that grows with usage |
| Who does it | Mostly model labs, plus teams that fine-tune | Almost every team that uses a model |
| What you change in code | Training code and data pipelines | Usually the model name, endpoint and prompt |
IBM describes LLM inference as running a pre-trained model to generate output tokens for new prompts without updating its learned parameters. The same page calls it a read-only, forward-only process, unlike training.
What happens during LLM inference?
Every request follows the same steps. NVIDIA's engineering blog (November 2023) splits the work into two phases, prefill and decode.
- The prompt is turned into tokens.
- In the prefill phase, the model processes the input tokens and computes intermediate states called keys and values. These produce the first new token.
- In the decode phase, the model generates output tokens one at a time until a stopping condition is met.
- The tokens are turned back into text and returned to you.
The keys and values are stored in a KV cache. NVIDIA names the model weights and the KV cache as the two main contributors to GPU memory use. It also calls the decode phase memory-bound.
In decode, NVIDIA says the time spent moving weights and cached values from memory dominates latency, more than the maths itself. NVIDIA's post covers optimisations aimed at this memory pressure, such as batching and KV cache management.
What is an inference engine?
An inference engine is the software that loads a model and runs it efficiently on your hardware. Its main job is to serve many requests at once without running out of memory.
Batching is the simplest lever. NVIDIA notes that batching improves GPU utilisation and throughput, but batch sizes can only grow until memory overflows.
Two research papers show how engines tackle this. Orca (Yu et al., OSDI 2022) schedules work at the level of each iteration, and each iteration generates one output token per request. That lets a server add new requests without waiting for old ones to finish.
The vLLM paper (Kwon et al., SOSP 2023) tackles memory. It manages KV cache memory the way an operating system pages memory, so almost none is wasted and more requests fit in a batch. Its authors reported that vLLM improved throughput by 2 to 4 times at the same latency, compared with FasterTransformer and Orca.
Common engines include vLLM, SGLang, llama.cpp, Hugging Face Transformers, MLX and NVIDIA TensorRT-LLM. They differ in the hardware they target and the features they offer. Benchmark the model you plan to run on the hardware you plan to use before you commit.
What is an inference provider?
An inference provider is a company that runs models for you. Most charge by usage, usually per token. You send a request to its API and it handles the engine, the GPUs and the scaling.
Hugging Face's Inference Providers is one example. As of October 2026, its documentation describes access to hundreds of models through partner providers, including Baseten, Cerebras, Fireworks, Groq and Together.
For many teams a hosted provider is the simpler choice. There is nothing to deploy, and you pay for what you use. Running your own inference makes sense when you need control over where it runs, which model version you serve, or how the engine is tuned.
The cost comparison depends on your volume. Our post on LLM inference cost, API versus self-hosted walks through it.
Does LLM inference need a GPU?
No. Inference can run without a GPU, but a GPU is usually what makes it fast enough to serve many people.
As of October 2026, the vLLM documentation lists support for NVIDIA GPUs, AMD GPUs and x86, ARM and PowerPC CPUs. So a CPU can run a model.
Whether a CPU is fast enough depends on the size of the model and how many requests arrive at once. Decode is memory-bound, as NVIDIA explains, so memory bandwidth often matters more than raw compute. A small model on a laptop can serve one person. A team sharing one endpoint will usually want a GPU for throughput.
If you are comparing options, choosing an LLM inference stack covers the engine, hardware and provider decisions together. Once a model is live, the production LLM inference checklist covers what to check.
How Xinference supports LLM inference
Xinference is an AI inference platform for enterprise teams. Dedicated deployments run in your own environment, whether that is your VPC, your data centre or your servers.
It puts one OpenAI-compatible API in front of every model. Moving from another OpenAI-compatible provider is a base URL and API key change, not a rewrite. If the term is new to you, read what an OpenAI-compatible API is.
You choose the serving engine per model when you deploy it. The options are vLLM, SGLang, llama.cpp, Transformers or MLX. Changing engine means redeploying the model.
Xinference offers more than 300 individual models across language, embedding, reranking, speech, image and multimodal tasks. Models in the same family are counted separately. Don't see the model you need? Talk to us about adding it.
If you want to run models in your own environment, see dedicated inference. If you would rather pay by the token, Model API gives you open models behind the same API. Contact sales for pricing.
Questions
What is the difference between LLM training and inference?
Training updates a model's weights by learning from data. Inference uses the finished model to answer new prompts without changing the weights.
Is ChatGPT inference?
Yes. A ChatGPT reply, like any chatbot reply, comes from running a trained model on your prompt. The model's weights do not change while it answers.
How much does LLM inference cost?
It depends on the number of tokens, the size of the model and how well you use your hardware. Our post on API versus self-hosted cost explains how to compare them.
Is LLM inference deterministic?
Not by default. With sampling settings such as temperature, the same prompt can give different outputs on different runs. If you need repeatable output, check what your engine documents about it.