Self-hosted LLM hardware requirements: which GPU do you need?

Self-hosted LLM hardware requirements start with GPU memory. Add the model's weights to the KV cache of every conversation running at once. By our arithmetic, Llama 3.3 70B at 8-bit needs about 71 GB for weights. One 96 GB card fits it, with room for fewer than ten 8,000-token conversations before overhead. Size for concurrent users.

How much VRAM do you need for an LLM?

You need enough GPU memory for the model's weights plus the KV cache. Weights equal parameters times bytes per parameter. A 16-bit model uses 2 bytes per parameter, an 8-bit model uses 1 byte and a 4-bit model uses 0.5 byte.

So Llama 3.3 70B needs about 141 GB at 16-bit, about 71 GB at 8-bit and at least 35 GB at 4-bit. Llama 3.1 8B needs about 16 GB, 8 GB or at least 4 GB.

The KV cache is the working memory the model keeps for each conversation, and it grows with every token. Per token, it is 2 (key and value) x layers x KV heads x head dimension x bytes. The layer counts, head counts and parameter counts come from each model's published files on Hugging Face: Llama 3.3 70B Instruct and Llama 3.1 8B Instruct.

• Llama 3.3 70B has 80 layers, 8 KV heads and a head dimension of 128. With a 16-bit cache, that is 2 x 80 x 8 x 128 x 2 = 327,680 bytes, about 0.33 MB per token.
• Llama 3.1 8B has 32 layers, 8 KV heads and a head dimension of 128. With a 16-bit cache, that is 2 x 32 x 8 x 128 x 2 = 131,072 bytes, about 0.13 MB per token.

The table is our worked arithmetic, not a benchmark. It uses decimal units (1 GB is a billion bytes) and leaves out runtime overhead, so real cards need some headroom. The cache is 16-bit in every row.

The weights are loaded once. Each extra conversation adds its own cache. For background on what the GPU is doing during a request, see what is LLM inference.

Why do concurrent users drive the memory you need?

Every active conversation holds its own KV cache, so memory use rises with the number of people using the model at the same time. Llama 3.3 70B with ten 8,000-token conversations needs about 26 GB of cache on top of its weights.

The vLLM paper (Kwon et al., SOSP 2023) describes the mechanism. It says high throughput requires batching many requests at a time. It also says the KV cache for each request is huge and grows and shrinks dynamically.

The vLLM optimisation guide says requests are preempted when there is not enough KV cache space. It notes that preemption and recomputation can hurt end-to-end latency. As of October 2026, it suggests raising gpu_memory_utilization or tensor_parallel_size to give the cache more memory.

A fast test with one user will not show this. Memory left after the weights sets the ceiling on how many people the card can serve.

Which GPU for LLM inference?

The right card is the one whose memory covers your weights plus the cache for your busy hour. The memory sizes below come from NVIDIA's product pages, as of October 2026. The "what fits" column is our arithmetic, before runtime overhead.

Llama 3.3 70B at 16-bit does not fit one RTX 5090, L40S or RTX PRO 6000. For 70B at 8-bit, two 48 GB cards with tensor parallelism give 96 GB in total, before each card's own overhead.

For a larger multi-card setup, see our earlier post, four lessons from a GPU cluster.

What hardware does a team of 5 to 10 need?

A team of 5 to 10 should size for how many people use the model at the same time, not for headcount.

1. Count simultaneous conversations. Ten people do not always prompt at the same moment, so estimate the busiest period and multiply by the cache figure for your model.
2. Expect users to share throughput. Batching means the engine runs several requests together, so people share the card's throughput. Test at your own load.
3. Include your longest real documents in that test.
4. Test before you buy. Run the exact model you want, with your real prompts, on rented or hosted GPUs. Then buy the card the test says you need.

By our arithmetic, Llama 3.3 70B at 8-bit on one 96 GB card holds fewer than ten 8,000-token conversations before overhead. Treat that as a hypothesis to test, not a promise.

Is a small card (6 to 8 GB) enough?

For one person running a small model, often yes. Llama 3.1 8B at 4-bit needs at least 4 GB for weights, so it fits on an 8 GB card.

A single 8,000-token conversation adds about 1.05 GB of cache. On an 8 GB card, that leaves room for about three conversations at most, before overhead. That is tight for a team, and a team is better served by a larger card.

Where Xinference fits

Xinference is an AI inference platform for enterprise teams. The self-hosted platform runs in your own environment and puts one OpenAI-compatible API in front of every model. You choose the serving engine per model: vLLM, SGLang, llama.cpp, Transformers or MLX. More on the Dedicated Inference page.

If you would rather not buy cards, the Model API is a separate hosted service for two other models, Kimi K3 and DeepSeek V4.1 Flash. As of October 2026, Kimi K3 costs US$3.00 input and US$15.00 output, and DeepSeek V4.1 Flash costs US$0.30 input and US$1.20 output, per 1M tokens. It is hosted in Australia on dedicated GPUs. The pricing page has the details.

Get API access. Don't see the model you need? Talk to us about adding it.