Self-hosting an LLM is worth it when traffic is steady, an open model meets your quality bar and you have staff to run it. For low or bursty traffic, an API is cheaper. On October 2026 AWS prices, one 8 x H100 server running Llama 3.3 70B matches Amazon Bedrock on compute at about 1.6 billion tokens a day.
Is self-hosting an LLM cheaper than an API?
Only when the GPUs stay busy. The cost trade-off between self-hosted and managed LLM inference is variable spend against fixed capacity. A pay-per-token API turns every token into a bill and charges nothing when idle. Self-hosting turns GPUs and staff into a fixed monthly cost.
| Cost factor | Pay-per-token API | Self-hosted |
|---|---|---|
| How you pay | Per input and output token | Per GPU hour or owned hardware, plus staff |
| Cost when idle | Nothing | The full capacity cost |
| Cost as traffic grows | Rises in line with tokens | Flat until a GPU is full, then a step up |
| Engineering effort | Integration only | Deployment, scaling, monitoring, upgrades, on-call |
| Data path | Requests go to the provider | Requests stay in your environment |
| Model choice | The provider's catalogue | Any open-weight or custom model you can run |
How do you compare self-hosted and API cost?
Start with one workload, not the whole AI budget. Pick a specific endpoint such as document extraction, support drafting or code review. Record four weeks of traffic if you can.
Collect these inputs:
- Daily input and output tokens, and request count by hour.
- Prompt-cache hit rate, retry rate and failure rate.
- Required model quality, P95 latency and availability target.
- Data residency or privacy constraints.
Do not average workloads together. That hides a strong self-hosting candidate next to a bursty one that belongs on an API. Then calculate both sides as cost per successful request. The API side is a token sum.
API cost = input tokens x input rate + output tokens x output rate + tool and service charges.
Add retries and fallback calls. Input and output rates can differ, so check both.
The self-hosted side is a capacity sum divided by the work it serves.
Monthly platform cost = GPU hours + CPU and memory + storage + network + software + operations.
Cost per successful request = monthly platform cost / successful production requests.
Idle GPUs raise this figure, because fewer requests share the same monthly cost. Higher utilisation lowers unit cost until queueing or latency forces you to add capacity.
What hidden costs change the answer?
The GPU bill is the easy part. People and headroom can change the answer as much.
- Engineering and on-call time. Model rollout, autoscaling, monitoring, driver compatibility, security updates and incident response.
- Idle headroom. You size for peak traffic and your availability target, not the average.
- Evaluation and upgrades. Each new model or engine version needs testing on your own prompts.
- Exit and switching. On an API, tuned prompts take effort to move. On your own hardware, commitments can outlast the workload.
Include all of these in the monthly platform cost. Counting only GPU hours against token prices favours self-hosting unfairly.
When does self-hosting break even? A worked example
On public AWS prices, one 8 x H100 server running Llama 3.3 70B costs the same as Bedrock for that model. That happens at about 1.6 billion tokens a day. That covers compute only. It is not a price quote, and it ignores staff, storage and network.
| Input | Value | Source |
|---|---|---|
| API price | Llama 3.3 70B on Amazon Bedrock, US East (N. Virginia): US$0.72 per 1M input tokens and US$0.72 per 1M output tokens. Batch: US$0.36 per 1M | Amazon Bedrock pricing, AWS price list dated 6 October 2026 |
| GPU price | p5.48xlarge, 8 x H100, US East (N. Virginia): US$47.757 per hour effective rate. Sydney is listed at the same rate | EC2 Capacity Blocks for ML pricing, checked 8 October 2026 |
Our arithmetic:
- GPU cost for 30 days, using the Capacity Block rate as a monthly proxy: US$47.757 x 720 hours = US$34,385.04.
- Tokens the API would bill for the same money: US$34,385.04 / US$0.72 per million = about 47.8 billion tokens a month.
- That is about 1.59 billion tokens a day, or about 18,400 tokens per second, around the clock.
Self-hosting wins on compute only if real traffic averages about 18,400 tokens per second, around the clock. One server must also serve it at your quality and latency targets. This counts input and output tokens together, so your input to output ratio changes how hard that target is. Measure throughput with your own prompts and engine settings. Staff and hidden costs push break-even higher.
The batch rate moves the line further. At US$0.36 per million tokens, the same US$34,385.04 buys about 95.5 billion tokens a month. That is about 3.2 billion tokens a day. If your work can wait for batch results, the API stays cheaper for longer.
AWS says Capacity Blocks prices are next scheduled to update in January 2027. Capacity Blocks are one purchase option. On-demand, reserved or owned hardware gives a different figure.
Why can't the crossover be one token number?
The API bill scales with tokens, while the GPU bill is mostly fixed. When a GPU hits a latency or memory limit, another replica adds a step to the cost curve.
The 1.6 billion figure comes from one model, one price list and one instance type. A different model, cache hit rate, GPU, purchase model or headroom target gives a different answer. Update the cost model quarterly, or when prices change.
Is cost the only reason to self-host?
No. Data control and model choice can matter as much as price. The OAIC says this about control: "Deploying a system locally or on premises is likely to be more privacy-preserving as it limits the risks of third party access to the data" (OAIC guidance).
The same guidance does not rule out the cloud. For cloud deployments, it says "it will be necessary to consider" security and privacy risks, including where servers are located.
So privacy can justify self-hosting even when the token maths favours an API. Check it against your own obligations.
Model choice is the other reason. Self-hosting lets you run any open-weight or custom model and pin its version. An API limits you to the provider's catalogue.
When should you use an API, self-hosting or both?
Keep a workload on an API when it is small, variable or needs a proprietary model. Evaluate self-hosting when traffic is steady and an open model passes your evaluation. If your portfolio has both kinds, use both.
A hybrid design assigns workloads rather than picking one provider for everything. Steady or sensitive requests run on private infrastructure. An API handles rare reasoning tasks, experiments, bursts or a fallback. Hybrid adds policy and observability work, so it does not automatically save money.
For lessons on running many models on private GPUs, read 4 lessons from a 50-model private GPU cluster. To compare platforms, see self-hosted inference platforms compared. For a wider cost model, see sovereign AI vs cloud TCO.
How Xinference fits
Xinference is an AI inference platform for enterprise teams. It runs open and custom models behind one OpenAI-compatible API in your own environment, whether that is your VPC, your data centre or your servers. You choose the serving engine for each model when you deploy it.
The Model API serves open models by the token through the same API, with pricing from sales. Every deployment option uses the same API, so moving a workload between them is a base URL change. See Dedicated Inference, the comparison pages and pricing.
Our Sovereign AI TCO Guide builds the cost model step by step. See also Open models or commercial APIs?
Sources: Amazon Bedrock pricing (AWS price list dated 6 October 2026), EC2 Capacity Blocks pricing (checked 8 October 2026) and OAIC guidance.
Questions
Is it worth it to self-host AI for a small team?
Usually not for cost alone. Small or uncertain traffic leaves GPUs idle, and an API charges nothing when unused. Self-host when privacy or model control matters and someone can run it.
Is a local LLM really worth it?
On a laptop or workstation, yes for private experiments, offline use and learning. For production traffic, apply the same test as for a server.
How much does it cost to self-host an LLM?
One 8 x H100 Capacity Block at US$47.757 an hour costs about US$34,385 for 30 days, before staff, storage and network. Source: AWS, checked 8 October 2026.
When does self-hosting an LLM break even?
In the example above, at about 1.6 billion tokens a day against Llama 3.3 70B at US$0.72 per million tokens. Batch pricing doubles that, and staff costs push it higher.
Can you self-host an LLM and still use an API?
Yes. Run steady or sensitive workloads on your own infrastructure and send bursts or frontier-model tasks to an API. If both sides use the same OpenAI-compatible API, the code change is a base URL and API key. You still need to re-evaluate the model on your prompts.