Enterprises running large language models choose between a managed API, such as Amazon Bedrock, Azure OpenAI or Together AI, and self-hosted inference on a platform like Xinference. Managed APIs win on speed to start. Self-hosting wins on control over data, models and the cost base at steady, high volume.
If the question is whether your team can use ChatGPT for sensitive files at all, see Can you use ChatGPT for confidential documents?. For the cost side in detail, with a worked example from public prices, see Self-hosted LLM vs API cost: how to compare.
What managed APIs do well
Managed inference APIs let a team call a model within minutes, with no infrastructure to buy or run. The provider handles scaling, patching and availability.
- Fast start: from sign-up to first API call in minutes
- Automatic scaling through traffic spikes
- No on-call duty for infrastructure failures
- The provider's own compliance reports, which your security team can review
This suits experiments, small or spiky workloads, and teams with a "no infrastructure" rule.
Where managed API costs grow
Most managed APIs charge per token, with separate input and output rates that vary by model, region and commitment. The bill grows in line with usage, so a workload that moves from pilot to every team in the company can grow its bill many times over in a few months.
Two things make this hard to manage:
- Forecasting. Token use depends on prompt length, retrieval context and how often agents call the model, which are hard to predict before launch.
- Model choice. If the catalogue offers only large models for a task, you pay large-model rates even where a smaller open model would do the job.
Self-hosting changes the cost from tokens to capacity: GPU hours or hardware you own, plus the people who run it. Whether that is cheaper depends on how busy the GPUs are. Our cost comparison shows how to work out the break-even point for your own workload.
Data control
With a managed API, prompts and outputs pass through the provider's systems. Providers offer regional endpoints, encryption and contractual terms, and for many teams that is enough.
Some organisations need more: their risk policy, sector rules or customer contracts require that model traffic stays inside infrastructure they run. Self-hosted inference meets that by design, because the model runs in your VPC, your data centre or on your own servers. For what this means in Australia, see Data sovereignty in Australia: what it means for AI.
Model choice and lock-in
Managed APIs give you the provider's catalogue. New open models arrive when the provider adds them, and some features rely on provider-specific SDKs, so moving workloads later can mean code changes.
Xinference serves 300+ open models behind one OpenAI-compatible API, plus your own fine-tunes. Applications written for the OpenAI SDK switch by changing the base URL. Models use standard formats, so you can move them between environments.
Comparison table
| Dimension | Managed APIs | Xinference (self-hosted) |
|---|---|---|
| Where it runs | Provider's cloud, regional endpoints | Your VPC, data centre or servers |
| Models | Provider's catalogue | 300+ open models and your fine-tunes |
| API | Provider SDK or OpenAI-compatible, depending on provider | One OpenAI-compatible API for every model |
| Cost structure | Per token, varies by model and region | GPU capacity plus the team to run it |
| Time to first call | Minutes | Hours to days, depending on infrastructure |
| Operations | Provider's job | Your team, with Xinference handling the serving layer |
| Best fit | Pilots, small or spiky workloads | Steady volume, strict data rules, many models |
When each approach wins
Choose a managed API if:
- the workload is experimental, small or very spiky
- your team cannot take on GPU operations
- the provider's catalogue covers your models
- your data rules allow third-party processing
Choose self-hosted inference with Xinference if:
- usage is steady and high enough to keep GPUs busy
- you need open-weight or custom models the provider does not offer
- policy or contracts require model traffic to stay in your infrastructure
- you want one API across many models and teams
Many organisations use both: a managed API for early pilots and a self-hosted platform once a workload proves itself.
What self-hosting takes
Self-hosted inference is real operational work. Your team, or a partner, needs to:
- choose and procure GPU capacity, in the cloud or on premises
- install Xinference and deploy models for your workload
- monitor GPU health, latency and application service levels
- plan capacity, apply updates and manage the model lifecycle
Xinference reduces the serving plumbing you would otherwise build, but the team still runs the environment underneath. Our production LLM inference checklist lists the tests to run before launch.
Questions
Is self-hosted LLM inference cheaper than a managed API?
It can be at steady, high volume, because you pay for capacity rather than tokens. At low or spiky volume a managed API is often cheaper. Compare both on your own token volume and GPU utilisation before deciding.
Can I keep using the OpenAI SDK with Xinference?
Yes. Xinference exposes an OpenAI-compatible API, so existing OpenAI SDK clients work by changing the base URL and API key.
Do I have to choose one approach?
No. Many teams pilot on a managed API and move steady workloads to self-hosted inference later. An OpenAI-compatible API on both sides keeps that move small.
How long does a self-hosted deployment take?
It depends on your infrastructure. With GPU capacity in place, deploying a model is quick; procurement, security review and integration usually take longer than the install.