There is no universal token volume at which self-hosting becomes cheaper than an API. The answer changes with the model, prompt mix, output length, GPU price, utilization, reliability target and engineering overhead.
That is why simple claims such as "self-host at 10 million tokens per day" age badly. A cached, low-output API workload can remain inexpensive at high volume. A steady extraction workload on a well-utilized open model may favor private capacity much earlier.
The useful comparison is cost per successful unit of work under the same quality and latency requirements.
Start with one workload, not the whole company
Choose a specific endpoint such as document extraction, customer-support drafting or code review. Record at least four weeks of traffic if possible.
Collect:
- Daily input and output tokens.
- Request count and concurrency by hour.
- Prompt-cache hit rate.
- Required model quality and evaluation score.
- P95 latency and availability target.
- Retry and failure rate.
- Data residency or privacy constraints.
Different workloads often produce different answers. Combining them into one average can hide a strong self-hosting candidate and a bursty workload that should stay on an API. This per-workload view matters more as the portfolio grows into a fleet of models rather than a single endpoint.
Path 1: managed model API
Managed APIs charge primarily for input and output tokens, with separate rates for caching, batch processing or priority service in some cases.
Use this simplified monthly model:
API cost = input tokens x input rate + output tokens x output rate + tool and service charges
Add the cost of retries and fallback calls. Output tokens often cost more than input tokens, so two workloads with the same total token count can have different bills.
Managed APIs are attractive when traffic is small or uncertain, a frontier model is required, or the team does not want to operate inference infrastructure. They also scale down to zero when unused.
The trade-offs are variable spend, an external data path and dependence on the provider's model lifecycle, quotas and service behavior.
Path 2: self-hosted inference
Self-hosted cost begins with capacity rather than tokens:
Monthly platform cost = GPU hours + CPU and memory + storage + network + software + operations
Convert that into a workload cost:
Cost per successful request = monthly platform cost / successful production requests
The denominator matters. Idle GPUs make each request expensive. Higher utilization can reduce unit cost until queueing or latency requires more capacity.
For a public reference point, AWS lists single H100 P5 capacity-block pricing by region, while Google Cloud publishes GPU and accelerator-optimized VM pricing. These figures do not include every infrastructure or staffing cost and can change. Use the region, commitment and availability that match the planned deployment.
Self-hosting becomes more attractive when traffic is steady, a suitable open-weight model meets the quality bar, existing GPU capacity can be shared and data must stay inside a controlled environment.
The operating burden includes model rollout, autoscaling, monitoring, security updates, driver compatibility, incident response and capacity planning. Price that work honestly.
Path 3: hybrid inference
A hybrid architecture assigns workloads rather than choosing one provider for everything.
Steady, high-volume or sensitive requests can run on private infrastructure. A managed API can handle rare reasoning tasks, experiments, temporary bursts or a fallback path. A unified API and routing layer keep that split out of application code.
Hybrid does not automatically save money. It adds policy, routing and observability requirements. It works when the workload split is explicit and teams can trace where each request ran and why.
An illustrative comparison
Assume a workload can use either a managed model API or one self-hosted GPU. The API bill scales with tokens. The GPU bill is mostly fixed while the instance is running.
At low utilization, the API is likely to cost less because the team pays only for use. As private capacity fills, its unit cost falls. Once the GPU reaches a latency or memory limit, another replica adds a step to the cost curve.
This crossover cannot be expressed as one token number without specifying:
- The API model and current rates.
- The open model and measured tokens per second.
- Input and output ratio.
- Cache behavior.
- GPU type, region and purchase model.
- Availability headroom.
- Engineering and platform costs.
- Evaluation score and retry rate.
If any of those inputs changes, recalculate.
The cost levers teams miss
Model choice is only one lever. Continuous batching, KV-cache reuse, quantization, prompt caching and model co-location can change the amount of useful work produced by each GPU hour.
Routing can also reduce cost by reserving larger models for requests that need them. Measure cost per successful request after routing, including any retry on a stronger model.
Finally, remove unused output. Shorter prompts and constrained responses may reduce spend on both APIs and private infrastructure while improving application consistency.
A better decision rule
Keep a workload on an API when it is small, highly variable or dependent on a proprietary capability. Evaluate self-hosting when traffic is steady, privacy matters, a suitable open model passes evaluation and the team can operate the service. Use a hybrid design when the portfolio contains both types.
Xinference runs open and custom models behind one OpenAI-compatible API across managed cloud, private cloud and on-prem environments. The ROI calculator on the product page can provide a starting estimate, but a production replay on your own prompts is the deciding test.
Build the model with editable assumptions and update it quarterly. Prices and model performance move too quickly for a one-time break-even claim.