Many enterprise AI teams start with frontier APIs and a retrieval-augmented generation (RAG) pipeline. RAG is a common first choice for good reason: it's cheap to start, fast to ship, and easy to keep current as source documents change. For a lot of internal search and document Q&A use cases, it's the right answer.

But RAG often gets a team to a demo without getting it to production economics. Fine-tuning is what teams add once RAG hits its ceiling.

RAG and fine-tuning do different jobs. RAG gives a model knowledge it doesn't already have. Add your policy documents, product manuals, and case files, and the model can quote them back accurately. Fine-tuning changes the model's behavior: how it reasons, what tone it uses, which patterns it recognizes, how consistent its output structure is.

Most enterprise workloads don't need both. Four situations are the exception, because prompt engineering and retrieval tuning alone don't get teams where they need to go.

Scenario 1: token costs at scale

The most common trigger for considering fine-tuning is the invoice. A customer support workflow processing two million tickets a month on a frontier API can easily run into seven figures a year. Costs stack up: you pay per token, for every retry, for context windows bloated by retrieval, and again at peak hours.

Not sure whether an open model can replace your API at all? Download our guide, Open models or commercial APIs, for a simple way to test it.

Run the same workload on a fine-tuned smaller open model, served in your own environment, and inference cost can drop by up to 70% per request. Latency at the 95th percentile often improves several times over, because you aren't waiting on someone else's queue. Throughput is limited by your own hardware budget instead of a provider's rate limit.

Model distillation is a common way to get that smaller model. Use the frontier model once to generate a large batch of high-quality reference answers for your specific workload, then fine-tune a small open model on that synthetic dataset. The small model won't write poetry like a frontier system, but on your narrow task it can match or beat one, at a fraction of the unit cost.

Scenario 2: output consistency is a product requirement

Some workloads can't accept "usually right": equipment fault diagnostics, code generation against an internal SDK, structured extraction for compliance review, automated coding of clinical or claims data. These workloads need output in a predictable schema every time, vocabulary anchored in your domain rather than a general-purpose corpus, and reasoning patterns that match how your specialists actually work.

You can push a general-purpose model toward this with elaborate prompts and few-shot examples, but the variance never fully goes away, and the prompt overhead grows every time the workload scales. Fine-tuning builds the behavior into the model itself, so the prompt gets shorter, the output gets more reliable, and the eval scores get tighter.

Scenario 3: your data is proprietary

A useful test for whether fine-tuning is worth it: could a competitor reproduce this model by reading public sources? If so, RAG is probably enough. If not, because the value lives in your internal incident history, your customer interaction logs, your proprietary process knowledge, or years of curated annotations, then fine-tuning converts that data asset into a model asset competitors can't replicate.

This matters most in regulated industries. Financial services, healthcare, and government teams weigh where the model and the data live as well as cost when they choose between RAG and fine-tuning.

Scenario 4: sovereignty and compliance

Sovereign data requirements shape procurement decisions across financial services regulation, outsourcing guidelines, and privacy frameworks. They all point the same way: core workloads need to run in environments the business controls.

A fine-tuned open model deployed on infrastructure you own keeps every byte of inference traffic inside your security perimeter, which a closed-source API can't do. In regulated industries this is often the deciding factor, before cost or quality even comes up.

Why fine-tuning is practical for more teams

Three developments make fine-tuning practical for teams that ruled it out before. Open models in the 7 to 32 billion parameter range are capable: on narrow, well-defined tasks, a fine-tuned mid-sized open model can land within a small margin of frontier quality, and often surpasses it on workload-specific evaluation sets. Parameter-efficient fine-tuning methods cut the cost sharply. They update a small fraction of the model's parameters rather than retraining the whole thing, so a fine-tune fits on a handful of GPUs in a day.

The third is the serving layer, which most teams underestimate. A great fine-tune served badly is worse than a mediocre API. Inference platforms can serve fine-tuned open models with high throughput, autoscaling, observability, and routing, in your own environment, on your own GPUs, behind a single endpoint.

A practical decision framework

Before fine-tuning anything, run the workload through three filters:

  1. Have you tried prompt engineering and RAG seriously? If not, start there. Most use cases don't need fine-tuning, and shipping faster matters.
  2. Is the workload repetitive at scale? Fine-tuning's economics only kick in when you're running the same task millions of times. One-off workloads don't justify the investment.
  3. Is the data yours, and does control over it matter? If yes on both, fine-tuning turns a data asset into a durable model asset.

If a workload clears all three filters, fine-tuning is an operations decision rather than a research project, and it often decides whether an AI product is sustainable in production.

Read next: Xinference Model API · How to choose an LLM inference stack