4 lessons from a 50-model private GPU cluster.

Over 18 months we helped a global financial services firm move a fragmented, multi-cloud, one-model-per-deployment setup onto a single private cluster of 72 NVIDIA GPUs spanning two hardware generations. The cluster serves 50 or more open-weight models concurrently across investment research, customer service, wealth advisory, and compliance.

It runs open models in several sizes and fine-tunes, an embedding model, a document-understanding OCR model, and a rotating set of fine-tuned variants for individual business lines. Daily traffic exceeds 360,000 requests across 6,000 or more active users, and four separate SLA tiers share the same physical hardware.

This post covers four lessons from the project. None of them came from a model benchmark, a serving-engine tutorial, or a cloud vendor reference architecture.

Lesson 1: match each model to the GPU class where it performs best

Model-to-hardware affinity matters more than raw model capability in most production scenarios. A 70-billion-parameter model on a single accelerator with naive tensor parallelism, serving long-context document workloads, can run measurably slower end to end than a well-tuned 32-billion-parameter model split across two accelerators with disaggregated prefill and decode and a properly sized KV cache budget. The bigger model has more capability per token, but that does not help when it produces tokens too slowly for the workload's latency budget.

The same pattern shows up across the cluster. Large open-weight variants run best on high-bandwidth GPUs with FP8 quantization. Embedding models are more cost-effective on smaller consumer-class cards, where bandwidth is not the bottleneck and price per unit of throughput is much better. A model with a vision encoder front end and a text decoder back end benefits from running those two stages on different instance types.

So do not pick hardware once and then squeeze every model onto it. Pick the routing layer first, and let the platform place each model on the GPU class where it performs best, with continuous re-benchmarking as engines and quantization recipes change. We built an automated benchmark-on-ingest pipeline: every new model is tested across every GPU class, every supported inference engine, and every viable quantization recipe before it sees production traffic. The routing manifest comes from that benchmark rather than from a human guess.

This cut time-to-first-token by 35 to 60% compared with the team's previous single-engine, hand-placed deployments.

Lesson 2: separate prefill and decode onto different instance pools

Separating prefill from decode is a well-known idea in inference research, and its operational implications are easy to miss. In a typical serving setup, prefill (processing the input prompt) and decode (generating output tokens) run on the same instance, sharing KV cache, scheduler queues, and memory bandwidth. That works when the workload is homogeneous and breaks down when it is not.

Want the research behind this? The Adaptive Inference Reader is an annotated list of 14 papers on efficient LLM inference, including prefill and decode splitting.

This customer's workload shows the problem clearly. Analysts upload 40-page documents for parsing, which means a very large prefill followed by a short, structured decode. At the same time, real-time advisory chat needs a P99 time-to-first-token budget under 200 milliseconds. When the two share a GPU, the long document prefill monopolizes compute and bandwidth for seconds at a stretch, and the chat session next to it sees latency spikes of several seconds.

The fix is structural: separate prefill and decode into distinct instance pools, each with its own autoscaling, with KV cache transferred between them over a fast interconnect at the boundary. The prefill pool scales on batch size and prompt length distribution. The decode pool scales on queue depth and active session count. The routing layer dispatches each request's prefill and decode to the right pool.

Real-time SLA compliance went from intermittent to consistent with no increase in hardware budget, even when document-parsing traffic spiked tenfold during quarterly research cycles.

Lesson 3: use idle overnight capacity with tidal scheduling

Most enterprise AI clusters are sized for peak business-hours load and sit at 20 to 30% utilization overnight and on weekends, while depreciation on the hardware runs around the clock.

We call the fix tidal scheduling: the same physical hardware serves different workload profiles at different times of day, and the cluster reconfigures itself automatically at the boundary. During trading hours, the full cluster is configured for real-time inference: advisory chat, customer service, on-demand research, with strict SLA tiers and disaggregated prefill and decode for the document pipeline. Off-hours, the cluster switches to larger batch sizes, embedding regeneration across the historical document corpus, fine-tuning jobs, compliance re-scoring, and benchmarks for newly ingested checkpoints.

The reconfiguration is automated and idempotent. At the close of trading hours, the platform drains real-time pools, swaps the scheduling policy, repartitions GPU memory, and starts pulling from batch queues. At the open, it reverses the process. No human is in the loop.

This raised overall GPU utilization by 30 to 45% without purchasing a single additional card.

Lesson 4: shorten model onboarding from days to hours

During the project, useful checkpoints landed on model hubs roughly once a week: frontier-quality multilingual models, domain-specialized fine-tunes, better OCR, better embeddings, and better long-context handling. If your platform takes five days to onboard a model into production, which is close to the industry norm, you stay one or two useful model generations behind teams that can do it in hours.

This customer's old onboarding cycle was a familiar one: download weights and run informal benchmarks, build a container image with the right engine version, hand-tune quantization, script deployment per environment, run integration tests, and roll out with manual monitoring. It typically took five days, and the team evaluated two models a month at best.

The new pipeline starts with a model URL submitted through the control plane. It then runs automated benchmarking across every GPU class and engine, an automated quantization sweep calibrated against the relevant business-line data, signed manifest generation, and a canary route with traffic-shadow comparison against the incumbent model. Promotion follows, with automatic rollback on any SLA regression.

As a result, onboarding time dropped from days to hours. The research team's model-evaluation throughput went up sixfold, and the platform team's manual effort per model dropped sharply.

The cumulative numbers

The numbers come from four architectural decisions taken together: hardware-aware routing, prefill and decode disaggregation, tidal scheduling, and automated onboarding. Any one of them alone improves things modestly. All four together produce the order-of-magnitude shift.

What we would tell a team starting a similar project

• Build the routing layer first. Everything else is downstream of it.
• Adopt prefill and decode disaggregation early, even if the initial workload looks homogeneous. It will not stay that way, and retrofitting it later costs roughly five times more.
• Treat onboarding as a product surface rather than an operational task. The team that gets models into production fastest gains a disproportionate advantage.
• Resist standardizing on a single inference engine. vLLM, TensorRT-LLM, SGLang, and llama.cpp each win on different workloads. The control plane should hide that choice from application teams.

Topologies and models will change. Routing, disaggregation, tidal scheduling, and automated onboarding remain the building blocks.

Read next: Production LLM inference checklist: 8 tests before launch