4 lessons from a 50-model private GPU cluster.

Over the past 18 months we helped a global financial services firm move a fragmented, multi-cloud, one-model-per-deployment setup onto a single private cluster of 72 NVIDIA GPUs spanning two hardware generations, serving 50 or more open-weight models concurrently across investment research, customer service, wealth advisory, and compliance.

The cluster runs a mix of leading open models in several sizes and fine-tunes, an embedding model, a document-understanding OCR model, and a rotating set of fine-tuned variants for individual business lines. Daily traffic runs past 360,000 requests across 6,000 or more active users, with four separate SLA tiers sharing the same physical hardware.

This post covers the four things we learned along the way that no model benchmark, no serving-engine tutorial, and no cloud vendor reference architecture told us up front.

Lesson 1: the wrong model on the right GPU is worse than the right model on the wrong GPU

Model-to-hardware affinity matters more than raw model capability in most production scenarios. A 70-billion-parameter model on a single accelerator with naive tensor parallelism, serving long-context document workloads, can run measurably slower end to end than a well-tuned 32-billion-parameter model split across two accelerators with disaggregated prefill and decode and a properly sized KV cache budget. The bigger model has more capability per token. It does not matter, because it produces tokens too slowly for the workload's latency budget.

The same pattern shows up across the cluster. Large open-weight variants prefer high-bandwidth GPUs with FP8 quantization. Embedding models run more cost-effectively on smaller consumer-class cards, where bandwidth is not the bottleneck and price per unit of throughput is dramatically better. A model with a vision encoder front end and a text decoder back end benefits from splitting those two stages across different instance types entirely.

The implication: do not pick hardware once and then squeeze every model into it. Pick the routing layer first, and let the platform place each model on the GPU class where it performs best, automatically, with continuous re-benchmarking as engines and quantization recipes evolve. We built an automated benchmark-on-ingest pipeline: every new model is tested across every GPU class, every supported inference engine, and every viable quantization recipe before it ever sees production traffic. The routing manifest comes from that benchmark, not from a human guess.

Result: 35 to 60% lower time-to-first-token compared with the team's previous single-engine, hand-placed deployments.

Lesson 2: prefill and decode should not share a room

This idea is now well known in the inference research community, but the operational implications are still underrated. In a typical serving setup, prefill (processing the input prompt) and decode (generating output tokens) run on the same instance, sharing KV cache, scheduler queues, and memory bandwidth. That works fine when the workload is homogeneous. It fails painfully when it is not.

This customer's workload is the textbook case. Analysts upload 40-page documents for parsing: a massive prefill followed by a short, structured decode. At the same time, real-time advisory chat needs a P99 time-to-first-token budget under 200 milliseconds. When the two share a GPU, the long document prefill monopolizes compute and bandwidth for seconds at a stretch, and the chat session sitting next to it sees latency spikes into multi-second territory.

The fix is structural: separate prefill and decode into distinct instance pools, each with its own autoscaling, with KV cache transferred between them over a fast interconnect at the boundary. The prefill pool scales on batch size and prompt length distribution. The decode pool scales on queue depth and active session count. The routing layer dispatches each request's prefill and decode to the right pool.

Result: real-time SLA compliance went from intermittent to consistent, with no increase in hardware budget, even when document-parsing traffic spiked tenfold during quarterly research cycles.

Lesson 3: your GPUs can work two shifts

Most enterprise AI clusters are sized for peak business-hours load and then sit at 20 to 30% utilization overnight and on weekends. Depreciation on that hardware runs around the clock. The work it does does not.

We call the fix tidal scheduling: the same physical hardware serves different workload profiles at different times of day, and the cluster reconfigures itself automatically at the boundary. During trading hours, the full cluster is configured for real-time inference: advisory chat, customer service, on-demand research, with strict SLA tiers and disaggregated prefill and decode for the document pipeline. Off-hours, the cluster shifts: larger batch sizes, embedding regeneration across the historical document corpus, fine-tuning jobs, compliance re-scoring, and benchmarks for newly ingested checkpoints.

The reconfiguration is automated and idempotent. At the close of trading hours, the platform drains real-time pools, swaps the scheduling policy, repartitions GPU memory, and starts pulling from batch queues. At the open, it reverses. No human in the loop.

Result: 30 to 45% higher overall GPU utilization, without purchasing a single additional card.

Lesson 4: model onboarding speed is a competitive advantage

In open-weight AI today, useful new checkpoints land on model hubs roughly once a week: frontier-quality multilingual models, domain-specialized fine-tunes, better OCR, better embeddings, better long-context handling. If your platform takes five days to onboard a new model into production, which is close to the industry norm, you are perpetually one or two useful model generations behind teams that can do it in hours.

This customer's old onboarding cycle looked familiar: download weights and run informal benchmarks, build a container image with the right engine version, hand-tune quantization, script deployment per environment, run integration tests, and roll out with manual monitoring. Elapsed time: typically five days. New models evaluated per month: two, on a good month.

The new pipeline starts with a model URL submitted through the control plane. From there: automated benchmarking across every GPU class and engine, an automated quantization sweep calibrated against the right business-line data, signed manifest generation, a canary route with traffic-shadow comparison against the incumbent model, and promotion with automatic rollback on any SLA regression.

Result: onboarding time dropped from days to hours, and the research team's model-evaluation throughput went up sixfold while the platform team's manual effort per model dropped sharply.

The cumulative numbers

None of these numbers come from a single optimization. They are the cumulative effect of four architectural decisions taken together: hardware-aware routing, prefill and decode disaggregation, tidal scheduling, and automated onboarding. Any one of them in isolation moves the needle modestly. All four together produce the order-of-magnitude shift.

What we would tell a team starting today

• Build the routing layer first. Everything else is downstream of it.
• Adopt prefill and decode disaggregation early, even if the initial workload looks homogeneous. It will not stay that way, and retrofitting it later costs roughly five times more.
• Treat onboarding as a product surface, not an operational task. The team that ships new models fastest wins disproportionately.
• Resist standardizing on a single inference engine. vLLM, TensorRT-LLM, SGLang, and llama.cpp each win on different workloads. The control plane should abstract over this so teams do not have to care.

The cluster topology will keep evolving. The models will keep changing. Routing, disaggregation, tidal scheduling, and automated onboarding are the primitives that compound.