Autoscale to Zero
The most common cloud AI bill we audit has one dominant line item, and it is not tokens. It's idle accelerators — GPU nodes held warm around the clock for traffic that arrives eight hours a day, five days a week. The fix is not a cheaper GPU. It's admitting that your traffic has a shape.
Know your three latencies
Every serving decision falls out of three numbers you should measure before choosing infrastructure:
- Tolerated first-token latency — what the user experience actually requires (a chat UI: ~1s; a batch pipeline: minutes).
- Cold-start time — container pull + weight load + warmup for your model on your node type. Measure it; don't guess. Ours ranged 40s–4min depending on weight size and cache locality.
- Traffic gap distribution — how long the quiet periods really are. If p50 gap < cold-start time, scale-to-zero saves nothing and costs latency.
The three-tier layout
Our default production layout routes each request to the cheapest tier that meets its latency budget:
TIER 1 — API frontier models (Claude / Gemini)
zero idle cost · per-token pricing · instant
→ interactive traffic, low volume, hardest queries
TIER 2 — managed autoscale pool (1..N GPU nodes, scale-to-zero off-hours)
open-weight workhorse, quantized · vLLM, paged KV
→ steady product traffic with SLA
TIER 3 — spot/batch pool (0..M preemptible nodes)
same weights as tier 2 · queue-driven, checkpointed
→ embeddings, backfills, evals, overnight jobsThe routing rule is boring on purpose: requests carry a latency budget and a cost class in their headers; the router does the rest. Nobody picks a model by vibes at 2 a.m.
Meter per token, or you're guessing
Every response in the fleet emits a cost event: model, tier, input/output tokens, GPU-seconds, and the internal cost rate. Those events roll up into a dashboard finance can read. Two effects follow within a month, every time: the team finds one endpoint quietly burning 40% of spend, and product decisions start citing cost-per-feature instead of the total bill.
| CHANGE (REAL AUDIT, 90 DAYS) | SPEND DELTA |
|---|---|
| Scale-to-zero off-hours on tier 2 | −31% |
| Routing easy queries off the frontier tier | −22% |
| Quantizing the workhorse (bf16 → int8) | −17% |
| Prompt caching on repeated prefixes | −9% |
| Buying a "better" GPU class | +4% |
Autoscale-to-zero is not an optimization pass you do later. It's an architecture you choose on day one — because the alternative architecture, a warm cluster shaped like your peak, will follow you around like a lease.