NOTE-01PRG-02 — LOCAL TRAINING

Fine-Tuning on a 4-GPU Rig

2026-06-18 · 7 MIN · FILED BY OPR-02

Every few weeks a prospect tells us they can't do local AI because they don't have a cluster. Then we point at the machine under the desk: four 24 GB GPUs, a used server chassis, roughly the price of one month of a mid-size cloud training bill. It has shipped eleven production adapters.

You do not need a cluster to adapt an open-weight model. You need discipline about four things, in this order: data, evals, memory, and only then hyperparameters.

Data beats everything

The single strongest predictor of adapter quality in our logs is not rank, learning rate, or base model — it's the ratio of curated to scraped examples. Two thousand examples written or reviewed by someone who understands the task routinely beat fifty thousand scraped ones. Our working rules:

Every example must be one a domain expert would sign. If nobody on the project can judge an example, it doesn't go in the set.
Deduplicate aggressively — near-duplicates teach the model to memorize, not generalize.
Hold out 10% before anyone looks at the data. That holdout is sacred; it's your only honest gauge.

Build the eval harness first

Before the first training run, we build the eval: a scored task suite that runs in minutes, not hours. If you can't measure "better," you will convince yourself that loss going down means the model improved. It often doesn't — especially on narrow domain tasks where the base model's general ability is doing the work.

RULE OF THE SHOPNo training run starts until the eval harness disagrees with itself by less than the improvement we're hoping to see.

QLoRA fits almost everything

On 4×24 GB, 4-bit QLoRA comfortably adapts models in the 30–70B class with sequence lengths that cover most enterprise documents. A typical launch configuration:

base:            open-weight 32B instruct
quantization:    nf4, double-quant, bf16 compute
adapter:         lora r=16, alpha=32, dropout=0.05
targets:         q_proj k_proj v_proj o_proj gate_proj up_proj down_proj
seq_len:         8192
batch:           micro 1 × grad-accum 16 (effective 64 across 4 GPUs)
lr:              1e-4 cosine, 3% warmup
epochs:          2–3, early-stop on eval, not loss

Things that moved our evals, ranked by how often they actually mattered:

CHANGE	EVAL IMPACT	HOW OFTEN IT MATTERS
Fixing label errors in data	LARGE	ALWAYS
Better prompt template match	LARGE	OFTEN
Sequence length ↑	MEDIUM	DOCUMENT TASKS
LoRA rank 16 → 64	SMALL	RARELY
Learning-rate sweeps	SMALL	RARELY

Serving is part of training

An adapter that evals well in bf16 and ships in 4-bit is a different model. We run the final eval pass on the quantized, merged artifact behind the same server we deploy — same sampler, same stop tokens, same context template. The eval report that goes to the client is generated from that artifact, not from the training checkpoint.

Total wall-clock for a typical engagement on the rig: two days of data work, four hours of training, one day of eval and packaging. The cluster is not the bottleneck. It never was.

← ALL FIELD NOTES