Small-Model Fine-Tuning Ablations — Isolating Dataset Quality on 1–3B Parameter Models
A controlled study of how much dataset quality alone moves the needle on small models — holding the training recipe fixed and varying only the data.
At a glance
Most fine-tuning comparisons change several things at once, which makes the result hard to attribute. This project fixes the recipe and varies only the dataset, so any difference in the evaluation can be traced back to data quality rather than hyperparameters or model choice.
The problem
A municipality, a clinic or a small firm with an obvious use for a language model — answering routine questions over its own documents, drafting standard correspondence, classifying incoming requests — usually runs into the same wall. The data cannot leave the building, and the advice on offer assumes it can.
- Sending citizen or patient records to a third-party API is a data-protection question before it is a technical one, and for many public bodies the answer is simply no.
- The received wisdom is use a bigger model, which converts the problem into a hardware and licensing bill these organizations do not have.
- Nobody tells them the useful thing: how much of the gap closes by curating a few hundred good examples instead of buying compute — or whether, for their case, it does not close at all.
How this helps
This study answers that question with a controlled experiment rather than an opinion. The training recipe is held fixed and only the dataset changes, so any movement in the evaluation is attributable to data quality — the one variable an organization can control with staff time instead of budget.
- Sized for hardware people already have. Models in the 1–3B range, trained with LoRA adapters in roughly 12–24 GPU hours and exported to GGUF so the result runs locally — no data leaves the network, no cluster is procured.
- A feasibility test before a purchase. Running the same ablation on an organization's own data says whether curation reaches an acceptable answer before anyone signs for compute or a vendor contract.
- The method is the transferable part. Fixing the recipe and varying one thing is what makes a result defensible in a procurement discussion, where we tried it and it seemed better is not enough.
- Cheap enough to repeat. An ablation set that fits in a day of GPU time can be re-run whenever the data changes, which turns it into an ongoing control rather than a one-time study.
Small models are not a substitute for a frontier model on open-ended work. The honest claim is narrower and more useful: for a bounded task over your own documents, data quality is the lever worth pulling first, and this is how you find out how far it takes you before spending anything.
Setup
- Fine-tuned three models in the 1–3B parameter range: Phi-3-mini, Qwen2.5-1.5B, and Gemma-2B.
- Built and curated dataset variants with Argilla and distilabel.
- Trained with LoRA / PEFT adapters rather than full fine-tuning, keeping each run cheap enough to repeat.
- Exported to GGUF with quantization for local inference.
- Total budget was roughly 12–24 GPU hours across the ablation set.
Why small models and why LoRA
The constraint drove the design. At 1–3B parameters with LoRA adapters, a full ablation sweep fits in a day of GPU time, which is what makes running several dataset variants feasible at all — a full fine-tune of a larger model would have allowed perhaps one run, and one run cannot separate data quality from noise. GGUF export closes the loop by making each resulting model runnable locally for evaluation.