Fine-tuning a Language Model: A Working Demo
What it is, when to use it, and why RAG might serve you better. Built with GPT-2, Bhagavad Gita text, and a Colab T4 GPU.
Ask the same question to base GPT-2 and the Gita model
Side-by-side responses reveal exactly what fine-tuning changes — and what it doesn't.
Models load from HuggingFace Hub. Generation runs on CPU — allow 15–30 s for the first response.
Opens in a new tab · Streamlit Cloud · free tier
The pipeline, start to finish
Data: The full text of the Bhagavad Gita from Project Gutenberg (#2388, Sir Edwin Arnold translation). Pre-processed into 100-word chunks, split 80/20 for training and validation.
Training: GPT-2 (124M parameters) fine-tuned using HuggingFace Transformers Trainer API. 3 epochs on a Google Colab T4 GPU — about 30 minutes total. The notebook is fully reproducible.
The side-by-side comparison is the whole point. Base GPT-2 drifts into generic internet text within a sentence. The fine-tuned model stays in the vocabulary of the Gita — dharma, karma, Arjuna — because that's the only text it saw during fine-tuning.
What the demo honestly shows: fine-tuning changes style and vocabulary. It does not give a model factual understanding. For accurate Q&A on a document, RAG is the right call.
Fine-tuning vs RAG — the decision framework
Most practitioners reach for fine-tuning when RAG would serve them better. Here's when each wins.
| Dimension | Fine-tuning | RAG |
|---|---|---|
| What it teaches | Style and tone | Nothing — retrieves facts |
| Factual accuracy | Can hallucinate | Grounded in source text |
| Source citations | No | Yes |
| Cost | One-time training cost | Inference + vector DB cost |
| Data updates | Requires retraining | Update the index |
| Latency | Fast (no retrieval) | Adds retrieval overhead |
| Best for | Style transfer, format enforcement | Q&A, fact lookup, live data |
When fine-tuning IS the right choice
RAG layer → retrieves the right passage for the question Fine-tuned model → formats and explains it in a specific voice Structured output → cited, formatted, on-brand
RAG handles what to say. Fine-tuning handles how to say it. Neither alone is sufficient for enterprise Q&A.
Everything open source
Let's talk about what fine-tuning, RAG, and agentic systems can actually do for your organisation.
Get in touch