You can build your own large language model (LLM) from scratch in about 10 months of part-time study, and the core of it runs on a free Google Colab GPU. ChatGPT, Llama and Gemini all rest on the same few ideas: the Transformer, tokenization, good data and a lot of training. Learn those ideas by building them yourself, and a home-grown LLM stops being a mystery.

This roadmap takes you from basic math to a deployed model in 9 phases. Each phase lists what to learn, the best free resources, and a milestone that tells you when you're ready to move on. It assumes you already know Python.

🗺️ Your roadmap at a glance

Each coloured block is one month (M1 = month 1). Phases overlap a little where one ends mid-month.

🤔 Is it realistic to build an LLM on your own?

Yes, if you pick the right size. A small model of 100 million to 1 billion parameters, trained on a focused dataset, is within reach of one person with a modest cloud GPU budget. A GPT-4-class model is not: it needs thousands of GPUs, a large team and hundreds of crores of rupees.

So the goal of this roadmap is simple: understand every piece by building it small, then scale up to the biggest model your budget allows.

🧮 Phase 1: Math foundations

 Phase 1   ⏱ Weeks 1–4   💰 Free 

You don't need a math degree. You need just enough to read papers and debug training.

• Linear algebra: vectors, matrices, matrix multiplication and dot products. Attention is mostly matrix multiplication plus softmax. Start with 3Blue1Brown's Essence of Linear Algebra.
• Calculus: derivatives, the chain rule and gradients. That's all backpropagation really is. 3Blue1Brown's Essence of Calculus covers it.
• Probability: probability distributions, softmax, cross-entropy loss and perplexity. An LLM simply predicts a probability distribution over the next token.

The two formulas you'll meet in every LLM. Softmax turns raw scores into probabilities:

$$\mathrm{softmax}(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}$$

Cross-entropy loss is small when the model gives the correct next token a high probability:

$$\mathcal{L} = -\log p(\text{correct token})$$

Try it yourself in NumPy:

import numpy as np

def softmax(z):
    e = np.exp(z - z.max())   # subtract max for numerical stability
    return e / e.sum()

def cross_entropy(probs, target):
    return -np.log(probs[target])

logits = np.array([2.0, 1.0, 0.1])   # scores for 3 possible next tokens
p = softmax(logits)
print(p)                     # [0.659 0.242 0.099]
print(cross_entropy(p, 0))   # 0.417  (low loss: token 0 was likely)

🎯 Milestone: you can code softmax and cross-entropy in NumPy and explain what they do.

🔥 Phase 2: Deep learning fundamentals

 Phase 2   ⏱ Weeks 5–10   💰 Free 

• PyTorch: tensors, autograd, nn.Module, optimizers, DataLoaders and running on a GPU. You'll use it for everything that follows. The official Learn the Basics tutorial is the best start.
• Neural networks from scratch: Andrej Karpathy's free Neural Networks: Zero to Hero series on YouTube is the best resource there is. It goes from micrograd (backpropagation by hand) to makemore (character-level models) to a full GPT. Type out every line yourself instead of copying.
• Training dynamics: learning rates, overfitting vs. underfitting, train/validation splits, batch size and reading a loss curve. Track every run with Weights & Biases.

🎯 Milestone: you've rebuilt micrograd from memory and trained a small network that you deliberately overfit, then fixed.

⚡ Phase 3: The Transformer architecture

 Phase 3   ⏱ Weeks 11–14   💰 Free 

The Transformer is the heart of every modern LLM. Here is how text flows through it:

• Read the paper: Attention Is All You Need (2017), with Jay Alammar's The Illustrated Transformer open beside it.
• Build every component yourself: token embeddings, positional encoding, self-attention, multi-head attention, feed-forward layers, layer norm, residual connections and causal masking.
• Learn the modern upgrades used in Llama, Qwen and Gemma: RoPE (rotary position embeddings), RMSNorm, the SwiGLU activation, grouped-query attention and the KV cache for fast inference.

The core equation of the whole architecture, scaled dot-product attention:

$$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

🎯 Milestone: you've built a GPT from scratch and trained it on a small text dataset, such as Shakespeare or a few Hindi books, on one GPU. Karpathy's video above and his nanoGPT repository walk you through exactly this.

✂️ Phase 4: Tokenization

 Phase 4   ⏱ Weeks 15–16   💰 Free 

• Learn Byte Pair Encoding (BPE) and build a tokenizer yourself with Karpathy's Let's build the GPT Tokenizer video. Then train one quickly with Hugging Face tokenizers or SentencePiece.
• Why it matters for Indian languages: tokenizers trained mostly on English break Hindi into many tiny pieces. That makes a model slower, more expensive and weaker in Hindi. A custom tokenizer trained on your own data is one of the biggest advantages a home-grown model can have.

🎯 Milestone: you've trained a tokenizer on Hindi and English text and measured how many fewer tokens it uses per Hindi word than GPT-2's tokenizer.

📚 Phase 5: Data, the most important part

 Phase 5   ⏱ Weeks 17–21   💰 Low 

At small scale, data quality matters more than model size.

• Sources: open datasets such as FineWeb and FineWeb-Edu for English web text, AI4Bharat corpora (IndicCorp, Sangraha) for Indian languages, and Wikipedia dumps.
• Cleaning pipeline: language detection, exact and near-duplicate removal (MinHash), quality filtering, removing spam, boilerplate and toxic text, and stripping personal data.
• Tools: Hugging Face datasets and datatrove process data at scale.
• How much you need: the Chinchilla rule of thumb is about 20 training tokens per parameter.

$$\text{training tokens} \approx 20 \times \text{parameters}$$

🎯 Milestone: a cleaned, tokenized dataset sharded into training files.

🚀 Phase 6: Pretraining at scale

 Phase 6   ⏱ Weeks 22–29   💰 GPU budget 

• Scaling laws: read the Chinchilla paper to understand how model size, data and compute trade off.
• Distributed training: mixed precision (bf16), gradient accumulation, gradient checkpointing, and multi-GPU training with PyTorch DDP and FSDP. DeepSpeed is another option.
• Training stability: learning-rate warmup and cosine decay, gradient clipping, regular checkpoints and catching loss spikes early.
• Compute: rent GPUs from RunPod, Lambda, Vast.ai or E2E Networks (an Indian provider), or apply for subsidized GPU access under the IndiaAI Mission's compute program.

💡 Tip: set a hard spending limit with your GPU provider before you start, and save checkpoints often so a crash never costs you more than an hour of training.

🎯 Milestone: you've pretrained a 100M–500M parameter base model on your own data. Karpathy's nanochat repository shows the full pipeline, from tokenizer to chat interface, for a small ChatGPT-style model at a cost of a few hundred dollars.

🎛️ Phase 7: Fine-tuning and alignment

 Phase 7   ⏱ Weeks 30–35   💰 Low to moderate 

A pretrained base model only continues text. This phase turns it into an assistant.

• Supervised fine-tuning (SFT): train on instruction-and-response pairs so the model learns to follow instructions and chat.
• Preference alignment: learn DPO (Direct Preference Optimization). It's simpler than RLHF and works well. Hugging Face's TRL library implements both.
• Parameter-efficient fine-tuning: LoRA and QLoRA let you fine-tune much larger models on a single GPU.

🎯 Milestone: your model follows instructions, and testers prefer its answers after DPO.

📏 Phase 8: Evaluation

 Phase 8   ⏱ Weeks 36–38   💰 Low 

• Standard benchmarks: perplexity on held-out data, plus HellaSwag, MMLU and ARC using EleutherAI's lm-evaluation-harness. For Indian languages, add IndicXTREME and similar benchmarks.
• Your own test set: a few hundred real questions from the people you're building for. It tells you more than any public benchmark.

Perplexity is just the exponential of the average loss, so lower is better:

$$\mathrm{PPL} = e^{\bar{\mathcal{L}}}$$

🎯 Milestone: scores logged for every model you've trained, and a clear answer to "is it good enough to ship?"

🌐 Phase 9: Inference and deployment

 Phase 9   ⏱ Weeks 39–42   💰 Low 

• Make it small and fast: quantize the model (GGUF, AWQ or GPTQ) and serve it with vLLM or llama.cpp.
• Ship it: wrap the model in an API with FastAPI and build a simple chat interface, or plug it into your existing app.

🎯 Milestone: real users are chatting with your model, and you're collecting their feedback.

💰 How long it takes and what it costs

At about 10 hours a week, the full roadmap takes 8–12 months.

🎯 Building for a startup or an Indian language? Go narrow

Don't try to beat ChatGPT at everything. Pick a narrow area where you have an edge: Hindi-medium education, board-exam maths and science, interview preparation, or a regional language that big models handle poorly. High-quality data from your own field is what gives a small model its advantage.

Many successful teams also take a hybrid approach. They learn by building small models from scratch, then fine-tune a strong open model such as Llama, Qwen or Gemma for the actual product, because that gives much better quality per rupee spent.

✅ Your first step this week

👉 Watch Karpathy's Let's build GPT: from scratch, in code, spelled out and train the model yourself on a free Colab GPU. Once it runs, you'll understand the core of every LLM. The rest of this roadmap is making it bigger and better.

Found this roadmap useful? Bookmark codingbattle.in for more guides on DSA, machine learning and placement preparation, and share this post with a friend who wants to build in AI.

❓ Frequently asked questions

Can I build an LLM from scratch without a GPU?

You can learn phases 1–4 on a laptop and free Google Colab GPUs. Pretraining a useful model needs rented cloud GPUs, but only for a few weeks.

How much does it cost to train a small LLM?

A 100M–500M parameter model costs from a few thousand rupees to about ₹2 lakh in GPU time, depending on its size and how much data you train on.

Do I need to know advanced math?

No. Basic linear algebra, the chain rule from calculus and basic probability are enough to start.

Which programming language do I need?

Python. Almost all LLM work uses Python and PyTorch.

Should I train from scratch or fine-tune an open model?

Train from scratch to learn how LLMs work. For a product, fine-tuning an open model like Llama or Qwen usually gives better results for less money.

Can I build an LLM for Hindi or other Indian languages?

Yes. Start with a custom tokenizer and Indian-language datasets from AI4Bharat. It's one of the best opportunities for independent builders in India.