Building a GPT-Style Language Model From Scratch, One Understandable Step at a Time

Large language models can feel mysterious because we usually meet them only after they have been trained on enormous datasets using equally enormous computers. FrankenGPT takes the opposite approach: make the model small enough to run locally, keep the implementation readable, and expose the complete path from a text file to generated text.

The result is not a replacement for ChatGPT, and that is not the goal. It is a compact, working GPT-style language model that lets us inspect the same essential ideas: tokenization, causal self-attention, next-token prediction, gradient-based optimization, validation, checkpoints, and autoregressive generation.

This article walks through that path without assuming that you already speak fluent Transformer.

Repository: JordiCorbilla/llm-from-scratch

What does "from scratch" mean here?

The phrase is important. In FrankenGPT's scratch workflow:

• no pretrained model weights are downloaded;
• no pretrained tokenizer is reused;
• the vocabulary is built from the text we provide;
• every model parameter starts with a random value; and
• every saved checkpoint is the result of our own training run.

The repository also contains an optional Hugging Face fine-tuning example, but that is a separate experiment and is clearly labelled as pretrained. Everything in this article uses the scratch model.

It is also worth being precise about the word large. FrankenGPT follows the architecture and training objective used by GPT-like language models, but our experiments range from roughly 90,000 to 2.9 million parameters. That is deliberately small by modern LLM standards. Small models are much easier to understand, test, and train on an ordinary computer.

The whole journey in one view

The model's job can be summarized as a pipeline:

corpus text
  -> tokenizer and vocabulary
  -> shifted input/target sequences
  -> token and position embeddings
  -> causal Transformer blocks
  -> next-token scores
  -> cross-entropy loss
  -> AdamW parameter updates
  -> checkpoint
  -> one-token-at-a-time generation

Training repeatedly moves from left to right until the model becomes better at predicting the next token. Generation then reuses the trained model one prediction at a time.

Step 1: Create a clean environment

FrankenGPT requires Python 3.10 or newer. A virtual environment keeps this experiment separate from other Python projects:

git clone https://github.com/JordiCorbilla/llm-from-scratch.git
cd llm-from-scratch

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

On macOS or Linux, activate the environment with:

source .venv/bin/activate

The editable installation gives us the frankengpt command while keeping the source code available to inspect and change.

Confirming GPU support

CPU training works everywhere and is useful for the first smoke test. An NVIDIA GPU makes the larger experiments much faster, but PyTorch must have been installed with CUDA support:

python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

A CUDA-enabled installation prints a version containing +cu... and then True. If it prints False, use the official PyTorch installation selector to choose the wheel that matches the machine.

FrankenGPT accepts --device auto, --device cpu, or --device cuda. On CUDA it automatically uses mixed-precision training: some operations use lower-precision numbers to improve GPU throughput and reduce memory use, while gradient scaling protects the optimization process from numerical underflow.

Step 2: Choose the training text

A language model learns statistical patterns from its corpus. It does not receive a grammar book or a database of facts; it only sees sequences of tokens and tries to predict what follows.

For the first experiment, FrankenGPT can download Mary Shelley's Frankenstein from Project Gutenberg:

frankengpt train --download --data data/pg84.txt ...

The loader removes the Project Gutenberg catalogue header and licence footer before training. It then makes a contiguous split:

• approximately 90% of the tokens are used for training;
• approximately 10% are held out for validation.

Those partitions do not overlap. The training portion updates the weights; the validation portion measures whether the learned patterns also work on unseen text.

For a broader experiment, the project can download several public-domain classics:

frankengpt fetch-data --output-dir data/classics

More varied data matters. A model trained on one novel mostly learns the vocabulary, punctuation, characters, and style of that novel. A multi-book corpus exposes it to more sentence structures and word combinations.

Step 3: Turn text into tokens

Neural networks operate on numbers, not strings. A tokenizer divides text into units and assigns an integer ID to each unit.

FrankenGPT includes two intentionally simple tokenizers:

Tokenizer	Example units	Trade-off
Character	["T", "h", "e"]	Small vocabulary and no unknown words, but long sequences
Word	["The", "creature", ","]	More readable output and shorter sequences, but a larger vocabulary

Both vocabularies are built only from the selected corpus. For example:

text:       The creature
tokens:     ["The", "creature"]
token IDs:  [418, 937]

The IDs themselves have no linguistic meaning. They are indexes into a learned embedding table. During training, the vector at each index gradually becomes useful for predicting tokens that tend to appear nearby.

Step 4: Create the question and the answer

The training objective is next-token prediction. Given a sequence of token IDs, the target is the same sequence shifted by one position:

token stream:  [12, 31, 7, 44, 9]
input:         [12, 31, 7, 44]
target:        [31, 7, 44, 9]

This produces several predictions in one forward pass:

• after 12, predict 31;
• after 12, 31, predict 7;
• after 12, 31, 7, predict 44; and
• after 12, 31, 7, 44, predict 9.

With batch size B and context length T, the input and target tensors both have shape [B, T].

Step 5: Pass the tokens through a GPT-style decoder

The model first combines two learned representations:

1. a token embedding, which represents the current token;
2. a position embedding, which represents where that token appears in the sequence.

The combined vectors pass through a stack of Transformer decoder blocks. Each block contains pre-layer normalization, multi-head causal self-attention, a GELU feed-forward network, dropout, and residual connections.

Causal self-attention, gently explained

Self-attention lets each position decide which earlier positions are useful. Internally, the model projects each token representation into queries, keys, and values:

queries, keys, values  [B, heads, T, head dimension]
attention scores       [B, heads, T, T]

The core operation is:

attention(Q, K, V) = softmax((Q K^T / sqrt(d_k)) + causal_mask) V

The notation is more compact than the idea. Q K^T measures which tokens are relevant to one another, the scale factor keeps the numbers stable, the mask removes future tokens, softmax turns the remaining scores into proportions, and those proportions select a mixture of the value vectors.

The causal mask blocks every future position. When predicting token 5, the model may use tokens 0 through 4, but it cannot peek at token 5's answer. Without this restriction, training loss would look excellent while generation would fail: the model would have learned to copy information that is unavailable when writing new text.

Residual connections preserve a direct path through the network, while the feed-forward layers transform each position independently. Repeating the block lets the model build progressively richer contextual representations.

Finally, a linear language-model head produces one score, or logit, for every item in the vocabulary:

hidden states  [B, T, model width]
logits         [B, T, vocabulary size]

FrankenGPT ties the output head to the token embedding table. The same learned token representations therefore support both reading input and scoring output, while reducing the parameter count.

Step 6: Measure the error and update the model

Cross-entropy loss compares the logits with the correct shifted targets. Lower loss means that the model assigned more probability to the actual next tokens.

One training update performs:

forward pass
  -> cross-entropy loss
  -> backpropagation
  -> gradient clipping
  -> AdamW update
  -> learning-rate schedule

The optimizer is AdamW. The learning rate warms up gradually and then follows a cosine decay. Gradient clipping limits unusually large updates, and weight decay discourages parameters from growing without bound.

There are two losses to watch:

• training loss is measured on text used to update the model;
• validation loss is measured on held-out text that never updates the model.

We want both to decrease. If training loss continues downward while validation loss begins to rise, the model is memorizing the training corpus more than it is learning patterns that transfer. That is overfitting.

Step 7: Prove the complete pipeline with a smoke run

Before spending time on a long training run, it is sensible to test every stage with a small CPU model:

frankengpt train --download --data data/pg84.txt --tokenizer word --max-vocab 2048 `
  --device cpu --output runs/readme-smoke --max-steps 10 --batch-size 8 `
  --context-length 16 --d-model 32 --n-heads 4 --n-layers 2 --eval-interval 5

This model has 91,520 parameters. In a verified run, training loss moved from 7.6218 to 7.5856, while validation loss moved from 7.6328 to 7.6039.

That is a small improvement, as expected after only ten updates. At this stage, success means that:

• data was loaded and tokenized;
• forward and backward passes completed;
• both losses were recorded;
• checkpoints were saved; and
• the checkpoint could be loaded for generation.

The generated text is not supposed to be fluent yet:

The earth interested does desires enemy affectionate desires disappeared figure figure
figure desires distinct noble.. the familiar valley valley safety disappeared disappeared
the

This output is useful precisely because it sets an honest baseline. Ten steps validate the machinery; they do not teach a language model to write.

Step 8: Train the larger model on a GPU

The more interesting run used the word tokenizer and a collection of public-domain classics:

frankengpt fetch-data --output-dir data/classics

frankengpt train --data data/classics/*.txt --tokenizer word --max-vocab 16384 `
  --device cuda --output runs/classics-word-v2 --max-steps 2000 `
  --batch-size 32 --context-length 64 --d-model 128 --n-heads 4 `
  --n-layers 4 --eval-interval 200

Its configuration was:

Setting	Value
Parameters	2,898,688
Vocabulary	16,384 word and punctuation tokens
Context length	64 tokens
Transformer blocks	4
Attention heads	4
Model width	128
Batch size	32
Training steps	2,000
Device	CUDA

On the test machine, an NVIDIA GeForce RTX 2060 ran the model through CUDA mixed precision. The recorded training segment processed approximately 44,868 tokens per second. Throughput is hardware- and software-dependent, so the loss trend is the more portable result.

Step 9: Save checkpoints and keep the best model

At each evaluation interval, FrankenGPT writes:

• checkpoint_last.pt, containing the newest state;
• checkpoint_best.pt, containing the lowest validation loss seen so far; and
• metrics.json, containing throughput and loss history.

The checkpoint includes the model configuration, tokenizer vocabulary, model parameters, optimizer, learning-rate scheduler, training step, and history. That makes it possible to resume training rather than restart:

frankengpt train --data data/classics/*.txt --tokenizer word --max-vocab 16384 `
  --device cuda --output runs/classics-word-v2 `
  --resume runs/classics-word-v2/checkpoint_last.pt `
  --max-steps 3000 --batch-size 32 --context-length 64 `
  --d-model 128 --n-heads 4 --n-layers 4 --eval-interval 200

Model shape and tokenizer settings must remain unchanged when resuming. Only the training budget and other compatible operational settings should change.

Load PyTorch .pt checkpoints only from trusted sources because they can contain pickled Python data.

What did the model learn?

The multi-book model's measured losses were:

Step	Training loss	Validation loss
100	6.6836	6.7742
1,000	5.1567	5.4137
2,000	5.0342	5.3361

Both values fell substantially, and validation loss was still improving at step 2,000. That is evidence that the model learned patterns that also applied to the held-out text.

Loss values from the word model should not be compared directly with the character model's losses. Their vocabularies and prediction problems are different: choosing among 16,384 words is not the same task as choosing among 93 characters. Loss trends within one run are what matter.

The eventual output

Generation begins with a prompt and repeatedly samples one next token:

frankengpt generate --checkpoint runs/classics-word-v2/checkpoint_best.pt `
  --prompt "I had worked hard" --max-new-tokens 80 `
  --temperature 0.7 --top-k 20 --device cuda

A fresh sample from the trained checkpoint produced:

I had worked hard, and all the house of a moment of my own friend, and that I am, with a
great. But I shall not not be no time, you shall have been no, it is all the door, my own,
to be so. What may not know. I see that I can do you will be a man, I cannot know you have
been as I can be so that my

The difference from the ten-step smoke output is visible. The larger model learned:

• plausible English punctuation;
• common phrase and clause structures;
• agreement between many neighbouring words; and
• a rough literary rhythm from the source corpus.

It also repeats itself, contradicts itself, and loses the thread. That is the expected boundary of this experiment, not a hidden failure. A 2.9-million-parameter model trained for 2,000 updates on a handful of books has learned local language patterns, not robust reasoning, factual knowledge, or long-range narrative planning.

Sampling settings change the style but cannot create knowledge the model did not learn. A lower temperature and smaller top-k make output safer and more repetitive. Higher values add variety along with more mistakes.

Step 10: Verify and benchmark the result

The repository includes automated checks for tokenization, shifted targets, causal masking, model shapes, loss handling, checkpoint loading, resumption, compilation fallbacks, and generation:

python -m ruff check .
python -m pytest

Benchmark a saved model separately:

frankengpt benchmark --checkpoint runs/classics-word-v2/checkpoint_best.pt --device cuda

Forward-pass throughput and autoregressive generation throughput are different measurements. A forward pass predicts many positions in parallel; generation must append one token and run the model again before it can choose the next.

What would improve it next?

The project is intentionally a foundation rather than an endpoint. The highest-impact next experiments are:

1. Use more varied text. Data diversity improves vocabulary and generalization.
2. Add a subword tokenizer. Subwords balance the long sequences of characters against the very large vocabulary and unknown words of a simple word tokenizer.
3. Increase context length. The model can then condition on more history, at a higher memory and compute cost.
4. Scale width and depth carefully. More parameters increase capacity only when the dataset and training budget grow with them.
5. Track validation first. More training is useful until held-out performance stops improving.
6. Evaluate multiple prompts and seeds. One attractive sample is not a complete model evaluation.

Closing thought

The most valuable output of FrankenGPT is not a perfect paragraph. It is the removal of the black box.

We can point to the corpus, vocabulary, shifted targets, embeddings, masked attention, loss, gradients, optimizer, checkpoint, and sampling loop. We can see the validation loss fall, resume the exact training state, move the computation to a GPU, and inspect what the model learned—and what it did not.

That is the real benefit of building a language model from scratch: every generated token has a path we can follow.