Language Model Lab

The Language Model Lab offers an interactive exploration of how sinmple language models predict and generate text.

Give it a beginning, inspect the probabilities for the next character, and watch a sequence develop one decision at a time. Compare three different models, adjust temperature, and explore how the available context changes their predictions.

Why it exists

Most of my work with AI happens around the model: products, workflows, retrieval, agents and evaluation. I wanted to revisit the mechanics underneath those systems and make them easier to examine.

Inspired by Andrej Karpathy’s makemore, the project starts with a deliberately small task: learning patterns from names. The limited scope makes it possible to follow the process from training data and parameter updates through to individual predictions.

The browser interface makes those mechanics accessible without having to read or run the Python code.

A few ideas shaped the project:

• Make prediction visible. Show the distribution of possible next characters, alongside the character actually selected.
• Make comparisons concrete. Give different architectures the same text and inspect how their predictions differ.
• Keep the mechanics inspectable. Implement the model equations, gradient calculations and training loop directly.
• Measure what matters. Evaluate on held-out examples rather than judge a model by a handful of convincing outputs.

How it’s built

The training code uses Python and NumPy. It includes a small automatic differentiation engine, an explicitly derived backward pass for the multilayer perceptron, and numerical checks of the gradients.

The broader repository contains five architectures: bigram, multilayer perceptron, recurrent neural network, gated recurrent unit and Transformer.

The browser demo focuses on three:

• Bigram — predicts from the previous character using a table of transition probabilities.
• Neural network — uses embeddings and a hidden layer to combine the most recent six characters.
• Transformer — uses positional information and two attention heads to combine information within the same six-character window.

Models are trained in Python, then their saved weights are exported for use in the browser. JavaScript implements the forward passes, with tests checking their predictions against the Python versions.

The interface uses plain HTML, CSS and JavaScript. Once the model files have loaded, prediction runs entirely in the browser, without an inference server or API key.

What you can explore

• Step-by-step generation — sample one character and see the probabilities update.
• Temperature — concentrate or flatten the sampling distribution without changing the learned weights.
• Context — see which characters are available to each model.
• Model comparisons — inspect the leading predictions from all three architectures for the current text.
• Attention — examine how the Transformer’s last position weights the available context within each head.
• Repeatable experiments — use a fixed random seed, undo a step or restart a sequence.
• Sampling history — review each selected character and its probability at the time it was chosen.

These are small teaching models trained on a limited list of names. They can produce awkward strings, repeat examples or stop early. Attention weights expose part of the computation, but they do not fully explain a prediction.