Chapter 18

Language Modeling

Autoregressive factorization, n-gram and neural language models, and perplexity.

A language model assigns probabilities to text, one next token at a time. That single skill drives pretraining, scoring, sampling, and perplexity evaluation for LLMs in 2026. This chapter builds the idea from a count table, then replaces the table by trainable NumPy models.

18.1 Autoregressive probabilities

For a token sequence $x_1,\ldots,x_T$ , the product rule from probability (Section 6.2) gives

p(x_1,\ldots,x_T)=\prod_{t=1}^{T}p(x_t\mid x_{<t}) .\tag{18.1}

An autoregressive language model estimates each factor on the right. At training time, every prefix in the dataset supplies a supervised example: context $x_{<t}$ , target $x_t$ . At generation time, the model samples or chooses the next token, appends it to the context, and repeats.

This training setup is called teacher forcing: the context comes from the data, not from tokens the model sampled a moment earlier. That makes all positions in a sequence trainable in parallel, because the correct previous tokens are already known. Generation is slower because each new token changes the next context.

The loss for one observed sequence is the negative log of (18.1):

\mathcal{L}=-\sum_{t=1}^{T}\log q_\theta(x_t\mid x_{<t}) .\tag{18.2}

Dividing by the number of predicted tokens gives cross-entropy per token. Exponentiating that average gives perplexity, the effective branching factor already defined in Section 7.6.

The factorization does not say how much history the model must use. A table can ignore almost all of it, an MLP can use a fixed suffix, and a transformer can use a long prefix. The loss only asks whether the probability assigned to the observed next token is high.

18.2 Count-based bigrams

A bigram model keeps only the previous token:

q(x_t=j\mid x_{t-1}=i)=\frac{c_{ij}}{\sum_k c_{ik}} .\tag{18.3}

Raw counts break when a row has no example of the observed next token. Add-alpha smoothing adds a small pseudo-count to every possible next token:

q(j\mid i)=\frac{c_{ij}+\alpha}{\sum_k c_{ik}+\alpha V}.\tag{18.4}

The denominator appears because adding $\alpha$ to each of $V$ columns adds $\alpha V$ total mass to the row. The probabilities still sum to one, and no next token receives zero probability.

Smoothing is a bias-variance trade. With little data, raw counts are brittle: one missing bigram means an infinite test loss if that bigram appears later. With too much smoothing, every row is pulled toward the uniform distribution and real patterns are washed out. The best $\alpha$ is therefore a validation choice, not a theorem.

Listing 18.1 A smoothed bigram table

def smoothed_bigram_probs(ids, vocab_size, alpha=1.0):
    """Estimate p(next | previous) with add-alpha smoothing."""
    counts = np.zeros((vocab_size, vocab_size), dtype=np.float64)
    previous, target = make_bigrams(ids)
    np.add.at(counts, (previous, target), 1.0)
    return (counts + alpha) / (counts.sum(axis=1, keepdims=True) + alpha * vocab_size)


def average_nll_from_probs(probs, previous, target):
    """Mean negative log-probability assigned to observed bigrams."""
    return -np.mean(np.log(probs[previous, target]))

The tiny corpus in the tests is synthetic: time flies like time fruit flies like fruit time flies. It is deliberately small enough that smoothing matters. The count model can memorize local regularities such as time → flies, but it has no representation of meanings, synonyms, or longer contexts.

The count table also cannot share evidence. Seeing fruit flies teaches only the row for fruit; it says nothing about time, even if a better model might learn that both can precede flies. Neural models earn their keep by replacing isolated table cells with shared parameters.

18.3 Neural bigrams

A neural bigram replaces the probability table by logits. With previous token $i$ , a one-hot vector selects row $i$ of a trainable matrix $\mW$ , then softmax turns that row into a distribution:

\vz_i=\boldsymbol{e}_i\mW,\qquad \vq_i=\softmax(\vz_i).\tag{18.5}

For a batch of $N$ bigrams, the cross-entropy loss is

\mathcal{L}=-\frac1N\sum_{n=1}^{N}\log q_{n,y_n}.\tag{18.6}

The softmax-cross-entropy gradient is $(\vq_n-\boldsymbol{e}_{y_n})/N$ for each row of logits. Since row $i$ of $\mW$ was gathered for examples whose previous token is $i$ , the matrix gradient scatter-adds those logit gradients into the selected rows. The tests finite-difference this backward pass.

This model is multinomial logistic regression with the previous token as the feature. It has the same information as a bigram count table, but training through cross-entropy introduces the optimization pattern used by larger networks: forward logits, softmax probabilities, a scalar loss, and reverse-mode gradients into parameters.

Listing 18.2 Neural bigram loss and gradient

def softmax(logits):
    shifted = logits - logits.max(axis=-1, keepdims=True)
    exp = np.exp(shifted)
    return exp / exp.sum(axis=-1, keepdims=True)


def neural_bigram_loss_and_grad(W, previous, target):
    """Cross-entropy loss and gradient for logits W[previous]."""
    logits = W[previous]
    probs = softmax(logits)
    n = len(target)
    loss = -np.mean(np.log(probs[np.arange(n), target]))
    grad_logits = probs
    grad_logits[np.arange(n), target] -= 1.0
    grad_logits /= n
    grad_W = np.zeros_like(W)
    np.add.at(grad_W, previous, grad_logits)
    return loss, grad_W

Training is now ordinary gradient descent. This model is still a bigram model: it can learn a better-smoothed row for each previous token, but it cannot condition on anything before that token.

The limitation is structural, not an optimization failure. No amount of training can make $q(x_t\mid x_{t-1})$ depend on $x_{t-2}$ , because $x_{t-2}$ never reaches the logits. To use more history, the architecture must receive more history.

Listing 18.3 Training a neural bigram

def train_neural_bigram(previous, target, vocab_size, steps=200, lr=2.0, seed=18):
    rng = np.random.default_rng(seed)
    W = (0.01 * rng.standard_normal((vocab_size, vocab_size))).astype(np.float32)
    losses = []
    for _step in range(steps):
        loss, grad_W = neural_bigram_loss_and_grad(W, previous, target)
        W -= lr * grad_W.astype(np.float32)
        losses.append(float(loss))
    return W, losses

18.4 Fixed-window neural language models

Bengio et al. introduced a neural probabilistic language model that embeds a fixed number of previous words, concatenates those embeddings, and feeds them through an MLP [bengio2003]. With context width $C$ ,

\vh_t=\tanh([\mE_{x_{t-C}};\ldots;\mE_{x_{t-1}}]\mW_1+\vb_1), \qquad \vq_t=\softmax(\vh_t\mW_2+\vb_2).\tag{18.7}

The embedding table lets related tokens share statistical strength: changing an embedding affects every context where that token appears. The hidden layer lets the model represent interactions among positions, unlike a count table. The fixed window is the limitation. Anything before $x_{t-C}$ is invisible, so the model must choose between a short memory or a rapidly growing input layer.

The model also treats each slot in the window separately. The parameters connected to the nearest previous token are different from the parameters connected to the farthest token, even if the same word appears in both slots. This is useful for word order, but it does not solve variable-length dependency: the relevant clue may be just outside the window.

Listing 18.4 A fixed-context MLP forward pass

def make_contexts(ids, width):
    """Return fixed-width histories and next-token targets."""
    ids = np.asarray(ids, dtype=np.int64)
    contexts = np.stack([ids[i:i + width] for i in range(len(ids) - width)])
    targets = ids[width:]
    return contexts, targets


def mlp_logits(params, contexts):
    """Bengio-style fixed-window MLP language model forward pass."""
    C, W1, b1, W2, b2 = params
    embedded = C[contexts].reshape(contexts.shape[0], -1)
    hidden = np.tanh(embedded @ W1 + b1)
    return hidden @ W2 + b2

Attention is the next answer to that limitation. Instead of compressing the past into a fixed-width concatenation, the current position will compare itself with all earlier positions and form a data-dependent weighted summary.

That change keeps the autoregressive objective intact. The target is still the next token and the loss is still cross-entropy; only the function that computes $q_\theta(x_t\mid x_{<t})$ changes. This separation between objective and architecture is why the same evaluation formula applies to count tables, MLPs, and transformers.

In practice

Modern decoder-only LLMs are still trained with the autoregressive next-token objective, but their conditional distribution is produced by a transformer rather than an n-gram table or fixed-window MLP [radford2019]. Cross-entropy is the training loss, while perplexity is useful only when the tokenizer and evaluation set are fixed. Scaling-law work reports loss as the central quantity because it is directly optimized and averages cleanly over tokens [kaplan2020scaling], [hoffmann2022training]. Fixed-window MLP language models are historically important because they introduced learned distributed word representations for language modeling [bengio2003].

Key equations

p(x_1,\ldots,x_T)=\prod_{t=1}^{T}p(x_t\mid x_{<t})

q(j\mid i)=\frac{c_{ij}+\alpha}{\sum_k c_{ik}+\alpha V}

\mathcal{L}=-\frac1N\sum_n\log q_{n,y_n}, \qquad \bar{\vz}_n=\frac{\vq_n-\boldsymbol{e}_{y_n}}{N}

\operatorname{PPL}=\exp\left(\frac{1}{N}\sum_n-\log q_{n,y_n}\right)

\vq_t=\softmax(\tanh([\mE_{x_{t-C}};\ldots;\mE_{x_{t-1}}]\mW_1+\vb_1)\mW_2+\vb_2)

18.5 Teach it

The one-sentence version. A language model is a probability rule for the next token, trained by minimizing average negative log-probability.

An analogy. It is autocomplete with calibrated uncertainty: not just one suggestion, but a full distribution over possible next tokens.

At the board.

Write the chain rule for time flies like.
Replace the full history by the previous token and fill a bigram count row.
Add $\alpha$ to every cell so unseen next tokens are not impossible.
Replace the row by neural logits, apply softmax, and backpropagate $\vq-\vy$ .

Misconceptions to address.

"A bigram model understands grammar." It only sees one previous token.
"Perplexity is comparable across tokenizers." It is not.
"The MLP solved long context." Its context is fixed before training.

Check for understanding. Why does assigning zero probability to one observed next token make the average loss infinite?

18.6 Exercises

Exercise 18.1 ★ Chain-rule sampling

Explain how (18.1) lets a model generate a sequence from left to right.

Solution in Appendix D

Exercise 18.2 ★★ Smoothing a row

Given counts $(3,0,1)$ for possible next tokens and $\alpha=1$ , compute the smoothed row and show it sums to one.

Solution in Appendix D

Exercise 18.3 ★★ Neural bigram gradient

Derive the gradient of (18.6) with respect to $\mW$ in the neural bigram model.

Solution in Appendix D

Exercise 18.4 ★★★ Fixed-window implementation

Create fixed-width contexts from a token-id array, run the MLP forward pass, and explain one dependency it cannot represent.

Solution in Appendix D

References

[radford2019] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. OpenAI technical report, 2019.
[bengio2003] Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin. A neural probabilistic language model. Journal of Machine Learning Research 3, 1137–1155, 2003.
[hoffmann2022training] J. Hoffmann et al. Training Compute-Optimal Large Language Models. 2022. arXiv:2203.15556
[kaplan2020scaling] J. Kaplan et al. Scaling Laws for Neural Language Models. 2020. arXiv:2001.08361