Next-Token Prediction

Strip away the chat interface, the system prompt, the tool calls, and the safety training, and a large language model is one thing: a function that reads a sequence of tokens and returns a probability distribution over which token comes next. That is the entire modeling objective — not "understand language," not "be helpful," just predict the next token. Everything an LLM appears to do, from translating French to writing Python to reasoning about a physics problem, is a side effect of getting very good at that single prediction task over a very large amount of text. This lesson builds the mental model that the rest of the course depends on: what the model is (a fixed function producing a distribution), how it was trained (maximize the likelihood of real text, equivalently minimize cross-entropy), and how that frozen function is turned into flowing text (an autoregressive sampling loop you can write in a few lines). Keeping the model and the loop distinct is the single most clarifying distinction for an engineer building on top of LLMs.

A language model is a distribution over the next token

Formally, a language model assigns a probability to a sequence of tokens, and it does so by factoring that sequence into a product of next-token predictions. For a sequence $w_1, w_2, \ldots, w_n$, the chain rule of probability gives:

$P(w_1, \ldots, w_n) = \prod_{t=1}^{n} P(w_t \mid w_1, \ldots, w_{t-1})$

Each factor $P(w_t \mid w_{<t})$ is "given everything so far, how likely is each possible next token." A language model is exactly a learned estimate of those conditional distributions. This is the definition Jurafsky & Martin open their language-modeling chapter with: models that assign probabilities to sequences of words, computed by predicting each word from the words that precede it (SLP3 ch. 3).

The output is a genuine probability distribution over the model's whole vocabulary — typically tens of thousands of possible tokens. Given the prompt The capital of France is, the model might put 0.7 on Paris, 0.05 on a, 0.01 on located, and minute probabilities on every other token, all summing to 1. The model never "decides" on a word. It reports odds. The mechanism that converts the raw scores (logits) into a distribution is the softmax function, and what to do with that distribution is a separate question entirely — that is the sampling loop, covered below and in detail in 05 · inference, sampling & context.

Note the word token, not word. Modern LLMs predict over sub-word tokens, not whole words — a detail with real cost and behavior implications that gets its own treatment in 03 · tokenization & embeddings. For this lesson, read "token" as "the unit the model predicts."

The training objective: maximize likelihood = minimize cross-entropy

How do you get a model whose distributions match real language? You show it enormous amounts of human text and adjust its parameters so it assigns high probability to the tokens that actually came next. This is maximum likelihood estimation: choose parameters $\theta$ that maximize the probability the model assigns to the training corpus. Taking the log and negating turns the product into a sum and a maximization into a minimization, giving the cross-entropy loss used to train essentially every LLM:

$\mathcal{L}(\theta) = -\frac{1}{N}\sum_{t=1}^{N} \log P_\theta(w_t \mid w_{<t})$

In words: for each position in the training text, look at the probability the model assigned to the actual next token, take its log, average over all positions, and negate. The loss is small when the model confidently predicted the right token and large when it was confident and wrong. Bengio et al.'s neural language model — the direct ancestor of today's LLMs — was trained on exactly this objective, learning "the joint probability function of sequences of words" via a distributed representation of words (Bengio et al., 2003).

This objective has a deep root. Shannon framed predicting the next symbol in English as a measurable quantity and tied it to entropy — the inherent unpredictability of a source — building successive "approximations to English" that read more naturally as the model conditioned on more prior context (Shannon, 1948). Cross-entropy loss is, almost literally, a measure of how surprised the model is by real text; its exponential, perplexity, is the standard intrinsic metric and can be read as the model's effective branching factor — how many tokens it is, on average, choosing among (SLP3 ch. 3).

The lineage: n-grams → neural LMs → Transformers

Next-token prediction long predates deep learning; what changed is how the conditional is estimated. Seeing the progression makes clear why today's models generalize where their predecessors could not.

N-gram models estimate $P(w_t \mid w_{<t})$ by counting: approximate the full history with the last $n-1$ tokens and use observed frequencies, e.g. a trigram model uses $P(w_t \mid w_{t-2}, w_{t-1})$ (SLP3 ch. 3). Simple and fast, but they drown in the curse of dimensionality: most plausible word sequences never appear in any finite corpus, so they get probability zero, and the model has no notion that "cat" and "dog" are similar.

Neural language models fixed this. Bengio et al. learned a distributed representation (an embedding) for each word and a neural network mapping the previous words' embeddings to a next-word distribution. Because semantically similar words get nearby vectors, the model assigns reasonable probability to sentences it never saw, "fighting the curse of dimensionality" by generalizing across similar contexts rather than memorizing exact strings (Bengio et al., 2003).

Transformers are the current answer to "what network architecture computes the conditional." They process the whole visible context in parallel with self-attention and scale to context windows of thousands of tokens, but the objective is unchanged — still next-token prediction trained with cross-entropy (SLP3 ch. 7). The architecture is the subject of 02 · transformers & attention; the point here is that the goalposts never moved, only the machinery for hitting them.

Why "just predict the next word" is deceptively powerful

It is tempting to dismiss next-token prediction as autocomplete. The reason it is not is that minimizing prediction error on a sufficiently broad corpus forces the model to acquire the structure that generates the text. To reliably predict the token after The opposite of "hot" is, a model must encode antonymy. To complete 2 + 2 = or def is_prime(n):, it must encode arithmetic and code structure. To finish a mystery novel's The murderer was, it would, in the limit, need a model of the plot. Karpathy makes this argument directly: the network is compressing the internet, and to predict the next token well it is compelled to learn a great deal about the world that produced the text (Karpathy, Intro to LLMs).

This is why a single, mechanically simple objective yields such broad competence: world knowledge, grammar, reasoning patterns, and style are not separate features bolted on — they are whatever it takes to lower the loss. The objective is unsupervised in the practical sense that no human labels the data; the "answer" at every position is simply the token that actually appeared next, which means the entire written record of humanity is training data with the labels already attached.

Two honest caveats. First, low loss means the model is good at predicting plausible continuations, which is not the same as being truthful — fluent, likely-sounding text can be wrong, the root of hallucination, examined in 06 · capabilities, limits & scaling. Second, the raw pretrained model is a document completer, not an assistant; turning "predict the next token" into "follow my instruction helpfully" takes additional training, the subject of 04 · training: pretraining to RLHF.

The model vs. the loop: a fixed function, run repeatedly

Here is the distinction engineers most often blur. The model is a fixed, deterministic function: same parameters, same input tokens → same output distribution, every time. It produces one distribution over the vocabulary and then it is done. It does not, on its own, produce a sentence, a paragraph, or even a second token.

Text comes from wrapping that function in an autoregressive generation loop: ask the model for the next-token distribution, sample one token from it, append that token to the input, and ask again — repeating until a stop condition (SLP3 ch. 7, Karpathy, Let's build GPT).

Everything you tune as an engineer at inference time lives in the sample step, not the model: greedy decoding (always take the most probable token, fully deterministic), or temperature and top-p/nucleus sampling to trade off coherence against diversity. The model's distribution is the same regardless; the loop decides how to consume it. This is also why the same prompt can yield different answers on repeated calls — the randomness is in sampling, not in the network. These controls are the whole of 05 · inference, sampling & context. Karpathy's "spelled out" build makes the separation concrete: a tiny Transformer is trained on the cross-entropy objective, then a short loop samples from it character by character to generate new text (Karpathy, Let's build GPT).

The same frozen function $P_\theta$ is used in two opposite regimes, and confusing them is a common source of muddled mental models. During training, the entire real sequence is already known: the model predicts every position in parallel, and at each step it is conditioned on the true prior tokens from the corpus, never on its own guesses. This is teacher forcing — the "answer so far" is always the ground-truth text, and the cross-entropy loss compares the model's distribution to the token that actually came next (SLP3 ch. 7). During inference there is no future text to lean on: the model must consume its own sampled outputs as the context for the next step, one token at a time. Same parameters, same function — but training feeds it ground truth and grades it, while inference feeds it its own predictions and lets them ride.

The generation loop in code

The loop is genuinely small. Stripped to its essence, ignoring batching and the KV-cache optimization (covered in 05), autoregressive generation is:

def generate(model, prompt_tokens, max_new_tokens, temperature=1.0):
    """Turn a fixed next-token model into a stream of text."""
    tokens = list(prompt_tokens)            # the running context
    for _ in range(max_new_tokens):
        logits = model(tokens)              # the MODEL: scores for every vocab token
        logits = logits[-1] / temperature   # we only care about the last position
        probs = softmax(logits)             # -> a probability distribution
        next_token = sample(probs)          # the LOOP's choice: greedy, top-p, etc.
        tokens.append(next_token)           # feed our own output back in
        if next_token == END_OF_TEXT:       # stop condition
            break
    return tokens

Three things to internalize from these lines. (1) model(tokens) is the entire learned object — it returns scores, never text. (2) The model consumes its own previous outputs as input on the next iteration; this self-feeding is precisely what "autoregressive" means, and it is why an early mistake can compound across a long generation. (3) sample() is policy, not model — swap it for argmax and generation becomes deterministic. The cost of each step scales with how much context the loop has accumulated, which is why long outputs and long prompts are expensive — the throughput and latency consequences are taken up in 05 · inference, sampling & context.

Sources

• Jurafsky & Martin — Speech and Language Processing, 3rd ed., ch. 3, N-gram Language Models — language model defined as assigning probabilities to sequences via next-word prediction; n-grams, perplexity, cross-entropy.
• Jurafsky & Martin — Speech and Language Processing, 3rd ed., ch. 7, Large Language Models — Transformer-based LMs and autoregressive generation under the same next-token objective.
• Bengio, Ducharme, Vincent & Jauvin — A Neural Probabilistic Language Model (JMLR, 2003) — learning the joint probability of word sequences via distributed representations; fighting the curse of dimensionality.
• Shannon — A Mathematical Theory of Communication (1948) — entropy, next-symbol prediction, and successive approximations to English; the information-theoretic root of cross-entropy.
• Karpathy — Let's build GPT: from scratch, in code, spelled out (2023) — trains a small Transformer on the cross-entropy objective, then samples autoregressively to generate text.
• Karpathy — [1hr Talk] Intro to Large Language Models (2023) — the "compressing the internet" framing: next-token prediction forces world knowledge.

Practice

Test your understanding — try each before reading the answers.

1. Model vs. loop, in one breath. A teammate says "the model wrote a three-paragraph reply." Why is that statement imprecise, and which part of the system actually produced three paragraphs rather than one token? Which single line of the generate snippet would you change to make every call return the identical reply, and why does that change nothing about the network?

2. Training vs. inference asymmetry. During training the model is fed the true prior tokens at every position (teacher forcing); during inference it is fed its own sampled tokens. Given that the parameters $\theta$ are identical in both regimes, explain why this difference makes an early sampling mistake able to "compound" at inference time but not during training.

3. Why the lineage matters. A trigram count model assigns probability zero to a perfectly grammatical sentence it never saw in its corpus. Why does that happen, and how do Bengio et al.'s distributed representations let a neural model assign it sensible probability instead? Use the term curse of dimensionality and explain what concretely changed between the two approaches.

4. The deceptive-power claim, with its caveat. Pick a task such as completing The chemical symbol for gold is or the last line of a proof, and argue in two or three sentences what the model must have internalized to predict that next token well. Then state the honest limit: why does a low cross-entropy loss not guarantee a true answer, and how does that connect to hallucination (06 · capabilities, limits & scaling)?

Answers:

1. The model is a fixed function that, on one call, returns one distribution over the vocabulary — it never emits text, let alone paragraphs. The three paragraphs come from the autoregressive loop wrapped around it (generate), which samples a token, appends it, and re-invokes the model until a stop condition. To force identical output every call, replace next_token = sample(probs) with a greedy argmax(probs) (equivalently, drive temperature toward 0): the randomness lives entirely in the sample step, so removing it leaves the model's weights and its produced distribution exactly the same — only the consumption of that distribution changes.

2. Under teacher forcing the context at every position is the ground-truth corpus, so a bad prediction at position $t$ does not corrupt the input the model sees at $t+1$ — each position is graded independently against the real next token. At inference there is no ground truth to fall back on: the loop feeds the model its own output, so a wrong or low-probability token sampled early becomes part of the context for every subsequent step, and the model then conditions on a premise it invented. That self-feeding (the literal meaning of autoregressive) is what lets errors accumulate over a long generation.

3. A trigram model estimates $P(w_t \mid w_{t-2}, w_{t-1})$ by counting occurrences in the corpus; an exact context it never observed has a count of zero, so it yields probability zero — and most plausible token sequences never appear in any finite corpus, the curse of dimensionality. Bengio et al. replaced exact-string counting with a distributed representation: each token maps to a learned embedding vector, and a neural network maps the previous tokens' embeddings to a next-token distribution. Because semantically similar tokens get nearby vectors, the model generalizes across similar contexts rather than memorizing exact strings, so a never-seen but reasonable sentence gets reasonable (nonzero) probability.

4. To complete The chemical symbol for gold is with Au, the network must have encoded a fact-like association between the element name and its symbol — the kind of world knowledge that lowering prediction loss over a large corpus compels it to absorb; to finish a proof it would need the inferential structure of the argument. The honest limit: minimizing cross-entropy makes the model good at producing plausible, likely-sounding continuations, which is not the same as true ones. Fluent text can be confidently wrong, and that gap between "high-probability under the data" and "correct" is the root of hallucination, taken up in 06 · capabilities, limits & scaling.

Hands-on: cross-entropy and perplexity by hand, then in code

Work the arithmetic first, then check it. For the prompt The sky is, suppose the model assigns these probabilities to the candidate next tokens: blue $= 0.5$, clear $= 0.2$, falling $= 0.04$, green $= 0.01$ (the rest of the vocabulary splits the remaining mass).

1. Per-token loss. The cross-entropy term for a position is $-\ln p(\text{actual token})$. Compute it for the case where the true next token is blue, and again for green. Confirm the confident-but-wrong case ( green) is penalized far more: $-\ln 0.5 \approx 0.693$ versus $-\ln 0.01 \approx 4.605$ nats.

2. Average over a sequence. Suppose across three positions the model assigned the actual tokens probabilities $0.5$, $0.2$, and $0.04$. Average their $-\ln p$ values to get the sequence cross-entropy, then exponentiate to get perplexity: $\exp\!\big(\tfrac{1}{3}(-\ln 0.5 - \ln 0.2 - \ln 0.04)\big)$. Interpret the result as the model's effective branching factor — roughly how many equally likely tokens it is choosing among per step (SLP3 ch. 3). Now reason about the limit cases: what is the perplexity of a model that puts probability $1.0$ on every actual token, and what does a lower perplexity say about how surprised the model is by the text?

Work the arithmetic in your head first, then run the cell below to check it. It computes the per-token loss for blue vs. the confident-but-wrong green, then the sequence cross-entropy and perplexity over the three probabilities above — all with the standard library only. Tweak the probs list to feel the metric move: push the probabilities toward $1.0$ and watch perplexity fall toward its floor of $1$ (a model never surprised by the text):

3. Inspect a real model's logits. Load any small open model with the Hugging Face transformers library, tokenize a prompt, and read the next-token distribution directly:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

ids = tok("The capital of France is", return_tensors="pt").input_ids
logits = model(ids).logits[0, -1]          # scores for the last position only
probs = torch.softmax(logits, dim=-1)      # -> a distribution over the vocab

topk = torch.topk(probs, 5)
for p, i in zip(topk.values, topk.indices):
    print(f"{tok.decode(i)!r:>12}  {p.item():.4f}")

Confirm three things the lesson predicted: the output is a full distribution that sums to ~1 across the vocabulary, the model returns scores/logits rather than text, and the top candidates are tokens (often with a leading space) — not necessarily whole words (03 · tokenization & embeddings). Then take argmax to do greedy decoding by hand, append the chosen token, and run the cell again to feel the autoregressive loop one step at a time.