How Large Language Models Actually Work

What an LLM is (and is not)

A large language model is a neural network trained on vast amounts of text to model the probability of the next token given prior tokens. “Large” refers to billions of adjustable parameters (weights) and training corpora that can span much of the public web, books, and licensed collections. The model does not store sentences in a lookup table; it compresses regularities of language—and many factual associations that appear in those regularities—into those weights.

Calling that “understanding” is a metaphor. Literally, the system computes a distribution over possible next tokens and picks one (or samples from the distribution). Behaviors that look like reasoning, empathy, or planning emerge when those behaviors were useful patterns to imitate in training data. That is powerful for drafting and explanation; it is not a guarantee of truth or of consistent logic.

Training data shapes behavior

Before architecture details, consider the diet. Pre-training exposes the model to diverse text: encyclopedic prose, forums, code repositories, news, fiction, and more. The model never “reads” with intent; it sees token sequences and learns to predict continuations. Whatever appears often—common APIs, famous dates, recurring code idioms—becomes statistically easy to reproduce. Rare, private, or recent facts may never appear strongly enough in weights to be retrieved reliably without external tools.

Tokenization: text to token IDs

Neural networks operate on numbers, not Unicode strings. A tokenizer splits raw text into tokens—pieces that might be whole words, subwords, or punctuation—and maps each piece to an integer ID from a fixed vocabulary (often tens of thousands of entries). Common English words like “the” may be a single token; rare words or typos split into smaller fragments so the model can still handle them without an infinite vocabulary.

Tokens versus words

Product UIs often show “words” or character limits, while the model bills and truncates by tokens. A short prompt in English might be ~1.3 tokens per word on average, but code, URLs, and non-Latin scripts can use more tokens per visible character. That mismatch explains why a “2000-word” essay fits in one product but overflows another, and why a JSON blob can consume context quickly even when it looks small on screen.

When you need exact counts for drafts or limits, use SmartKitio’s word counter and character counter on the actual string. The model’s internal token count is authoritative for the model—but opaque in most chat UIs.

Embeddings: tokens as vectors

Each token ID is converted into a high-dimensional vector (an embedding). Early in the network, related tokens sit in similar directions: plural forms near singulars, programming keywords near one another, and so on. These vectors are learned during training, not hand-written. They let the model generalize: if “json” and “JSON” share statistical contexts, their representations can align even when the tokenizer treats them differently.

The transformer and self-attention

Modern LLMs are dominated by the transformer architecture introduced by Vaswani et al. (2017). Stacked layers apply self-attention: each token’s representation is updated by weighted combinations of all other tokens in the current context. Attention weights are computed from learned projections—roughly, “how much should token B influence token A for this prediction task?”—rather than from fixed rules of grammar.

Why attention matters

Older sequence models struggled with long-range dependencies because information had to pass through many sequential steps. Attention lets distant tokens interact directly within a layer, which helps with coreference (“Alice … she”), long code blocks, and multi-step instructions in a single prompt. Depth (many layers) and width (large hidden sizes) let the model build hierarchical features: local syntax in early layers, broader discourse and topic structure in later ones.

Context window: the visible past

At inference time, the model can only attend to tokens currently loaded in the context window—a fixed maximum length that varies by product (thousands to millions of tokens in specialized systems). Everything outside that window is invisible unless you summarize it elsewhere or use retrieval to inject excerpts. Long conversations therefore need compression, sliding windows, or external memory; otherwise the model literally forgets the opening instructions.

Next-token prediction as the core objective

During pre-training, the model sees a sequence and learns to predict each next token. Wrong guesses produce a loss signal; optimization adjusts weights to reduce that loss across billions of examples. Grammar, style, factual snippets, and coding patterns are all side effects of doing this job well on diverse text. There is no separate module labeled “facts” or “ethics” unless later training stages explicitly add objectives (for example, preference tuning on human-rated outputs).

Inference, temperature, and decoding

Training teaches probabilities; inference uses them to produce text. Greedy decoding always picks the highest-probability next token—deterministic but sometimes repetitive. Sampling draws from the distribution, which adds variety. Temperature scales logits before softmax: lower temperature sharpens the distribution (safer, duller continuations); higher temperature flattens it (more creative, more erratic). Top-p (nucleus) sampling truncates the tail of low-probability tokens to reduce nonsense without freezing the output entirely.

Context window size caps how many prior tokens enter attention on each step. A long system prompt plus retrieved docs plus chat history all compete for the same budget; when the window fills, the earliest tokens drop out of view unless the product summarizes them. Decoding is autoregressive: each newly generated token is appended and fed back in, so a wrong early token can steer the rest of the completion—not because the model “changed its mind,” but because later steps condition on the mistake.

Training versus inference

Why models can appear intelligent

Human language already encodes reasoning steps, caveats, and dialogue. A model trained to predict such text will sometimes emit step-by-step explanations because those patterns reduced prediction error on training documents. Scale increases the fidelity of those patterns: more parameters can store more nuanced associations. That can feel like a tutor or a pair programmer. It remains pattern completion unless paired with tools that ground outputs in checks—compilers, calculators, retrieval, or databases.

Hallucinations and limitations (preview)

When the model lacks evidence, the same machinery still produces fluent tokens—hence hallucinations. It also inherits biases and blind spots from training data, can mishandle counterfactuals, and may confabulate libraries or URLs that look plausible. Context limits, tokenization quirks, and sampling randomness add further failure modes. These are architectural consequences, not occasional bugs; mitigations live in process design (retrieval, verification, human review), not in wishing the model were a database.

A simplified worked example

Working with structured text

LLMs can propose JSON shapes and reformat messy payloads, but they may drop commas, duplicate keys, or invent fields. For machine-readable output, treat the model as a draft assistant and run the result through a formatter or validator. SmartKitio’s JSON formatter helps inspect structure locally; it does not replace schema validation against your real API contract.

Fine-tuning and alignment (briefly)

After pre-training, many products apply further stages: supervised fine-tuning on curated dialogues, preference optimization from human ratings, or tool-use training that teaches the model to call search or code interpreters. These stages change tone, safety refusals, and willingness to cite uncertainty—they do not replace the core next-token mechanism. Tool-augmented assistants feel more reliable because external programs fetch facts or run calculations, not because the base transformer gained a conscience.