The Sequence Opinion - Issue 914: From Prompt to Token: How AI Inference Really Works
A field guide to prefill, decode, KV caches, and the systems that turn model weights into a responsive product.
Training gets the headlines. Inference gets the invoice.
A model may spend months learning on a giant cluster, but after training it enters a stranger world. Production traffic arrives asynchronously. Prompts have different lengths. Some users ask for one sentence; others ask for a small novel. Everyone wants the first token immediately, the rest smoothly, and the whole thing cheaply.
This is why “inference” is a misleadingly small word. It sounds like one forward pass. A modern inference system is closer to a miniature operating system wrapped around a token factory. It assembles context, tokenizes text, routes requests, schedules GPU work, manages memory, executes transformer kernels, samples outputs, and streams text—while serving thousands of users at different stages.
To see the machinery, follow one request: a 4,000-token prompt asking for a 300-token answer.