datartha
LLM inference · Part 1 of 2

What happens when you ask an LLM a question? Context, tokens and attention

Before an LLM can generate an answer, it must assemble the available context, turn text into tokens and work out which parts matter most.

Vinay C G AI 4 minute read

1. What happens when you ask an LLM a question?

When we type a question into an LLM, it looks like a simple question-and-answer process. Behind the scenes, the model goes through several steps before we see the response.

At a high level, it takes the available context, breaks the text into tokens, processes those tokens and starts predicting what should come next. The important part is this: the full answer is not created at once. It is built one token at a time.

Overview of an LLM turning a user question into a response through context, tokenization, attention, next-token prediction and autoregressive decoding.
The complete inference flow, from the user’s question to the final response.

2. It starts with context

Before the model can generate an answer, it needs context. This includes your question, previous messages in the conversation, instructions given to the model and any other information provided to it.

Example prompt: “Explain checkpoint in Oracle”

That question becomes part of the context the model uses to generate its answer. Simply put, context is the information the model has available while answering you.

Diagram showing an LLM context assembled from the user prompt, previous messages, model instructions and other available information.
The model generates its response using all relevant context available to it.

3. Text becomes tokens

LLMs do not work directly with sentences in the same way we read them. The text is first broken into smaller pieces called tokens.

Explain  |  checkpoint  |  in  |  Oracle

The exact split depends on the tokenizer used by the model. A token can be a full word, part of a word, punctuation or another small piece of text. These tokens are what the model processes.

Tokenization diagram splitting the sentence Explain checkpoint in Oracle into four illustrative tokens.
An illustrative token split; the exact result depends on the model’s tokenizer.

4. Attention helps connect the context

Now the model needs to understand how the tokens relate to each other. This is where attention plays an important role. In “Explain checkpoint in Oracle,” the words “checkpoint” and “Oracle” are closely related to what the user is asking.

Attention helps the model assign different importance to different parts of the available context while processing the text. This helps it use the right information at the right time.

Self-attention diagram showing relationships between the tokens Explain, checkpoint, in and Oracle, with a stronger connection between checkpoint and Oracle.
Attention helps determine which parts of the available context matter for each token.

Part 1 takeaway: Context, tokens and attention prepare the model to generate - but the answer has not been written yet. Part 2 follows the prediction loop that builds it.