LLMs & Generative AI
Tokenization & Context Windows
LLMs operate on tokens, not raw characters or words directly. The context window is the maximum token budget the model can condition on during one request.
- A token is a model-facing chunk of text
- Context is working memory, not permanent memory
- Long context has real costs
- Ordering and formatting matter
- Different languages and code tokenize differently
| Context ingredient | Why include it | Risk if overused |
|---|---|---|
| Instructions | Define behavior and constraints | Long policies become hard to follow |
| Examples | Teach output pattern | Too many examples crowd evidence |
| Retrieved chunks | Ground answers in facts | Noise causes wrong grounding |
| Chat history | Preserve state | Old state conflicts with current task |
| Tool outputs | Fresh external data | Large raw outputs hide key facts |