As asked
Explain what a context window is in an LLM, how it affects what you can build, and what product engineering decisions you make differently depending on whether you have 8k versus 128k versus 1M token contexts.
Sample answer outline
The context window is the maximum amount of text the model can attend to in a single call, including both the input and the output. With 8k tokens you need careful chunking and retrieval to handle long documents. With 128k tokens you can fit most single documents but still cannot load an entire codebase. With 1M tokens you can fit large repositories but the cost per call scales with input tokens, so you still want smart retrieval for cost reasons. Product decisions affected: chat history truncation strategy, document chunking and retrieval for RAG, whether to use an embedding search step at all, and latency since time-to-first-token grows with context length. A strong answer also notes the lost-in-the-middle problem: models attend less reliably to content in the middle of a very long context.
Expect these follow-ups
- How do you handle a chat product where conversation history grows beyond the context window over time?