As asked
Explain how KV caching and batching affect throughput and latency in transformer inference.
Sample answer outline
KV cache stores prior key and value tensors so decoding each next token does not recompute attention over the full prefix. It improves decode efficiency but consumes memory proportional to batch size, sequence length, layers, and hidden dimensions. Batching improves GPU utilisation by running multiple requests together, but excessive batching adds queueing delay and can hurt p95 latency. Strong answers distinguish prefill, which is compute-heavy and parallel, from decode, which is often memory-bandwidth constrained. The common mistake is talking about tokens per second without considering time to first token and tail latency.
Expect these follow-ups
- Why can long prompts reduce the number of concurrent requests per GPU?
- What is continuous batching?
- How would prefix caching change product latency?
kv-cachebatchingtransformers