LLM ops engineer domain knowledge interview questions

As asked

State the time and memory complexity of standard multi-head attention as a function of sequence length N and hidden dimension D. Explain which term dominates at long versus short contexts and why this drives the adoption of Flash Attention.

Sample answer outline

Time complexity is O(N^2 * D) for the attention matrix multiply plus O(N * D^2) for the projection matrices. Memory is O(N^2) for the attention score matrix. At short context (N much less than D), the projection dominates; at long context (N much greater than D), the N^2 attention score matrix dominates both memory and time. Flash Attention eliminates the O(N^2) memory materialization by computing in tiles, making long context tractable.

Expect these follow-ups

• At what sequence length does the N^2 term start to dominate for a 4096 hidden-dim model?
• How does multi-query attention reduce memory bandwidth cost during decode?

As asked

Explain what temperature, top-p (nucleus sampling), and top-k do to the token probability distribution at inference time. When would you set temperature to 0, and what are the failure modes of high temperature in a production API?

Sample answer outline

Temperature scales the logits before softmax: values below 1 sharpen the distribution (more predictable), values above 1 flatten it (more diverse). Top-k truncates all tokens outside the top-k highest logits. Top-p (nucleus) selects the smallest set of tokens whose cumulative probability exceeds p. Temperature 0 means greedy decoding (argmax), producing fully deterministic outputs. High temperature in production causes inconsistent output quality, more hallucinations, and outputs that drift far from the intended task, which is why most production APIs default to temperature 0.7 or less.

Expect these follow-ups

• Why does temperature 0 still not guarantee identical outputs across different serving systems?
• What is the interaction between top-p and temperature when both are set?

As asked

A product team wants to launch an LLM feature that can summarise uploaded legal, financial, and HR documents. Walk me through your red-team threat model before launch.

Sample answer outline

Identify assets first: confidential documents, model outputs, user identities, audit logs, and any downstream actions triggered by summaries. Then enumerate abuse paths: cross-tenant data leakage, prompt injection from uploaded files, hallucinated legal claims, unsafe advice, and adversaries using the system to process stolen material. The answer should prioritise risks by likelihood and impact rather than producing a generic list. Strong candidates propose concrete tests, such as tenant-boundary probes, adversarial documents, output policy checks, and privacy-preserving logging review. Interviewers listen for clear reporting language that product teams can act on, not theatrical exploit naming.

Expect these follow-ups

• What launch blocker would make you recommend delaying the release?
• How would you test cross-tenant leakage without touching real customer data?
• What belongs in the risk acceptance memo?

As asked

Explain arithmetic intensity and how you would calculate it for a single decode step of a large language model. What does this tell you about whether the workload is memory-bandwidth-bound or compute-bound, and how does this affect your hardware selection?

Sample answer outline

Arithmetic intensity is FLOPs per byte of memory traffic. For a single decode step at batch size 1, the model reads all parameters from HBM (2 bytes per param in BF16) and does one matrix-vector multiply per layer (2 * hidden_dim^2 FLOPs per layer). The arithmetic intensity is roughly 1 FLOP/byte for small batches, far below the H100 SXM5 ridge point of roughly 295 FLOPs/byte (dense BF16 tensor core TFLOPS divided by HBM3 bandwidth), meaning decode is heavily memory-bandwidth-bound. This means FLOP count does not matter for decode; HBM bandwidth does, which is why A100 and H100 with high HBM bandwidth are the right hardware.

Expect these follow-ups

• At what batch size does a decode step become compute-bound on an H100?
• How does this analysis change for the prefill phase versus the decode phase?

As asked

Explain the byte-pair encoding tokenization algorithm. How does tokenization affect inference cost, and what are the operational implications when switching tokenizers between model versions?

Sample answer outline

BPE starts with individual bytes as the vocabulary and iteratively merges the most frequent adjacent pair, building a vocabulary of subword units. At inference time, the tokenizer maps text to token IDs before the model sees it. Tokenization affects cost because pricing and latency scale with token count, not character count. Switching tokenizers between model versions can silently change how inputs are tokenized, causing the same string to produce different token counts and different outputs, which can break downstream integrations that cache tokenized inputs or rely on fixed token counts.

Expect these follow-ups

• How would you detect that a tokenizer change is causing output differences in a model upgrade?
• What is the token efficiency difference between tiktoken and the Llama tokenizer for English text?

As asked

Explain the low-rank decomposition used in LoRA. Given a weight matrix W of shape (d_in, d_out), what are the shapes of the LoRA matrices, and what is the additional FLOPs cost per forward pass as a function of the rank r?

Sample answer outline

LoRA replaces the full weight update with W + BA, where B has shape (d_in, r) and A has shape (r, d_out), with r much less than min(d_in, d_out). The additional FLOPs per layer are 2 * d_in * r + 2 * r * d_out versus 2 * d_in * d_out for the full layer, which is a fraction 2r / min(d_in, d_out) of the full layer cost. For r=16 and d=4096, the LoRA overhead is about 0.8% per layer, making it nearly free at inference if the adapter is merged into the base weights before serving.

Expect these follow-ups

• When is it better to merge the LoRA weights into the base model versus keeping them separate at serve time?
• What is the rank needed to fully represent the difference between a fine-tuned and base model in theory?

As asked

Explain grouped-query attention (GQA) and multi-query attention (MQA). How do they change the size of the KV cache relative to standard multi-head attention, and what is the quality tradeoff?

Sample answer outline

In standard MHA, each head has its own K and V projections. In MQA, all query heads share a single K and V head, reducing KV cache by a factor of n_heads. GQA is in between: G groups of query heads each share one K and V head, reducing KV cache by n_heads/G. Llama-3 70B uses GQA with 8 groups, reducing KV cache by 8x versus MHA. The quality tradeoff is minimal for GQA at moderate group counts but more visible for pure MQA, which is why GQA has largely replaced MQA in modern models.

Expect these follow-ups

• How does GQA affect the tensor parallel communication pattern compared to MHA?
• Given 80GB of GPU memory and a 70B GQA model, how much memory is left for KV cache?

As asked

List and explain the standard approaches to filtering LLM outputs before returning them to end users. What are the latency costs and failure modes of each approach?

Sample answer outline

Main approaches: keyword blocklists (zero latency, low recall for obfuscated content), regex pattern matching (near-zero latency, brittle), small classifier models running on CPU (10 to 30ms, better recall, requires maintenance), LLM-as-judge scoring (50 to 500ms, highest accuracy, highest cost), and embedding-based similarity to known harmful outputs (fast but requires curated reference set). The key failure modes are false positives (blocking legitimate outputs) and adversarial bypasses. Production systems typically layer multiple approaches with different cost-accuracy tradeoffs.

Expect these follow-ups

• How would you measure the false-positive rate of your output filter without user complaints?
• What changes to the filtering pipeline would you make for a streaming response versus a non-streaming one?

As asked

List the memory types available on a modern GPU (e.g., A100 or H100), their approximate sizes and bandwidths, and explain which level each part of the LLM inference computation uses.

Sample answer outline

The hierarchy from fastest to slowest: registers (per-thread, roughly 256KB per SM, roughly 20 TB/s effective), shared memory/L1 cache (per-SM, 164 to 228KB on H100, roughly 20 TB/s), L2 cache (shared across SMs, 50MB on H100, roughly 10 TB/s), HBM (80GB on H100, roughly 3.35 TB/s). Model weights live in HBM and are loaded into registers/shared memory for each matrix multiply. Flash Attention keeps the attention computation in shared memory to avoid HBM round trips. KV cache lives in HBM. The decode step is bounded by HBM bandwidth because it reads all weights once per step.

Expect these follow-ups

• How does shared memory bank conflict affect GEMM kernel performance?
• What is the purpose of the L2 cache in the context of KV cache access patterns?

As asked

RLHF with PPO and Direct Preference Optimization both align models with human preferences. From a serving and ops perspective, what differences would you expect to see in model behavior, output length distribution, and infrastructure requirements?

Sample answer outline

PPO-trained models use a reward model at training time, which does not affect inference infrastructure. However, PPO models often produce longer outputs due to reward hacking (the model learns that longer answers get higher rewards). DPO models tend to produce more concise outputs and can be more calibrated. From a serving perspective, PPO models may have higher average output token count (and thus cost), and their refusal behavior can be more erratic at temperature 0. Both require the same inference stack; the differences are behavioral rather than architectural.

Expect these follow-ups

• How would you detect reward hacking in a deployed model using production monitoring?
• What changes to your token budget configuration would you make for a PPO-trained model known to produce verbose outputs?

As asked

Explain RoPE (Rotary Position Embedding) and how it encodes position information. Why is RoPE used in most modern LLMs, and what happens operationally when you try to extend a RoPE model beyond its trained context length?

Sample answer outline

RoPE encodes position by rotating query and key vectors in the complex plane by position-dependent angles, which makes attention scores naturally sensitive to relative position. It is preferred over learned absolute embeddings because it generalizes better to unseen positions. When inference exceeds the trained context length, the rotation angles fall outside the range seen during training, causing attention patterns to degrade. Context extension techniques (YaRN, NTK-aware scaling) re-scale the base frequency of the rotation to fit a longer context into the trained angular range without fine-tuning, at the cost of reduced precision for positions near the maximum.

Expect these follow-ups

• How does YaRN extend RoPE context length and what is the tradeoff in perplexity at long versus short ranges?
• What would you observe in model outputs if you serve a 4K-context model with a 16K prompt without any context extension?