LLM ops engineer role-specific interview questions

As asked

Walk me through how vLLM's PagedAttention allocates and manages KV cache memory at runtime. How is it different from the contiguous pre-allocation that Hugging Face transformers uses, and what problems does it actually solve?

Sample answer outline

A strong answer explains that standard inference pre-allocates a contiguous KV cache block proportional to max sequence length, wasting memory for short sequences. PagedAttention divides the KV cache into fixed-size logical blocks mapped to non-contiguous physical pages, like virtual memory in an OS. This lets multiple sequences share the physical memory of prefilled prompt tokens (prefix sharing), dramatically increasing GPU utilization and throughput under mixed-length traffic.

Expect these follow-ups

• How does prefix caching build on top of PagedAttention?
• What happens to the block table when a request is preempted and later resumed?

As asked

Explain the difference between static batching and continuous batching in an LLM serving system. What is the core scheduling loop in a continuous batching engine, and where does it still fall short?

Sample answer outline

Static batching waits for a full batch to assemble before starting, leaving GPU idle while slow sequences hold a slot. Continuous batching inserts new requests into the batch as soon as any sequence emits an end-of-sequence token, keeping GPU utilization high. The scheduler runs at every decode step, checking for finished requests and admitting new ones. The main remaining gap is head-of-line blocking when a very long sequence occupies a slot that a short request could have finished in.

Expect these follow-ups

• How does speculative decoding interact with continuous batching scheduling?
• What metrics would you watch to detect that your scheduler is starving short requests?

As asked

Describe how you would design an evaluation pipeline to gate model updates before they go to production. What benchmarks, custom evals, and automation would you include, and how do you handle the fact that LLM quality is hard to measure automatically?

Sample answer outline

A strong answer includes standard benchmarks (MMLU, HellaSwag, task-specific evals) as fast-failing smoke tests, a curated golden-set of real user queries with reference answers graded by an LLM-as-judge, latency and throughput regression tests, and a small shadow traffic comparison against the current production model. The pipeline gates on statistically significant quality drop, not just raw score, to avoid blocking good models on noise.

Expect these follow-ups

• How do you prevent the judge model from having systematic biases that mask real regressions?
• What would you do if a new model scores better on benchmarks but users report worse answers?

As asked

List the specific metrics you would instrument and expose when operating a production vLLM or TGI deployment. For each, explain what alerting threshold you would set and why.

Sample answer outline

Core metrics include: GPU SM utilization (alert below 60% sustained for idle detection), GPU memory usage (alert above 90% to give head-room for eviction), KV cache usage percentage (alert above 80% which signals imminent eviction storms), request queue depth (alert above N x expected concurrency), time-to-first-token p99, and tokens-per-second per GPU. For SLO compliance, also track the fraction of requests exceeding latency targets.

Expect these follow-ups

• How would you instrument a vLLM deployment that runs behind an autoscaler?
• What metric would you use as the scale-out signal for your HPA in Kubernetes?

As asked

Walk me through how you would calculate and then reduce the cost-per-output-token for a self-hosted LLM deployment on A100 GPUs. What are the main levers and what are their risks?

Sample answer outline

Cost-per-token starts from GPU-hours divided by total tokens generated. Increasing throughput (via continuous batching, higher max batch size, prefix caching) reduces cost by keeping the GPU busy. Quantization to INT4/INT8 reduces memory pressure and allows larger batches, with a small quality risk. Model distillation or switching to a smaller model that meets quality thresholds is the largest lever. The risk of each is quality degradation, which must be caught by the eval pipeline before deploy.

Expect these follow-ups

• How would you measure the ROI of prefix caching specifically?
• What would change in your cost model if you moved from on-demand A100 instances to reserved capacity?

As asked

Why is there an inherent tension between time-to-first-token and throughput in an LLM serving system, and how do you tune the serving stack for each objective independently?

Sample answer outline

TTFT is dominated by the prefill phase, which is compute-bound and must complete before the first token is returned. Throughput is maximized by batching many decode steps together. Large batches improve decode throughput but delay new request prefill (head-of-line blocking). Tuning for TTFT means capping max batch size and prioritizing prefill scheduling. Tuning for throughput means maximizing batch size and tolerating higher TTFT. Chunked prefill is the mechanism that lets you interleave prefill and decode to soften this tradeoff.

Expect these follow-ups

• How would you set different SLOs for streaming versus non-streaming endpoints on the same cluster?
• At what traffic level does chunked prefill start paying for its added scheduling complexity?

As asked

You are replacing the production model with a new fine-tune. Walk me through the rollout process including how you split traffic, what you measure during the canary phase, and what your rollback trigger is.

Sample answer outline

Start with 1 to 5% traffic to the new model, using request duplication or a weighted routing rule. Measure quality via LLM-as-judge on real traffic outputs (without user seeing both), latency p50/p99, error rates, and any task-specific business metrics (e.g., downstream conversion). Rollback triggers are: judge score dropping more than a defined threshold versus the baseline, error rate spike above 1%, or latency p99 regression above 20%. Increase traffic in steps (5%, 20%, 50%, 100%) with a holding period and automated alerting at each step.

Expect these follow-ups

• How do you detect quality regressions on free-form generations where there is no ground truth?
• What would you do if the canary model is better on aggregate quality metrics but worse for one specific user cohort?

As asked

Three days after deploying a new fine-tune, you start getting user reports that the model is making up facts more frequently. Your eval metrics showed no regression. How do you investigate, and what is your decision process for rolling back?

Sample answer outline

First, collect specific failing examples from user reports and check if they cluster around a particular topic or prompt pattern not covered in the eval golden set. Then run targeted evals on those failure patterns to quantify the regression your golden set missed. If confirmed, roll back immediately while investigating why the eval missed it. The root cause is usually a distribution gap between the eval golden set and actual user traffic. The fix is both a rollback and an update to the eval pipeline to cover the new failure mode.

Expect these follow-ups

• How do you reliably collect high-quality hallucination examples from users without revealing internal model details?
• What changes to your eval pipeline would you make after this incident?

As asked

What telemetry would you add to an LLM application so on-call can debug latency spikes, cost overruns, and quality regressions?

Sample answer outline

Capture request count, error rate, provider status, queue time, time to first token, total latency, input tokens, output tokens, cache hit rate, and estimated cost by route, tenant, model, and prompt version. Quality telemetry should include eval score distributions, user feedback, refusal rate, tool-call failure rate, and structured-output validation failures. Use traces to connect retrieval, model call, tool calls, and post-processing into one request path. Watch cardinality carefully because raw prompt text and user ids do not belong in metric labels. Candidates often provide infrastructure metrics but miss product-level quality signals.

Expect these follow-ups

• Which labels would you forbid on metrics?
• How do you correlate a prompt change with a quality regression?
• What alert would page someone versus create a ticket?

As asked

Your LLM inference bill has doubled in a month while traffic grew by only 20 percent. How would you investigate and control cost?

Sample answer outline

Break cost down by model, route, tenant, prompt tokens, completion tokens, cache hit rate, retries, and hardware utilisation. Look for longer prompts, lower batching efficiency, runaway agent loops, increased fallback to larger models, and retry storms after provider or gateway errors. Strong answers propose guardrails such as token budgets, model routing, semantic or prefix caching, prompt compression, and per-tenant cost dashboards. The candidate should avoid blind quality regression by pairing cost changes with evals and customer-impact checks. A common miss is treating unit cost as fixed when application behaviour often drives the bill.

Expect these follow-ups

• How do you detect an agent loop from billing data?
• When would you route a request to a smaller model?
• What cost metric should product managers see weekly?

As asked

Your vLLM deployment is running near capacity and the KV cache is full. Walk me through the eviction strategies available, the tradeoffs between recompute and swap, and how you would tune the policy for a latency-sensitive API.

Sample answer outline

A strong answer covers the three main options: recompute the evicted sequence from scratch (highest latency spike, zero memory overhead), swap to CPU RAM (moderate latency, requires PCIe bandwidth headroom), and chunked prefill to reduce the likelihood of hitting the limit in the first place. For latency-sensitive workloads, setting a low watermark on cache usage and failing early with a 503 is often better than suffering a recompute stall mid-generation.

Expect these follow-ups

• How does chunked prefill affect time-to-first-token versus throughput?
• What vLLM configuration knobs would you change to lower the probability of eviction in the first place?

As asked

Compare AWQ and GPTQ quantization methods. How do they differ in how they decide which weights to quantize, what data they need, and what the resulting quality versus speed tradeoffs are in practice?

Sample answer outline

GPTQ is a post-training quantization method that minimizes layer-wise reconstruction error using a small calibration set, processing columns of the weight matrix with second-order information. AWQ identifies the 1% of weights that are most salient based on activation magnitude and protects them from quantization, compressing the rest more aggressively. AWQ tends to produce better INT4 models with less calibration data and is faster at inference because it was co-designed with efficient GPU kernels. GPTQ may edge ahead on perplexity at INT3 but requires more careful calibration set curation.

Expect these follow-ups

• What is the role of the calibration dataset in GPTQ, and what happens if it is domain-mismatched?
• How does GGUF differ from these two in terms of where the quantization happens?

As asked

Describe how speculative decoding works mechanically. What guarantees does it give about output quality, what determines how much speedup you actually get, and when does it hurt rather than help?

Sample answer outline

A small draft model generates K tokens speculatively, then the target model verifies them all in a single forward pass using parallel evaluation. The rejection sampling step mathematically guarantees the same output distribution as the target model alone. Speedup depends on the acceptance rate of draft tokens, which drops when the draft and target models diverge in distribution (domain shift, long-tail vocabulary). Speculative decoding actually hurts latency if the draft model is too slow or the acceptance rate falls below roughly 0.7.

Expect these follow-ups

• How does Medusa heads approach compare to using a separate draft model?
• What would you monitor in production to detect when speculative decoding is net-negative?

As asked

You need to serve a 70B parameter model across 8 H100s. Walk me through the tradeoffs between tensor parallelism and pipeline parallelism for this use case. What does each strategy do to latency, and what failure modes are unique to each?

Sample answer outline

Tensor parallelism splits individual weight matrices across GPUs, requiring all-reduce communication at every transformer layer but keeping all GPUs busy during both prefill and decode. Pipeline parallelism splits layers across GPUs and passes activations between stages, which reduces all-reduce volume but introduces pipeline bubbles during prefill. For serving, tensor parallelism is usually preferred because it has lower per-request latency, while pipeline parallelism can scale to more GPUs when a single-node tensor parallel group does not fit. Both require careful placement: pipeline stages must communicate over NVLink or InfiniBand, otherwise communication becomes the bottleneck.

Expect these follow-ups

• How does sequence parallelism help with very long context inference?
• What would you check first if you noticed one GPU in your tensor-parallel group was consistently underutilized?

As asked

Explain the core insight behind Flash Attention. Why does the standard attention implementation use O(N^2) memory, and how does the tiling approach in Flash Attention reduce that without changing the mathematical result?

Sample answer outline

Standard attention materializes the full N x N attention score matrix in HBM, which is O(N^2) memory and bandwidth. Flash Attention tiles the Q, K, V matrices into blocks that fit in SRAM, computes attention locally with an online softmax correction (using the log-sum-exp trick), and accumulates the output without ever writing the full score matrix to HBM. The result is identical to standard attention but uses O(N) HBM memory and achieves near-peak FLOP utilization because compute is now memory-bound only on much smaller SRAM loads.

Expect these follow-ups

• How does Flash Attention 2 differ from Flash Attention 1 in its parallelism strategy?
• What are the implications of Flash Attention for backward pass memory during fine-tuning?

As asked

You are deploying Llama-3 70B on a node with 4 x A100 80GB. Walk me through the TGI flags you would set, the reasoning behind each, and what you would validate before taking traffic.

Sample answer outline

A strong answer sets --num-shard 4 for tensor parallelism across the 4 GPUs, selects an appropriate dtype (bfloat16 fits in roughly 140GB across 4 x A100 80GB with room for KV cache), configures --max-input-length and --max-total-tokens based on expected traffic patterns, and enables Flash Attention. Validation includes running a sweep of concurrent requests to check throughput and TTFT, checking nvidia-smi to confirm balanced memory usage across all 4 GPUs, and verifying outputs match a reference run.

Expect these follow-ups

• What would change if you only had 2 x A100 80GB and still needed 70B?
• How do you handle TGI's warmup period in a Kubernetes liveness/readiness probe?

As asked

Your serving fleet starts throwing CUDA out-of-memory errors after running for several hours, but nvidia-smi shows only 85% memory usage. What would you investigate, and how do you distinguish fragmentation from a memory leak?

Sample answer outline

The symptom of OOM at 85% usage strongly suggests fragmentation: the allocator cannot find a contiguous block large enough for the next allocation even though total free memory exists. Investigation would use torch.cuda.memory_stats() to compare reserved vs allocated bytes, look at the allocation histogram for large one-off tensors, and check whether the OOM correlates with specific request patterns (very long prompts or large batch sizes). A memory leak would show monotonically increasing reserved bytes over time; fragmentation shows high reserved-vs-allocated ratio with stable totals.

Expect these follow-ups

• How would you configure the CUDA caching allocator to reduce fragmentation?
• What change to your request routing or max sequence length cap would reduce the probability of this happening?

As asked

You have one base model and 50 customer-specific LoRA adapters. Describe the architecture you would use to serve all 50 adapters without loading 50 copies of the base model, and what the operational challenges are.

Sample answer outline

vLLM and TGI both support dynamic LoRA loading, where the base model weights stay resident in GPU memory and adapter weights are swapped in per-request. A strong answer describes the adapter registry, how requests are routed to the correct adapter by name, and the memory overhead of keeping hot adapters pinned versus evicting cold ones to CPU. Operational challenges include adapter version management, the cold-start latency when an adapter is not currently loaded, and ensuring eval pipelines cover per-adapter quality.

Expect these follow-ups

• How does the rank of the LoRA adapter affect inference latency and GPU memory overhead?
• What would you change about the architecture if 3 of the 50 adapters handle 95% of traffic?

As asked

What are the concrete security and operational reasons to prefer safetensors over .pt or .bin PyTorch checkpoints when loading models in a production serving environment?

Sample answer outline

Legacy PyTorch .pt files use Python's object serialization format under the hood, which can execute arbitrary code at load time, making them a code injection vector if the file is untrusted. Safetensors are a flat binary format with no executable code, safe to load from third-party sources. Operationally, safetensors also support zero-copy memory-mapped loading, which reduces peak RAM during model loading and speeds up cold start on large models.

Expect these follow-ups

• How would you audit a checkpoint from an external model provider before loading it in production?
• What is the safetensors validation step, and does it protect against all possible malicious payloads?

As asked

You operate a fleet of models (70B, 13B, and a fine-tuned 7B) and want to route incoming /v1/chat/completions requests to the cheapest model that can handle the task. Walk me through the routing architecture, the routing signals you would use, and the fallback strategy.

Sample answer outline

The router sits in front of all model instances and makes a fast routing decision based on signals like declared model name in the request, prompt complexity heuristics (token count, keyword classifiers), user tier (premium users always get 70B), and current queue depth per model. Fallback routes to the next tier up if the preferred model is saturated. A strong answer also covers how to handle streaming responses through the router and how to propagate errors cleanly without leaking internal model identities.

Expect these follow-ups

• How do you prevent the routing classifier itself from becoming a latency bottleneck?
• What changes to the router are needed if you add a third-party API model as a fallback of last resort?

As asked

Standard Kubernetes HPA based on CPU or memory does not work well for LLM pods. What metrics would you use as the scale-out signal, and how do you handle the slow startup time of large model pods without dropping traffic during scale-up?

Sample answer outline

The right scale-out signals are queue depth (pending requests) and KV cache utilization, both exposed via Prometheus and consumed by a custom HPA or KEDA scaler. To handle slow startup (model loading can take 2 to 5 minutes for a 70B model), use aggressive scale-out lead time: start new pods before the existing pods are saturated. Pre-warmed spare pod pools or node pool warm-up groups on cloud providers reduce effective startup time. Readiness probes should gate traffic until the model has completed its warm-up forward pass.

Expect these follow-ups

• How would you handle scale-to-zero for cost savings on off-peak hours?
• What is the interaction between Kubernetes pod anti-affinity and your GPU node pool during rapid scale-out?

As asked

From your experience running quantized models, at what point does INT4 quantization start producing noticeably worse outputs compared to INT8, and what is the per-layer or per-head diagnostic you would run to find the culprit?

Sample answer outline

INT4 degradation is most visible on tasks requiring precise factual recall, long-form reasoning, and code generation, typically measurable as a 2 to 5% perplexity increase on standard benchmarks. The diagnostic is per-layer weight quantization error: layers with high outlier activation magnitude degrade more because uniform quantization bins are too coarse for high-dynamic-range tensors. Tools like LM-Eval harness on task-specific benchmarks and per-layer sensitivity analysis (e.g. GPTQ's Hessian-based importance scores) pinpoint which layers to protect with mixed-precision or keep at INT8.

Expect these follow-ups

• What is SmoothQuant and how does it help with the outlier problem before quantizing?
• How would you design an A/B test to measure the business impact of a quality regression from INT8 to INT4?

As asked

A customer-facing LLM API you operate starts producing outputs that suggest users are injecting instructions to override the system prompt. What detection mechanisms would you put in place, and what mitigations would you implement at the infrastructure layer rather than the model layer?

Sample answer outline

Detection: input and output classifiers that flag known injection patterns, output monitors that check for system-prompt leakage, and anomaly detection on output token distribution drift. Infrastructure mitigations include strict input length caps on the user turn, structurally separating system prompt from user content using sentinel tokens or multi-turn conversation structuring supported by the model, output filtering pipelines, and rate-limiting users who trigger repeated classifier alerts. The model layer (fine-tuning for instruction hierarchy) is complementary but not sufficient alone.

Expect these follow-ups

• How would you handle a zero-day injection pattern that bypasses your classifier?
• What logging would you need to conduct a post-incident forensic analysis?

As asked

Explain how constrained decoding works to guarantee valid JSON output from a model that was not specifically trained for it. What are the limitations of this approach in a production serving system?

Sample answer outline

Constrained decoding maintains a finite-state machine or a LALR parser for the target schema and, at each decode step, masks the logits of all tokens that would violate the grammar. Only valid next tokens get non-zero probability. This guarantees syntactic validity but not semantic correctness (the model can emit a valid JSON object with wrong field values). Limitations in production: the FSM compilation from complex JSON Schema is not instant (cold-start latency for rare schemas), recursive or very deep schemas can make the state space explode, and the masking adds CPU overhead per decode step.

Expect these follow-ups

• How does Outlines or Guidance implement this, and what is the performance cost at 100 tokens/second?
• What would you do if a customer's required schema is too complex for the FSM and causes timeouts?

As asked

A model supports 128K context. What are the concrete operational problems that appear at that context length compared to 4K, and how do you handle them at the infrastructure level without simply refusing long requests?

Sample answer outline

The KV cache for a single 128K request is massive (GBs for a 70B model), which can evict many shorter concurrent requests. Prefill time scales quadratically with sequence length in standard attention, so a 128K prefill can stall the serving loop for seconds. Solutions include flash attention to reduce prefill memory, chunked prefill to interleave with decode, per-request KV cache memory limits with an explicit rejection policy, and dedicated long-context server pools so large requests do not starve normal traffic. Monitoring needs per-request KV allocation tracking to detect a single bad request crowding out the fleet.

Expect these follow-ups

• How does ring attention help with 128K inference across multiple GPUs?
• What changes to the autoscaler would you make if 1% of requests are 128K and 99% are under 2K?