LLM ops engineer interview questions

As asked

Design a RAG feature that answers customer support questions using internal documentation. How do you keep answers grounded and know when the model should refuse?

Sample answer outline

A good answer separates retrieval quality from generation quality. Chunk documents around semantic boundaries, attach metadata such as product, version, and freshness, and retrieve with hybrid search plus reranking for high precision. The generation prompt should require citations to retrieved passages and should refuse when evidence is missing or contradictory. Evaluation needs golden questions, adversarial no-answer cases, citation accuracy checks, and human review for high-impact categories. Candidates often focus on vector databases while ignoring stale docs, permissions, and the model's tendency to fill gaps.

Expect these follow-ups

• How do you prevent a user from retrieving documents they cannot access?
• What metric tells you retrieval is the bottleneck?
• How do you handle two documents that disagree?

As asked

Design an internal LLM gateway used by ten product teams. It must support provider fallback, prompt versioning, cost controls, and audit logging.

Sample answer outline

The gateway should be a policy and observability layer, not just an HTTP proxy. It needs typed request contracts, prompt and model version identifiers, tenant-level budgets, retries with idempotency, and provider fallback rules that are explicit about quality and compliance tradeoffs. Audit logs should capture inputs, outputs, metadata, and redaction state according to data policy. The answer should include latency budgets and failure modes, such as a fallback model producing different JSON shape or safety behaviour. Strong candidates avoid automatic fallback for high-risk flows unless the evals prove equivalence.

Expect these follow-ups

• How do you prevent fallback from masking a provider outage?
• What data do you store when prompts contain personal data?
• How do teams override defaults without bypassing governance?

As asked

Walk me through how vLLM's PagedAttention allocates and manages KV cache memory at runtime. How is it different from the contiguous pre-allocation that Hugging Face transformers uses, and what problems does it actually solve?

Sample answer outline

A strong answer explains that standard inference pre-allocates a contiguous KV cache block proportional to max sequence length, wasting memory for short sequences. PagedAttention divides the KV cache into fixed-size logical blocks mapped to non-contiguous physical pages, like virtual memory in an OS. This lets multiple sequences share the physical memory of prefilled prompt tokens (prefix sharing), dramatically increasing GPU utilization and throughput under mixed-length traffic.

Expect these follow-ups

• How does prefix caching build on top of PagedAttention?
• What happens to the block table when a request is preempted and later resumed?

As asked

Explain the difference between static batching and continuous batching in an LLM serving system. What is the core scheduling loop in a continuous batching engine, and where does it still fall short?

Sample answer outline

Static batching waits for a full batch to assemble before starting, leaving GPU idle while slow sequences hold a slot. Continuous batching inserts new requests into the batch as soon as any sequence emits an end-of-sequence token, keeping GPU utilization high. The scheduler runs at every decode step, checking for finished requests and admitting new ones. The main remaining gap is head-of-line blocking when a very long sequence occupies a slot that a short request could have finished in.

Expect these follow-ups

• How does speculative decoding interact with continuous batching scheduling?
• What metrics would you watch to detect that your scheduler is starving short requests?

As asked

Describe how you would design an evaluation pipeline to gate model updates before they go to production. What benchmarks, custom evals, and automation would you include, and how do you handle the fact that LLM quality is hard to measure automatically?

Sample answer outline

A strong answer includes standard benchmarks (MMLU, HellaSwag, task-specific evals) as fast-failing smoke tests, a curated golden-set of real user queries with reference answers graded by an LLM-as-judge, latency and throughput regression tests, and a small shadow traffic comparison against the current production model. The pipeline gates on statistically significant quality drop, not just raw score, to avoid blocking good models on noise.

Expect these follow-ups

• How do you prevent the judge model from having systematic biases that mask real regressions?
• What would you do if a new model scores better on benchmarks but users report worse answers?

As asked

List the specific metrics you would instrument and expose when operating a production vLLM or TGI deployment. For each, explain what alerting threshold you would set and why.

Sample answer outline

Core metrics include: GPU SM utilization (alert below 60% sustained for idle detection), GPU memory usage (alert above 90% to give head-room for eviction), KV cache usage percentage (alert above 80% which signals imminent eviction storms), request queue depth (alert above N x expected concurrency), time-to-first-token p99, and tokens-per-second per GPU. For SLO compliance, also track the fraction of requests exceeding latency targets.

Expect these follow-ups

• How would you instrument a vLLM deployment that runs behind an autoscaler?
• What metric would you use as the scale-out signal for your HPA in Kubernetes?