LLM ops engineer behavioral interview questions

As asked

Walk me through a production incident on an LLM serving system you operated. What were the symptoms, how did you diagnose it, and what did you change afterward to prevent recurrence?

Sample answer outline

A strong answer gives a specific incident (GPU OOM, latency spike from KV cache eviction storm, model producing garbage after a bad deployment), describes the exact diagnostic steps taken (which metrics surfaced the issue, how the root cause was isolated), and specifies concrete preventive changes (alerting threshold added, configuration change, deployment gate added). Vague answers about working well under pressure without technical specifics are weak.

Expect these follow-ups

• What monitoring would have caught this 30 minutes earlier?
• How did you communicate the impact to stakeholders during the incident?

As asked

Describe a time when you discovered a significant model quality regression close to a deadline. What data did you use to quantify the impact, how did you communicate it, and what was the outcome?

Sample answer outline

A strong answer shows that the regression was quantified with specific metrics (not just 'it felt worse'), describes honest communication of risk to stakeholders with data rather than opinion, and shows that the decision to delay or proceed was based on the magnitude of impact on real user tasks. A candidate who rolled back despite pressure and prevented a bad user experience is demonstrating good judgment.

Expect these follow-ups

• How did you decide which eval metrics were the right ones to present to a non-technical stakeholder?
• Would you have made the same call if the regression was 1% instead of 5%?

As asked

Tell me about a situation where multiple teams needed GPU capacity at the same time and you had to make allocation decisions that disappointed someone. How did you approach it?

Sample answer outline

A strong answer shows that the decision was based on transparent criteria (business impact, SLAs, time-sensitivity), that the disappointed team received a clear explanation and a timeline for when their needs would be met, and that the candidate worked proactively to increase total capacity rather than just arbitrating the current shortage. Answers that show passive avoidance or blame are weak.

Expect these follow-ups

• What process did you put in place after that to prevent the same conflict from recurring?
• How did you handle a team that went around you and provisioned their own GPU instances?

As asked

Has there been a situation where a business stakeholder or engineering team wanted to deploy an aggressively quantized model to cut costs, and you had concerns about quality? What did you do and what was the outcome?

Sample answer outline

A strong answer shows that the concern was backed by eval data on the specific tasks the model was used for (not generic perplexity), that the candidate proposed a middle ground (INT8 instead of INT4, mixed-precision, or a smaller higher-quality model), and that the decision was ultimately driven by evidence rather than intuition. Candidates who just complied without raising a data-backed concern are demonstrating weak professional judgment.

Expect these follow-ups

• What eval tasks did you run to quantify the quality risk?
• How would you design a shadow deployment to measure real-user impact before committing?

As asked

Describe your experience being on-call for an LLM or ML serving system. What was the most surprising failure mode you encountered, and what did you change in the system or runbook because of it?

Sample answer outline

A strong answer names a specific failure mode that was not anticipated in the original design (KV cache storm from long prompts, model producing different outputs after a dependency upgrade, GPU driver crash causing a silent degradation), describes the diagnostic process, and shows what was added to the runbook or monitoring to make the next occurrence faster to resolve. Candidates who have not been on-call for serving systems should be honest and describe adjacent experience.

Expect these follow-ups

• What alert would have woken you up 20 minutes earlier?
• How did you reduce on-call burden after that incident?

As asked

Tell me about a specific project where you reduced the per-request or per-token cost of an LLM serving system. What was the baseline, what did you change, and how did you measure the impact?

Sample answer outline

A strong answer names the specific interventions (quantization, batching tuning, prefix caching, smaller model routing), gives concrete before-and-after numbers (e.g., $4 per 1M tokens to $1.10), and shows that quality was measured and maintained. Answers that mention vague optimizations without numbers or that cannot explain the mechanism of the cost reduction are weak.

Expect these follow-ups

• How did you validate that the cost reduction did not affect user-facing quality?
• What was the next cost reduction opportunity you identified but did not have time to implement?

As asked

Have you had a disagreement with a model training or research team about whether a model was ready to deploy, or how it should be served? How did you resolve it?

Sample answer outline

A strong answer shows that the disagreement was resolved with shared data rather than authority, that the candidate was able to articulate the production risk in terms the research team cared about (not just operational metrics), and that the outcome was a collaborative decision with clear success criteria. Answers showing that the candidate just deferred to the research team without raising valid production concerns are weak.

Expect these follow-ups

• What did you learn about how to frame production concerns for a research audience?
• How did the relationship with that team change after the disagreement?

As asked

Describe a real situation where improving throughput or reducing GPU cost came at the expense of latency, and you had to make a decision about which to prioritize. What was the context and how did you make the call?

Sample answer outline

A strong answer shows that the decision was grounded in user research or product context (interactive chat cares about TTFT, bulk document processing does not), that the tradeoff was quantified (e.g., increasing max batch size from 8 to 32 increased throughput by 3x but raised p99 TTFT from 800ms to 2.1s), and that the candidate knew which metric the business actually cared about. Answers that optimized for the wrong objective without checking with product are weak.

Expect these follow-ups

• How did you communicate this tradeoff to a non-engineering stakeholder?
• What monitoring did you put in place to catch if the latency degraded further than expected?

As asked

Have you inherited or worked with an eval setup that was giving false confidence, where it showed no regression but real users were experiencing worse outputs? What did you change?

Sample answer outline

A strong answer identifies the specific gap in the eval (benchmark distribution mismatch, judge model bias, cherry-picked golden set), describes what signals from production revealed the real problem (user feedback, downstream metric drop, error classification), and shows what concrete additions to the eval pipeline prevented the same blind spot in future. Showing that they treated the eval pipeline itself as a product that needs maintenance is the key signal.

Expect these follow-ups

• How do you keep your golden set fresh as user behavior changes over time?
• What is your process for deciding when to add a new eval task versus updating an existing one?

As asked

Have you built a tool, script, or internal platform component for LLM ops that other teams started using without being asked to? What problem did it solve, and how did adoption happen?

Sample answer outline

A strong answer shows that the tool addressed a real pain point observed across multiple teams (not just the candidate's own workflow), that it was designed with usability in mind rather than just solving a one-off problem, and that adoption was driven by the tool being genuinely useful rather than mandated. Examples include eval harnesses, model benchmarking dashboards, deployment scripts, or prompt testing utilities.

Expect these follow-ups

• How did you decide what to document versus leave implicit in the tool's API?
• What would you have built differently if you had known how widely it would be used?

As asked

Describe how you learned a new GPU architecture (e.g., Hopper, Ada) or a new serving framework (e.g., vLLM, TGI) when your team needed to adopt it. What resources did you use and how long did it take to be productive?

Sample answer outline

A strong answer shows a structured learning approach (reading the paper or architecture guide, running small benchmarks to validate understanding, finding the production-relevant gotchas that documentation skips), an honest timeline, and what they specifically did to go beyond basic usage to operational fluency. Candidates who only cite documentation without practical validation are less convincing.

Expect these follow-ups

• What was the most unexpected thing you discovered that was not in the official documentation?
• How did you share what you learned with the rest of the team?