AI infrastructure engineer behavioral interview questions

As asked

Tell me about a time when a long-running training job failed unexpectedly at a critical point, such as near the end of a multi-day run. What did you do, how did you communicate with the team, and what did you change to prevent it from happening again?

Sample answer outline

Strong answers cover: immediate diagnosis steps (check logs, identify the last good checkpoint, determine whether restart is safe), stakeholder communication (who was notified and in what form, how urgency was conveyed), and a post-incident review that produced concrete changes like more frequent checkpointing, adding monitoring alerts for early warning signals, and documenting the failure mode. Avoid generic answers; look for specific technical details about what the failure was.

Expect these follow-ups

• What monitoring or alerting would have caught this failure earlier, and did you implement it afterwards?

As asked

Give me an example of a time when training throughput dropped significantly between two runs that appeared identical. Walk me through how you found the root cause and how long it took.

Sample answer outline

Strong answers demonstrate a systematic diagnostic approach: first rule out configuration changes (batch size, seq len, num workers), then profile with Nsight Systems or PyTorch Profiler to identify where time went, check NCCL logs for communication degradation, and check hardware health (ECC errors, thermal throttle). The candidate should name the specific tools used and the actual root cause found, not a hypothetical one.

Expect these follow-ups

• What did you add to your monitoring or CI system after this incident to catch similar regressions faster?

As asked

Describe a significant infrastructure project you owned from design through production. How did you scope it, handle unexpected technical challenges during implementation, and measure whether it was successful after launch?

Sample answer outline

Look for a clear narrative arc: problem definition with data, design choices and tradeoffs considered, a specific technical challenge that required pivoting, and quantitative success metrics (latency improvement, cost reduction, reliability). Candidates who cannot quantify the impact of their work or who describe a project they contributed to rather than owned should be probed further.

Expect these follow-ups

• What would you do differently if you were starting this project over today?

As asked

Describe a time when a research team wanted to run experiments in a way that conflicted with your infrastructure constraints or reliability requirements. How did you handle the disagreement and what was the outcome?

Sample answer outline

Look for candidates who approached the conflict with data (showing the researcher the cost or risk), proposed alternatives that preserved the research goal while respecting constraints, and maintained a positive working relationship. Avoid candidates who either caved immediately without explanation or blocked the researcher without exploring options. The best answers describe a specific compromise and what was learned.

Expect these follow-ups

• Looking back, do you think the constraint you enforced was the right call, or would you handle it differently now?

As asked

Tell me about a time you were on-call and faced a cascading or multi-system failure in a GPU cluster or serving infrastructure. What was the sequence of events, how did you triage, and what did you do to stabilize the system?

Sample answer outline

Strong answers describe clear triage priority (stop the bleeding first, understand root cause second), describe how the candidate kept stakeholders informed during the incident, and show they knew when to escalate vs. handle alone. Look for specific technical details about what failed, what the symptoms were, and how the candidate distinguished correlation from causation in a complex distributed system.

Expect these follow-ups

• What runbook or automation did you create after this incident so the next on-call would handle it faster?

As asked

Tell me about a time your code change caused a failure in a training run or serving system. How did you identify that your change was the cause, how did you fix it, and how did you communicate with the team?

Sample answer outline

Look for honesty and ownership: the candidate should describe the bug clearly, explain how they confirmed causality (bisect, revert, logs), and describe any impact on the team. Candidates who blame tooling, downstream systems, or minimize the incident are flags. The best answers describe a specific systemic fix (added a test, improved CI, added a rollback gate) that prevented recurrence.

Expect these follow-ups

• What test would have caught this bug before it reached production?

As asked

Give me an example of a time you had to choose between shipping a new capability the team was asking for and investing in reliability or observability improvements. How did you make the call, and how did it turn out?

Sample answer outline

Strong answers show the candidate can quantify the risk of deferring reliability work (how often does this failure mode occur, what is the blast radius) and articulate the opportunity cost of not shipping the feature. Candidates who always default to reliability or always default to features without situational reasoning are weaker. Look for data-driven decision making and alignment with the team after the decision.

Expect these follow-ups

• How did you get buy-in from the team or management for the path you chose?

As asked

Describe a time you helped a junior engineer debug a complex distributed training or inference issue. How did you approach teaching them the debugging process rather than just fixing the problem yourself?

Sample answer outline

Look for candidates who describe a deliberate teaching approach: asking guiding questions rather than giving answers, explaining the mental model behind each diagnostic step, and setting up the junior engineer to recognize similar issues independently in the future. Strong answers describe a specific issue and the actual debugging steps, not a generic mentoring philosophy.

Expect these follow-ups

• After the debugging session, what did you suggest they add to their toolkit or daily workflow?

As asked

Give me an example of a piece of infrastructure technical debt you identified and championed paying down. How did you justify the investment, and what resistance did you encounter?

Sample answer outline

The best answers quantify the cost of the debt (incidents per quarter, engineering hours spent working around it, reliability impact) and compare it against the cost of the fix. Candidates should show they can make a business case to non-technical stakeholders, not just to peers. Resistance could come from timeline pressure or not understanding the risk; strong candidates describe how they addressed each objection specifically.

Expect these follow-ups

• After completing the fix, how did you validate that it actually addressed the problem you identified?

As asked

You have one week of cluster time and ten plausible research ideas. How do you choose what to run?

Sample answer outline

A strong answer ranks experiments by expected information value, cost, dependency structure, and reversibility. Start with cheap sanity checks that can kill weak ideas quickly before committing to full-scale runs. Preserve at least one control and one replication, because a week of compute spent on untrustworthy results is worse than fewer experiments. Good candidates explain how they align with the research lead while still pushing back on vague or unfalsifiable proposals. The trap is treating compute allocation as a queue rather than a portfolio of bets.

Expect these follow-ups

• What would you refuse to run?
• How do you handle a senior researcher pushing for the most expensive idea?
• What does your experiment tracker need to show at the end of the week?

As asked

Tell me about a model release that failed or nearly failed in production. What happened, and what changed afterwards?

Sample answer outline

Strong answers include the model's intended change, the production symptom, detection path, mitigation, and durable prevention work. The interviewer wants evidence of ownership across ML and engineering boundaries, not blame on the data scientist or platform team. Good candidates mention rollback readiness, shadow traffic, feature compatibility, and how they communicated uncertainty during the incident. The lesson should be specific, such as adding a canary gate, replay test, or input schema contract. A vague story about metrics going down without operational detail is not enough for an MLOps role.

Expect these follow-ups

• How fast could you roll back?
• Which pre-production check would have caught it?
• How did you decide whether to retrain or revert?

As asked

A cloud region loses a large fraction of your GPU capacity during peak traffic. What do you do in the first hour?

Sample answer outline

A strong answer starts with customer-impact triage: preserve critical inference paths, shed or degrade lower-priority workloads, and communicate current impact. Operational steps include shifting traffic across regions, enabling smaller fallback models, reducing max output tokens, disabling non-essential batch jobs, and checking whether queues are draining. The candidate should discuss pre-agreed degradation modes because inventing them during an incident is risky. Good answers include post-incident capacity planning, provider diversification, and game days. The common trip-up is focusing on cluster mechanics while ignoring product-level priority decisions.

Expect these follow-ups

• Which workloads do you shed first?
• What degradation mode would customers notice least?
• How much spare GPU capacity is worth paying for?

As asked

Tell me about a time you needed to escalate a critical infrastructure issue to a hardware vendor (NVIDIA, Mellanox, etc.) or cloud provider and get it resolved quickly. How did you prepare for the escalation and what did you do when the initial response was inadequate?

Sample answer outline

Strong answers describe gathering the right data before escalating (reproduction steps, logs, firmware versions, syscall traces), framing the business impact clearly, and escalating through the right channel (TAM, P1 ticket, named account support). When initial response is slow, candidates should describe persistence with escalation paths, not just waiting. Look for candidates who understand vendor support processes and know how to make their case compelling.

Expect these follow-ups

• After the vendor resolved the issue, what did you do to ensure you would catch this class of problem earlier in the future?

As asked

Describe a situation where two training runs with the same hyperparameters produced significantly different results, and you were responsible for investigating why. What did you find?

Sample answer outline

Strong answers probe the usual suspects systematically: RNG seeds not fixed across all libraries, non-deterministic CUDA operations (cudnn benchmark mode, async allreduce), different data order due to shuffling, different hardware causing different floating point rounding, or a subtle code path that changed. The candidate should describe a concrete root cause and the fix, not a list of hypotheticals.

Expect these follow-ups

• What do you now include in every experiment launch to prevent this from happening again?