AI engineer behavioural interview questions

As asked

Tell me about a time you had to ship an LLM feature under significant time pressure. What corners did you cut, which ones did you not cut, and what did you do to manage risk?

Sample answer outline

Strong answers identify a concrete project with a real deadline. The candidate is specific about what they cut (e.g., a reranker, A/B testing, async logging) versus what they refused to cut (evals, input validation, a cost cap). They describe how they communicated the known gaps to stakeholders and what the post-launch cleanup plan looked like. Weak answers are vague or claim they cut nothing.

Expect these follow-ups

• Did any of the compromises you made come back to bite you?
• What would you do differently if you had two extra weeks?

As asked

Tell me about a time you had to manage a non-technical stakeholder's expectations about what an LLM feature could and could not do. What made it difficult and what did you actually say?

Sample answer outline

A good answer describes a concrete gap between stakeholder expectation (e.g., 100% accuracy, zero hallucination) and reality, the candidate's approach to communicating in terms the stakeholder cared about (user impact, error rate as a user experience metric), and how they reached a shared definition of success. Weak answers are vague or describe only internal team conflicts.

Expect these follow-ups

• How did you handle it when the stakeholder pushed back on your assessment?
• What metrics did you agree on to track progress?

As asked

Tell me about a time an evaluation you built caught a problem before it reached users. What was the problem, how did the eval surface it, and what would have happened if you had not caught it?

Sample answer outline

The candidate describes a concrete failure scenario (model update changed behavior, prompt change introduced hallucination, reranker scored irrelevant docs highly) and a specific metric or test case that fired. They quantify the impact in terms the business cares about and explain the fix. The best answers also describe what they added to the eval suite as a result.

Expect these follow-ups

• How did you balance the cost of running that eval suite versus its coverage?
• Was there anything the eval missed that you later found in production?

As asked

Describe a time the LLM in a system you owned started returning factually wrong or hallucinated outputs in production. How did you identify the root cause and fix it?

Sample answer outline

Strong answers walk through their debugging process: reproducing the failing cases, tracing the retrieval to identify whether the bad context was missing or the model ignored good context, checking whether a model or prompt change preceded the issue, and confirming the fix with an automated metric. They are specific about the root cause and honest if it took several hypotheses to get there.

Expect these follow-ups

• What did you add to your monitoring so you would catch this faster next time?
• How did you communicate the issue to users or stakeholders while you were debugging?

As asked

Tell me about a time you argued against using an LLM for a task where a simpler approach was better. How did you make the case and what happened?

Sample answer outline

The candidate describes the original proposal, explains the specific reasons an LLM was the wrong tool (latency, cost, determinism requirements, simple rule-based problem), proposes the alternative, and describes the outcome. Strong answers show intellectual honesty about when LLMs are not the answer and the ability to influence without authority.

Expect these follow-ups

• Was there a case where you were initially wrong about whether an LLM was needed?
• How do you frame that argument without sounding dismissive of AI?

As asked

Tell me about a time you significantly reduced the cost of an LLM system. What were the before and after numbers, what changed, and what quality trade-offs did you manage?

Sample answer outline

The candidate gives concrete numbers (e.g., $40k/month to $8k/month) and a specific set of changes (model downgrade for simpler calls, prompt caching, output length cap, request batching). They describe how they measured the quality impact and what regression they accepted. The best answers describe a prioritized sequence of changes rather than doing everything at once.

Expect these follow-ups

• How did you decide which optimization to do first?
• Did you have a case where a cost reduction broke something unexpected?

As asked

Walk me through the most complex prompt engineering problem you have solved. What was the task, what did you try, what failed, and what finally worked?

Sample answer outline

Strong answers describe a specific task with measurable success criteria, walk through at least two or three failed approaches with reasons why they failed, and describe the final working prompt with an explanation of which specific change made the difference. They mention how they tracked improvement (eval metric, human rating). Vague answers that say 'I kept iterating until it worked' are weak.

Expect these follow-ups

• How did you avoid overfitting the prompt to your eval set?
• Would you approach the same problem differently now?

As asked

Tell me about a time you had to demonstrate the business impact of an LLM feature you built. What metrics did you choose, how did you collect the data, and what did you find?

Sample answer outline

Strong answers identify a north-star business metric (task completion rate, time saved, revenue attributed, support ticket deflection) and separate it from proxy LLM metrics like faithfulness. The candidate describes their experiment setup (A/B test or pre/post comparison), sample sizes, and what they found. They are honest if the impact was smaller than expected and describe how they communicated that.

Expect these follow-ups

• How did you isolate the impact of the LLM feature from other changes happening at the same time?
• What would you do differently in the measurement if you could run the experiment again?

As asked

Describe a time you deployed an open-source LLM rather than using a hosted API. What drove that decision, what infrastructure challenges did you face, and would you make the same call again?

Sample answer outline

The candidate gives the business context (data privacy, cost at scale, latency requirements), describes the infrastructure choices (vLLM/TGI on GPU instances, quantization for memory), and the operational challenges they hit (GPU memory fragmentation, KV cache tuning, autoscaling cold start). They give an honest retrospective on whether self-hosting was worth it versus using a hosted provider.

Expect these follow-ups

• What was the performance gap between the open-source model and GPT-4 for your task?
• How did you handle GPU availability during a demand spike?

As asked

Tell me about a time you prepared a dataset for fine-tuning and found the data was lower quality than expected. What did you do about it and how did it affect the project?

Sample answer outline

Strong answers describe the specific quality problems found (label noise, duplicates, distribution mismatch, inconsistent formatting), the cleaning steps taken (deduplication, LLM-assisted relabeling, stratified sampling), the impact on timeline, and the final model quality. The candidate also reflects on what data quality checks they would add earlier in the pipeline next time.

Expect these follow-ups

• At what point did you decide the data was good enough rather than continuing to clean?
• How did poor data quality in training show up in the fine-tuned model's behavior?

As asked

Have you ever faced a security incident specific to an LLM system, such as a prompt injection or data leak? Walk me through what happened and how you responded.

Sample answer outline

Strong answers describe a specific incident with a root cause (injected instructions in a retrieved document, overly permissive tool access, leaked system prompt via adversarial query), the detection method, the immediate mitigation, and the systematic fix. The candidate explains how they communicated the incident internally and what they added to prevent recurrence. Candidates without direct experience should describe how they would respond to a hypothetical scenario based on a red-team finding.

Expect these follow-ups

• How did you balance the urgency of mitigation with the need to understand the full scope before patching?
• What changes did you make to your threat model as a result?