Prompt engineer behavioral interview questions

As asked

Tell me about a time you inherited or built a RAG pipeline where the answer quality was unacceptably low. What did you diagnose, what did you change, and how did you measure improvement?

Sample answer outline

Strong answers isolate which component failed (retrieval, context assembly, or generation), describe concrete measurement before and after changes, mention a specific metric that drove the decision, and explain what they learned that changed how they design future pipelines.

Expect these follow-ups

• What would you do differently if you were starting from scratch?

As asked

Tell me about a time you received an ambiguous or conflicting requirement for a prompt you were building. How did you resolve the ambiguity and what did you do when you couldn't get a clear answer?

Sample answer outline

Strong answers show proactive clarification rather than guessing, use of concrete examples to expose the ambiguity ('here are three inputs and three different outputs I could produce, which do you want?'), and a documented decision when the stakeholder was unavailable.

Expect these follow-ups

• How did you document the decision so future team members could understand it?

As asked

Tell me about a time you were pressured to ship a prompt or AI feature that you felt wasn't ready. What did you do?

Sample answer outline

Look for: data-driven communication of the specific risks (this prompt fails 12% of the time on edge case X), a proposed alternative path (ship with scope reduction, add a human fallback), and ultimate outcome. Red flag: candidate always caved without pushing back or always refused to compromise.

Expect these follow-ups

• Looking back, do you think you made the right call?

As asked

Describe a situation where you had to build an evaluation suite for a task with no existing benchmark. How did you design it, who did you involve, and how did you validate that the eval was trustworthy?

Sample answer outline

Look for: clear articulation of what the eval was measuring and why those metrics, the annotation or data collection process, inter-annotator agreement or some other quality check, and a story about how the eval caught a real regression or improvement that mattered.

Expect these follow-ups

• How long did the first version of the eval take to build, and what would you have done faster?

As asked

Walk me through a time you found and fixed a specific jailbreak vulnerability in a deployed system. What was the attack, how did you discover it, what did you change, and how do you know it's fixed?

Sample answer outline

A strong answer names the specific attack pattern (not just 'jailbreak'), describes the discovery method (user report, automated red team, internal audit), distinguishes between prompt-based and model-level fixes, and explains the regression test added to prevent recurrence.

Expect these follow-ups

• Did your fix introduce any false positives that blocked legitimate users?

As asked

Tell me about a time you had to convince a non-technical stakeholder to delay a launch based on eval results. How did you communicate the risk, and what happened?

Sample answer outline

Look for: ability to translate technical metrics (F1, hallucination rate, refusal rate) into business risk (customer complaints, liability, brand damage), a clear threshold that justified the delay, and the outcome. Bonus: candidate adapted the communication to the specific stakeholder's concerns.

Expect these follow-ups

• What would you have done if the stakeholder overruled you and launched anyway?

As asked

Describe a specific classification problem where prompt engineering with few-shot examples meaningfully improved accuracy. What made the example selection non-obvious, and how did you figure it out?

Sample answer outline

Look for: understanding that boundary cases are more informative than easy positives/negatives, iterative testing of example sets, and a metric that showed the improvement. Bonus: candidate describes deliberate handling of class imbalance in the example set.

Expect these follow-ups

• How many iterations did it take to land on the final set of examples?

As asked

Tell me about a time a prompt chain produced a completely wrong final output and you had to trace the failure back to its source. What did you find and how did you fix it?

Sample answer outline

Strong answers describe a systematic isolation process (testing each step independently), identify the specific failure mode (schema mismatch, wrong extraction, model confusion between steps), and implement a fix at the correct layer rather than patching downstream. They also describe the monitoring added to catch future regressions.

Expect these follow-ups

• Was the failure deterministic or flaky? How did that affect your debugging process?

As asked

Describe a project where you had to significantly reduce the token cost of a production prompt. What levers did you pull, what did you measure, and what was the outcome?

Sample answer outline

Look for: a structured approach (measure cost first, identify the biggest token consumers, test reductions against an eval), specific techniques used (caching, compression, smaller model routing, example pruning), and honest reporting of any quality tradeoffs accepted.

Expect these follow-ups

• Did you have to accept any quality degradation? How did you communicate that tradeoff?

As asked

Walk me through a time when an LLM was supposed to return structured output but kept producing malformed responses at an unacceptable rate. How did you bring the failure rate down?

Sample answer outline

Strong answers identify the root cause (ambiguous schema description, model not capable enough, missing example, no constrained decoding available), describe the specific technique used to fix it, and quantify the improvement with a before/after failure rate.

Expect these follow-ups

• What was your fallback strategy when the model still failed after your fix?

As asked

Tell me about a time you had to migrate a set of production prompts to a new model version. How did you plan the migration, test it, and handle regressions?

Sample answer outline

Key elements: running the full eval suite on the new model before any changes, identifying which prompts regressed and why, deciding whether to fix prompts or roll back, shadow-mode comparison before full cutover, and stakeholder communication about the timeline.

Expect these follow-ups

• Were there any prompts that improved without changes on the new model? What does that tell you?

As asked

Describe a time you disagreed with a colleague about how to design a prompt or evaluation. How did you handle the disagreement and what was the outcome?

Sample answer outline

Look for: use of data or experiments to resolve the disagreement rather than opinion-based debate, willingness to be proven wrong, and constructive handling of the conflict. Bonus: candidate proposed a small experiment to settle the question empirically.

Expect these follow-ups

• Were you right? If not, what did you take away from it?

As asked

Describe a time a prompt change you made caused an unexpected problem in production. How did you respond, and what process changes did you make afterward?

Sample answer outline

Key elements: honest account of what went wrong (not blaming the model), rapid containment (rollback or hotfix), a post-mortem that identifies the root cause, and specific process changes (regression test added, review checklist updated) that prevent recurrence.

Expect these follow-ups

• How long did it take to detect the problem after the change was deployed?

As asked

Tell me about a project where you had to work closely with both an ML engineer (who controlled the model) and a product manager (who defined the task). How did you align all three perspectives?

Sample answer outline

Strong answers show: translating between technical constraints (what the model can and can't do) and product goals (what users need), creating shared artifacts (an eval set that both sides agreed on), and unblocking yourself when the ML engineer was busy by designing experiments you could run independently.

Expect these follow-ups

• What was the biggest misalignment between the ML engineer's mental model and the product manager's expectations?

As asked

Describe a situation where you had to design a prompt for a use case that had no existing examples or prior work you could reference. How did you approach it?

Sample answer outline

Look for: first-principles decomposition of the task, rapid prototyping with a small eval set, learning from early failures, and a structured iteration process rather than random trial and error.

Expect these follow-ups

• What was the first thing you tried and why did you try that first?

As asked

Tell me about a time you had to dramatically improve a prompt's performance in a very short window. What was your strategy for iterating quickly without losing quality?

Sample answer outline

Strong answers describe: quickly defining a minimal but representative eval set, prioritizing changes with the highest expected impact, running parallel experiments, and knowing when to stop iterating. Red flag: candidate describes hours of manual spot-checking without any systematic eval.

Expect these follow-ups

• What did you sacrifice to hit the deadline, and are you comfortable with that tradeoff in hindsight?

As asked

Describe a time when annotators on your eval dataset disagreed significantly. How did you diagnose the disagreement and what did you do about it?

Sample answer outline

Strong answers distinguish between random noise (annotators weren't paying attention) and genuine ambiguity (the task is underdefined), describe updating the annotation guide, running a calibration session, and possibly re-annotating or treating highly contested examples as ambiguous rather than assigning a single label.

Expect these follow-ups

• Did the disagreements reveal a fundamental ambiguity in the task definition?

As asked

Tell me about a time an LLM agent you built took an action you didn't anticipate. How did you find out, what did the agent do, and how did you redesign the guardrails?

Sample answer outline

Look for: clear description of what the agent was supposed to do versus what it did, how the failure was discovered (monitoring, user report, or audit), the specific guardrail that failed or was missing, and the redesign. Bonus: candidate describes how they improved monitoring so similar issues surface faster in the future.

Expect these follow-ups

• Would you have allowed the agent to take that action with human approval?

As asked

Tell me about a time you took over responsibility for a prompt system that had no documentation, no eval suite, and no clear owner. What did you do first?

Sample answer outline

Strong candidates describe: reverse-engineering intent by reading the prompt and user feedback, building a minimal eval set as the first priority, documenting what they found, and identifying the highest-risk gaps before making any changes.

Expect these follow-ups

• How long did it take before you felt confident enough to make your first change?

As asked

Have you ever been asked to build a prompt for a use case you had concerns about? What were the concerns and what did you do?

Sample answer outline

Look for: ability to articulate the specific concern clearly (not just discomfort), willingness to raise it through appropriate channels, and a constructive path forward (modified scope, additional safeguards, escalation). Red flag: candidate either never pushes back or refuses all ambiguous requests without nuance.

Expect these follow-ups

• What would you have done if your concern was dismissed by leadership?

As asked

Describe a time you mentored or helped a colleague who was new to prompt engineering. What were the most common mistakes they made, and how did you help them fix those habits?

Sample answer outline

Look for: specific anti-patterns identified (vague instructions, no few-shot examples, no eval), teaching approach that was practical rather than theoretical, and follow-through to confirm the colleague improved.

Expect these follow-ups

• What's the single most important habit you try to instill in new prompt engineers?