Agent SDK engineer behavioural interview questions

As asked

Tell me about a specific incident where an agent you built failed in production in a way you did not anticipate. What happened, how did you diagnose it, and what did you change to prevent it from recurring?

Sample answer outline

Strong answers give a concrete incident with clear before/after metrics, show structured debugging (pulled traces, identified the exact failure mode), demonstrate systems thinking in the fix (not just patching the symptom), and describe a process change like adding a new eval case, a monitor, or a circuit breaker. Candidates who cannot name a real failure are a red flag.

Expect these follow-ups

• What monitoring would have caught this before users reported it?
• Did you build any automated tests to cover this failure mode after the fact?

As asked

Describe a situation where you disagreed with your team's choice of model or agent framework. How did you make your case, and how did it resolve?

Sample answer outline

Strong answers show the candidate built evidence (ran benchmarks, estimated costs, prototyped alternatives) rather than arguing from opinion, communicated tradeoffs clearly to non-technical stakeholders, respected the final decision while documenting their position, and revisited the decision when new data emerged. Candidates should avoid sounding like they always capitulate or always fight to the end.

Expect these follow-ups

• If the team chose the option you disagreed with and it later proved to be a problem, what did you do?
• How do you decide when a technical disagreement is worth escalating versus accepting the team's decision?

As asked

Walk me through a time you identified that an agent was costing far more than expected and what you did to bring the cost down without degrading quality.

Sample answer outline

Strong answers identify a concrete cost reduction (percentage or dollar figure), describe the diagnosis (which model, which tool, which prompt was the culprit), the intervention (smaller model for simple steps, prompt caching, fewer tool calls, caching deterministic results), and the measurement of quality after the change. Candidates should show they balanced cost against quality rather than blindly cutting.

Expect these follow-ups

• How did you measure quality before and after the optimization to make sure you had not regressed?
• What would you have done if the only way to hit the cost target was to accept a quality degradation?

As asked

Your agent depends on a tool API owned by another team that keeps changing its contract without notice. How have you handled a situation like this in the past?

Sample answer outline

Strong answers describe establishing a contract (JSON Schema, OpenAPI spec) with the owning team, adding integration tests that run against their staging environment, implementing an adapter layer to isolate your agent from their API shape, setting up a monitoring alert for schema drift, and negotiating a deprecation notice period. Candidates should show they collaborated rather than complained.

Expect these follow-ups

• What do you do when the other team refuses to maintain a stable contract?
• How do you test your adapter layer for a tool API that is difficult to mock?

As asked

Tell me about a time you had to design an evaluation strategy for an agent from scratch with no prior golden dataset. How did you bootstrap it?

Sample answer outline

Strong answers describe generating seed cases from product requirements and user stories, using adversarial red-teaming to find edge cases, running a small user study to collect real inputs, labeling outputs manually for the first N cases, and setting up an LLM-as-judge for ongoing scoring. Candidates should show they iterated on the eval design rather than trying to perfect it upfront.

Expect these follow-ups

• How did you decide when the eval suite was good enough to start making decisions from?
• How did you handle cases where human labelers disagreed on whether an output was correct?

As asked

You are on call and get paged at 2am because an agent is making hundreds of tool calls per minute and burning through the monthly API budget in hours. Walk me through exactly what you do.

Sample answer outline

Strong answers describe immediate triage (kill switch or rate limit clamp to stop the bleeding), pulling traces to identify which agent or task is looping, identifying the root cause (missing termination, bad tool response), a temporary fix (deploy a step limit or hotfix), communication to stakeholders, and a post-incident review. Candidates should show calm prioritization: stop damage first, diagnose second.

Expect these follow-ups

• How do you implement a kill switch that can be triggered without a deployment?
• What post-incident changes would prevent this class of runaway loop in future?

As asked

Have you ever had to convince a product manager or engineering leadership that investing in agent evaluations was worth the time? How did you make the case?

Sample answer outline

Strong answers quantify the risk of shipping without evals (a concrete past regression or a hypothetical cost), frame evals as an enabler of shipping speed rather than a slowdown, propose a minimal viable eval suite rather than a perfect one, and show a prototype that proves the value in a short time. Candidates should demonstrate business communication skills, not just technical arguments.

Expect these follow-ups

• How did you measure the return on investment of the eval suite once it was built?
• What would you do if leadership agreed in principle but kept deprioritizing the eval work?

As asked

Tell me about a time a prompt change that seemed minor caused a quality regression that was only noticed days later. What did you learn from that?

Sample answer outline

Strong answers describe the specific change (a wording tweak, an instruction removal, a persona shift), how long it took to detect (user complaints, a metric drop), what the regression looked like, and the process change that came out of it (CI eval gate, shadow testing, canary rollout with automatic rollback). Candidates should show they now treat prompts with the same discipline as code.

Expect these follow-ups

• How do you track the history of prompt changes so you can audit what changed and when?
• How do you run a canary rollout for a prompt change across a diverse user population?

As asked

A user configures their agent to send emails on their behalf automatically. Three weeks later they claim the agent sent a customer an email with incorrect pricing. How do you approach this situation?

Sample answer outline

Strong answers address both the immediate user problem (retrieve the email record from the audit log, identify the tool call and the LLM output that generated the content) and the systemic question (should this class of action require confirmation by default, how do you communicate that risk to users in the product). Candidates should show empathy for the user while being clear about what the system was configured to do.

Expect these follow-ups

• Would you make send-email an action that requires confirmation by default? Why or why not?
• How do you log the content of outgoing emails in a way that is useful for debugging but respects user privacy?

As asked

You notice in your traces that for a certain class of user request, the agent almost always calls tool A when it should call tool B. The task descriptions are different but the agent does not see the distinction. What do you do?

Sample answer outline

Strong answers describe pulling a sample of the bad traces, analyzing what the tools' descriptions say versus what the user is asking, rewriting the tool descriptions to make the distinction explicit, adding a clarifying example in the description, running an eval to confirm the fix works, and adding a test case to the golden set to prevent recurrence. Candidates should not jump straight to prompt changes without diagnosis.

Expect these follow-ups

• What if rewriting the tool description fixes one class of request but breaks another?
• At what point would you consider splitting the confused tool into two more specific tools?

As asked

An agent is halfway through a complex multi-step task when it starts hitting context length errors. What do you do in the moment, and how do you redesign to prevent this?

Sample answer outline

Strong answers triage based on whether the current run can be salvaged (summarize history, trim old tool results, continue) versus needs to restart from a checkpoint, then redesign to add context monitoring before each step with automatic summarization when utilization exceeds a threshold. Candidates should discuss what information is safe to compress versus what must be preserved verbatim.

Expect these follow-ups

• How do you write a summary that loses as little task-relevant information as possible?
• How do you test that an agent with context compression behaves identically to one with full context?

As asked

The provider releases a new model version and you upgrade. Within an hour you get alerts that 20% of tool calls are failing to parse. What do you do?

Sample answer outline

Strong answers immediately roll back to the previous model version to restore service, then diagnose by comparing raw API responses between versions to find the format change (argument quoting, nested object format, new finish reason), fix the parser or update to the new format, add a test that would catch this in future, and implement a shadow mode that runs the new model in parallel before promoting it.

Expect these follow-ups

• How do you roll back to the previous model without a code deployment?
• How do you set up shadow testing so a new model version can be validated before it handles live traffic?

As asked

After a model provider updates their model, you notice agents are no longer following key instructions in the system prompt, such as always responding in a specific format or staying within a defined scope. What do you do?

Sample answer outline

Strong answers run controlled comparisons between old and new model versions on a fixed set of prompts, identify which instructions are being ignored, try reinforcing them with stronger wording or repetition in the user turn, consider falling back to the previous model version if the degradation is severe, and open a support ticket with the provider with reproducible examples. Candidates should not assume the model is at fault without evidence.

Expect these follow-ups

• How do you distinguish a model behavior change from a prompt-sensitivity issue caused by a recent prompt edit?
• At what point do you consider switching providers permanently versus waiting for a fix?

As asked

A customer reports that if they phrase their request a certain way, your agent ignores its intended scope and starts doing arbitrary web searches unrelated to the product. How do you handle this?

Sample answer outline

Strong answers reproduce the issue with the exact phrasing, identify whether this is prompt injection from user input or a model generalization failure, add an input validation layer or a scope guardrail prompt that checks whether the request falls within the intended domain, test the guardrail against known adversarial inputs, and consider restricting tool access to only what the agent legitimately needs. Candidates should also consider whether this is a safety concern that needs reporting.

Expect these follow-ups

• How do you balance making the guardrail strict enough to block abuse versus flexible enough to serve legitimate requests?
• How do you build a regression suite specifically for adversarial inputs?

As asked

Have you ever shipped an agent or tool-use feature that later turned out to have a security vulnerability? What was it, how was it discovered, and what did you do?

Sample answer outline

Strong answers describe the vulnerability concretely (prompt injection, over-privileged tool, unsanitized output), whether it was found internally or by a user, the immediate mitigation (disabling the feature or adding a guard), the root cause analysis, and the systemic change (threat modeling in the design phase, security review step in the release process). Candidates who have never thought about security in agent systems are a concern.

Expect these follow-ups

• What is your standard process now for thinking through security before shipping an agent feature?
• How did you communicate the issue to affected users or stakeholders?

As asked

Describe a time you migrated a production agent or tool-use system from one LLM provider to another. What were the hardest technical challenges and how did you validate the migration was successful?

Sample answer outline

Strong answers describe running both providers in shadow mode, comparing outputs on the same inputs, handling API format differences (message schema, tool call format, streaming events), discovering capability gaps (context length, tool count limits), and using eval scores as the migration gate. Candidates should mention rollback planning and gradual traffic shifting.

Expect these follow-ups

• How did you handle prompts that were tuned for the old provider and did not behave the same on the new one?
• What would have caused you to abort the migration midway?

As asked

While reviewing traces you discover that a search tool is occasionally returning documents that contain other users' PII, which is then injected into the LLM context. What do you do?

Sample answer outline

Strong answers treat this as a security incident: immediately disable or sandbox the search tool, notify the security team and DPO, preserve the affected traces for investigation, scrub PII from logs, add a PII detection filter at the tool output boundary as a temporary control, and fix the root cause (missing access control in the search index). Candidates should not treat this as just a bug fix.

Expect these follow-ups

• How do you implement a PII filter that is fast enough to run inline on every tool result without adding significant latency?
• How do you notify affected users whose PII may have been exposed to another user's agent session?

As asked

A third-party API your agents rely on as a tool changes its pricing and your monthly bill goes from $200 to $8000 overnight. You had no warning. What do you do now, and what process change do you make?

Sample answer outline

Strong answers cover immediate steps (rate-limit or gate the tool calls to control spend, check if a cached or cheaper alternative exists), communication (engineering and finance stakeholders), and process changes (set up cost alerts on third-party API spend, add estimated cost per tool call to the tool's metadata, evaluate alternatives). Candidates should show they take ownership rather than waiting for someone else to act.

Expect these follow-ups

• How do you build tool cost into the agent's decision-making so it naturally prefers cheaper tools for simple tasks?
• How do you evaluate a cheaper alternative tool without degrading agent quality?