Agent SDK engineer domain knowledge interview questions

As asked

Explain how function calling works in LLM APIs at the protocol level: what you send in the request, what the model returns, and how you execute the function and close the loop.

Sample answer outline

Strong answers describe sending a tools array with each tool's name, description, and parameter schema, the model returning a tool_use content block with the chosen name and arguments JSON, the caller executing the function locally and returning a tool_result message, and the model then generating its final text response. Candidates should know the difference between Anthropic's and OpenAI's naming conventions for these fields.

Expect these follow-ups

• What does the model's output look like when it wants to call two tools at once?
• How does the model signal that it has finished using tools and wants to give its final answer?

As asked

Explain what RAG is, how it differs from fine-tuning, when you would choose one over the other, and what the main failure modes of a naive RAG implementation are.

Sample answer outline

Strong answers describe RAG as fetching relevant documents at inference time and injecting them into the prompt, versus fine-tuning which bakes knowledge into weights, and argue RAG is better for frequently updated or private data while fine-tuning is better for style or format adaptation. Failure modes include retrieval returning irrelevant chunks, context overflow, and the model ignoring retrieved content to hallucinate anyway.

Expect these follow-ups

• How do you evaluate retrieval quality separately from generation quality?
• What is the difference between RAG and a tool-augmented agent that calls a search tool?

As asked

Describe the ReAct pattern for agent reasoning. What does the loop look like step by step, and why does alternating between reasoning and action tend to produce better results than just prompting for an answer?

Sample answer outline

Strong answers describe the Thought, Action, Observation cycle where the model reasons about what to do, takes an action (tool call), observes the result, and repeats until it can answer. They explain that externalizing reasoning into the context helps the model stay on track on multi-step tasks and gives the engineer visibility into where the model went wrong. Candidates should know the original ReAct paper.

Expect these follow-ups

• What are the downsides of the ReAct pattern for very long tasks?
• How does chain-of-thought prompting relate to the ReAct pattern?

As asked

What are the possible stop reasons an LLM API can return, and what should your agent loop do in response to each one?

Sample answer outline

Strong answers cover end_turn or stop (model finished normally, check for tool calls or return to user), max_tokens (response was truncated, handle partial output carefully), tool_use or function_call (model wants to call a tool, dispatch it), stop_sequence (a custom stop marker was hit), and provider-specific reasons like content_filter. Candidates should note that on end_turn with no tool calls the loop should terminate.

Expect these follow-ups

• What do you do when you get max_tokens mid-tool-arguments, meaning the JSON was truncated?
• How do you distinguish an intentional stop from an unintentional truncation?

As asked

You are adding computer-use actions to an agent SDK so agents can operate web apps. What sandboxing, permissions, and audit controls do you require?

Sample answer outline

Run agents in isolated browser or desktop sessions with scoped credentials, network controls, and no ambient access to the user's local files unless explicitly granted. Separate observation from action and require confirmation for sensitive actions such as purchases, messages, deletes, or permission changes. Store an audit trail of observations, actions, screenshots where allowed, and user approvals. The SDK should expose policy hooks so product teams can block actions by domain, selector, data class, or user role. Strong answers treat model behaviour as untrusted automation inside a controlled runtime.

Expect these follow-ups

• What actions should always require confirmation?
• How do you redact secrets from screenshots and traces?
• How would you test that policies cannot be bypassed through prompt injection?

As asked

Explain what temperature, top-p, and top-k do to the token sampling distribution. For an agent making tool calls, what setting would you typically use and why?

Sample answer outline

Strong answers describe temperature as scaling logits before softmax (low = more deterministic, high = more random), top-p as nucleus sampling that picks from tokens covering cumulative probability p, top-k as restricting to the k most likely tokens, and argue that for tool-calling agents you want low temperature (around 0) for consistent, parseable JSON and higher temperature only for creative generation tasks.

Expect these follow-ups

• What is the risk of using temperature 0 for all parts of an agent run?
• How does temperature interact with a constrained decoding approach for structured output?

As asked

Explain how prompt caching works in providers that support it, what the cost and latency benefits are, and what the conditions are for a cache hit.

Sample answer outline

Strong answers describe caching as reusing the KV activations for a prefix of the prompt that has not changed, reducing both compute cost and time-to-first-token, note that the prefix must be identical byte-for-byte to get a hit, and give practical examples like caching the system prompt and tool schemas that are constant across requests while varying the user message. Candidates should know approximate price differences between cache hits and misses on major providers.

Expect these follow-ups

• How do you structure your messages array to maximize cache hits across requests in a multi-turn conversation?
• What happens to your cache hit rate if you add a timestamp to the system prompt?

As asked

What are reasoning or thinking tokens in models that support extended thinking? How do they affect latency and cost, and when would you enable them for an agent?

Sample answer outline

Strong answers explain that thinking tokens are generated before the visible response and contain the model's internal chain of thought, that they increase both latency and cost (billed at the same or higher rate as output tokens), and that they improve accuracy on complex multi-step reasoning tasks but are overkill for simple retrieval or formatting tasks. Candidates should clarify that thinking blocks are returned in the API response content array and are fully accessible to the developer, but the product decision of whether to display them to end users is separate. Strong answers also note that in Anthropic's API the thinking budget is set via a budget_tokens parameter and the model may use fewer tokens than the budget allows.

Expect these follow-ups

• How do you decide per step in an agent loop whether to enable extended thinking?
• Can the model access tool call results within its thinking block?

As asked

Compare HNSW and IVF indexes for approximate nearest neighbor search. Which would you choose for a real-time agent retrieval tool with 10M vectors and why?

Sample answer outline

Strong answers describe HNSW as a graph-based index with fast query time and no training step but higher memory usage (storing the graph), IVF as a clustering approach that partitions vectors into cells and is more memory-efficient but requires a training step and tuning nprobe for recall, and recommend HNSW for real-time retrieval where latency and recall matter more than memory cost. Candidates should mention that pgvector supports both.

Expect these follow-ups

• What is the tradeoff between recall and query speed in HNSW, and which knobs control it?
• How do you handle adding new vectors to an HNSW index without rebuilding it from scratch?

As asked

Most LLM APIs offer tool_choice parameters with modes like auto, any, and a specific tool name. What does each mode do, and when would you use each in an agent?

Sample answer outline

Strong answers describe auto as letting the model decide whether to call a tool or generate text, any as forcing the model to call at least one tool from the list, and specifying a tool name as forcing a specific tool call. Candidates should give concrete use cases: auto for general agents, any for structured data extraction where you always want a schema-conforming output, and specific tool for one-shot tool invocations.

Expect these follow-ups

• What is the risk of always using tool_choice: any in an agent loop?
• How do you force the model to produce a final answer instead of calling another tool when you want the loop to end?

As asked

When should you use cosine similarity versus dot product for measuring embedding similarity in a retrieval tool, and how does the choice depend on whether embeddings are normalized?

Sample answer outline

Strong answers explain that cosine similarity normalizes by magnitude so it measures direction only (good when embeddings vary in magnitude), dot product is equivalent to cosine when vectors are L2-normalized (many modern embedding models output normalized vectors), and dot product is faster to compute. Most production embedding models produce normalized vectors so dot product is preferred for efficiency. Candidates should know that pgvector's <=> uses cosine distance.

Expect these follow-ups

• How do you verify whether the embedding model you are using outputs normalized vectors?
• What does it mean when two documents have identical cosine similarity but very different magnitudes?