AI red team engineer interview questions

As asked

You are asked to red team a tool-using support agent that reads web pages, calls internal APIs, and drafts customer replies. How would you test it for indirect prompt injection?

Sample answer outline

Start by mapping the trust boundaries: user instructions, retrieved web content, tool outputs, system prompts, and privileged internal data. Build payloads where hostile content appears in external pages or documents, then check whether the agent treats that content as instructions rather than data. A strong answer separates exfiltration, unauthorised tool use, data corruption, and policy bypass as distinct failure classes. Good candidates discuss both manual probing and an eval harness that measures attack success rate across model and prompt versions. The common trap is testing only direct jailbreaks in chat and missing the real agentic path through retrieved content.

Expect these follow-ups

• What signals prove the model followed the web page rather than the user task?
• How would you turn one successful exploit into a regression test?
• Which mitigations belong in prompting and which belong in the tool layer?

As asked

Users are trying to make a policy assistant ignore its instructions and reveal restricted guidance. How do you improve the prompt and the surrounding system?

Sample answer outline

A senior answer does not claim prompt wording alone solves jailbreaks. Put policy and tool permissions outside the user-controlled text, use clear instruction hierarchy, and make refusal behaviour specific to the domain. Add retrieval filters, output checks, and tool allowlists so the model cannot access or reveal restricted material just because the prompt was persuasive. Build an adversarial eval set from real attempts and track bypass rate by category. Candidates trip up when they write longer scolding instructions but leave the same unsafe tools and documents available.

Expect these follow-ups

• What belongs in the system prompt versus application code?
• How do you measure jailbreak resistance without overfitting?
• When should the assistant escalate to a human?

As asked

You are adding computer-use actions to an agent SDK so agents can operate web apps. What sandboxing, permissions, and audit controls do you require?

Sample answer outline

Run agents in isolated browser or desktop sessions with scoped credentials, network controls, and no ambient access to the user's local files unless explicitly granted. Separate observation from action and require confirmation for sensitive actions such as purchases, messages, deletes, or permission changes. Store an audit trail of observations, actions, screenshots where allowed, and user approvals. The SDK should expose policy hooks so product teams can block actions by domain, selector, data class, or user role. Strong answers treat model behaviour as untrusted automation inside a controlled runtime.

Expect these follow-ups

• What actions should always require confirmation?
• How do you redact secrets from screenshots and traces?
• How would you test that policies cannot be bypassed through prompt injection?

As asked

A product team wants to launch an LLM feature that can summarise uploaded legal, financial, and HR documents. Walk me through your red-team threat model before launch.

Sample answer outline

Identify assets first: confidential documents, model outputs, user identities, audit logs, and any downstream actions triggered by summaries. Then enumerate abuse paths: cross-tenant data leakage, prompt injection from uploaded files, hallucinated legal claims, unsafe advice, and adversaries using the system to process stolen material. The answer should prioritise risks by likelihood and impact rather than producing a generic list. Strong candidates propose concrete tests, such as tenant-boundary probes, adversarial documents, output policy checks, and privacy-preserving logging review. Interviewers listen for clear reporting language that product teams can act on, not theatrical exploit naming.

Expect these follow-ups

• What launch blocker would make you recommend delaying the release?
• How would you test cross-tenant leakage without touching real customer data?
• What belongs in the risk acceptance memo?

As asked

How would you build and maintain an evaluation set for jailbreak resistance in a customer-facing AI assistant?

Sample answer outline

Separate broad coverage from high-signal regressions: one set should cover known jailbreak families, while another should contain failures your product has actually seen. Label expected behaviour precisely, because vague labels make eval scores meaningless. Include multi-turn attacks, role-play, encoding tricks, tool-use attempts, and benign prompts that look suspicious so the system does not become over-refusing. Strong answers mention versioning, holdout sets, scorer calibration, and review by policy or legal stakeholders when categories are sensitive. A weak answer treats jailbreak resistance as one static prompt list.

Expect these follow-ups

• How do you prevent overfitting to the public jailbreak set?
• What metric would you report to executives?
• How do you handle prompts where reviewers disagree on the correct response?

As asked

Your model scores unusually well on a public benchmark. How would you investigate possible data contamination?

Sample answer outline

First check whether benchmark examples, paraphrases, or answer keys appear in pretraining, fine-tuning, retrieval corpora, or synthetic data generation prompts. Use exact matching, fuzzy matching, embedding similarity, and n-gram overlap because contamination is often transformed rather than copied. A strong answer distinguishes accidental exposure from memorisation and from legitimate generalisation. The candidate should propose a clean holdout or newly authored evaluation to validate the result. Interviewers want to hear that high scores are treated as a debugging signal, not only as good news.

Expect these follow-ups

• How do you handle contamination discovered after a paper is submitted?
• What if you cannot inspect the full pretraining corpus?
• How would you design a benchmark that ages better against contamination?