AI red team engineer domain knowledge interview questions

As asked

Users are trying to make a policy assistant ignore its instructions and reveal restricted guidance. How do you improve the prompt and the surrounding system?

Sample answer outline

A senior answer does not claim prompt wording alone solves jailbreaks. Put policy and tool permissions outside the user-controlled text, use clear instruction hierarchy, and make refusal behaviour specific to the domain. Add retrieval filters, output checks, and tool allowlists so the model cannot access or reveal restricted material just because the prompt was persuasive. Build an adversarial eval set from real attempts and track bypass rate by category. Candidates trip up when they write longer scolding instructions but leave the same unsafe tools and documents available.

Expect these follow-ups

• What belongs in the system prompt versus application code?
• How do you measure jailbreak resistance without overfitting?
• When should the assistant escalate to a human?

As asked

You are adding computer-use actions to an agent SDK so agents can operate web apps. What sandboxing, permissions, and audit controls do you require?

Sample answer outline

Run agents in isolated browser or desktop sessions with scoped credentials, network controls, and no ambient access to the user's local files unless explicitly granted. Separate observation from action and require confirmation for sensitive actions such as purchases, messages, deletes, or permission changes. Store an audit trail of observations, actions, screenshots where allowed, and user approvals. The SDK should expose policy hooks so product teams can block actions by domain, selector, data class, or user role. Strong answers treat model behaviour as untrusted automation inside a controlled runtime.

Expect these follow-ups

• What actions should always require confirmation?
• How do you redact secrets from screenshots and traces?
• How would you test that policies cannot be bypassed through prompt injection?

As asked

A product team wants to launch an LLM feature that can summarise uploaded legal, financial, and HR documents. Walk me through your red-team threat model before launch.

Sample answer outline

Identify assets first: confidential documents, model outputs, user identities, audit logs, and any downstream actions triggered by summaries. Then enumerate abuse paths: cross-tenant data leakage, prompt injection from uploaded files, hallucinated legal claims, unsafe advice, and adversaries using the system to process stolen material. The answer should prioritise risks by likelihood and impact rather than producing a generic list. Strong candidates propose concrete tests, such as tenant-boundary probes, adversarial documents, output policy checks, and privacy-preserving logging review. Interviewers listen for clear reporting language that product teams can act on, not theatrical exploit naming.

Expect these follow-ups

• What launch blocker would make you recommend delaying the release?
• How would you test cross-tenant leakage without touching real customer data?
• What belongs in the risk acceptance memo?