Prompt engineer machine learning interview questions

As asked

A product team wants a generative assistant to answer support questions in the company's tone using private documentation. When would you fine-tune, use RAG, or rely on prompting?

Sample answer outline

Use RAG when the main requirement is fresh private knowledge and citations, because fine-tuning is a poor way to memorise changing facts. Use fine-tuning when you need a consistent behaviour, format, style, or task policy that prompting cannot reliably maintain at scale. Prompting alone is suitable when the task is simple, low-risk, and changes frequently. Strong answers mention evaluation before and after each approach, plus the operational costs of data curation, retraining, retrieval quality, and latency. The common error is treating fine-tuning as a magic upgrade rather than a targeted behavioural intervention.

Expect these follow-ups

• What failure tells you the retriever is the problem?
• What data would you need for a useful fine-tune?
• How would you measure whether tone improved without making factuality worse?

As asked

How would you build and maintain an evaluation set for jailbreak resistance in a customer-facing AI assistant?

Sample answer outline

Separate broad coverage from high-signal regressions: one set should cover known jailbreak families, while another should contain failures your product has actually seen. Label expected behaviour precisely, because vague labels make eval scores meaningless. Include multi-turn attacks, role-play, encoding tricks, tool-use attempts, and benign prompts that look suspicious so the system does not become over-refusing. Strong answers mention versioning, holdout sets, scorer calibration, and review by policy or legal stakeholders when categories are sensitive. A weak answer treats jailbreak resistance as one static prompt list.

Expect these follow-ups

• How do you prevent overfitting to the public jailbreak set?
• What metric would you report to executives?
• How do you handle prompts where reviewers disagree on the correct response?