Prompt engineer analytics interview questions

As asked

A PM says prompt B sounds better than prompt A for an AI writing feature. How do you decide which one to ship?

Sample answer outline

Start by defining the user outcome, such as acceptance rate, edit distance after generation, task completion, or user-rated usefulness. Offline evals should use a representative dataset with blind human preference labels and rubric-based checks for factuality, tone, policy, and formatting. Online tests need guardrails: latency, cost, complaint rate, regeneration rate, and downstream retention if the feature is material. A strong answer explains why subjective taste alone is not enough but also why automatic judge scores need calibration. Candidates trip up by optimising for win rate against a narrow prompt set and missing product impact.

Expect these follow-ups

• When would you trust an LLM-as-judge?
• How do you avoid overfitting the prompt to the eval set?
• What if prompt B wins quality but doubles latency?

As asked

You own prompts for a sales-email assistant. What evaluation harness do you build so prompt changes can ship safely?

Sample answer outline

The harness should run every candidate prompt against a fixed and growing set of real tasks, including good-fit, edge, and should-refuse examples. Score dimensions separately: factual grounding, tone, personalisation, compliance, formatting, and unnecessary claims. Use human review for a labelled sample and calibrated LLM judges for cheaper ongoing checks. Track regressions by customer segment and email type rather than only a single average score. Good candidates describe release gates, prompt versioning, and how production failures become new eval cases.

Expect these follow-ups

• How do you calibrate an LLM judge?
• What is your policy for changing the eval set?
• How do you avoid rewarding generic but safe emails?