As asked
A PM says prompt B sounds better than prompt A for an AI writing feature. How do you decide which one to ship?
Sample answer outline
Start by defining the user outcome, such as acceptance rate, edit distance after generation, task completion, or user-rated usefulness. Offline evals should use a representative dataset with blind human preference labels and rubric-based checks for factuality, tone, policy, and formatting. Online tests need guardrails: latency, cost, complaint rate, regeneration rate, and downstream retention if the feature is material. A strong answer explains why subjective taste alone is not enough but also why automatic judge scores need calibration. Candidates trip up by optimising for win rate against a narrow prompt set and missing product impact.
Expect these follow-ups
- When would you trust an LLM-as-judge?
- How do you avoid overfitting the prompt to the eval set?
- What if prompt B wins quality but doubles latency?