As asked
You own prompts for a sales-email assistant. What evaluation harness do you build so prompt changes can ship safely?
Sample answer outline
The harness should run every candidate prompt against a fixed and growing set of real tasks, including good-fit, edge, and should-refuse examples. Score dimensions separately: factual grounding, tone, personalisation, compliance, formatting, and unnecessary claims. Use human review for a labelled sample and calibrated LLM judges for cheaper ongoing checks. Track regressions by customer segment and email type rather than only a single average score. Good candidates describe release gates, prompt versioning, and how production failures become new eval cases.
Expect these follow-ups
- How do you calibrate an LLM judge?
- What is your policy for changing the eval set?
- How do you avoid rewarding generic but safe emails?
evalsprompt-versioningquality