LLM ops engineer machine learning interview questions

As asked

Design a RAG feature that answers customer support questions using internal documentation. How do you keep answers grounded and know when the model should refuse?

Sample answer outline

A good answer separates retrieval quality from generation quality. Chunk documents around semantic boundaries, attach metadata such as product, version, and freshness, and retrieve with hybrid search plus reranking for high precision. The generation prompt should require citations to retrieved passages and should refuse when evidence is missing or contradictory. Evaluation needs golden questions, adversarial no-answer cases, citation accuracy checks, and human review for high-impact categories. Candidates often focus on vector databases while ignoring stale docs, permissions, and the model's tendency to fill gaps.

Expect these follow-ups

• How do you prevent a user from retrieving documents they cannot access?
• What metric tells you retrieval is the bottleneck?
• How do you handle two documents that disagree?

As asked

You are upgrading the model behind a customer-facing assistant. What regression suite do you run before and after rollout?

Sample answer outline

Use a frozen eval set covering common tasks, edge cases, safety cases, no-answer cases, and historically bad production examples. Score with a mix of deterministic validators, human review for a stratified sample, and calibrated LLM judges for rubric checks. Compare not only aggregate score but slices by language, customer tier, task type, and retrieval source. Run shadow traffic before canarying, then watch online metrics such as escalation rate, thumbs-down rate, cost, latency, and refusal rate. The trap is declaring victory on average quality while a critical slice regresses.

Expect these follow-ups

• How do you choose examples for the frozen eval set?
• When should a single failing example block release?
• How do you stop the suite becoming stale?