Observability engineer interview questions

As asked

A team adds user_id and request_id labels to Prometheus metrics and the metrics backend starts falling over. How do you fix it without losing useful debugging signal?

Sample answer outline

Explain that each unique label set creates a time series, so unbounded labels like user_id, request_id, email, or URL path parameters can explode storage and query cost. Remove the high-cardinality labels from metrics and move per-request details to logs or traces, linked by trace_id. Keep bounded labels such as service, route template, status class, region, and dependency. Add instrumentation review, cardinality budgets, and alerts on series growth so this is caught before ingestion fails. Strong candidates preserve the debugging use case rather than just saying 'delete the label'.

Expect these follow-ups

• Which labels would you allow on an HTTP server duration histogram?
• How do exemplars help connect metrics to traces?
• What query patterns become dangerous after cardinality spikes?

As asked

What telemetry would you add to an LLM application so on-call can debug latency spikes, cost overruns, and quality regressions?

Sample answer outline

Capture request count, error rate, provider status, queue time, time to first token, total latency, input tokens, output tokens, cache hit rate, and estimated cost by route, tenant, model, and prompt version. Quality telemetry should include eval score distributions, user feedback, refusal rate, tool-call failure rate, and structured-output validation failures. Use traces to connect retrieval, model call, tool calls, and post-processing into one request path. Watch cardinality carefully because raw prompt text and user ids do not belong in metric labels. Candidates often provide infrastructure metrics but miss product-level quality signals.

Expect these follow-ups

• Which labels would you forbid on metrics?
• How do you correlate a prompt change with a quality regression?
• What alert would page someone versus create a ticket?

As asked

Your team has dozens of drift dashboards and nobody trusts the alerts. How would you redesign model drift monitoring?

Sample answer outline

Start by separating input drift, prediction drift, label drift, and business metric degradation because each has a different owner and response. Alert only on monitored signals with an agreed action, such as retraining, feature pipeline investigation, or business review. Use population slices for important cohorts rather than only global averages, since failures often hide in small but valuable segments. Strong candidates discuss delayed labels, seasonality, alert thresholds, and annotation of known events such as campaigns or outages. A weak answer adds more charts instead of reducing noise and tying alerts to playbooks.

Expect these follow-ups

• What drift signal would you page on?
• How do you monitor a model whose labels arrive 30 days later?
• When is drift expected and harmless?

As asked

Teams keep opening platform tickets for production issues in services they own. How would you clarify service ownership without abandoning teams?

Sample answer outline

Define ownership in operational terms: who is on call, who approves changes, who owns SLOs, and who fixes production bugs. The platform team should own shared tooling, templates, and guardrails, not become the default operations team for every product service. Strong answers include an escalation path, office hours, runbooks, and platform-provided diagnostics so teams can solve common issues themselves. The candidate should recognise that ownership changes require enablement, not a memo. The trap is either accepting every ticket forever or abruptly pushing work back without giving teams the capability to succeed.

Expect these follow-ups

• How do you handle a team that refuses to take on-call?
• What belongs in a service catalogue?
• When should the platform team take direct ownership of an incident?

As asked

Design alerting for an API with a 99.9 percent monthly availability SLO. How would you avoid noisy alerts while still catching fast outages?

Sample answer outline

Start from the error budget: 99.9 percent availability gives roughly 43 minutes of monthly unavailability, so alerting should trigger on budget burn, not isolated error spikes. Use multi-window, multi-burn-rate alerts, for example a fast page on high burn over 5 minutes and 1 hour, and a ticket on slower burn over 6 hours and 3 days. Measure availability from user-facing successful requests, not just pod health. Include routing, runbooks, and silence policy so alerts lead to action. Candidates often miss that paging on every 500 rate spike creates fatigue and makes the SLO less useful.

Expect these follow-ups

• How do you define a good request versus a bad request for this API?
• What if latency violates the user promise but success rate stays high?
• How do error budgets change release decisions?

As asked

Your tracing bill is too high, but engineers rely on traces to debug rare checkout failures. How would you change the sampling strategy?

Sample answer outline

Separate head sampling, which decides early and is cheap, from tail sampling, which can keep traces based on outcome after the request completes. Keep all error traces, slow traces, and traces for important business flows, while sampling routine successful traffic at a lower rate. Propagate sampling decisions consistently across services so one request does not produce broken partial traces. Use attributes with bounded cardinality to drive retention rules, and validate with engineers that rare failure modes remain findable. A weak answer simply lowers the global sampling percentage and loses the exact traces people need during incidents.

Expect these follow-ups

• What metadata must be propagated for tracing to work across services?
• When is tail sampling too expensive or too late?
• How would you prove the new sampling policy is not hiding production bugs?