Observability engineer role-specific interview questions

As asked

What telemetry would you add to an LLM application so on-call can debug latency spikes, cost overruns, and quality regressions?

Sample answer outline

Capture request count, error rate, provider status, queue time, time to first token, total latency, input tokens, output tokens, cache hit rate, and estimated cost by route, tenant, model, and prompt version. Quality telemetry should include eval score distributions, user feedback, refusal rate, tool-call failure rate, and structured-output validation failures. Use traces to connect retrieval, model call, tool calls, and post-processing into one request path. Watch cardinality carefully because raw prompt text and user ids do not belong in metric labels. Candidates often provide infrastructure metrics but miss product-level quality signals.

Expect these follow-ups

• Which labels would you forbid on metrics?
• How do you correlate a prompt change with a quality regression?
• What alert would page someone versus create a ticket?

As asked

Your team has dozens of drift dashboards and nobody trusts the alerts. How would you redesign model drift monitoring?

Sample answer outline

Start by separating input drift, prediction drift, label drift, and business metric degradation because each has a different owner and response. Alert only on monitored signals with an agreed action, such as retraining, feature pipeline investigation, or business review. Use population slices for important cohorts rather than only global averages, since failures often hide in small but valuable segments. Strong candidates discuss delayed labels, seasonality, alert thresholds, and annotation of known events such as campaigns or outages. A weak answer adds more charts instead of reducing noise and tying alerts to playbooks.

Expect these follow-ups

• What drift signal would you page on?
• How do you monitor a model whose labels arrive 30 days later?
• When is drift expected and harmless?

As asked

A team adds user_id and request_id labels to Prometheus metrics and the metrics backend starts falling over. How do you fix it without losing useful debugging signal?

Sample answer outline

Explain that each unique label set creates a time series, so unbounded labels like user_id, request_id, email, or URL path parameters can explode storage and query cost. Remove the high-cardinality labels from metrics and move per-request details to logs or traces, linked by trace_id. Keep bounded labels such as service, route template, status class, region, and dependency. Add instrumentation review, cardinality budgets, and alerts on series growth so this is caught before ingestion fails. Strong candidates preserve the debugging use case rather than just saying 'delete the label'.

Expect these follow-ups

• Which labels would you allow on an HTTP server duration histogram?
• How do exemplars help connect metrics to traces?
• What query patterns become dangerous after cardinality spikes?

As asked

Your tracing bill is too high, but engineers rely on traces to debug rare checkout failures. How would you change the sampling strategy?

Sample answer outline

Separate head sampling, which decides early and is cheap, from tail sampling, which can keep traces based on outcome after the request completes. Keep all error traces, slow traces, and traces for important business flows, while sampling routine successful traffic at a lower rate. Propagate sampling decisions consistently across services so one request does not produce broken partial traces. Use attributes with bounded cardinality to drive retention rules, and validate with engineers that rare failure modes remain findable. A weak answer simply lowers the global sampling percentage and loses the exact traces people need during incidents.

Expect these follow-ups

• What metadata must be propagated for tracing to work across services?
• When is tail sampling too expensive or too late?
• How would you prove the new sampling policy is not hiding production bugs?