Observability engineer behavioural interview questions

As asked

Describe a real incident where you realized mid-incident that you were missing the metric, log, or trace you needed to diagnose the problem. What did you do, and what did you build afterward?

Sample answer outline

A strong answer follows STAR format and names the specific gap (e.g., no database query timing metrics, no trace IDs in logs, no histogram for tail latency). The candidate should describe the workaround used during the incident (log parsing, manual profiling), the post-incident work to add the missing instrumentation, and how they validated the new instrumentation would have found the problem faster. The best answers show they updated the runbook and added a synthetic failure test.

Expect these follow-ups

• How did you decide which gap to fix first when there were multiple missing signals?
• How do you prevent the same observability gap from appearing in a new service?

As asked

Walk me through a situation where your team was overwhelmed by alert noise. What analysis did you do, what changes did you make, and how did you measure success?

Sample answer outline

The candidate should describe a structured approach: analyzing alert history to find high-frequency low-action-rate alerts, classifying alerts into symptom vs cause, making specific changes (raising thresholds, adding 'for' clauses, converting to informational channels, using inhibition), and measuring the before-and-after ratio of actionable pages. Strong answers also discuss how they got buy-in from other teams to stop defending their noisy alerts.

Expect these follow-ups

• How did you handle pushback from a team that was attached to their alert even though it was firing 90% false positives?
• What metric did you use to define 'alert fatigue' objectively?

As asked

Product engineers often see instrumentation as overhead that slows them down. Tell me about a time you had to get a skeptical team to add meaningful observability to their service. What was your approach?

Sample answer outline

A strong answer shows empathy with the team (instrumentation is extra work), a concrete ROI argument (the last incident took 4 hours to diagnose and would have taken 20 minutes with traces), and practical steps taken to make instrumentation easy (auto-instrumentation, shared libraries, templates). The candidate should describe following up to verify the instrumentation was maintained, not just added once.

Expect these follow-ups

• How did you prioritize which services to instrument first?
• What did you do when the team added instrumentation but used incorrect semantic conventions?

As asked

Observability costs can spiral quickly. Describe a time you identified and executed on a meaningful cost reduction in your metrics, logs, or traces spend without degrading visibility.

Sample answer outline

The best answers quantify the before and after cost and explain what analysis was done (cardinality breakdown, log volume by service, sampling rate analysis). Specific actions might include dropping high-cardinality labels, increasing sampling rates for low-value traces, tiering logs to cold storage, or switching vendors. The candidate should describe how they validated that the changes did not degrade visibility by reviewing incident response quality before and after.

Expect these follow-ups

• How did you decide what to cut without risking missing a future incident?
• What was the hardest conversation you had about removing a signal that someone cared about?

As asked

Tell me about a time you migrated from one observability platform to another, for example from one APM vendor to OTel, or from ELK to Loki. What was the plan, what went wrong, and how did you keep the team running during the transition?

Sample answer outline

A strong answer covers the motivation (cost, vendor lock-in, features), the migration strategy (parallel running, cut-by-service, rollback plan), the specific technical challenges (schema mapping, alert migration, dashboard rebuild), and how they handled the human side (teams using the old platform resisting change). The candidate should discuss how they validated the new platform was equivalent before cutting over and what was lost or gained.

Expect these follow-ups

• How did you handle the period where some teams were on the old platform and some on the new one?
• What would you do differently if you ran this migration again?

As asked

Describe a situation where you believed a team's SLO was set too loose or measured the wrong thing. How did you raise it, and what was the outcome?

Sample answer outline

A strong answer demonstrates the candidate can articulate the specific problem (SLO measured uptime but not user-perceived latency, or the threshold was 95% when users expected 99.9%), escalated through data (showing actual user complaints correlated with times when SLO was green), and collaborated with the team rather than imposing a change. The outcome should show either a changed SLO or a principled decision not to change it with documented reasoning.

Expect these follow-ups

• How do you handle a team that sets a deliberately easy SLO to avoid being held accountable?
• What is the right process for setting SLOs in a new service with no historical data?

As asked

Give me an example of an incident post-mortem where the outcome was a lasting change to how your team thinks about observability. What was the incident, what was the learning, and how did you make the change stick?

Sample answer outline

A strong answer describes a specific incident (e.g., a silent data corruption that lasted 6 hours because there were no business metric alerts, only infrastructure metrics), the post-mortem finding about the observability gap, the specific change made (e.g., adding golden signal checks to the deployment checklist, requiring trace IDs in all logs), and the mechanism used to make it stick (a linter, a checklist in the PR template, or a periodic audit). Durability of the change is key.

Expect these follow-ups

• How long did the change take to fully roll out across all services?
• How do you check that new services follow the standard?

As asked

We all have external dependencies we cannot instrument directly. Tell me about a time a third-party service caused an incident and you had limited visibility into it. How did you approach the problem?

Sample answer outline

Strong answers describe instrumenting the client side (measuring latency and errors of calls to the third party from your own code), using synthetic probes to detect third-party availability independently, correlating your client-side metrics with the vendor's status page, and setting up SLO-based alerting on the dependency rather than alerting on internal metrics. The candidate should describe the communication process with the vendor and how they set expectations internally when the vendor's SLA was insufficient.

Expect these follow-ups

• How do you build a circuit breaker and make the tripped state visible in your observability stack?
• How do you handle a vendor that provides no status API or webhook?

As asked

Tell me about a situation where you used metrics or trend analysis to make a capacity recommendation before a problem occurred. How did you present the data and what was the result?

Sample answer outline

The candidate should describe identifying a trend in metrics (memory usage growing 10% per week, disk IOPS approaching limit, connection pool saturation) before it became an incident, extrapolating to a projected breach date, presenting a clear cost-vs-risk tradeoff to stakeholders, and what action was taken. Strong answers include describing how they validated the trend was real and not an anomaly, and how they communicated uncertainty.

Expect these follow-ups

• How do you distinguish a temporary spike from a genuine long-term growth trend?
• What Grafana features or PromQL functions did you use for trend extrapolation?

As asked

Describe a production incident that involved multiple teams blaming each other. How did you use distributed tracing to establish the actual root cause and facilitate the conversation?

Sample answer outline

A strong answer describes the multi-team blame scenario, how the candidate used a trace waterfall to show exactly where latency or errors originated (specific span, specific service, specific database query), how they presented this objectively without blaming, and what the outcome was. The best answers show the candidate mediating the technical conversation using data rather than opinion.

Expect these follow-ups

• What did you do when the trace showed the problem was in your own team's service?
• How did you handle a service that was not instrumented and had a gap in the trace?

As asked

Walking into a company without a consistent SLO review process, how did you set one up? What cadence, what attendees, what output did you produce, and how did you keep it from becoming a pointless meeting?

Sample answer outline

The candidate should describe a weekly or monthly error budget review where engineers (not just managers) attend and the output is a decision: invest in reliability if budget is low, ship faster if budget is healthy. An effective meeting focuses on trend (is the budget improving or degrading?) rather than point-in-time status, includes a clear action item format, and has a rule that SLOs breached two months in a row require a reliability investment.

Expect these follow-ups

• How did you handle a team that consistently had 0% budget remaining but no one cared?
• What automated report did you send before the meeting so the discussion was focused?

As asked

You joined a company where every team was doing observability differently: some used Datadog, some used CloudWatch, some used nothing. How did you approach standardizing instrumentation and tooling, and what was the resistance you faced?

Sample answer outline

Strong answers describe a phased approach: audit the current state, identify common pain points, propose a standard with input from teams (not a mandate), start with a reference implementation in one service, provide shared libraries and templates, and measure adoption. The candidate should describe the specific resistance (teams not wanting to rewrite working instrumentation, vendor preferences, cost concerns) and how they addressed each objection. The outcome should include a measurable improvement in incident response time or cross-service debugging capability.

Expect these follow-ups

• How did you handle a team that refused to migrate and had executive support for staying on their current tool?
• What did you put in the developer onboarding guide to make the standard easy to adopt?