Observability engineer domain knowledge interview questions

As asked

Name the four golden signals from the Google SRE book, explain what each measures, and give me an example of a specific metric for each in a typical web service.

Sample answer outline

Latency (time to serve a request: http_request_duration_seconds), Traffic (demand on the system: http_requests_total rate), Errors (rate of failed requests: http_requests_total where status=5xx), Saturation (how full the system is: cpu_usage_seconds rate vs limit, or queue depth). They were chosen because they are the signals most directly correlated with user experience without being so low-level that they require deep system knowledge to interpret. A strong answer notes that saturation is the hardest to measure because it requires knowing capacity as well as usage.

Expect these follow-ups

• Which golden signal is most likely to predict a problem before users notice it?
• How do the four golden signals map to RED and USE methods?

As asked

Define metric cardinality in the Prometheus context. At what cardinality level does a Prometheus instance typically start to struggle, and what are the three most common sources of cardinality explosion in a real microservices environment?

Sample answer outline

Cardinality is the number of unique time series, determined by the number of unique label value combinations per metric. A Prometheus instance starts to slow down around 1 to 2 million active series and struggles badly beyond 5 million. The three most common sources: user or request IDs as label values (unbounded), URL paths not templated (infinite paths from REST APIs with IDs embedded), and version strings or build hashes on every process restart (unbounded churn). A strong candidate mentions that cardinality churn (series that come and go frequently) is even more damaging than high but stable cardinality.

Expect these follow-ups

• What is the difference between high cardinality and high churn, and which is worse for Prometheus?
• How do you check total active series on a running Prometheus instance?

As asked

Explain the concrete operational benefits of structured logging (JSON or key-value pairs) compared to unstructured printf-style log lines, and describe the problems that arise when you mix both in the same system.

Sample answer outline

Structured logs allow querying on any field without regex parsing, enable direct aggregation (count errors by user_id), correlate with traces via trace_id field, and can be indexed efficiently by backends like Elasticsearch. Printf logs require fragile regex parsers that break when the log format changes. Mixing both in the same pipeline forces you to maintain two parsers and means some log lines are unsearchable by field. A strong answer mentions that structured logs must have consistent field names across services (using semantic conventions) to be useful at scale.

Expect these follow-ups

• What is the maximum safe size for a single structured log line in production?
• How do you handle a third-party library that only emits printf-style logs?

As asked

Explain why average latency is misleading for user-facing SLOs and why you would use P95 or P99 instead. Give me a concrete numerical example where the average looks fine but the P99 indicates a serious problem.

Sample answer outline

Average latency is pulled down by the majority of fast requests and hides the tail. Example: 990 requests in 10ms and 10 requests in 10 seconds gives an average of about 110ms but a P99 of 10 seconds. The 10ms average feels healthy but 1% of users are waiting 10 seconds. For user-facing SLOs, you care about the worst experience a real percentage of users has, which is captured by percentiles. A strong answer also explains that P99.9 may be more appropriate for latency-sensitive paths and that percentile calculation from histograms requires well-chosen bucket boundaries.

Expect these follow-ups

• What is the Apdex score and how does it differ from a percentile-based SLI?
• How does Prometheus histogram_quantile estimation error affect your SLO calculations?

As asked

What is the USE method and the RED method? For a PostgreSQL database, which one would you apply and what specific metrics does it give you?

Sample answer outline

USE (Utilization, Saturation, Errors) applies to resources like CPU, memory, disk, and network. RED (Rate, Errors, Duration) applies to services and request flows. For PostgreSQL as a resource: USE gives CPU utilization, connection pool saturation, and disk I/O errors. For PostgreSQL as a service (from the client's perspective): RED gives query rate, error rate, and query duration. A strong answer explains that you need both for a database: USE to understand the machine and RED to understand the user experience.

Expect these follow-ups

• What USE metric would you check first when a database is slow?
• How do the USE and RED methods relate to the four golden signals?

As asked

What does a W3C traceparent header look like? Break down each field, what it contains, and what the trace-flags byte controls.

Sample answer outline

The traceparent header has the format: version-traceid-parentid-traceflags. For example: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. Version (2 hex chars, always 00 currently), trace ID (32 hex chars, 128-bit), parent span ID (16 hex chars, 64-bit), trace flags (2 hex chars where bit 0 is the sampling flag: 01 means sampled, 00 means not sampled). A strong answer explains that the sampling flag tells downstream services whether this trace was selected for sampling, so they do not need to make their own decision independently.

Expect these follow-ups

• What is the tracestate header and how does it differ from traceparent?
• What happens if an intermediate proxy strips the traceparent header?

As asked

Define Mean Time To Detect and Mean Time To Recover. How would you compute both from your alerting and incident management system, and what should you prioritize improving first?

Sample answer outline

MTTD is the time from when a problem starts until it is detected (alert fires or user reports it). MTTR is the time from detection to full recovery. MTTD is measured as alert_fired_time minus incident_start_time (often approximated from the point the metric crossed the threshold). MTTR is measured from first alert to incident resolved. Most teams should improve MTTD first because a problem you detect in 2 minutes gives you 28 minutes to fix it within a 30-minute SLO window. A strong answer mentions that MTTD improvements come from better alerting and monitoring, while MTTR improvements come from better runbooks, faster rollback, and incident process.

Expect these follow-ups

• How does your choice of SLO error budget affect the acceptable MTTD?
• What is the relationship between alert 'for' clause duration and MTTD?

As asked

List all the telemetry signals OpenTelemetry supports, which ones are marked stable in the spec as of 2024, and which are still experimental. What does signal stability mean for production use?

Sample answer outline

OTel currently defines four telemetry signals: traces, metrics, logs, and profiles. Traces and metrics reached stable (v1.0) spec status first. The Logs signal (including the Logs Bridge API and SDK) reached stable status in OTel 1.x as well. Profiles are still experimental. Events in OTel are not a separate signal type; they are log records with a specific event.name attribute, emitted through the Logs API. Signal stability means the API and SDK interfaces will not have breaking changes, making it safe to use in production. The candidate should distinguish between the spec stability and the SDK language stability, since some language SDKs trail the spec.

Expect these follow-ups

• What is the difference between the OTel Logs API and the OTel Logs Bridge API?
• How do you use the OTel Logs Bridge API with an existing logging framework like Log4j?

As asked

A team is choosing between using the OpenTelemetry SDK plus OTel Collector versus their current Datadog APM agent. What are the concrete tradeoffs in terms of vendor lock-in, features, operational overhead, and data quality?

Sample answer outline

OTel advantages: vendor neutral (data can be routed to any backend), open standard with broad ecosystem support, no per-host agent license cost. Proprietary APM advantages: tighter integration with the vendor's UI (automatic correlation of metrics, traces, and logs in one dashboard), more out-of-the-box integrations and less configuration, better support for language-specific profiling and exception tracking. OTel trade-offs: requires operational expertise to run the Collector fleet, data quality depends on correct SDK configuration, semantic convention compliance needs enforcement. A strong candidate mentions that OTel and Datadog are not mutually exclusive: Datadog accepts OTel data, so you can use OTel for instrumentation and Datadog as the backend.

Expect these follow-ups

• What data does a proprietary APM agent collect that the OTel SDK cannot?
• How do you migrate from a proprietary agent to OTel without a big bang cutover?

As asked

Prometheus uses a pull model and Datadog uses a push model. Explain the architectural difference, when each model causes problems, and what the Prometheus Pushgateway is for.

Sample answer outline

Pull: Prometheus scrapes targets on a schedule. Targets must be reachable (problematic for short-lived jobs, behind firewalls, or in different network segments) but the scraper knows immediately if a target is down. Push: agents send metrics to a central endpoint. Scales across network boundaries more easily but the collection system cannot tell if a silent target is down or just not generating metrics. The Pushgateway is a bridge for short-lived jobs (batch jobs, cron) that exit before Prometheus scrapes them, allowing them to push final metrics. A strong answer notes that Pushgateway is not for general use (it aggregates last-value per job, not per instance, and is a single point of failure).

Expect these follow-ups

• How does remote_write change the Prometheus pull model?
• When would you use the Pushgateway versus a recording rule on a counter?