Observability engineer system design interview questions

As asked

Design alerting for an API with a 99.9 percent monthly availability SLO. How would you avoid noisy alerts while still catching fast outages?

Sample answer outline

Start from the error budget: 99.9 percent availability gives roughly 43 minutes of monthly unavailability, so alerting should trigger on budget burn, not isolated error spikes. Use multi-window, multi-burn-rate alerts, for example a fast page on high burn over 5 minutes and 1 hour, and a ticket on slower burn over 6 hours and 3 days. Measure availability from user-facing successful requests, not just pod health. Include routing, runbooks, and silence policy so alerts lead to action. Candidates often miss that paging on every 500 rate spike creates fatigue and makes the SLO less useful.

Expect these follow-ups

• How do you define a good request versus a bad request for this API?
• What if latency violates the user promise but success rate stays high?
• How do error budgets change release decisions?

As asked

We have 200 microservices, each with independently defined SLOs. Design a system that lets any team see their current error budget, burn rate, and projected budget exhaustion date, and that enables platform engineering to see aggregate reliability across the fleet.

Sample answer outline

A strong answer covers the data model (per-service SLO config stored as code, SLI definitions as recording rules), a central Prometheus federation or Thanos layer to aggregate metrics, a purpose-built SLO service (Sloth, pyrra, or custom) that generates recording rules from SLO definitions, Grafana dashboards with per-team views and a fleet-level summary, and alerting on burn rate. The candidate should discuss how to handle services with different SLO windows, the difference between calendar and rolling windows, and how to expose this to non-engineers via a simple UI.

Expect these follow-ups

• How do you prevent one team's SLO definition mistake from flooding the entire rule evaluation system?
• How would you implement SLO-based deployment gates that block a release if the error budget is low?

As asked

Design the storage and query layer for a distributed tracing system that ingests 10 million spans per second with a requirement to query any trace by ID in under 500ms and support tag-based search across the last 24 hours.

Sample answer outline

A strong answer separates the write path (ingest to Kafka, process with streaming jobs, write to Cassandra or ScyllaDB keyed by trace ID for point lookups) from the search path (write span tags to an inverted index like Elasticsearch or ClickHouse for tag-based search). The candidate should address sampling at ingest (you cannot store all 10M spans), TTL strategies, and the difference between trace-ID lookup (O(1) from Cassandra) vs tag search (inverted index with time partitioning). Tempo uses Parquet in object storage with a local index for this.

Expect these follow-ups

• How does Grafana Tempo approach this problem differently from Jaeger?
• What is the cost difference between storing traces in object storage vs Cassandra at this scale?

As asked

Our on-call engineers receive 300 alerts per week, of which they say 70% are noise. Design an alerting platform and process that reduces actionable alert ratio to below 20% noise without missing real incidents.

Sample answer outline

A strong answer covers alert classification (SLO-based alerts vs symptom vs cause), correlation to deduplicate related alerts from the same root cause, ML-based noise scoring trained on historical alert-to-action data, progressive notification (Slack first, then page only on no-ack), and a weekly review process to retire high-noise alerts. The candidate should mention tools like Alertmanager for grouping and inhibition, and third-party platforms like PagerDuty or Grafana OnCall for scheduling. Key insight: most alert noise comes from alerts on causes (CPU high) rather than symptoms (user-facing error rate high).

Expect these follow-ups

• How do you measure whether your changes actually reduced actionable alert ratio?
• What is the risk of being too aggressive in suppressing alerts?

As asked

We have 2,000 pods across 50 namespaces all sending telemetry. Design the OTel Collector fleet: how many tiers, what DaemonSet vs Deployment, how you handle tail sampling, and how you ensure no single Collector failure drops data.

Sample answer outline

A three-tier architecture: DaemonSet Collectors (one per node) receive OTLP from pods on the same node via local socket or localhost, perform lightweight transforms, and forward to a gateway Collector Deployment. The gateway tier performs tail sampling (requiring trace-consistent routing via consistent hashing on trace ID) and fans out to backends. A third optional tier handles heavy processing. For resilience, DaemonSet collectors buffer to local disk, gateway deployments are autoscaled behind a load balancer, and Kubernetes readiness probes ensure traffic is not routed to unhealthy collectors.

Expect these follow-ups

• How do you handle the routing problem for tail sampling when the gateway fleet scales up?
• What metrics do you expose on the Collectors themselves to know if they are healthy?

As asked

Design a synthetic monitoring system that probes our checkout flow from 10 geographic regions every minute, detects regional outages within 2 minutes, and distinguishes between real user impact and transient probe failures.

Sample answer outline

The candidate should describe a fleet of probes (Grafana Synthetic Monitoring or Datadog Synthetics style) running scripted browser or API checks from each region, emitting success/failure and latency to a central metrics store. Detection requires comparing probe results across regions (isolating regional vs global issues), using consecutive failure thresholds rather than single-probe failure, and publishing structured results as Prometheus metrics for alerting. The system must avoid false pages from probe node failures by comparing results from at least two probes per region.

Expect these follow-ups

• How do you keep synthetic test credentials secure and rotate them without breaking the probes?
• How would you extend this to also measure core web vitals using a headless browser?

As asked

We have five Kubernetes clusters across three cloud regions. Design a Prometheus architecture that provides a unified query interface, per-cluster local alerting, and long-term metric storage at acceptable cost.

Sample answer outline

A strong answer proposes per-cluster Prometheus instances for local alerting (no cross-cluster dependency for critical alerts), remote_write to a centralized Thanos Receive or Grafana Mimir for long-term storage and cross-cluster querying, and Thanos Query or Mimir's query frontend as the unified query layer. Object storage (S3/GCS) handles long-term data cheaply. The candidate should address deduplication via replica external labels, cross-region network cost of remote_write, and the risk of losing local alerting if the Prometheus instance itself fails (mitigated by HA pairs with deduplication).

Expect these follow-ups

• How do you handle a cluster that is network-isolated from the central storage during a cloud region outage?
• What is the cost of remote_write at 1 million samples per second across regions?

As asked

You need to build a metrics pipeline that accepts Prometheus remote_write from 50 teams, enforces per-team cardinality limits, allocates costs back to teams, and provides a self-service portal for teams to see their usage. Design it.

Sample answer outline

The answer should describe a gateway layer (Cortex/Mimir distributor or a custom proxy) that authenticates remote_write by team token, enforces per-tenant cardinality limits by tracking active series in a shared Redis or in-process counter, drops or rejects writes that exceed limits, emits metering events to a cost DB, and exposes a dashboard. Strong answers mention Grafana Mimir's native tenant isolation, how to handle bursting gracefully (soft limits with alerting vs hard limits with drops), and how to communicate limits to teams before they hit them.

Expect these follow-ups

• How would you handle a team that legitimately needs to exceed their cardinality limit during a product launch?
• What happens to existing data when a team is over-limit and you drop new writes?

As asked

We process credit card payments and our logs may contain PAN data, CVVs, and cardholder names. Design a log pipeline that scrubs this data before logs are stored, is auditable, and does not introduce more than 50ms of latency.

Sample answer outline

The candidate should propose scrubbing at the source (structured logging with explicit field marking for sensitive data, never logging raw request bodies) as the first line, combined with a real-time scrubbing processor in the OTel Collector or Fluent Bit using regex patterns for PAN (Luhn check optional), CVV, and name formats. Audit trail requires logging what was scrubbed (the fact of scrubbing, not the data). Latency is managed by running scrubbing synchronously in the pipeline before buffering, since 50ms is achievable with regex at the byte rate of a log stream. The candidate should mention that redaction is not enough; logs must never be written to disk pre-scrubbed.

Expect these follow-ups

• How do you handle structured JSON logs where the card number might be nested at any depth?
• How do you test that scrubbing actually works without using real PAN data?

As asked

Error rates spike on some deployments but not others. Design a system that automatically correlates metric anomalies with deployment events, configuration changes, and infrastructure changes to help engineers find root cause faster.

Sample answer outline

The design needs an event ingest layer (webhooks from CI/CD, Kubernetes events, config management) writing to a time-series event store, an anomaly detection layer on metrics (e.g., percent change alerts or ML-based), and a correlation engine that, when an anomaly is detected, queries events in the preceding N minutes and ranks them by proximity and label overlap (same service, same cluster). Grafana annotations are a simple version of this. A strong answer discusses the challenge of false correlations (many events happen continuously) and proposes a scoring model based on temporal proximity and service overlap.

Expect these follow-ups

• How would you build this without creating a tightly coupled dependency between your CI/CD and observability systems?
• How do you handle the case where the deployment happened 3 hours before the failure?

As asked

We want to add continuous profiling (CPU, heap, goroutine) to 30 Go services and be able to correlate a slow trace span with the CPU profile taken during that exact time window. Design the collection, storage, and query flow.

Sample answer outline

The candidate should describe using the pprof HTTP endpoint on each Go service, collected by a profiling agent (Grafana Pyroscope or Parca) on a configurable interval (e.g., every 10 seconds). Profiles are stored in object storage with labels matching the service, pod, and time range. Span-to-profile correlation works by embedding the profiling labels in the trace span as attributes and using the span's timestamp to query the profile store for the overlapping window. A strong answer covers the overhead of profiling (2 to 5% CPU for CPU profiles), sampling rate tradeoffs, and how to gate profiling via a feature flag per service.

Expect these follow-ups

• What is the difference between always-on profiling and on-demand profiling triggered by a slow trace?
• How does Pyroscope's eBPF profiler differ from pprof and what are its tradeoffs?