Site reliability engineer system design interview questions

As asked

You join a startup with 50 microservices and no observability. What do you put in place in the first 90 days?

Sample answer outline

Three pillars but in the right order. Logs first: ship structured logs to a single store (Loki, Elastic, or a vendor) with a trace ID field. Metrics next: Prometheus with a service-level dashboard per team and four golden signals (latency, traffic, errors, saturation). Tracing last because it requires app instrumentation: OpenTelemetry SDKs in every service, sampling at 1 to 10 percent at the head, tail sampling for errors. Tie it all together with the trace ID propagated through logs and metrics. Get an on-call rota and a paging policy in place at the same time.

Expect these follow-ups

• How do you sell the observability investment to a CEO who wants features?
• What is your retention policy and why?
• When is OpenTelemetry not the right choice?

As asked

Design the load balancer layer for a service that must keep serving 99 percent of traffic during a full AZ outage.

Sample answer outline

Regional cross-zone load balancer (ALB or GLB) with backends in three AZs and health checks that drop unhealthy AZ targets within seconds. Connection draining for graceful failover. Capacity headroom: each AZ must be able to absorb 50 percent of total traffic with the third AZ down, so plan to N+1 at AZ granularity. DNS TTLs short enough that a regional failure can be cut over by routing changes. The honest answer notes that 'survive an AZ outage' is mostly about capacity planning, not load balancer cleverness.

Expect these follow-ups

• How does this change for a stateful service like a database?
• What is the failure mode you do not catch with health checks?
• When would you put a global load balancer in front of regional ones?

As asked

You run the URL shortener that powers links in a marketing campaign. One link goes viral and redirect traffic spikes 50x in minutes. As the on-call SRE, how is the system designed to absorb it and how do you respond if it starts to degrade?

Sample answer outline

Reliability framing first: the redirect path should be almost entirely cacheable, so a viral link is a cache-friendly workload, one hot key served from the edge. Define SLOs for redirect availability and latency, with burn-rate alerts so a spike that erodes the budget pages you on real impact. Capacity: the edge tier should scale horizontally and the origin should be protected by request coalescing so a cold key does not stampede the database. In the moment: confirm the hot key is being served from cache, watch origin saturation and the database connection pool, and shed non-critical load such as synchronous analytics writes before the redirect path suffers. The honest point is that absorbing a viral spike is mostly cache design and capacity headroom decided well before the incident.

Expect these follow-ups

• What SLI best captures user pain for a redirect service?
• How does request coalescing prevent a cold-key stampede?
• What would you shed first to protect the redirect path?

As asked

Design alerting for an API with a 99.9 percent monthly availability SLO. How would you avoid noisy alerts while still catching fast outages?

Sample answer outline

Start from the error budget: 99.9 percent availability gives roughly 43 minutes of monthly unavailability, so alerting should trigger on budget burn, not isolated error spikes. Use multi-window, multi-burn-rate alerts, for example a fast page on high burn over 5 minutes and 1 hour, and a ticket on slower burn over 6 hours and 3 days. Measure availability from user-facing successful requests, not just pod health. Include routing, runbooks, and silence policy so alerts lead to action. Candidates often miss that paging on every 500 rate spike creates fatigue and makes the SLO less useful.

Expect these follow-ups

• How do you define a good request versus a bad request for this API?
• What if latency violates the user promise but success rate stays high?
• How do error budgets change release decisions?