Site reliability engineer role-specific interview questions

As asked

A new Deployment rollout is stuck at 3 out of 10 pods Ready. Walk me through how you would debug it in production.

Sample answer outline

Start with kubectl rollout status and kubectl describe deployment. Look at the events: ImagePullBackOff, CrashLoopBackOff, FailedScheduling all point in different directions. Check pod events and pod logs for the failing replicas. Common causes: readiness probe failing because the app needs longer to start, resource requests too high for the available nodes, a config map or secret reference that does not exist, or a PodDisruptionBudget blocking eviction of old pods. Resist the urge to delete pods until you know the cause.

Reference implementation (bash)

# Triage sequence
kubectl rollout status deployment/api -n prod
kubectl describe deployment/api -n prod
kubectl get pods -n prod -l app=api
kubectl describe pod <pending-pod> -n prod
kubectl logs <crashlooping-pod> -n prod --previous
kubectl get events -n prod --sort-by=.lastTimestamp | tail -20

Expect these follow-ups

• What if the readiness probe passes but the service is still returning 503s?
• How do you set sane defaults for readiness and liveness probes?
• When would you use a startup probe?

As asked

You own the checkout service for an e-commerce site. Define a meaningful set of SLOs and explain the error budget policy you would put in place.

Sample answer outline

Pick SLIs that match what the user experiences: availability (proportion of checkout submissions that return a 2xx within 5 seconds), latency (95th percentile under 800ms), and correctness (no duplicate charges). Set the SLO from the SLI: 99.9 percent availability over 28 days gives a 43-minute error budget. Error budget policy: if we burn the budget, freeze feature work on checkout and direct effort to reliability. Spell out the burn-rate alerts (fast: 2 percent of budget in 1 hour, slow: 10 percent in 6 hours) so on-call gets paged on real impact, not noise.

Expect these follow-ups

• What do you do when the business says the SLO is too strict and they want to ship features?
• Why is a 99.99 percent SLO usually wrong for a product like this?
• How do you measure the correctness SLI in practice?

As asked

A team adds user_id and request_id labels to Prometheus metrics and the metrics backend starts falling over. How do you fix it without losing useful debugging signal?

Sample answer outline

Explain that each unique label set creates a time series, so unbounded labels like user_id, request_id, email, or URL path parameters can explode storage and query cost. Remove the high-cardinality labels from metrics and move per-request details to logs or traces, linked by trace_id. Keep bounded labels such as service, route template, status class, region, and dependency. Add instrumentation review, cardinality budgets, and alerts on series growth so this is caught before ingestion fails. Strong candidates preserve the debugging use case rather than just saying 'delete the label'.

Expect these follow-ups

• Which labels would you allow on an HTTP server duration histogram?
• How do exemplars help connect metrics to traces?
• What query patterns become dangerous after cardinality spikes?

As asked

Walk me through how you would set up multi-window burn rate alerts for an error budget. Why use two windows instead of one, and what values would you choose for a 30-day SLO?

Sample answer outline

Strong answers explain that a fast burn rate window (like 1 hour) catches sudden outages while a slow window (like 6 hours) filters noise and confirms the burn is sustained. The candidate should know the math: a 14.4x burn rate over 1 hour consumes 2% of a 30-day budget. They should mention the Google SRE book's recommended thresholds (2% budget consumed in 1h triggers page, 5% in 6h triggers ticket). They should also explain that single-window alerts either fire too late or produce too many false positives.

Expect these follow-ups

• How do you decide what SLO target to set in the first place, and who signs off on it?
• If your error budget is exhausted three weeks into a sprint, what concrete actions do you take?

As asked

Pods keep getting OOMKilled in production. Describe your full debugging process, from the first kubectl command through to the root-cause fix.

Sample answer outline

The candidate should start with `kubectl describe pod` to confirm the OOMKilled exit code and see the last restart reason, then check resource requests and limits. They should look at container memory usage via metrics-server or Prometheus `container_memory_working_set_bytes`. The fix side should cover setting correct limits backed by load-test data, checking for memory leaks in application code, enabling VPA or HPA appropriately, and understanding the difference between the memory limit (cgroup) and the OOM killer priority.

Expect these follow-ups

• What is the difference between a container's memory limit and the node's allocatable memory?
• When would you deliberately set no memory limit on a container, and what risk does that carry?

As asked

You had a two-hour production outage last night that cost the company. Walk me through how you would run the postmortem meeting, what document you would produce, and how you would make sure action items actually get done.

Sample answer outline

A good answer covers the blameless mindset (systems, not people), a structured timeline reconstructed from logs and alerts, five-why or contributing-factors analysis rather than root cause (which implies one cause), and distinguishing immediate fixes from longer-term systemic improvements. Action items should have owners, deadlines, and be tracked in a project-management tool. The candidate should mention a postmortem review meeting separate from the writing session and how to share learnings across teams.

Expect these follow-ups

• How do you handle it when the postmortem reveals that a specific engineer made a mistake that caused the outage?
• What makes a postmortem action item worth doing versus one that gets ignored?

As asked

Your on-call engineer is getting paged 15 times per night and most alerts resolve themselves. How do you audit and fix this situation without lowering the bar on what gets caught?

Sample answer outline

Good answers start with alert audit: categorize alerts by actionability (did the on-call actually do anything, or did it self-resolve?). Alerts that never require action should be demoted to tickets or deleted. The candidate should discuss symptom-based alerting (alert on SLO burn rate, not on CPU usage) and the difference between alerting and dashboarding. They should also cover tuning thresholds with historical data, adding deduplication and grouping in the alertmanager, and tracking toil metrics (hours of on-call interruption per week).

Expect these follow-ups

• What is the difference between a symptom-based alert and a cause-based alert, and which should you prefer?
• How do you handle alerts that are genuinely noisy but have occasionally flagged real incidents?

As asked

Implement and explain a circuit breaker that wraps calls to a downstream service. Cover the state machine, configuration, and what metrics you expose.

Sample answer outline

States: closed (requests pass through), open (requests fail fast), half-open (a small probe is allowed to test recovery). Transitions driven by a rolling error rate over a window, with thresholds for opening and a cooldown before half-open. Configuration knobs: failure threshold, rolling window size, cooldown duration, half-open probe count. Metrics: state changes, fail-fast count, downstream success rate. The trap to avoid: opening the breaker on the first error spike, which causes a thundering herd when the breaker tries to close.

Reference implementation (typescript)

type State = "closed" | "open" | "half-open";

class CircuitBreaker {
  private state: State = "closed";
  private failures = 0;
  private openedAt = 0;

  constructor(
    private readonly threshold: number,
    private readonly cooldownMs: number,
  ) {}

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === "open") {
      if (Date.now() - this.openedAt < this.cooldownMs) throw new Error("circuit-open");
      this.state = "half-open";
    }
    try {
      const result = await fn();
      this.failures = 0;
      this.state = "closed";
      return result;
    } catch (err) {
      this.failures += 1;
      if (this.failures >= this.threshold) {
        this.state = "open";
        this.openedAt = Date.now();
      }
      throw err;
    }
  }
}

Expect these follow-ups

• How do you avoid the synchronised retry storm when the breaker closes?
• When would you not use a circuit breaker?
• How would you implement this in a service mesh vs in application code?

As asked

p99 latency on a service has tripled over the last hour. Median is flat. Error rate is flat. CPU and memory look fine. Where do you look?

Sample answer outline

Tail latency without errors usually means contention or saturation somewhere off the obvious dashboards. Check: GC pauses (especially long minor GCs on the JVM), thread pool or connection pool saturation, downstream service slowness on a subset of requests, lock contention, slow database queries on a path used only by some requests, noisy neighbours on shared hosts, network microbursts. Use traces to find the slow requests and look at their span breakdown. The p99 is a different population from the median, so segment by route, customer, region until the slow subset is named.

Reference implementation (promql)

# PromQL: segment p99 by route to find the slow subset
histogram_quantile(
  0.99,
  sum by (le, route) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

# Compare against p50 to confirm tail is diverging
histogram_quantile(0.50, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))

# Check connection pool saturation (pgx/pgbouncer exporter)
max by (instance) (pgbouncer_pools_server_active_connections / pgbouncer_pools_server_max_connections)

Expect these follow-ups

• How would you set up an alert for tail latency without spamming the team?
• What is your strategy when the slow subset is concentrated on one customer?
• When is a head-of-line blocking the answer?

As asked

A Prometheus instance starts OOMing every few hours. You suspect high cardinality. Walk me through how you would diagnose which metrics are the problem and what you would do to fix it without losing the data you actually need.

Sample answer outline

A good answer covers querying `prometheus_tsdb_symbol_table_size_bytes` and `topk(20, count by (__name__)({__name__=~".+"}))` to find the worst offenders. The candidate should mention dropping high-cardinality labels like user IDs, request IDs, or unbounded URL paths at the scrape level using `metric_relabel_configs`. They should discuss recording rules to pre-aggregate before storage, and possibly sharding Prometheus or using a remote write backend like Thanos or Cortex for long-term storage.

Expect these follow-ups

• What is the difference between metric_relabel_configs and relabel_configs and when do you use each?
• How does Thanos handle global querying across multiple Prometheus instances?

As asked

Your p99 API latency is 4 seconds but p50 is 80ms. The obvious dashboards show nothing. How do you use distributed tracing to find where that tail latency is coming from, and what tools would you reach for?

Sample answer outline

A strong answer describes filtering traces by duration (only traces above the p99 threshold), then comparing span breakdowns between slow and fast traces. The candidate should mention Jaeger, Tempo, or Honeycomb for trace storage and how to correlate trace IDs with logs. They should bring up common tail latency culprits: GC pauses, thread pool exhaustion, mutex contention, slow downstream fans-out, and head-of-line blocking. They should also mention exemplars in Prometheus 2.x for linking metrics to traces.

Expect these follow-ups

• How do you sample traces without losing the slow ones you care most about?
• Explain tail-based sampling versus head-based sampling and their tradeoffs.

As asked

Your team wants to run its first game day to test resilience. Walk me through how you would design, run, and measure the outcome of a chaos experiment for a microservices-based checkout service.

Sample answer outline

Strong answers cover hypothesis-driven experiments (define steady state first), blast radius control (start in staging, then canary in prod), and tooling like Chaos Monkey, Gremlin, or LitmusChaos for Kubernetes. The candidate should describe what failure modes to inject (pod kill, latency injection, dependency failure) and what metrics to watch during the experiment. They should also cover rollback criteria and what to do if the steady state is violated before the planned experiment window ends.

Expect these follow-ups

• What is the difference between fault injection and chaos engineering?
• How do you get organizational buy-in to run chaos experiments in production?

As asked

A marketing campaign is going live in two weeks and traffic is expected to be 5x normal. How do you approach capacity planning? What data do you pull, what assumptions do you validate, and what could still go wrong?

Sample answer outline

A strong answer starts with baselining current resource utilization at peak and identifying the bottleneck service. The candidate should discuss load testing to find the actual breaking point, not just back-of-envelope math. They should cover auto-scaling configuration (HPA in Kubernetes, ASG policies on AWS), pre-warming caches, coordinating with the database team on connection pool limits, and having a go/no-go process with a runbook ready. They should also identify risks like third-party dependencies that cannot be scaled.

Expect these follow-ups

• How do you load test a service that has downstream dependencies you cannot easily mock?
• What would you do if the load test reveals your database is the bottleneck at 3x traffic?

As asked

Postgres is showing query latency spikes at peak hours. You have access to pg_stat_statements and pg_stat_activity. Walk me through how you identify the bad queries and what you look at to understand why they are slow under load but fast otherwise.

Sample answer outline

A strong answer queries pg_stat_statements ordered by mean_exec_time or total_exec_time, then uses EXPLAIN ANALYZE on the worst offenders. Under load specifically, the candidate should look at lock waits in pg_stat_activity (wait_event_type = 'Lock'), autovacuum activity causing bloat, and connection pool exhaustion via `select count(*) from pg_stat_activity`. They should also mention pg_locks for deadlock detection and the impact of missing indexes on sequential scans that are cheap on small tables but deadly at scale.

Expect these follow-ups

• What is a hot standby and how do you use read replicas to reduce load on the primary?
• When does adding an index make write performance worse and how do you decide if it is worth it?

As asked

A network partition splits your Kafka cluster into two groups. Describe what happens to producers and consumers, how Kafka's leader election responds, and what your SRE responsibilities are during and after the partition.

Sample answer outline

Strong answers cover that Kafka uses ZooKeeper or KRaft for leader election, partitions without a leader in the minority group become unavailable (producers with acks=all will fail), and consumers from the minority side will see lag increase. The candidate should explain unclean leader election risk (data loss if enabled), ISR (in-sync replicas) behavior, and how min.insync.replicas protects durability. SRE duties include monitoring consumer lag, coordinating with the network team, and running a partition reassignment after recovery.

Expect these follow-ups

• What is the difference between acks=1, acks=all, and acks=0 in Kafka and when would you choose each?
• How does Kafka handle split-brain if ZooKeeper itself is partitioned?

As asked

A teammate manually changed a security group rule in AWS to unblock an incident. Now Terraform state is out of sync with reality. How do you reconcile this safely, and how do you prevent it from happening again?

Sample answer outline

The candidate should describe running `terraform plan` to see the drift, then deciding between `terraform import` to bring the manual change into state or reverting the AWS resource to match the code. They should stress that the decision depends on whether the manual change was the right fix or a workaround. Prevention includes IAM policies that deny direct console changes to prod, drift detection via `terraform plan` in CI on a schedule, and locking the state file during applies.

Expect these follow-ups

• How does terraform state locking work with S3 and DynamoDB?
• What is the risk of running terraform import on a live resource?

As asked

Your company has no formal incident severity levels. Design a classification system from scratch, including what criteria separate SEV1 from SEV2, how it maps to response expectations, and the edge cases that always cause disagreement.

Sample answer outline

A strong answer defines SEV1 as complete loss of a revenue-generating or safety-critical customer-facing feature requiring immediate response and executive notification. SEV2 is significant degradation affecting a subset of users. SEV3 and below are non-urgent. The candidate should address the typical edge case: is a 100% failure of an internal tool a SEV1? They should also discuss how severity drives on-call wake-up rules, escalation paths, and postmortem requirements.

Expect these follow-ups

• How do you handle severity escalation during an incident, and who has the authority to change it?
• What happens when business stakeholders consistently disagree with the engineering team's severity assessment?

As asked

A VP asks you to improve system reliability. They want to increase MTBF. Make the case for why you would also focus on MTTR, and explain how each metric drives different engineering investments.

Sample answer outline

The candidate should explain MTBF (Mean Time Between Failures) measures how often failures occur and requires investment in prevention: testing, redundancy, change management. MTTR (Mean Time To Recover) measures how fast you recover and requires investment in detection, diagnosis, and automated remediation. SRE philosophy leans heavily on MTTR because distributed systems will fail regardless. They should connect this to error budget math: a single 4-hour outage consumes four times the budget of a 1-hour outage caused by the same class of failure, so improving MTTR directly extends how long teams can keep shipping without burning the budget.

Expect these follow-ups

• How would you define MTTD (Mean Time To Detect) and where does it fit in the incident lifecycle?
• What is the difference between availability and reliability, and can a system have high availability but low reliability?

As asked

A downstream payment service is timing out and causing cascading failures across your checkout API. Explain how a circuit breaker pattern helps, what states it has, and how you would tune the thresholds for this specific case.

Sample answer outline

A strong answer covers the three states: closed (normal, requests flow through), open (failure threshold exceeded, requests fail fast without waiting), and half-open (one probe request to test recovery). They should discuss threshold choices: error count versus error rate, and the evaluation window. For the payment service specifically they should note that failing fast with a 503 is better than queuing requests and exhausting thread pools. They should mention libraries like resilience4j or Hystrix and the difference between circuit breaker and retry with backoff.

Expect these follow-ups

• When does a circuit breaker do more harm than good?
• How do you expose circuit breaker state in your observability stack so the on-call can see when one is tripped?

As asked

You are logging every HTTP request at 50,000 requests per second across 200 pods. Describe the full pipeline from application to query, the retention and cost tradeoffs, and how you avoid losing logs during high traffic.

Sample answer outline

A strong answer covers structured JSON logs from the app, Fluentd or Fluent Bit as a node-level DaemonSet for buffering and batching, and a backend like Loki (low cost, label-based) or Elasticsearch (rich full-text search, higher cost). The candidate should discuss log sampling for debug-level logs, the risk of back-pressure causing pod latency if the log buffer is full, and using file-based buffering on disk rather than in-memory to survive pod restarts. They should also address log retention tiers: hot storage for recent queries, cold storage (S3) for compliance.

Expect these follow-ups

• How do you correlate logs with traces when you have sampling enabled on both?
• What is the difference between Loki's LogQL and Elasticsearch query DSL in terms of indexing strategy?

As asked

Kubernetes is evicting pods from a node even though the container memory limits are well below the node capacity. Explain why this happens and what you would check.

Sample answer outline

The candidate should explain that kubelet eviction is driven by actual node-level memory usage, not the sum of container limits. System processes and kubelet itself consume memory. The eviction thresholds (memory.available < 100Mi by default) are configurable but not part of the container resource model. They should mention that setting requests too low means the scheduler overcommits the node. The fix involves setting appropriate requests that reflect actual usage, not just limits.

Expect these follow-ups

• What is the difference between a Pod's Quality of Service class (Guaranteed, Burstable, BestEffort) and how does it affect eviction priority?
• How does the Kubernetes scheduler use resource requests versus limits when placing pods?

As asked

Your team debates whether to measure your availability SLI from synthetic probes or from real user traffic. Describe the tradeoffs and when you would use each or both.

Sample answer outline

Synthetic probes (Blackbox Exporter, Datadog Synthetics) run on a fixed schedule from known locations and detect issues even in low-traffic periods. They are consistent and easy to reason about, but they probe specific paths and miss real user variety. Real-user SLIs measured from access logs or client-side RUM reflect actual customer experience but require traffic volume to be statistically meaningful and can have noisy signals from bot traffic or retries. Best practice is to use real-user data as the official SLI for the error budget and synthetics as an early detection layer.

Expect these follow-ups

• How do you handle SLI measurement when a significant fraction of your traffic is from automated clients or bots?
• How do you count a request that times out on the client side but succeeded on the server?

As asked

Not all production changes carry the same risk. Describe a change risk scoring system you have used or would design, and how it maps to required approval gates or deployment windows.

Sample answer outline

A good answer describes factors: change size (lines of code, number of services affected), change type (config versus code versus schema migration), recency of test coverage, time since last incident on the affected service, and whether it is reversible. Higher-risk changes require more approvers, mandatory staging bake time, and deployment during business hours with the author on-call. Low-risk changes (config flag flips) can deploy via automated pipeline with no human gate. The candidate should mention that schema migrations and dependency upgrades are almost always high-risk.

Expect these follow-ups

• How do you enforce this scoring in practice without it becoming a checkbox exercise?
• What is your policy for deploying to production during a freeze window for something that fixes an active SEV2?

As asked

Your AWS bill doubled overnight and no major features launched. Walk me through how you would identify what caused it and the categories of things that most commonly cause sudden cost spikes.

Sample answer outline

A strong answer covers AWS Cost Explorer with daily granularity, filtering by service and then by usage type. Common culprits: NAT Gateway data transfer spike from a new log sink or runaway S3 sync, EC2 auto-scaling that did not scale back down, RDS snapshot replication to a new region, and CloudWatch Logs retention not set (everything goes to forever). The candidate should mention setting a cost anomaly detection alert in AWS and using resource tags to attribute costs to teams.

Expect these follow-ups

• How do you enforce resource tagging at creation time in AWS so cost attribution is always accurate?
• What is the difference between Reserved Instances, Savings Plans, and Spot Instances in terms of cost and risk?

As asked

You track that your SRE team spends 55% of its time on operational work rather than engineering. Google's SRE book recommends keeping this under 50%. What is the process you would use to identify the highest-value toil to eliminate and how would you justify the engineering investment?

Sample answer outline

The candidate should describe toil tracking (time log or ticket tagging), grouping by category (manual restarts, config changes, certificate rotation, access requests), and finding the highest frequency or highest time-cost item. Justification uses the math: if a manual task takes 30 minutes 20 times per month, automating it saves 10 engineer-hours per month. They should also mention that not all operational work is toil: genuine engineering investigation is not toil. They should describe working with the product team to carve out dedicated toil-reduction sprint capacity.

Expect these follow-ups

• How do you prevent the toil backlog from growing faster than you can automate it away?
• What is your approach when the product team resists giving SRE engineering time because incidents are low right now?