Site reliability engineer interview questions

As asked

A new Deployment rollout is stuck at 3 out of 10 pods Ready. Walk me through how you would debug it in production.

Sample answer outline

Start with kubectl rollout status and kubectl describe deployment. Look at the events: ImagePullBackOff, CrashLoopBackOff, FailedScheduling all point in different directions. Check pod events and pod logs for the failing replicas. Common causes: readiness probe failing because the app needs longer to start, resource requests too high for the available nodes, a config map or secret reference that does not exist, or a PodDisruptionBudget blocking eviction of old pods. Resist the urge to delete pods until you know the cause.

Reference implementation (bash)

# Triage sequence
kubectl rollout status deployment/api -n prod
kubectl describe deployment/api -n prod
kubectl get pods -n prod -l app=api
kubectl describe pod <pending-pod> -n prod
kubectl logs <crashlooping-pod> -n prod --previous
kubectl get events -n prod --sort-by=.lastTimestamp | tail -20

Expect these follow-ups

• What if the readiness probe passes but the service is still returning 503s?
• How do you set sane defaults for readiness and liveness probes?
• When would you use a startup probe?

As asked

Tell me about the worst on-call shift you have personally had. What broke, what did you do in the moment, and what changed afterwards?

Sample answer outline

Pick a real story with measurable user impact. Cover detection (was it self-detected or customer-reported), the diagnosis path (especially the dead ends), the mitigation, and the postmortem actions that actually landed. Interviewers are listening for ownership without blame, calm under pressure, and process changes that prevent the next one. Avoid 'we just restarted it' answers, dig into the root cause.

Expect these follow-ups

• What did the postmortem find as the contributing factors?
• Did the action items actually ship, or did they slip?
• What would you do differently in the first 10 minutes if it happened again?

As asked

You own the checkout service for an e-commerce site. Define a meaningful set of SLOs and explain the error budget policy you would put in place.

Sample answer outline

Pick SLIs that match what the user experiences: availability (proportion of checkout submissions that return a 2xx within 5 seconds), latency (95th percentile under 800ms), and correctness (no duplicate charges). Set the SLO from the SLI: 99.9 percent availability over 28 days gives a 43-minute error budget. Error budget policy: if we burn the budget, freeze feature work on checkout and direct effort to reliability. Spell out the burn-rate alerts (fast: 2 percent of budget in 1 hour, slow: 10 percent in 6 hours) so on-call gets paged on real impact, not noise.

Expect these follow-ups

• What do you do when the business says the SLO is too strict and they want to ship features?
• Why is a 99.99 percent SLO usually wrong for a product like this?
• How do you measure the correctness SLI in practice?

As asked

Walk me through the most recent incident where you were the incident commander. What was the impact, how did you run it, and what fell out of the postmortem?

Sample answer outline

Set the impact in concrete terms (users affected, revenue, duration) up front. Show the roles you assigned: IC, comms, ops, scribe. Walk through the timeline with the key decisions and the rationale for each. The strong answer is honest about the dead ends and the wasted time. Postmortem: blameless review, contributing factors (not 'root cause', plural), and action items with owners and dates. Mention which action items actually landed and which slipped.

Expect these follow-ups

• How did you keep stakeholders informed without distracting the responders?
• Did the team trust your decisions in the moment? How do you build that trust?
• Which action item would you not assign next time?

As asked

A team adds user_id and request_id labels to Prometheus metrics and the metrics backend starts falling over. How do you fix it without losing useful debugging signal?

Sample answer outline

Explain that each unique label set creates a time series, so unbounded labels like user_id, request_id, email, or URL path parameters can explode storage and query cost. Remove the high-cardinality labels from metrics and move per-request details to logs or traces, linked by trace_id. Keep bounded labels such as service, route template, status class, region, and dependency. Add instrumentation review, cardinality budgets, and alerts on series growth so this is caught before ingestion fails. Strong candidates preserve the debugging use case rather than just saying 'delete the label'.

Expect these follow-ups

• Which labels would you allow on an HTTP server duration histogram?
• How do exemplars help connect metrics to traces?
• What query patterns become dangerous after cardinality spikes?

As asked

You join a startup with 50 microservices and no observability. What do you put in place in the first 90 days?

Sample answer outline

Three pillars but in the right order. Logs first: ship structured logs to a single store (Loki, Elastic, or a vendor) with a trace ID field. Metrics next: Prometheus with a service-level dashboard per team and four golden signals (latency, traffic, errors, saturation). Tracing last because it requires app instrumentation: OpenTelemetry SDKs in every service, sampling at 1 to 10 percent at the head, tail sampling for errors. Tie it all together with the trace ID propagated through logs and metrics. Get an on-call rota and a paging policy in place at the same time.

Expect these follow-ups

• How do you sell the observability investment to a CEO who wants features?
• What is your retention policy and why?
• When is OpenTelemetry not the right choice?