Site reliability engineer behavioral interview questions

As asked

Tell me about the worst on-call shift you have personally had. What broke, what did you do in the moment, and what changed afterwards?

Sample answer outline

Pick a real story with measurable user impact. Cover detection (was it self-detected or customer-reported), the diagnosis path (especially the dead ends), the mitigation, and the postmortem actions that actually landed. Interviewers are listening for ownership without blame, calm under pressure, and process changes that prevent the next one. Avoid 'we just restarted it' answers, dig into the root cause.

Expect these follow-ups

• What did the postmortem find as the contributing factors?
• Did the action items actually ship, or did they slip?
• What would you do differently in the first 10 minutes if it happened again?

As asked

Walk me through the most recent incident where you were the incident commander. What was the impact, how did you run it, and what fell out of the postmortem?

Sample answer outline

Set the impact in concrete terms (users affected, revenue, duration) up front. Show the roles you assigned: IC, comms, ops, scribe. Walk through the timeline with the key decisions and the rationale for each. The strong answer is honest about the dead ends and the wasted time. Postmortem: blameless review, contributing factors (not 'root cause', plural), and action items with owners and dates. Mention which action items actually landed and which slipped.

Expect these follow-ups

• How did you keep stakeholders informed without distracting the responders?
• Did the team trust your decisions in the moment? How do you build that trust?
• Which action item would you not assign next time?

As asked

Tell me about a time you were the primary responder for a major production outage. Walk me through what happened, how you communicated under pressure, what you did to resolve it, and what you changed afterward.

Sample answer outline

A strong answer covers the STAR structure: the situation (severity, customer impact), the task (what they owned), the actions (concrete debugging steps, communications cadence, escalation decisions), and the result (time to resolution, business impact). The candidate should show calm decision-making under pressure, proactive communication with stakeholders, and concrete postmortem improvements that stuck.

Expect these follow-ups

• What is the single thing you would do differently if you could redo that incident?
• How did you communicate with non-technical stakeholders during the outage?

As asked

Give me a specific example of operational toil you identified, how you measured its impact, what you built or changed to eliminate it, and how you made sure the improvement lasted over time.

Sample answer outline

A good answer names a specific repeating task (certificate rotation, manual restart runbook, access request handling), quantifies the time cost, describes the automation or process change, and explains how it was operationalized (runbook updated, code in the standard toolchain, team trained). Sustainability is key: the improvement should not require the original engineer to stay around.

Expect these follow-ups

• How did you get the team to adopt the new process rather than falling back to the old way?
• Was there any toil that you decided not to automate, and why?

As asked

It is 2am. Your checkout service is down, affecting 100% of users. You can see that the database is healthy, the load balancer is healthy, but all API pods are in a crash loop. New pods come up and crash within 30 seconds. Deployments have been frozen for 12 hours. What do you do in the first 10 minutes?

Sample answer outline

The candidate should describe immediate triage: check the crash loop reason (OOMKilled? configuration error? dependent service unreachable?). If no recent deploy, look for external changes: is a ConfigMap or Secret different? Is a downstream dependency returning an unexpected response that the service cannot parse? They should also mention enabling traffic bypass to a static maintenance page to reduce customer impact while debugging. Escalating to the team lead if not resolved in 5 minutes is appropriate.

Expect these follow-ups

• How do you decide when to roll back versus when to push a forward fix?
• What is your communication cadence to stakeholders during a 2am incident?

As asked

You get paged at 2am for an alert called `payment_processor_saturation_high` that has no runbook and that you have never seen before. The service appears to be handling traffic. Walk me through your first 15 minutes.

Sample answer outline

The candidate should describe checking the alert definition in Prometheus or Alertmanager to understand the query, looking at the metric graph to see if it is actually exceeding a threshold or a flapping alert, checking the payment processor dashboard for error rate and latency, looking at recent changes to the service, and waking the service owner if there is genuine customer impact. They should also flag that a missing runbook is a postmortem action item regardless of outcome.

Expect these follow-ups

• At what point do you escalate to the service owner versus trying to handle it yourself?
• How do you document what you learned from this page so the next on-call does not start from zero?

As asked

Describe a situation where you believed a planned launch was not ready from a reliability standpoint and you pushed back against schedule pressure. What did you do, how did engineering leadership respond, and what was the outcome?

Sample answer outline

Strong answers show that the candidate grounded their concern in data (SLO headroom, missing runbooks, untested failure modes) rather than gut feeling, proposed a concrete risk mitigation rather than just saying no, and escalated appropriately. The outcome does not have to be that they won. What matters is that they communicated risk clearly and documented it.

Expect these follow-ups

• What would you do if management overruled you and the launch went ahead?
• How do you quantify reliability risk in terms that resonate with business stakeholders?

As asked

Tell me about a time when you disagreed with a development team about a reliability standard or practice. How did you handle the disagreement, what was the root cause of the conflict, and how was it resolved?

Sample answer outline

A strong answer shows that the candidate tried to understand the developer's perspective first (they may have had real constraints), used data and business impact to make the reliability case, and found a compromise rather than an all-or-nothing standoff. The best answers describe the relationship after resolution: ideally the dev team became an ally for reliability, not a reluctant follower.

Expect these follow-ups

• What would you do if the conflict never got resolved and the dev team continued the practice you objected to?
• How do you build trust with development teams who see SRE as a gatekeeper rather than a partner?

As asked

Describe a time you established SLOs for a service or product that had no reliability targets defined. How did you decide on the SLI, what data did you use to set the target, and how did you get buy-in from stakeholders?

Sample answer outline

Strong answers describe starting from customer expectations (what does a bad experience look like to the user?), picking an SLI that actually measures that experience (not a proxy like CPU), using historical performance data to set an achievable baseline target, and iterating with product and engineering leadership until everyone agreed the target was meaningful. They should mention how the SLO was documented and where it lives.

Expect these follow-ups

• How did you handle it when historical data showed you were already below the proposed SLO target?
• What was the first concrete action you took after the SLO was defined?

As asked

Tell me about a time when you recognized that the on-call rotation was burning people out. What signals told you it was a problem, what did you do about it, and what was the result for the team?

Sample answer outline

Good answers track signals like pages per shift, overnight pages, and time spent on toil during on-call. The candidate should describe how they quantified the problem (making it real for management), what they proposed (alert pruning, runbook improvements, adding more people to the rotation, shifting some on-call to a separate tier), and what the outcome was in terms of measured improvement and team morale.

Expect these follow-ups

• How do you measure engineer satisfaction with on-call without relying purely on anecdotes?
• What would you do if management acknowledged the problem but did not prioritize fixing it?

As asked

Postmortem action items often get de-prioritized after the urgency of an incident fades. Tell me about a time you had to advocate for completing important reliability work that was getting delayed. How did you approach it and what was the outcome?

Sample answer outline

A strong answer shows the candidate tracked the action items actively (not just leaving them in a doc), escalated with data (the risk if this is not fixed), and found ways to make the work easier to pick up (breaking it into smaller tasks, offering to pair). They should acknowledge the tension between reliability work and feature velocity and describe how they navigated it.

Expect these follow-ups

• When is it acceptable to close a postmortem action item as won't-do?
• How do you track the cumulative risk of stalled postmortem actions across many incidents?

As asked

Tell me about a time you embedded SRE practices (SLOs, error budgets, blameless postmortems, or runbook culture) into a development team that had none of those things. What did you teach them, how did you make it stick, and how did you measure reliability improvement over time?

Sample answer outline

Good answers show the candidate met the developers where they were (not lecturing from the SRE book), used a real incident or near-miss from their own service to make it concrete, embedded practices into their workflow (adding SLO dashboards to the team's review process, running blameless postmortems with them rather than for them), and tracked measurable improvement over a quarter.

Expect these follow-ups

• How do you handle a developer who is resistant to adopting reliability practices?
• What is the most common reliability mistake you see development teams make?

As asked

Your team has burned through 100% of the 30-day error budget by day 18 due to three separate incidents. What do you do for the rest of the month, and how do you have the conversation with the product team about feature releases?

Sample answer outline

A good answer covers freezing new feature releases as per the SLO policy, redirecting engineering capacity to reliability improvements, communicating the situation transparently to stakeholders with the specific incidents listed, and using the remaining time to implement postmortem action items. The candidate should know that the error budget policy should be agreed in advance so this is not a surprise conversation. They should also discuss whether to grant an exception if a release is critical for business reasons.

Expect these follow-ups

• What do you do if the product team refuses to freeze releases despite the budget being gone?
• How do you handle a situation where the error budget was exhausted by infrastructure failures outside the team's control?

As asked

You notice a burn rate alert firing 20 minutes after a new deployment started, which used a rolling update strategy and is about 60% complete. Error rate went from 0.1% to 3%. What do you do?

Sample answer outline

The candidate should immediately pause or roll back the deployment (kubectl rollout undo or the equivalent), not wait for it to finish. They should check whether the 3% errors are coming from the new pods versus the old pods using the pod label in Prometheus. If the rollback brings errors back to baseline, they should confirm the rollback is complete and file a bug. If errors persist after rollback, the bug is in data or configuration, not just the code.

Expect these follow-ups

• How do you distinguish between an error spike that will resolve itself (cold start) and one that requires rollback?
• What deployment strategy would have made this less risky?

As asked

A VP of Engineering tells you they want five nines of availability (99.999%) for a service that currently sits at 99.9%. You have a small team and a constrained budget. How do you respond?

Sample answer outline

A strong answer translates the numbers: 99.999% is 5 minutes of downtime per year, which requires fully redundant infrastructure with sub-second automated failover, no single points of failure, and extensive testing. They should ask what problem the VP is actually trying to solve (have there been customer complaints? Is there a contractual reason?) and present the cost of each additional nine. They should also propose starting with understanding current failure modes rather than chasing a number.

Expect these follow-ups

• What does it cost in engineering time and infrastructure to go from 99.9% to 99.99%?
• How do you measure availability when your monitoring infrastructure itself can go down?

As asked

A critical third-party payment API your service depends on has been down for 45 minutes with no ETA from the vendor. You have no fallback. What do you do from an operational and communication standpoint?

Sample answer outline

The candidate should describe implementing a maintenance/degraded mode for the payment flow rather than showing users generic errors, communicating proactively to customers via a status page, escalating to the vendor's enterprise support channel, and notifying internal stakeholders with a timeline. They should also use this incident to prioritize adding a fallback or circuit breaker to prevent future single-point-of-failure dependency. They should make clear that the user communication is their responsibility even if the failure is not.

Expect these follow-ups

• How do you contractually protect against recurring third-party downtime?
• What does a good vendor SLA look like from an SRE perspective?

As asked

A junior engineer who joined two weeks ago deployed a change that caused a 30-minute production outage affecting 100% of users. They bypassed the canary step in the deployment pipeline because they did not know it was required. They are clearly shaken. How do you handle the situation as the on-call senior SRE, both during and after the incident, and what systemic gaps does this reveal?

Sample answer outline

The candidate should describe focusing on recovery first (the engineer's feelings are secondary during the incident), then after resolution explicitly separating the person from the problem, helping the junior engineer through the postmortem as a learning experience rather than a blame session, and using the incident to identify the systemic gaps (why could a two-week engineer deploy to production without review? were there missing safeguards?). The blameless culture piece is key.

Expect these follow-ups

• What specific safeguards would you add to prevent a similar mistake from a new engineer in the future?
• How would you handle the postmortem if the engineer tried to minimize or hide what they did?

As asked

Tell me about a time you introduced chaos engineering or fault injection to a system that had never had it before. How did you get organizational buy-in, what did you test first, and what did you find?

Sample answer outline

A strong answer shows the candidate started with a hypothesis based on a known weakness (a dependency the team never tested losing), ran the first experiment in staging with a small blast radius, communicated the plan and abort conditions in advance, and shared the findings regardless of whether the system passed. They should describe what they found (either a confirmed resilience or a real gap) and what changed as a result.

Expect these follow-ups

• How do you maintain team confidence in chaos engineering after an experiment goes wrong and causes an actual outage?
• What is the hardest part of getting buy-in for chaos experiments in production?

As asked

Describe a situation where your capacity planning predicted you had enough headroom but the service still fell over under load. What did you miss, how did you respond in the moment, and how did you update your planning process?

Sample answer outline

A good answer shows intellectual honesty about what was missed (a hidden dependency that was the real bottleneck, a caching assumption that did not hold under load, a non-linear scaling behavior). The response during the incident should show fast mitigation. The follow-up should describe a concrete change to the planning process that would catch that class of mistake in the future.

Expect these follow-ups

• How do you account for dependencies you do not own in your capacity planning?
• What is the difference between a load test and a capacity test, and how do you design each?

As asked

Describe a situation where you discovered or were paged about a potential security incident during an on-call shift. What was your first response, how did you coordinate with the security team, and what did you learn about the gap between reliability incidents and security incidents?

Sample answer outline

A strong answer shows the candidate immediately involved the security team rather than trying to handle it alone, preserved evidence before making changes (took snapshots, saved logs), followed established incident response procedures, and understood the difference between remediating a reliability issue (bring it back up fast) and a security incident (understand the full scope before changing anything). They should describe what they changed about their runbooks afterward.

Expect these follow-ups

• How do you balance the SRE goal of fast recovery with the security team's need to preserve forensic evidence?
• What signals would lead you to suspect a reliability incident is actually security-related?

As asked

You want to automate a manual restart runbook to reduce toil. The runbook works fine when followed by a human, but one of the steps involves deleting pods in sequence and if done in the wrong order it can cause data loss. How do you approach automating this safely?

Sample answer outline

The candidate should first document the ordering invariant and understand why it matters. The automation should include guard rails: verify pre-conditions before each dangerous step, add a dry-run mode, include a confirmation prompt or require an operator flag for the dangerous step, and add logging with full audit trail. They should test in staging with deliberately induced wrong-order scenarios and get a review from the team before running in production.

Expect these follow-ups

• When is it better to leave a runbook manual rather than automating it?
• How do you version control runbooks alongside the code they operate on?