Site reliability engineer coding interview questions

As asked

Write a retry wrapper in Go that retries a function call with exponential backoff and full jitter. It should respect a maximum number of retries and a maximum delay cap. Explain why jitter matters for SRE.

Sample answer outline

The candidate should implement base delay doubling each attempt, capped at maxDelay, with jitter as `rand.Int63n(currentDelay)` to spread retries across the retry window. They should explain that without jitter, all clients back off to the same delay and then send a synchronized burst (thundering herd), which can overwhelm a recovering service. Context cancellation should also be respected.

Reference implementation (go)

package main

import (
    "context"
    "math/rand"
    "time"
)

func RetryWithBackoff(
    ctx context.Context,
    maxRetries int,
    baseDelay time.Duration,
    maxDelay time.Duration,
    fn func() error,
) error {
    // TODO: implement exponential backoff with full jitter
    return nil
}

Expect these follow-ups

• What is the difference between full jitter and decorrelated jitter?
• When should you NOT retry, even with backoff?

As asked

You have a Prometheus histogram metric `http_request_duration_seconds` with labels `service` and `endpoint`. Write a PromQL expression for the p99 latency per endpoint over the last 5 minutes, and explain why you cannot use a simple average of histogram buckets.

Sample answer outline

The correct expression uses `histogram_quantile(0.99, sum by (le, endpoint) (rate(http_request_duration_seconds_bucket[5m])))`. The candidate should explain that histogram_quantile interpolates within a bucket and that you must aggregate `_bucket` counters with `rate` before passing to histogram_quantile. Averaging histogram values directly loses the distribution. They should also mention that native histograms in Prometheus 2.40+ fix some of these limitations.

Reference implementation (promql)

# Metric shape:
# http_request_duration_seconds_bucket{le="0.1", service="checkout", endpoint="/pay"} 1234
# http_request_duration_seconds_bucket{le="0.5", service="checkout", endpoint="/pay"} 2345
# http_request_duration_seconds_bucket{le="1.0", service="checkout", endpoint="/pay"} 2400
# http_request_duration_seconds_bucket{le="+Inf", service="checkout", endpoint="/pay"} 2401

# Write the PromQL for p99 per endpoint:
# YOUR QUERY HERE

Expect these follow-ups

• What is the difference between a Prometheus histogram and a summary metric and when would you choose each?
• How do you estimate the error in histogram_quantile's output and what drives it?

As asked

Implement a sliding window rate limiter that allows at most N requests per window of W seconds per key. Use Redis as the backing store. The implementation should be atomic and handle the case where Redis is unavailable.

Sample answer outline

A strong answer uses a Lua script executed atomically on Redis with ZADD to add the current timestamp, ZREMRANGEBYSCORE to expire old entries, and ZCARD to count. If Redis is unavailable, the candidate should decide on a fail-open or fail-closed policy and explain the tradeoff. They should also mention that a fixed window counter is cheaper but can allow 2x burst at window boundaries, which is why sliding window is preferred for strict rate limiting.

Reference implementation (python)

import redis
import time

def is_allowed(r: redis.Redis, key: str, max_requests: int, window_seconds: int) -> bool:
    now = time.time()
    window_start = now - window_seconds
    pipe = r.pipeline()
    # TODO: implement atomically
    # 1. Remove entries older than window_start
    # 2. Count remaining entries
    # 3. If count < max_requests, add current timestamp and return True
    # 4. Otherwise return False
    pass

Expect these follow-ups

• How would you implement this without Redis using only in-process memory if you have multiple replicas?
• What is the token bucket algorithm and how does it differ from sliding window?

As asked

Write a function that checks health of N services concurrently with a timeout per check, and returns an overall status: healthy if all pass, degraded if some fail, unhealthy if all fail. It must complete within the timeout even if some checks hang.

Sample answer outline

The candidate should use goroutines with a context timeout for each check, collect results over a channel, and aggregate. All goroutines must be collected (no goroutine leak). The three-way status enum is a common pattern in readiness probes. They should also handle the edge case where N is zero and make the timeout configurable per check rather than global.

Reference implementation (go)

package main

import (
    "context"
    "time"
)

type Status string
const (
    Healthy   Status = "healthy"
    Degraded  Status = "degraded"
    Unhealthy Status = "unhealthy"
)

type CheckFn func(ctx context.Context) error

func AggregateHealth(
    ctx context.Context,
    checks []CheckFn,
    timeout time.Duration,
) Status {
    // TODO: run checks concurrently, return aggregate status
    return Healthy
}

Expect these follow-ups

• How would you expose this as a Kubernetes readiness versus liveness probe endpoint, and what is the difference in behavior?
• What happens to in-flight requests if your readiness probe fails?

As asked

Given a stream of HTTP access log lines (combined log format), write a Python function that returns the error rate (5xx responses / total responses) for a given service over the last N seconds, reading from stdin. Optimize it for low memory usage on a large log file.

Sample answer outline

The candidate should parse each line with a regex or split, extract the status code and timestamp, and use a deque or circular buffer keyed by time to drop entries older than N seconds. Memory is O(requests in the window) not O(total log file). They should handle malformed lines gracefully and note that a sliding window deque is better than sorting or storing all entries.

Reference implementation (python)

import sys
import re
from collections import deque
from datetime import datetime

# Combined log format example:
# 127.0.0.1 - - [10/Oct/2000:13:55:36 -0700] "GET /index.html HTTP/1.1" 200 2326

LOG_RE = re.compile(r'\[(.*?)\].*" (\d{3}) ')

def error_rate(lines, window_seconds: int) -> float:
    # TODO: parse lines, return 5xx rate over last window_seconds
    pass

Expect these follow-ups

• How would you make this run as a continuous tail rather than reading a static file?
• How would you test this function with a synthetic log that has a known error rate?

As asked

Write a script that reads messages from an SQS dead-letter queue and replays them to the original queue with rate limiting, idempotency handling, and a dry-run mode. Explain the risks of naive replay.

Sample answer outline

The candidate should implement polling with long-poll (WaitTimeSeconds=20), rate limiting via a token bucket or simple sleep, and idempotency via a deduplication ID or checking a processed-IDs store. Dry-run mode logs what would be sent without actually calling SQS SendMessage. Risks of naive replay: flooding the downstream system, replaying messages that failed because the system state has changed making them invalid, and creating duplicate side effects if the consumer is not idempotent.

Reference implementation (python)

import boto3
import time

def replay_dlq(
    dlq_url: str,
    target_url: str,
    rate_per_second: int = 10,
    dry_run: bool = True,
) -> None:
    sqs = boto3.client('sqs')
    # TODO:
    # 1. Poll DLQ with long polling
    # 2. Rate limit sends
    # 3. In dry_run mode, print but do not send
    # 4. Delete from DLQ only after successful send
    pass

Expect these follow-ups

• How does SQS message deduplication ID work for FIFO queues and what is its retention period?
• When would you choose to replay only a subset of DLQ messages rather than all of them?

As asked

Write a script that polls the Kubernetes API every 30 seconds and fires a webhook alert if any pod in a given namespace has restarted more than 3 times in the last 10 minutes. Handle the case where a pod is replaced by a new pod with a different name.

Sample answer outline

The candidate should use the kubernetes Python client to list pods and read `status.containerStatuses[].restartCount` and `status.containerStatuses[].lastState.terminated.finishedAt`. They should track restart deltas per pod (not absolute count, which grows forever) and handle pod replacement by tracking the owning ReplicaSet or Deployment rather than the pod name. The webhook call should be idempotent and the alert should include pod name, namespace, restart count, and last exit code.

Reference implementation (python)

from kubernetes import client, config
import requests
import time
from datetime import datetime, timedelta, timezone

def watch_restarts(
    namespace: str,
    max_restarts: int,
    window_minutes: int,
    webhook_url: str,
) -> None:
    config.load_kube_config()
    v1 = client.CoreV1Api()
    restart_baseline: dict[str, int] = {}

    while True:
        # TODO: poll pods and detect anomalous restarts
        time.sleep(30)

Expect these follow-ups

• How would you avoid sending duplicate alerts for the same pod during multiple polling cycles?
• What is the difference between CrashLoopBackOff and OOMKilled as exit reasons and how do you tell them apart programmatically?

As asked

Implement a consistent hash ring for distributing load across a set of backend nodes. It should support adding and removing nodes with minimal key remapping, and use virtual nodes to improve balance. Show how you would use it for routing in a caching layer.

Sample answer outline

A strong answer uses a sorted list of (hash, node) tuples with virtual nodes (e.g., 150 replicas per real node). Adding a node inserts virtual points; removing a node deletes them. Key lookup does a binary search for the next clockwise point. They should discuss how virtual nodes reduce hotspots versus a naive single-point ring. They should note that a sorted structure with bisect in Python or sort.Search in Go achieves O(log N) lookup.

Reference implementation (python)

import hashlib
import bisect

class ConsistentHashRing:
    def __init__(self, replicas: int = 150):
        self.replicas = replicas
        self.ring: dict[int, str] = {}
        self.sorted_keys: list[int] = []

    def _hash(self, key: str) -> int:
        return int(hashlib.md5(key.encode()).hexdigest(), 16)

    def add_node(self, node: str) -> None:
        pass  # TODO

    def remove_node(self, node: str) -> None:
        pass  # TODO

    def get_node(self, key: str) -> str:
        pass  # TODO

Expect these follow-ups

• What is the expected fraction of keys that remap when you add one node to a ring of N nodes?
• How does consistent hashing compare to rendezvous hashing (HRW) for cache routing?

As asked

Write a function that takes a list of pod names and their assigned nodes (with zone labels) and checks whether a deployment's topology spread constraint of maxSkew=1 across zones is satisfied. Return which zones are out of balance.

Sample answer outline

The candidate should count pods per zone, find max and min counts, and check that max minus min is less than or equal to maxSkew. They should return the over-represented zones clearly. Edge cases include pods with no zone label, an empty pod list, and zones with zero pods. They should note that Kubernetes actually enforces this at scheduling time and this function would simulate what the scheduler checks.

Reference implementation (python)

from collections import Counter

def check_spread(
    pods: list[dict],  # [{"name": "pod-1", "zone": "us-east-1a"}, ...]
    max_skew: int = 1,
) -> dict:
    """
    Returns {"balanced": bool, "zone_counts": Counter, "over_zones": list[str]}
    """
    # TODO
    pass

Expect these follow-ups

• How do topology spread constraints interact with pod affinity and anti-affinity rules?
• What happens to spread constraints when a node or zone goes down and the constraint cannot be satisfied?

As asked

You receive latency measurements as a stream and need to report approximate p99 latency every 60 seconds without storing every measurement. Implement a solution using a T-Digest or reservoir sampling approach and explain the accuracy tradeoff.

Sample answer outline

The candidate should describe reservoir sampling (keep a fixed-size random sample and sort it when queried, giving exact percentile of the sample) or T-Digest (compress data into centroid clusters, more accurate at tails). Reservoir sampling is simpler to implement but loses accuracy proportional to sample size. T-Digest is more accurate for tail percentiles specifically. They should implement reservoir sampling cleanly and discuss error bounds.

Reference implementation (python)

import random

class ReservoirSampler:
    def __init__(self, size: int = 1000):
        self.size = size
        self.reservoir: list[float] = []
        self.count = 0

    def add(self, value: float) -> None:
        # TODO: implement reservoir sampling (Algorithm R)
        pass

    def percentile(self, p: float) -> float:
        # p in [0, 1]
        if not self.reservoir:
            return 0.0
        sorted_r = sorted(self.reservoir)
        idx = int(p * len(sorted_r))
        return sorted_r[min(idx, len(sorted_r) - 1)]

Expect these follow-ups

• How does Prometheus histogram_quantile achieve a similar goal without storing individual measurements?
• How would you implement a sliding window version of reservoir sampling?

As asked

You are building an alert deduplication layer that receives potentially millions of alert events per day. Implement a Bloom filter in Python that determines whether an alert (identified by its fingerprint) has already been seen in the last 24 hours. Explain the false positive rate tradeoff and how you handle the rolling window.

Sample answer outline

The candidate should implement a Bloom filter with k hash functions and a bit array, explain the false positive rate formula p = (1 - e^(-kn/m))^k, and choose appropriate m and k for the expected daily volume. For the rolling 24-hour window, they should describe using two Bloom filters in rotation (swap every 12 hours, clear the older one) rather than trying to delete individual entries. They should also note that Redis's BF.ADD command offers a production-ready alternative.

Reference implementation (python)

import hashlib
import math
from bitarray import bitarray

class BloomFilter:
    def __init__(self, capacity: int, error_rate: float = 0.01):
        # Calculate m (bits) and k (hash functions)
        self.m = math.ceil(-capacity * math.log(error_rate) / (math.log(2) ** 2))
        self.k = math.ceil((self.m / capacity) * math.log(2))
        self.bits = bitarray(self.m)
        self.bits.setall(0)

    def _hashes(self, item: str) -> list[int]:
        # TODO: return k hash positions in [0, m)
        pass

    def add(self, item: str) -> None:
        pass

    def might_contain(self, item: str) -> bool:
        pass

Expect these follow-ups

• Can you delete an element from a Bloom filter, and if not, what data structure allows deletion?
• How would you use Redis Bloom filter module instead of a custom implementation in production?