DevOps engineer coding interview questions

As asked

Write a Bash function called retry that takes a command as its arguments and retries it up to 5 times with exponential backoff starting at 2 seconds, printing the attempt number each time and returning a non-zero exit code if all attempts fail.

Sample answer outline

The function should use a loop from 1 to MAX_ATTEMPTS, run the command with $@, check the exit code, sleep for 2^attempt seconds on failure using arithmetic expansion, and return 1 after all attempts. A strong answer handles quoting of arguments correctly and avoids swallowing the exit code.

Reference implementation (bash)

retry() {
  local max_attempts=5
  local attempt=1
  local delay=2
  until "$@"; do
    if (( attempt == max_attempts )); then
      echo "All $max_attempts attempts failed."
      return 1
    fi
    echo "Attempt $attempt failed. Retrying in ${delay}s..."
    sleep "$delay"
    (( attempt++ ))
    (( delay = delay * 2 ))
  done
}

Expect these follow-ups

• How would you add a jitter to the backoff to avoid thundering herd when many scripts retry at the same time?
• How would you make this function work correctly when the command contains pipes or redirections?

As asked

You have a file where each line is a JSON log entry with fields level, service, and message. Write a Python script that reads the file from stdin, counts ERROR entries per service, and prints them sorted by count descending.

Sample answer outline

Strong answers use sys.stdin, json.loads per line, a collections.Counter or defaultdict to accumulate counts, and sorted with key=lambda to sort descending. Handle malformed lines with try/except. Output format should be clear. A good candidate also mentions that in practice they would use jq for one-liners but Python for anything that needs more logic.

Reference implementation (python)

import sys, json
from collections import defaultdict

counts = defaultdict(int)
for line in sys.stdin:
    try:
        entry = json.loads(line)
        if entry.get('level') == 'ERROR':
            counts[entry.get('service', 'unknown')] += 1
    except json.JSONDecodeError:
        pass

for service, count in sorted(counts.items(), key=lambda x: -x[1]):
    print(f"{service}: {count}")

Expect these follow-ups

• How would you change this to stream a real-time log feed from a Kubernetes pod instead of reading a file?
• How would you extend this to also track the first and last occurrence timestamp of each error per service?

As asked

Write a Go function that takes a slice of URLs and makes HTTP GET requests to all of them concurrently with a 3-second timeout per request. Return a map from URL to bool indicating whether the request returned a 200 status code.

Sample answer outline

Strong answers use a goroutine per URL, a sync.WaitGroup to wait for all, a buffered channel or mutex-protected map to collect results, and http.Client with a timeout set via context.WithTimeout or the client's Timeout field. The candidate should handle the case where the response body must be closed to avoid connection leaks even when only checking the status code.

Reference implementation (go)

func checkURLs(urls []string) map[string]bool {
    results := make(map[string]bool, len(urls))
    var mu sync.Mutex
    var wg sync.WaitGroup
    client := &http.Client{Timeout: 3 * time.Second}

    for _, u := range urls {
        wg.Add(1)
        go func(url string) {
            defer wg.Done()
            resp, err := client.Get(url)
            ok := err == nil && resp.StatusCode == 200
            if err == nil { resp.Body.Close() }
            mu.Lock()
            results[url] = ok
            mu.Unlock()
        }(u)
    }
    wg.Wait()
    return results
}

Expect these follow-ups

• How would you limit concurrency to at most 10 simultaneous requests using a semaphore pattern?
• What happens if you forget to close the response body and this function is called thousands of times?

As asked

Write a Python script using the official kubernetes client library that lists all pods across all namespaces and prints the name and namespace of any pod where a container's restart count exceeds 5.

Sample answer outline

Strong answers import kubernetes.client and kubernetes.config, call config.load_kube_config() or config.load_incluster_config() depending on where the script runs, then use CoreV1Api().list_pod_for_all_namespaces() and iterate over items, checking container_statuses for restart_count. A good candidate handles None container_statuses for pods in Pending state and adds the --all-namespaces equivalent by not passing a namespace parameter.

Reference implementation (python)

from kubernetes import client, config

config.load_kube_config()
v1 = client.CoreV1Api()
pods = v1.list_pod_for_all_namespaces(watch=False)

for pod in pods.items:
    statuses = pod.status.container_statuses or []
    for cs in statuses:
        if cs.restart_count > 5:
            print(f"{pod.metadata.namespace}/{pod.metadata.name}: {cs.restart_count} restarts")

Expect these follow-ups

• How would you modify this to watch for new crash-loop events in real time rather than polling?
• How do you authenticate when running this script as a pod inside the cluster rather than from your laptop?

As asked

Write a Bash one-liner or short script that finds all .log files under /var/log that are older than 7 days, compresses them with gzip, and removes the original uncompressed files, without touching any .gz files that are already compressed.

Sample answer outline

The canonical answer uses find with -name '*.log' -not -name '*.gz' -mtime +7 and either -exec gzip {} + for efficiency or a while read loop. The candidate should know that gzip by default removes the original file, making the deletion redundant, but should explain the -k flag if they want to keep it. A strong answer also mentions dry-run testing with -print before -exec.

Reference implementation (bash)

find /var/log -name '*.log' -not -name '*.gz' -mtime +7 -exec gzip {} +

Expect these follow-ups

• How would you schedule this as a cron job and where would you put the cron entry?
• How would you parallelize this if there are thousands of files to compress?

As asked

You have a map of environment names to CIDR blocks and you want to create one VPC per environment using Terraform's for_each. Write the resource block and show how you would reference an output from one of those VPCs using the key.

Sample answer outline

Strong answers use for_each on the map variable, reference each.key and each.value inside the resource, and access outputs using aws_vpc.main["production"].id. The candidate should explain why for_each is preferred over count for maps (stable keys vs index-based addressing which breaks on element removal) and how to convert a set to a map if needed.

Reference implementation (hcl)

variable "envs" {
  default = {
    production  = "10.0.0.0/16"
    staging     = "10.1.0.0/16"
  }
}

resource "aws_vpc" "main" {
  for_each   = var.envs
  cidr_block = each.value
  tags = {
    Name = each.key
  }
}

output "vpc_ids" {
  value = { for k, v in aws_vpc.main : k => v.id }
}

Expect these follow-ups

• What happens to the other VPCs if you remove one entry from the map and run terraform apply?
• How is for_each different from count and why does count create problems when you delete an element from the middle of a list?

As asked

You are on a Linux server and need to find which ports a process named nginx is listening on, without root access. Write the command and explain the output.

Sample answer outline

Strong answers use ss -tlnp | grep nginx or lsof -i -P -n | grep LISTEN | grep nginx. Without root, ss may not show the process name for processes owned by other users, so the candidate should try both ss and /proc/net/tcp parsing as alternatives. They should explain the column format of ss output (state, recv-q, send-q, local address:port, peer address:port, process) and that -t is TCP, -l is listening, -n is numeric, and -p is process.

Reference implementation (bash)

# With ss (preferred)
ss -tlnp | grep nginx

# With lsof
lsof -i -P -n | grep LISTEN | grep nginx

# If no process names visible, get the PID first
pgrep nginx | xargs -I{} ss -tlnp | grep pid={}

Expect these follow-ups

• How would you script a check that alerts if a specific port stops listening?
• What is the difference between ss and netstat and why is ss preferred on modern Linux?

As asked

Write a Python function using boto3 that takes a parameter name and a new secret value, writes the new version to SSM Parameter Store as a SecureString, and returns the new version number.

Sample answer outline

Strong answers use boto3.client('ssm').put_parameter with Type='SecureString', Overwrite=True, and handle the case where the parameter does not exist yet (first write does not need Overwrite). The function should return response['Version']. A strong candidate mentions that the KMS key ID can be specified via KeyId and that Overwrite=True is required for updating an existing parameter.

Reference implementation (python)

import boto3

def rotate_secret(parameter_name: str, new_value: str, kms_key_id: str = 'alias/aws/ssm') -> int:
    client = boto3.client('ssm')
    response = client.put_parameter(
        Name=parameter_name,
        Value=new_value,
        Type='SecureString',
        KeyId=kms_key_id,
        Overwrite=True
    )
    return response['Version']

Expect these follow-ups

• How would you also update the Kubernetes secret that mounts this value, without restarting the pod?
• How does AWS Secrets Manager differ from SSM Parameter Store for secret rotation?

As asked

Write a Go main function for a long-running server that listens for SIGTERM and SIGINT, waits for in-flight requests to complete with a 30-second deadline, then exits cleanly. Show how you would integrate this with an http.Server.

Sample answer outline

Strong answers use signal.NotifyContext or signal.Notify with a channel, call server.Shutdown with a context that has a 30-second deadline, and use a goroutine for the ListenAndServe call. The candidate should know that Shutdown disallows new connections, waits for active handlers, then closes the listener. They should also handle the ErrServerClosed error which ListenAndServe returns after Shutdown.

Reference implementation (go)

func main() {
    srv := &http.Server{Addr: ":8080", Handler: myHandler()}
    go func() {
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            log.Fatal(err)
        }
    }()
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    <-quit
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    if err := srv.Shutdown(ctx); err != nil {
        log.Fatalf("Forced shutdown: %v", err)
    }
    log.Println("Server exited cleanly")
}

Expect these follow-ups

• Why does Kubernetes send SIGTERM before SIGKILL and how long does it wait between them by default?
• How would you ensure database connections are also closed cleanly during shutdown?

As asked

Write a Python function that parses a Prometheus text format exposition string and returns a dict mapping metric name to a list of (labels_dict, value) tuples. Assume no histogram or summary types, just gauges and counters.

Sample answer outline

Strong answers use a line-by-line approach, skip lines starting with # (comments and TYPE/HELP), split on the last space to separate labels block from value, use regex or string parsing to extract label key-value pairs from inside curly braces, and handle metric names without labels. The candidate should be comfortable with regex groups and handle edge cases like escaped quotes in label values.

Reference implementation (python)

import re
from collections import defaultdict

def parse_metrics(text):
    result = defaultdict(list)
    pattern = re.compile(r'^(\w+)(\{.*?\})?\s+([\d.e+\-]+)')
    label_pattern = re.compile(r'(\w+)="([^"]*?)"')
    for line in text.splitlines():
        if line.startswith('#') or not line.strip():
            continue
        m = pattern.match(line)
        if m:
            name, labels_str, value = m.groups()
            labels = dict(label_pattern.findall(labels_str or ''))
            result[name].append((labels, float(value)))
    return dict(result)

Expect these follow-ups

• How would you handle the histogram bucket lines that include the +Inf bucket?
• Why does the Prometheus text format put labels in curly braces and what is the OpenMetrics format trying to improve?

As asked

Write a Bash script that takes a service account name and namespace as arguments, extracts its token and the cluster's CA certificate, and outputs a kubeconfig file that can be used by a CI tool to deploy to the cluster.

Sample answer outline

Strong answers use kubectl get secret to find the service account token secret (or in Kubernetes 1.24+ use kubectl create token for a bound token), extract the token and ca.crt with -o jsonpath, and assemble the kubeconfig with base64-encoded CA data and the cluster API server URL from kubectl cluster-info. The candidate should note the difference between auto-generated secrets (pre-1.24) and manually created token requests (1.24+).

Reference implementation (bash)

#!/bin/bash
SA="$1"; NS="$2"
SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
TOKEN=$(kubectl create token "$SA" -n "$NS" --duration=8760h)
CA=$(kubectl get secret -n "$NS" \
  $(kubectl get sa "$SA" -n "$NS" -o jsonpath='{.secrets[0].name}') \
  -o jsonpath='{.data.ca\.crt}' 2>/dev/null || \
  kubectl config view --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')
kubectl --kubeconfig=/tmp/ci-kubeconfig config set-cluster default --server="$SERVER"
kubectl --kubeconfig=/tmp/ci-kubeconfig config set credentials ci --token="$TOKEN"
kubectl --kubeconfig=/tmp/ci-kubeconfig config set-context default --cluster=default --user=ci
kubectl --kubeconfig=/tmp/ci-kubeconfig config use-context default

Expect these follow-ups

• How do you scope this service account's RBAC permissions to only allow deploying to one namespace?
• What changed in Kubernetes 1.24 regarding auto-mounted service account tokens?