Platform engineer coding interview questions

As asked

Write a Go HTTP server that exposes /healthz for liveness and /readyz for readiness probes. The readiness probe should return 503 during startup and during shutdown. Implement graceful shutdown that gives in-flight requests 30 seconds to complete after SIGTERM before the process exits.

Sample answer outline

The solution uses a sync/atomic or channel-based ready flag that starts as false and flips to true after initialization. The /readyz handler checks the flag. On SIGTERM, the flag flips to false (so Kubernetes stops routing traffic), then http.Server.Shutdown is called with a 30-second context. Strong answers handle the race between Kubernetes removing the endpoint and the pod receiving SIGTERM by adding a pre-stop sleep.

Reference implementation (go)

package main

import (
	"context"
	"net/http"
	"os/signal"
	"sync/atomic"
	"syscall"
	"time"
)

var ready atomic.Bool

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
	})
	mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
		if ready.Load() {
			w.WriteHeader(http.StatusOK)
		} else {
			w.WriteHeader(http.StatusServiceUnavailable)
		}
	})
	srv := &http.Server{Addr: ":8080", Handler: mux}
	// TODO: implement graceful shutdown on SIGTERM
	_ = signal.NotifyContext
	_ = syscall.SIGTERM
	_ = time.Second
	_ = context.Background
	srv.ListenAndServe()
}

Expect these follow-ups

• What is the difference between liveness and readiness in terms of Kubernetes behavior when each probe fails?
• How do you add a startup probe and why is it useful for slow-starting JVM applications?

As asked

You have two maps representing the desired state (from Git) and the live state (from the API server) of Kubernetes resources, keyed by 'namespace/name/kind'. Write a function that returns three slices: resources to create (in desired but not live), to delete (in live but not desired), and to update (in both but with differing spec fields).

Sample answer outline

A clean solution iterates the desired map once to find creates and candidate updates, then iterates the live map to find deletes. The update check compares the spec field (a nested map or JSON string) for equality. Strong answers note that labels/annotations added by Kubernetes (like last-applied-configuration) must be excluded from the comparison to avoid false positives.

Reference implementation (go)

type Resource struct {
	Namespace string
	Name      string
	Kind      string
	Spec      map[string]interface{}
}

func diffResources(
	desired map[string]Resource,
	live    map[string]Resource,
) (toCreate, toDelete, toUpdate []Resource) {
	// TODO: implement
	return
}

Expect these follow-ups

• How would you extend this to produce a JSON patch instead of just flagging an update?
• How does kubectl diff implement this comparison under the hood?

As asked

Our internal CI dispatch API must allow each team a burst of 10 jobs and then a sustained rate of 2 jobs per second per team. Implement a token bucket rate limiter in Python that is thread-safe and can handle multiple teams concurrently. The function should return True if the job is allowed and False if it is rejected.

Sample answer outline

Each team gets its own bucket with a capacity of 10 tokens and a refill rate of 2 per second. On each request, compute elapsed time, add (elapsed * rate) tokens capped at capacity, then check if a token is available. A threading.Lock per team ensures thread safety. Strong candidates use time.monotonic() for drift resistance and consider using a single lock with a dict of per-team state to minimize lock contention.

Reference implementation (python)

import time
import threading
from dataclasses import dataclass, field

@dataclass
class Bucket:
    tokens: float
    capacity: float
    rate: float  # tokens per second
    last_refill: float = field(default_factory=time.monotonic)
    lock: threading.Lock = field(default_factory=threading.Lock)

class RateLimiter:
    def __init__(self, capacity: float = 10, rate: float = 2):
        self.capacity = capacity
        self.rate = rate
        self.buckets: dict[str, Bucket] = {}
        self._global_lock = threading.Lock()

    def allow(self, team_id: str) -> bool:
        # TODO: implement
        pass

Expect these follow-ups

• How would you extend this to be distributed across multiple API server instances using Redis?
• What is the difference between a token bucket and a leaky bucket? Which is better for bursty CI workloads?

As asked

Helm merges values files with last-write-wins semantics but does a shallow merge on nested maps. Write a Python function that performs a deep merge of two values dicts where the override dict takes precedence at every nesting level, matching Helm's actual behavior for lists (override replaces, not appends).

Sample answer outline

A recursive function that iterates the override dict: if both base and override have a key whose value is a dict, recurse; otherwise, set base[key] = override[key]. Lists are overwritten entirely, not merged. Base keys not in override are preserved. Strong answers handle None values and ensure the function does not mutate the original dicts.

Reference implementation (python)

base = {
    "image": {"tag": "v1", "pullPolicy": "IfNotPresent"},
    "replicas": 2,
    "env": ["A=1"]
}
override = {
    "image": {"tag": "v2"},
    "env": ["A=2", "B=3"]
}
# Expected: {"image": {"tag": "v2", "pullPolicy": "IfNotPresent"}, "replicas": 2, "env": ["A=2", "B=3"]}

def deep_merge(base: dict, override: dict) -> dict:
    # TODO: implement
    pass

Expect these follow-ups

• How does Helm's --set flag handle nested keys using dot notation, and how does that interact with values files?
• When would you want list merging instead of list replacement? How could you implement that?

As asked

Write a Go function that retries a Kubernetes API call with exponential backoff and full jitter. The function should accept the operation as a func() error, a maximum number of retries, a base delay, and a maximum delay cap. It must not retry on non-retriable errors (e.g. 404 Not Found).

Sample answer outline

The backoff delay for attempt n is min(cap, base * 2^n). Full jitter means the actual sleep is rand.Int63n(delay). The function checks the error type using errors from k8s.io/apimachinery/pkg/api/errors or a custom IsRetriable function: 404 and 409 conflicts might not be retriable, while 429 and 5xx are. The loop exits early on a nil error or a non-retriable error.

Reference implementation (go)

package main

import (
	"math"
	"math/rand"
	"time"
)

func withRetry(
	op func() error,
	maxRetries int,
	baseDelay time.Duration,
	maxDelay time.Duration,
	isRetriable func(error) bool,
) error {
	// TODO: implement exponential backoff with full jitter
	return nil
}

// hint: delay = min(maxDelay, baseDelay * 2^attempt)
// jitter: sleep = time.Duration(rand.Int63n(int64(delay)))
_ = math.Pow
_ = rand.Int63n

Expect these follow-ups

• What is the difference between full jitter and decorrelated jitter? When does decorrelated jitter perform better under thundering herd conditions?
• How does controller-runtime's ratelimiter implement backoff for the work queue?

As asked

You have a custom resource MyApp and need to update its status.phase field to 'Ready' without overwriting other status fields. Show how you patch the status subresource using the controller-runtime client in Go, and explain why you must use the status subresource patch rather than a regular update.

Sample answer outline

Use client.Status().Patch() with a MergeFrom patch that contains only the changed status fields. Using a full Update on the main resource would overwrite the spec field and cause a conflict because the spec is managed by a different controller path. Patching only the status subresource requires the CRD to have subresources.status enabled. Strong answers also handle the OptimisticLock conflict error by re-fetching the object and retrying.

Reference implementation (go)

import (
	"context"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
)

type MyAppReconciler struct {
	client.Client
}

func (r *MyAppReconciler) setReady(ctx context.Context, app *MyApp) error {
	patch := client.MergeFrom(app.DeepCopy())
	app.Status.Phase = "Ready"
	// TODO: patch status subresource correctly
	return nil
}
_ = ctrl.Log

Expect these follow-ups

• What is the difference between SSA (Server-Side Apply) and a merge patch when updating a resource?
• How do you avoid a reconcile loop where your operator updates status, which triggers a new reconcile, which updates status again?

As asked

A CI system defines jobs with dependencies as a directed acyclic graph. Write a Python function that takes a dict mapping job names to lists of dependency job names and returns an ordered list of lists, where each inner list contains jobs that can run in parallel. The function should detect cycles and raise an error.

Sample answer outline

Use Kahn's algorithm: compute in-degrees for all nodes, enqueue nodes with in-degree 0, process them as a batch (one parallel level), decrement neighbors' in-degrees and enqueue any that reach 0. A cycle is detected when the processed count is less than the total node count at the end. Each batch represents jobs that can run concurrently.

Reference implementation (python)

def level_order_schedule(
    jobs: dict[str, list[str]]
) -> list[list[str]]:
    """
    jobs = {
      'test': ['build'],
      'build': [],
      'lint': [],
      'deploy': ['test', 'lint']
    }
    Expected: [['build','lint'], ['test'], ['deploy']]
    """
    # TODO: implement Kahn's algorithm returning levels
    pass

Expect these follow-ups

• How do you handle a diamond dependency (A depends on B and C, both B and C depend on D) to ensure D only runs once?
• How would you extend the scheduler to respect resource limits so only N jobs can run in parallel at any time?

As asked

You are building a platform observability tool that reads the OTel Collector's JSON log output piped from a Kubernetes node DaemonSet. Write a Python function that reads newline-delimited JSON log lines from stdin and outputs, for each service.name attribute, the total request count and error rate (percentage of lines where http.status_code >= 500), sorted by error rate descending. Handle malformed JSON lines by skipping them.

Sample answer outline

Read lines from sys.stdin in a loop, try json.loads and skip on exception. Accumulate a dict keyed by service with total and errors counters. After processing, compute error_rate = errors/total*100 and sort descending. A strong answer uses collections.defaultdict and formats the output cleanly. Bonus: handles missing 'service' or 'status' fields with .get() defaults.

Reference implementation (python)

import sys, json
from collections import defaultdict

# Input line format:
# {"service": "auth", "status": 200, "latency_ms": 45}
# {"service": "auth", "status": 503, "latency_ms": 1200}

def aggregate_errors():
    stats = defaultdict(lambda: {"total": 0, "errors": 0})
    for line in sys.stdin:
        # TODO: parse and aggregate
        pass
    # TODO: print sorted output
    pass

if __name__ == "__main__":
    aggregate_errors()

Expect these follow-ups

• How would you extend this to compute p99 latency per service using only a single pass through the data?
• What is a T-Digest and when would you use it instead of keeping all samples in memory?

As asked

Given a list of container image tags (strings like '1.2.3', 'v2.0.0-rc1', 'latest', 'sha-abc123'), write a Python function that returns only valid semver tags sorted from newest to oldest, ignoring non-semver tags like 'latest' or commit SHAs.

Sample answer outline

Parse each tag by stripping an optional leading 'v', then splitting on '-' to separate the pre-release from the version core, and splitting the core on '.' to get major/minor/patch integers. Filter out tags that do not match this pattern. Sort using a tuple key of (major, minor, patch, pre-release handling) where stable versions sort above pre-releases. Using the semver library is acceptable as long as the candidate explains what it is doing.

Reference implementation (python)

tags = [
    "latest", "v1.2.3", "2.0.0", "v2.0.0-rc1",
    "sha-abc123", "1.10.0", "1.9.0", "v2.0.0-alpha"
]

def sorted_semver_tags(tags: list[str]) -> list[str]:
    """Return valid semver tags sorted newest-first."""
    # Expected: ['2.0.0', '2.0.0-rc1', '2.0.0-alpha', '1.10.0', '1.9.0', '1.2.3']
    pass

Expect these follow-ups

• How do you determine which tags to clean up from a registry to keep only the last 5 minor versions of each major?
• How does semantic versioning handle pre-release precedence? Which is higher: 1.0.0-alpha or 1.0.0-beta?

As asked

Given the output of terraform graph (a DOT-format directed graph), write a Python function that, given a module name, returns all resource names that transitively depend on that module. This is useful for planning the blast radius of a module change.

Sample answer outline

Parse the DOT file to extract edges (source -> target). Build an adjacency list. Run a BFS or DFS starting from all edges that point TO the module, collecting all nodes that have the module as a transitive dependency. Note: the direction of arrows in terraform graph means an arrow from A to B means B depends on A, so candidates must get the direction right.

Reference implementation (python)

import re
from collections import defaultdict, deque

# Sample DOT input (terraform graph output):
# digraph {
#   "module.vpc" -> "module.eks"
#   "module.eks" -> "aws_iam_role.node"
#   "module.vpc" -> "aws_subnet.main"
# }

def dependents_of(dot_graph: str, module_name: str) -> set[str]:
    """Return all nodes that transitively depend on module_name."""
    # TODO: parse edges and BFS/DFS
    pass

Expect these follow-ups

• How would you compute the critical path through the Terraform dependency graph to estimate maximum parallelism time?
• What is the terraform graph -type=plan-destroy option and when is it useful?

As asked

Two replicas of your operator run for high availability. Write the core of a leader election loop in Python using the Kubernetes Lease API: acquire the lease by creating/updating it with a holder identity, renew it every 10 seconds while leading, and step down if renewal fails. Show how you detect that a stale leader's lease has expired.

Sample answer outline

The Lease object has spec.holderIdentity, spec.leaseDurationSeconds, and spec.renewTime. To acquire: if no Lease exists, create it with your identity; if it exists and the renewTime is older than leaseDuration seconds, update it with your identity (compare-and-swap using resourceVersion for optimistic locking). While leading, update renewTime every leaseDuration/3 seconds. On failure to renew, stop acting as leader and wait for the lease to expire before attempting re-acquisition.

Reference implementation (python)

from kubernetes import client, config
from datetime import datetime, timezone, timedelta
import socket, time

HOLDER = socket.gethostname()
NAMESPACE = "platform"
LEASE_NAME = "my-operator-leader"
LEASE_DURATION = 30  # seconds

def try_acquire_or_renew(coord_v1) -> bool:
    """Return True if this instance holds the lease after this call."""
    # TODO: implement acquire/renew logic
    pass

def run_leader_loop():
    config.load_incluster_config()
    coord_v1 = client.CoordinationV1Api()
    while True:
        if try_acquire_or_renew(coord_v1):
            do_leader_work()
        time.sleep(LEASE_DURATION // 3)

def do_leader_work(): pass

Expect these follow-ups

• How does controller-runtime's leaderelection package implement this, and what lock types does it support beyond Lease?
• What is the split-brain scenario in leader election and how do you mitigate it when the Kubernetes API is partitioned?