DevOps engineer system design interview questions

As asked

Walk me through how you would design a blue/green deployment pipeline for a stateful API that owns a Postgres database and accepts long-lived WebSocket connections.

Sample answer outline

Stand up two identical environments behind a router (ALB, Envoy, or a service mesh). Migrate the schema in expand/contract phases so blue and green can read and write at the same time. Drain WebSockets gracefully: stop accepting new connections on blue, let clients reconnect to green via DNS or sticky-session routing. Hold the cutover with a small canary slice (1 to 5 percent) and watch SLO burn, error rates, and connection re-establishment metrics. Keep blue warm for the rollback window, then tear it down.

Expect these follow-ups

• How does this change if the schema migration is destructive and not backwards compatible?
• What if you cannot afford to double-provision the database?
• How would you handle a partial cutover failure where 30 percent of traffic is on green?

As asked

You join a startup with 50 microservices and no observability. What do you put in place in the first 90 days?

Sample answer outline

Three pillars but in the right order. Logs first: ship structured logs to a single store (Loki, Elastic, or a vendor) with a trace ID field. Metrics next: Prometheus with a service-level dashboard per team and four golden signals (latency, traffic, errors, saturation). Tracing last because it requires app instrumentation: OpenTelemetry SDKs in every service, sampling at 1 to 10 percent at the head, tail sampling for errors. Tie it all together with the trace ID propagated through logs and metrics. Get an on-call rota and a paging policy in place at the same time.

Expect these follow-ups

• How do you sell the observability investment to a CEO who wants features?
• What is your retention policy and why?
• When is OpenTelemetry not the right choice?

As asked

Design a CI/CD platform that 50 product teams can use to deploy their services independently. Each team should be able to onboard without waiting for the platform team, have their own isolated pipelines, and the platform should enforce security and compliance policies automatically.

Sample answer outline

Strong answers cover a golden path template approach (standardized pipeline templates in a shared repo, teams fork or reference them), GitOps with ArgoCD ApplicationSets for automatic onboarding via a team config file. Security policies enforced via OPA Gatekeeper or Kyverno admission controllers, Vault for secrets, image scanning in pipeline gates. Platform team owns the control plane; product teams own their manifests. Discuss observability (per-team dashboards), cost attribution via tags, and a breaking-glass process for platform emergencies.

Expect these follow-ups

• How do you prevent one team's misconfigured pipeline from consuming all shared runner capacity?
• How would you handle a breaking change to the shared pipeline template that all 50 teams depend on?

As asked

Design a system that allows schema migrations on a Postgres database with 50 million rows and 200 QPS without any application downtime. How do you handle long-running migrations, lock contention, and rollback?

Sample answer outline

Strong answers cover the expand-contract (parallel change) pattern: add a new column as nullable first, backfill in batches during off-peak hours using pg_repack or manual UPDATE WHERE id BETWEEN, make the application write to both old and new columns, then cut over reads, then drop the old column. Use tools like pg_repack to rebuild tables without exclusive locks. Discuss lock_timeout and statement_timeout to avoid blocking, and feature flags to gate the code change independently from the schema change. Rollback involves feature flag flips, not schema revert.

Expect these follow-ups

• How does pg_repack work and why is it better than VACUUM FULL for reclaiming space without downtime?
• How do you handle a migration that needs a unique index on a 50-million-row table?

As asked

Design a disaster recovery setup for a SaaS application running on AWS us-east-1 that must achieve an RTO of under 1 hour and an RPO of under 5 minutes. What AWS services do you use and what does the failover process look like?

Sample answer outline

Strong answers cover Aurora Global Database for sub-5-minute RPO via sub-second replication lag, Route 53 health checks with failover routing policies, cross-region S3 replication, and pre-warmed Auto Scaling groups or Fargate tasks in the secondary region that can take traffic within minutes. The failover process involves Route 53 DNS cutover (with low TTL), promoting the Aurora secondary, and validating the secondary region with synthetic monitoring. Discuss runbook automation to reduce human steps and planned failover drills.

Expect these follow-ups

• What is the difference between a warm standby and a pilot light DR strategy?
• How do you test your DR plan without causing an actual outage?

As asked

Design a logging pipeline that ingests structured logs from 100 microservices at 500 MB per second, makes them searchable within 30 seconds of emission, retains them for 90 days, and costs under $5000 per month.

Sample answer outline

Strong answers cover a collector tier (Fluent Bit as DaemonSet, lightweight), a streaming tier (Kafka or Kinesis for buffering and fan-out), a processing tier (Logstash or Vector for parsing and enrichment), and a storage tier. For cost at 500 MB/s, OpenSearch or Loki are good options: Loki stores only compressed index plus object storage (S3) which is much cheaper than Elasticsearch. Discuss index lifecycle management, hot/warm/cold tiers, log sampling for verbose debug logs, and alerting on ingestion lag.

Expect these follow-ups

• How does Loki's label-based indexing differ from Elasticsearch's inverted index and what are the cost implications?
• How do you handle a service that emits 100x normal log volume during an incident without letting it fill the pipeline?

As asked

Design the repository and workspace layout for Terraform that manages dev, staging, and production environments across two AWS regions, with shared modules and environment-specific configuration, ensuring a prod apply cannot accidentally modify dev resources.

Sample answer outline

Strong answers propose separate state files per environment (either Terraform Cloud workspaces or S3 prefixes like envs/prod/terraform.tfstate), a modules/ directory for reusable components, and env-specific variable files. Separate AWS accounts per environment enforced by IAM boundaries prevent cross-environment mistakes. A CI pipeline applies dev on merge to main, staging on a tag, and prod after manual approval. The candidate should discuss the monorepo vs per-environment repo tradeoff and module pinning for stability.

Expect these follow-ups

• What is Terraform Cloud's workspace concept and how does it enforce isolation between environments?
• How do you handle a module change that needs to be applied in dev first, validated, then promoted to prod?

As asked

Design a pipeline where a Docker image built in CI is the exact same binary that goes to production. No rebuilds at deploy time. How do you track provenance, enforce environment promotion gates, and handle rollbacks?

Sample answer outline

Strong answers describe building once and tagging with the Git commit SHA, pushing to a private registry (ECR or Artifact Registry), and promoting the immutable digest through environments by updating a deployment manifest in a GitOps repo. Provenance is captured with SLSA attestations and image signing via cosign. Promotion gates are implemented as CI steps that require automated test passage and sometimes manual approval via CI environment protection rules. Rollback is re-deploying the previous SHA manifest in the GitOps repo.

Expect these follow-ups

• What is a SLSA provenance attestation and what problem does it solve?
• How does content-addressable image referencing by digest rather than tag prevent the 'latest' tag problem?

As asked

You are setting up on-call for an engineering team of 20. Design the rotation, escalation policy, alert routing, and runbook system so that any engineer can handle an incident with the runbook within 5 minutes of being paged.

Sample answer outline

Strong answers cover PagerDuty or OpsGenie for rotation management with weekly primary and secondary coverage, severity tiers (SEV1 pages primary immediately, SEV2 gives 15 minutes before escalation), alert routing from Prometheus Alertmanager with deduplication and inhibition rules, and runbooks stored in a wiki with deep links embedded in alert descriptions. New engineers shadow the on-call before going solo. Discuss reducing alert noise as the most impactful reliability investment and measuring MTTA and MTTR.

Expect these follow-ups

• How do you prevent alert fatigue when a noisy metric generates 200 pages in one night?
• How do you measure the health of your on-call rotation over time?

As asked

Design an autoscaling strategy for a Kubernetes cluster that runs both latency-sensitive API services and large bursty batch jobs that need hundreds of GPUs for minutes at a time. How do you avoid the batch jobs evicting the API pods?

Sample answer outline

Strong answers use node taints and tolerations to put batch jobs on dedicated GPU node groups, and HPA or KEDA for the API layer. For GPU scaling, Cluster Autoscaler with node group configuration, or Karpenter which provisions nodes faster and supports arbitrary instance types. Priority classes prevent batch pods from evicting Guaranteed-class API pods. Use Spot instances for batch (with interruption handling) and On-Demand for API. Discuss overprovisioning buffers for faster scale-out and bin-packing to reduce waste.

Expect these follow-ups

• How does Karpenter differ from Cluster Autoscaler and when would you choose one over the other?
• How do you handle Spot instance interruptions gracefully for a long-running batch job?

As asked

Design a secrets management system for 200 microservices running on Kubernetes. Services need to access database credentials, API keys, and TLS certificates. Secrets must be rotated automatically and access must be audited.

Sample answer outline

Strong answers use HashiCorp Vault with Kubernetes auth (pod service account JWT validates against Vault, Vault returns a token scoped to the pod's policy). Dynamic secrets for Postgres rotate credentials on each lease renewal. The Vault agent sidecar or Vault Secrets Operator syncs secrets into Kubernetes Secrets or injects them as files. Audit logging in Vault captures every read. For TLS, cert-manager with Vault PKI Secrets Engine issues short-lived certificates. Discuss Vault HA with Raft storage and backup strategies.

Expect these follow-ups

• How does the Vault Kubernetes auth method work at the protocol level?
• What happens to running pods when a Vault-issued database credential is rotated and the old credential is revoked?

As asked

Your security team requires that pods can only make outbound calls to approved external endpoints. Design a system to enforce, monitor, and update this egress allowlist without requiring pod restarts.

Sample answer outline

Strong answers cover options at multiple layers: Kubernetes NetworkPolicies for intra-cluster traffic but they only work with IPs and cannot do DNS-based egress filtering; Cilium's FQDN-based policies allow rules like toFQDNs: [{matchPattern: '*.amazonaws.com'}]; a dedicated egress gateway (Istio egress gateway or a NAT instance with IP allowlist) as a centralized enforcement point; and Squid or Envoy as a forward proxy for HTTP/S traffic. Dynamic updates via GitOps mean adding a hostname to a configmap or policy triggers an automated policy push without pod restarts.

Expect these follow-ups

• How does Cilium's FQDN-based egress policy handle DNS TTL and IP rotation for services like S3?
• What is the tradeoff between network-layer egress control and application-layer proxy-based egress control?