DevOps engineer role-specific interview questions

As asked

A new Deployment rollout is stuck at 3 out of 10 pods Ready. Walk me through how you would debug it in production.

Sample answer outline

Start with kubectl rollout status and kubectl describe deployment. Look at the events: ImagePullBackOff, CrashLoopBackOff, FailedScheduling all point in different directions. Check pod events and pod logs for the failing replicas. Common causes: readiness probe failing because the app needs longer to start, resource requests too high for the available nodes, a config map or secret reference that does not exist, or a PodDisruptionBudget blocking eviction of old pods. Resist the urge to delete pods until you know the cause.

Reference implementation (bash)

# Triage sequence
kubectl rollout status deployment/api -n prod
kubectl describe deployment/api -n prod
kubectl get pods -n prod -l app=api
kubectl describe pod <pending-pod> -n prod
kubectl logs <crashlooping-pod> -n prod --previous
kubectl get events -n prod --sort-by=.lastTimestamp | tail -20

Expect these follow-ups

• What if the readiness probe passes but the service is still returning 503s?
• How do you set sane defaults for readiness and liveness probes?
• When would you use a startup probe?

As asked

An engineer commits an AWS access key to a public GitHub repo. Walk me through what you do in the next hour.

Sample answer outline

Treat the key as compromised. Rotate immediately, do not wait. Revoke the old key, generate a new one, push it via the secret manager so dependent services pick it up. Check CloudTrail for any usage of the key in the window between commit and rotation, especially from unfamiliar IPs. Remove the key from history (git filter-repo or BFG) but understand that GitHub may have already cached it and bots scrape new commits within seconds, so rotation is what matters, not rewriting history. Talk to the engineer without blame, and add pre-commit hooks (gitleaks, trufflehog) so it does not happen again.

Expect these follow-ups

• Why is rewriting history less important than rotating?
• How would you scan all current repos for past leaks?
• What is your policy for repeat offenders?

As asked

Two engineers run terraform apply at the same time against the same workspace. What happens with state locking, what backend are you using, and what do you do if a lock is stuck after a failed apply?

Sample answer outline

Strong answers explain that backends like S3 with DynamoDB or Terraform Cloud implement state locking via DynamoDB conditional writes. The candidate should describe the lock record format, how to use terraform force-unlock with the lock ID found in the error output, and why force-unlocking without confirming no apply is running can corrupt state. Good answers also mention remote backends vs local and the risk of partial state writes.

Expect these follow-ups

• What does a corrupted terraform state file look like and how do you recover from it?
• How would you structure workspaces and state files for a multi-environment setup?

As asked

How do you inject secrets into a GitHub Actions pipeline without exposing them in logs or baking them into container images? Walk me through your approach from secret storage to the running container.

Sample answer outline

Strong answers cover GitHub Actions encrypted secrets or OIDC-based short-lived credentials to pull from AWS Secrets Manager or HashiCorp Vault at runtime. The candidate should explain why secrets must not be printed with echo, that GitHub masks registered secrets in logs but this can be defeated with base64 encoding, and how to use build-time ARGs versus runtime ENV correctly to avoid baking secrets into image layers. Mention secret scanning tools like truffleHog or GitGuardian as a defense layer.

Expect these follow-ups

• How do you rotate a secret that is already in use by a running Kubernetes deployment?
• What is OIDC federation and why is it preferable to long-lived AWS IAM access keys in CI?

As asked

A developer asks for broad S3 access for a Lambda function. Walk me through how you would design the IAM policy correctly, what tools you would use to discover the minimum permissions needed, and how you would audit it over time.

Sample answer outline

Strong answers start with denying all and using IAM Access Analyzer to generate policies from CloudTrail logs of actual API calls made during testing. The candidate should know resource-level permissions in S3 (object vs bucket ARN), the difference between identity-based and resource-based policies, and how to use conditions like aws:SourceVpc or s3:prefix to narrow scope. Mention periodic IAM Access Analyzer findings review and automated policy validation in CI with tools like Cloudsplaining or Checkov.

Expect these follow-ups

• What is the difference between IAM Access Analyzer and IAM policy simulator?
• How do service control policies in AWS Organizations differ from IAM policies?

As asked

You are on-call and a production service is returning 5xx errors for 20 percent of traffic. Walk me through your first 15 minutes: what you check, in what order, and how you decide when to escalate.

Sample answer outline

Strong answers demonstrate a structured approach: first declare the incident and set severity, then look at dashboards to identify the error rate trend and whether it is worsening, check recent deploys with kubectl rollout history or CI/CD logs, look at pod logs and events for crash loops or OOM kills. The candidate should explain blast radius assessment, when to roll back versus investigate further, and escalation criteria based on customer impact. Mention post-incident reviews and blameless culture.

Expect these follow-ups

• How do you communicate status to stakeholders during an active incident?
• What is your process for writing a post-mortem and how do you ensure action items actually get fixed?

As asked

Your Docker builds in CI take 8 minutes because npm install runs on every push. Walk me through how Docker layer caching works and how you would restructure the Dockerfile and CI pipeline to get build time under 2 minutes.

Sample answer outline

Strong answers explain that Docker invalidates a layer and all layers below it when any file in that COPY instruction changes. The fix is to copy package.json and package-lock.json first, run npm install, then copy the source code so the install layer is only invalidated when dependencies change. For CI, candidates should mention BuildKit inline cache or registry cache modes, using --cache-from to pull the previous image, and GitHub Actions cache action with Docker layer caching. Mention multi-stage builds to keep the final image small.

Expect these follow-ups

• What is BuildKit and how does its parallelization differ from classic Docker builds?
• How does a multi-stage build help with both build time and image security?

As asked

Explain the difference between readiness and liveness probes in Kubernetes. What happens if you misconfigure them and what are the failure modes you have seen or would expect in production?

Sample answer outline

Strong answers explain that readiness probes control whether a pod receives traffic from a Service (it is removed from Endpoints if failing) while liveness probes cause the kubelet to restart the container. Misconfigurations include: liveness probes that are too aggressive causing restart loops, readiness probes checking dependencies that are temporarily slow causing all pods to drop from load balancing simultaneously (thundering herd on recovery), and startup probes being absent causing liveness to kill a slow-starting container during initialization.

Expect these follow-ups

• When would you use a startup probe instead of a liveness probe?
• What happens during a rolling update if the new pods fail their readiness probe?

As asked

Write Terraform that provisions a VPC across three availability zones with public and private subnets and a NAT gateway per AZ. Explain the cost tradeoffs.

Sample answer outline

Use a module that takes a CIDR and a count of AZs and emits subnet pairs per AZ. One NAT per AZ keeps egress availability isolated to the AZ but triples the NAT bill. A single shared NAT is cheaper but means an AZ outage on the NAT side takes down outbound traffic for everyone. Tag everything with owner and cost-centre so the bill is debuggable. Use remote state with locking, plan in CI, apply manually for shared environments.

Reference implementation (hcl)

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "prod"
  cidr = "10.0.0.0/16"

  azs             = ["eu-west-2a", "eu-west-2b", "eu-west-2c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway     = true
  single_nat_gateway     = false
  one_nat_gateway_per_az = true

  tags = { owner = "platform", env = "prod" }
}

Expect these follow-ups

• How would you swap the NAT gateway for a NAT instance and when is that worth it?
• Where do VPC endpoints fit into the cost story?
• How do you handle drift when someone edits the console by hand?

As asked

You have a TypeScript monorepo with 30 packages and a CI pipeline that takes 45 minutes. How would you restructure GitHub Actions to bring it under 10 minutes without dropping coverage?

Sample answer outline

Start with measurement: where is the time going. Common wins: matrix-parallelise tests by package, cache the package manager store and the build output keyed by lockfile and source hash, and only run tests for packages whose graph was actually touched (turbo, nx, pnpm filter). Move integration and e2e tests off the PR critical path into a nightly or post-merge job. Pin runner sizes for hot jobs. If the queue is the bottleneck, use larger runners or self-hosted on spot.

Expect these follow-ups

• How do you keep flaky tests from blocking PRs without ignoring them?
• What is your retry policy for transient infrastructure failures?
• When is a self-hosted runner worth the operational cost?

As asked

You have 30 services running in one AWS account. Design an IAM strategy that gives each service least privilege and keeps the policies maintainable.

Sample answer outline

One IAM role per service, assumed via the workload identity (IRSA for EKS, instance profile for EC2, task role for ECS). Policies generated from infrastructure code, not hand-edited. Use IAM Access Analyzer to flag overly permissive policies and unused permissions. Avoid wildcard resource ARNs. Separate the prod and non-prod accounts to remove the worst blast-radius failure modes. Service control policies at the org level for guardrails like 'no public S3'. Rotate credentials never used (CIS benchmarks). Treat IAM as code with review.

Reference implementation (json)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOwnConfigOnly",
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::prod-config/api/*"
    },
    {
      "Sid": "PublishToOwnTopic",
      "Effect": "Allow",
      "Action": ["sns:Publish"],
      "Resource": "arn:aws:sns:eu-west-2:123456789012:api-events"
    },
    {
      "Sid": "DenyAllOtherS3",
      "Effect": "Deny",
      "NotAction": ["s3:GetObject"],
      "Resource": "*"
    }
  ]
}

Expect these follow-ups

• How do you handle a service that needs cross-account access?
• What is your strategy for human access (engineers) to production?
• How would you scope a policy down from a wildcard that someone shipped 3 years ago?

As asked

Your monthly cloud forecast jumps by 45 percent over a weekend. What do you check first, and how do you stop it happening again?

Sample answer outline

Break the bill down by service, region, account, tag, and usage type before guessing. Common causes are runaway data transfer, NAT gateway traffic, oversized compute, forgotten test clusters, log ingestion, or a warehouse job scanning far more data than expected. Stop the bleeding with quotas, budgets, anomaly alerts, and temporary scaling caps where they are safe. Then fix ownership: mandatory tags, showback or chargeback, and pull-request visibility for expensive infrastructure changes. The interviewer wants pragmatism, not just 'buy savings plans', because commitments can lock in waste.

Expect these follow-ups

• Which cost category is hardest to allocate back to a team?
• When do savings plans make the problem worse?
• How would you explain this incident to finance without hiding the engineering details?

As asked

A node in your cluster is under memory pressure and starts evicting pods. Walk me through exactly what happens at the kubelet level, how eviction thresholds work, and what you would tune to prevent critical workloads from being evicted.

Sample answer outline

Strong answers cover soft and hard eviction thresholds, the kubelet's eviction manager polling loop, and the role of QoS classes (Guaranteed, Burstable, BestEffort) in eviction order. Candidates should explain setting resource requests and limits precisely, using PodDisruptionBudgets, and configuring --eviction-hard and --eviction-soft flags. Bonus for mentioning node conditions like MemoryPressure and DiskPressure propagating to the API server.

Expect these follow-ups

• How do PodDisruptionBudgets interact with eviction, and what happens if a PDB blocks eviction entirely?
• How would you use priority classes to protect critical system pods from eviction?

As asked

Your team wants to reduce deployment risk for a stateful service with a Postgres database. Compare blue/green and canary deployments in this context. Which do you recommend and what problems do you have to solve around schema migrations?

Sample answer outline

Strong answers explain that blue/green gives instant rollback but requires double capacity and is tricky with stateful services because you cannot easily run two database versions simultaneously. Canary releases traffic gradually but the same schema must work for both old and new code simultaneously, requiring backward-compatible migrations (expand-contract pattern). The candidate should mention tools like Flyway or Liquibase, feature flags for code-level gates, and the risk of long-lived migrations locking tables.

Expect these follow-ups

• How do you handle a migration that needs to rename a column without downtime?
• What Kubernetes constructs would you use to implement a canary without a service mesh?

As asked

Your Prometheus instance is OOMing and scrape latency is climbing. You suspect a cardinality explosion. How do you diagnose which metric or label is causing it, and how do you fix it without losing the data you care about?

Sample answer outline

Strong answers start with the Prometheus TSDB status endpoint at /api/v1/status/tsdb which surfaces the highest-cardinality metric names and label values. The candidate should explain that cardinality comes from unbounded label values like user IDs or request paths, the fix being relabeling rules in the scrape config to drop or replace the offending label. They should also know that reducing cardinality after the fact requires deleting series with the admin API and that recording rules can pre-aggregate high-cardinality data.

Expect these follow-ups

• How does Prometheus TSDB store time series on disk and why does high cardinality cause memory pressure specifically?
• Walk me through writing a relabeling rule that drops a label called user_id before ingestion.

As asked

A pod cannot reach a service in a different namespace and you suspect a NetworkPolicy is blocking it. Walk me through your debugging process step by step.

Sample answer outline

Strong answers start with kubectl exec and a curl or nc test to confirm the connection fails, then check all NetworkPolicy objects in both namespaces with kubectl get netpol. The candidate must explain that NetworkPolicies are additive and a missing ingress rule on the target denies all traffic from outside its namespace. They should know how to read selector logic and namespace selectors, use tools like netshoot or cilium hubble for packet-level tracing, and check whether the CNI plugin actually enforces NetworkPolicies.

Expect these follow-ups

• What happens when no NetworkPolicy selects a pod? Is it open or isolated?
• How does Cilium's eBPF-based enforcement differ from iptables-based CNIs for NetworkPolicy?

As asked

You maintain a shared Terraform module used by 20 teams. A breaking change is needed in the module. How do you version, publish, and migrate teams without causing outages?

Sample answer outline

Strong answers cover semantic versioning with Git tags and the Terraform registry, pinning module sources to a specific version constraint in each consumer. The candidate should explain the expand-contract pattern: publish a new major version, update the migration guide, run both versions in parallel, and deprecate the old one after teams migrate. Bonus for mentioning automated tests on the module itself using Terratest and using depends_on or moved blocks for state migration.

Expect these follow-ups

• How do you test a Terraform module in isolation before publishing a new version?
• What is the moved block in Terraform and when would you use it?

As asked

You are designing RBAC for a Kubernetes cluster shared by five product teams. Each team should be able to deploy to their own namespace but not see or affect other namespaces. Walk me through the Role, ClusterRole, RoleBinding, and ClusterRoleBinding objects you would create.

Sample answer outline

Strong answers distinguish between Role (namespace-scoped) and ClusterRole (cluster-wide), explaining that a Role in namespace A bound to a team gives them rights only in that namespace. The candidate should create a Role with verbs like get, list, create, update, delete on resources like pods, deployments, services within the team namespace, and a RoleBinding to a group in their identity provider. Platform engineers get a ClusterRole for read-only cluster-wide visibility. Mention aggregated ClusterRoles for composing permissions and audit logging to track access.

Expect these follow-ups

• How would you give a CI service account permission to push images and update deployments without cluster-admin?
• What is the aggregated ClusterRole pattern and when is it useful?

As asked

Someone manually changed a security group rule in the AWS console and now terraform plan shows a diff. What is your process for detecting and remediating drift, and how do you prevent it from happening again?

Sample answer outline

Strong answers describe terraform plan as the primary drift detector, and mention running it on a schedule in CI (drift detection pipelines) with notifications on non-empty plans. Remediation options are importing the manual change into state with terraform import, reverting it with terraform apply, or using terraform apply -refresh-only to pull remote reality into state without changing resources (the modern replacement for the deprecated terraform refresh command). Prevention involves SCPs or IAM deny policies that block direct console changes to managed resources, and using AWS Config rules to detect out-of-band changes. Mention that terraform apply is the source of truth, not the console.

Expect these follow-ups

• What is the terraform import command and what does it do to state?
• How does the lifecycle ignore_changes argument interact with drift detection?

As asked

You have a GitOps deployment where a ConfigMap, a database migration Job, and the application Deployment must apply in that exact order. How do you enforce ordering in ArgoCD?

Sample answer outline

Strong answers explain ArgoCD sync waves using the argocd.argoproj.io/sync-wave annotation, where lower wave numbers apply first and ArgoCD waits for each wave to be healthy before proceeding. The migration Job would be in wave 1 and the Deployment in wave 2, with the Job needing a hook deletion policy to clean up. Candidates should also mention sync hooks (PreSync, Sync, PostSync) and the difference between resource health checks and hooks for blocking progress.

Expect these follow-ups

• How do you handle a sync wave where the Job fails? Does ArgoCD roll back the previous wave?
• What is the difference between a sync hook and a sync wave in ArgoCD?

As asked

How do you integrate container image vulnerability scanning into your CI pipeline so that a critical CVE blocks deployment but you do not break the build on every minor vulnerability?

Sample answer outline

Strong answers describe tools like Trivy, Grype, or Snyk integrated as a CI step after the docker build. The candidate should explain severity thresholds (block on CRITICAL and HIGH, warn on MEDIUM), VEX documents or .trivyignore files for accepting known false positives or non-exploitable CVEs, and how to keep base images pinned by digest rather than tag to prevent silent upstream changes. Mention image signing with cosign and policy enforcement at admission time with OPA Gatekeeper or Kyverno.

Expect these follow-ups

• How does image signing with cosign and keyless signing via Sigstore work?
• What is a VEX document and how does it help suppress false positive CVE findings?

As asked

Walk me through how you would define a service level objective for an API, calculate the error budget, and write a Prometheus alerting rule that pages you before the budget is exhausted rather than after.

Sample answer outline

Strong answers define an SLO (e.g. 99.9% of requests return 2xx within 500ms over 30 days), calculate the error budget as 0.1% of total requests or 43.8 minutes of downtime. Multi-window multi-burn-rate alerting is the key: alert when the burn rate over the last 1 hour AND 5 minutes exceeds a threshold that would exhaust the budget in 1 hour, avoiding slow burns and alert storms. The candidate should mention Pyrra or Sloth for generating these rules and why simple error rate alerts miss slow error budget drain.

Expect these follow-ups

• What is a burn rate and how is it different from an error rate?
• Why do you need both a short window and a long window in multi-window alerting?

As asked

What are the most common mistakes people make when writing Helm charts, and how do you test and validate a chart before shipping it to a shared chart repository?

Sample answer outline

Strong answers mention common mistakes: not quoting values that could be interpreted as numbers (e.g. port: 8080 without quotes in YAML), missing default values causing template panics, not using include over template for composable helpers, and hardcoding image tags instead of using chart values. Validation steps include helm lint, helm template to render manifests and pipe through kubeval or kubeconform, ct lint (chart-testing) for changed charts in a PR, and deploying to a kind or minikube cluster in CI with ct install.

Expect these follow-ups

• What is the difference between include and template in Helm's Go template language?
• How does Helm handle upgrades when a new version changes a Kubernetes resource type, such as migrating from Deployment to StatefulSet?

As asked

You are running NGINX Ingress Controller on Kubernetes and need to automate TLS certificate provisioning and renewal for 50 subdomains. Walk me through the cert-manager setup, including how ACME challenges work.

Sample answer outline

Strong answers describe cert-manager's ClusterIssuer or Issuer resource pointing to Let's Encrypt, how the Certificate resource triggers cert-manager to request a certificate, and the two ACME challenge types: HTTP-01 (requires the ingress to serve a token at /.well-known/acme-challenge/) and DNS-01 (adds a TXT record to the DNS zone, required for wildcard certificates). The candidate should explain the tls stanza in the Ingress spec and that cert-manager stores the cert in a Secret. Mention monitoring cert expiry with the x509 Prometheus exporter.

Expect these follow-ups

• When would you use DNS-01 over HTTP-01 for ACME challenges?
• How do you handle certificate rotation for a service that caches TLS connections?

As asked

Your AWS bill jumped 30 percent last month and the team has no idea why. Walk me through how you would investigate, identify the culprit, and build a process to prevent uncontrolled cost growth.

Sample answer outline

Strong answers start with AWS Cost Explorer with grouping by service, linked account, and tag, looking for the month-over-month delta. The candidate should mention Cost Anomaly Detection for automated alerting, tag enforcement policies to attribute costs, and EC2 right-sizing recommendations from Compute Optimizer. Remediation includes switching to Savings Plans or Reserved Instances for baseline load, using Spot instances for batch workloads, and setting AWS Budgets with alerts. Mention CloudWatch alarms on estimated charges.

Expect these follow-ups

• How do Savings Plans differ from Reserved Instances and when would you choose each?
• How do you enforce tagging policies for cost allocation without blocking deployments?