DevOps engineer role-specific interview questions

As asked

Write Terraform that provisions a VPC across three availability zones with public and private subnets and a NAT gateway per AZ. Explain the cost tradeoffs.

Sample answer outline

Use a module that takes a CIDR and a count of AZs and emits subnet pairs per AZ. One NAT per AZ keeps egress availability isolated to the AZ but triples the NAT bill. A single shared NAT is cheaper but means an AZ outage on the NAT side takes down outbound traffic for everyone. Tag everything with owner and cost-centre so the bill is debuggable. Use remote state with locking, plan in CI, apply manually for shared environments.

Reference implementation (hcl)

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "prod"
  cidr = "10.0.0.0/16"

  azs             = ["eu-west-2a", "eu-west-2b", "eu-west-2c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway     = true
  single_nat_gateway     = false
  one_nat_gateway_per_az = true

  tags = { owner = "platform", env = "prod" }
}

Expect these follow-ups

• How would you swap the NAT gateway for a NAT instance and when is that worth it?
• Where do VPC endpoints fit into the cost story?
• How do you handle drift when someone edits the console by hand?

As asked

You have a TypeScript monorepo with 30 packages and a CI pipeline that takes 45 minutes. How would you restructure GitHub Actions to bring it under 10 minutes without dropping coverage?

Sample answer outline

Start with measurement: where is the time going. Common wins: matrix-parallelise tests by package, cache the package manager store and the build output keyed by lockfile and source hash, and only run tests for packages whose graph was actually touched (turbo, nx, pnpm filter). Move integration and e2e tests off the PR critical path into a nightly or post-merge job. Pin runner sizes for hot jobs. If the queue is the bottleneck, use larger runners or self-hosted on spot.

Expect these follow-ups

• How do you keep flaky tests from blocking PRs without ignoring them?
• What is your retry policy for transient infrastructure failures?
• When is a self-hosted runner worth the operational cost?

As asked

Your production database password has leaked in a log file. Walk me through rotating it with zero downtime across a fleet of 200 application pods.

Sample answer outline

Two passwords on the database account, or a dual-credential pattern (Vault dynamic credentials or AWS RDS IAM auth). Sequence: add the new credential, push it to the secret store, trigger a rolling restart so pods pick it up via the projected secret or a sidecar, verify all pods are on the new credential, then revoke the old one. Audit logs to confirm no connections used the old credential before revoking. Treat the leak itself separately: rotate, then investigate scope.

Reference implementation (bash)

# Postgres supports a window where two passwords are valid (via pgbouncer auth_query
# or by issuing both old and new for the same role). For RDS IAM auth, no static
# password is needed at all.

# 1. Stage new credential in the secret manager
aws secretsmanager update-secret \
  --secret-id prod/api/db \
  --secret-string '{"username":"api","password":"NEW_PASSWORD"}'

# 2. Trigger a rolling restart so pods pick up the new secret
kubectl rollout restart deployment/api -n prod
kubectl rollout status deployment/api -n prod --timeout=10m

# 3. Confirm no active sessions are using the old credential
psql -c "select usename, count(*) from pg_stat_activity group by 1;"

# 4. Revoke the old credential on the database
psql -c "alter user api with password 'NEW_PASSWORD';"

Expect these follow-ups

• What if your database does not support two simultaneous passwords?
• How do you prove the leak did not lead to data exfiltration?
• How would you automate this so the next rotation is one click?

As asked

A new Deployment rollout is stuck at 3 out of 10 pods Ready. Walk me through how you would debug it in production.

Sample answer outline

Start with kubectl rollout status and kubectl describe deployment. Look at the events: ImagePullBackOff, CrashLoopBackOff, FailedScheduling all point in different directions. Check pod events and pod logs for the failing replicas. Common causes: readiness probe failing because the app needs longer to start, resource requests too high for the available nodes, a config map or secret reference that does not exist, or a PodDisruptionBudget blocking eviction of old pods. Resist the urge to delete pods until you know the cause.

Reference implementation (bash)

# Triage sequence
kubectl rollout status deployment/api -n prod
kubectl describe deployment/api -n prod
kubectl get pods -n prod -l app=api
kubectl describe pod <pending-pod> -n prod
kubectl logs <crashlooping-pod> -n prod --previous
kubectl get events -n prod --sort-by=.lastTimestamp | tail -20

Expect these follow-ups

• What if the readiness probe passes but the service is still returning 503s?
• How do you set sane defaults for readiness and liveness probes?
• When would you use a startup probe?

As asked

Your CTO drops a 200k monthly AWS bill on your desk and asks for a 30 percent cut. Where do you start?

Sample answer outline

Get cost broken down by service and by team via tags. The biggest wins are usually in three places: idle or oversized EC2/EKS nodes (right-size with metrics, move to Graviton or spot for non-critical pools), under-utilised RDS or Redshift (look at CPU and IO, downsize or move to serverless variants), and egress (NAT gateway costs and inter-region transfer often hide). Reserved instances and savings plans for the predictable baseline. S3 lifecycle policies for cold data. Talk to teams before cutting, a 30 percent cut by surprise breaks trust.

Expect these follow-ups

• How do you handle a team that pushes back on right-sizing?
• What is your strategy for spot instance interruptions in a production workload?
• When does multi-cloud save money and when does it cost more?

As asked

Show me how you would configure a canary deployment with Argo Rollouts that progresses based on Prometheus-derived success rate.

Sample answer outline

Argo Rollouts replaces the Deployment resource with a Rollout that has explicit canary steps and analysis. Define the canary weight schedule (5, 25, 50, 100 percent), with pauses between steps. Add an AnalysisTemplate that queries Prometheus for the success rate over the canary version and fails the rollout if it drops below the threshold. The rollout aborts and rolls back automatically. Wire the dashboard and the alert so humans can see canary progress in real time.

Reference implementation (yaml)

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 5m }
        - analysis:
            templates: [{ templateName: success-rate }]
        - setWeight: 25
        - pause: { duration: 10m }
        - setWeight: 50
        - pause: { duration: 10m }
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: success-rate }
spec:
  metrics:
    - name: success-rate
      successCondition: result[0] >= 0.99
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{status!~"5.."}[2m]))
            / sum(rate(http_requests_total[2m]))

Expect these follow-ups

• How do you handle a slow-burn regression that only shows up after 30 minutes?
• What if your traffic split cannot be controlled at the service-mesh level?
• When would you pick Flagger over Argo Rollouts?

As asked

You have 30 services running in one AWS account. Design an IAM strategy that gives each service least privilege and keeps the policies maintainable.

Sample answer outline

One IAM role per service, assumed via the workload identity (IRSA for EKS, instance profile for EC2, task role for ECS). Policies generated from infrastructure code, not hand-edited. Use IAM Access Analyzer to flag overly permissive policies and unused permissions. Avoid wildcard resource ARNs. Separate the prod and non-prod accounts to remove the worst blast-radius failure modes. Service control policies at the org level for guardrails like 'no public S3'. Rotate credentials never used (CIS benchmarks). Treat IAM as code with review.

Reference implementation (json)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOwnConfigOnly",
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::prod-config/api/*"
    },
    {
      "Sid": "PublishToOwnTopic",
      "Effect": "Allow",
      "Action": ["sns:Publish"],
      "Resource": "arn:aws:sns:eu-west-2:123456789012:api-events"
    },
    {
      "Sid": "DenyAllOtherS3",
      "Effect": "Deny",
      "NotAction": ["s3:GetObject"],
      "Resource": "*"
    }
  ]
}

Expect these follow-ups

• How do you handle a service that needs cross-account access?
• What is your strategy for human access (engineers) to production?
• How would you scope a policy down from a wildcard that someone shipped 3 years ago?

As asked

An engineer commits an AWS access key to a public GitHub repo. Walk me through what you do in the next hour.

Sample answer outline

Treat the key as compromised. Rotate immediately, do not wait. Revoke the old key, generate a new one, push it via the secret manager so dependent services pick it up. Check CloudTrail for any usage of the key in the window between commit and rotation, especially from unfamiliar IPs. Remove the key from history (git filter-repo or BFG) but understand that GitHub may have already cached it and bots scrape new commits within seconds, so rotation is what matters, not rewriting history. Talk to the engineer without blame, and add pre-commit hooks (gitleaks, trufflehog) so it does not happen again.

Expect these follow-ups

• Why is rewriting history less important than rotating?
• How would you scan all current repos for past leaks?
• What is your policy for repeat offenders?

As asked

Your monthly cloud forecast jumps by 45 percent over a weekend. What do you check first, and how do you stop it happening again?

Sample answer outline

Break the bill down by service, region, account, tag, and usage type before guessing. Common causes are runaway data transfer, NAT gateway traffic, oversized compute, forgotten test clusters, log ingestion, or a warehouse job scanning far more data than expected. Stop the bleeding with quotas, budgets, anomaly alerts, and temporary scaling caps where they are safe. Then fix ownership: mandatory tags, showback or chargeback, and pull-request visibility for expensive infrastructure changes. The interviewer wants pragmatism, not just 'buy savings plans', because commitments can lock in waste.

Expect these follow-ups

• Which cost category is hardest to allocate back to a team?
• When do savings plans make the problem worse?
• How would you explain this incident to finance without hiding the engineering details?