DevOps engineer at Snowflake

As asked

A new Deployment rollout is stuck at 3 out of 10 pods Ready. Walk me through how you would debug it in production.

Sample answer outline

Start with kubectl rollout status and kubectl describe deployment. Look at the events: ImagePullBackOff, CrashLoopBackOff, FailedScheduling all point in different directions. Check pod events and pod logs for the failing replicas. Common causes: readiness probe failing because the app needs longer to start, resource requests too high for the available nodes, a config map or secret reference that does not exist, or a PodDisruptionBudget blocking eviction of old pods. Resist the urge to delete pods until you know the cause.

Reference implementation (bash)

# Triage sequence
kubectl rollout status deployment/api -n prod
kubectl describe deployment/api -n prod
kubectl get pods -n prod -l app=api
kubectl describe pod <pending-pod> -n prod
kubectl logs <crashlooping-pod> -n prod --previous
kubectl get events -n prod --sort-by=.lastTimestamp | tail -20

Expect these follow-ups

• What if the readiness probe passes but the service is still returning 503s?
• How do you set sane defaults for readiness and liveness probes?
• When would you use a startup probe?

As asked

Tell me about the worst on-call shift you have personally had. What broke, what did you do in the moment, and what changed afterwards?

Sample answer outline

Pick a real story with measurable user impact. Cover detection (was it self-detected or customer-reported), the diagnosis path (especially the dead ends), the mitigation, and the postmortem actions that actually landed. Interviewers are listening for ownership without blame, calm under pressure, and process changes that prevent the next one. Avoid 'we just restarted it' answers, dig into the root cause.

Expect these follow-ups

• What did the postmortem find as the contributing factors?
• Did the action items actually ship, or did they slip?
• What would you do differently in the first 10 minutes if it happened again?

As asked

An engineer commits an AWS access key to a public GitHub repo. Walk me through what you do in the next hour.

Sample answer outline

Treat the key as compromised. Rotate immediately, do not wait. Revoke the old key, generate a new one, push it via the secret manager so dependent services pick it up. Check CloudTrail for any usage of the key in the window between commit and rotation, especially from unfamiliar IPs. Remove the key from history (git filter-repo or BFG) but understand that GitHub may have already cached it and bots scrape new commits within seconds, so rotation is what matters, not rewriting history. Talk to the engineer without blame, and add pre-commit hooks (gitleaks, trufflehog) so it does not happen again.

Expect these follow-ups

• Why is rewriting history less important than rotating?
• How would you scan all current repos for past leaks?
• What is your policy for repeat offenders?

As asked

Tell me about the biggest incident you handled that lived in the platform layer: a broken deploy pipeline, an infrastructure change gone wrong, a certificate expiry, or a cluster failure. Walk me through detection, mitigation, and what changed in the platform afterwards.

Sample answer outline

Frame it through a platform lens, where the blast radius is every team that depends on you. Describe the impact across consumers, not just one service. Detection: what alerted you, and whether it was your monitoring or a downstream team that noticed first. Mitigation: the rollback or break-glass procedure, and whether it existed before the incident or had to be improvised. The strong answer ends with platform-level prevention: a guardrail in the pipeline, a pre-deploy check, an expiry alert, automated rollback. Interviewers listen for ownership of shared infrastructure and the discipline to turn one painful event into a control that protects every team.

Expect these follow-ups

• Did a self-service guardrail exist, or did you have to build one after?
• How did you communicate with the many teams affected at once?
• What pipeline or infrastructure check would have caught this earlier?

As asked

Two engineers run terraform apply at the same time against the same workspace. What happens with state locking, what backend are you using, and what do you do if a lock is stuck after a failed apply?

Sample answer outline

Strong answers explain that backends like S3 with DynamoDB or Terraform Cloud implement state locking via DynamoDB conditional writes. The candidate should describe the lock record format, how to use terraform force-unlock with the lock ID found in the error output, and why force-unlocking without confirming no apply is running can corrupt state. Good answers also mention remote backends vs local and the risk of partial state writes.

Expect these follow-ups

• What does a corrupted terraform state file look like and how do you recover from it?
• How would you structure workspaces and state files for a multi-environment setup?

As asked

How do you inject secrets into a GitHub Actions pipeline without exposing them in logs or baking them into container images? Walk me through your approach from secret storage to the running container.

Sample answer outline

Strong answers cover GitHub Actions encrypted secrets or OIDC-based short-lived credentials to pull from AWS Secrets Manager or HashiCorp Vault at runtime. The candidate should explain why secrets must not be printed with echo, that GitHub masks registered secrets in logs but this can be defeated with base64 encoding, and how to use build-time ARGs versus runtime ENV correctly to avoid baking secrets into image layers. Mention secret scanning tools like truffleHog or GitGuardian as a defense layer.

Expect these follow-ups

• How do you rotate a secret that is already in use by a running Kubernetes deployment?
• What is OIDC federation and why is it preferable to long-lived AWS IAM access keys in CI?