Platform engineer role-specific interview questions

As asked

You inherit a cloud estate where every team uses admin-like roles in a shared account. How would you redesign access without blocking delivery?

Sample answer outline

Start by separating environments and blast radii: production from non-production, shared platform from application accounts, and human access from workload identity. Define roles around actions people actually need, then enforce least privilege with permission boundaries, short-lived credentials, and break-glass access that is audited. Use infrastructure-as-code and policy-as-code so access changes are reviewed, repeatable, and testable. Strong answers mention migration sequencing because cutting everyone over in one move usually breaks deployments. Candidates often trip up by describing perfect IAM theory without a path from the current unsafe state.

Expect these follow-ups

• How do you handle a senior engineer who insists they need admin for debugging?
• Where would you use service control policies or organisation policies?
• What audit signals tell you the new model is working?

As asked

Walk me through exactly what happens inside the Kubernetes scheduler from the moment a pod is created until it is bound to a node. What are the two main phases, what plugins run in each, and where would you intervene if pods were stuck in Pending?

Sample answer outline

Strong answers cover the two-phase model: filtering (predicates) removes ineligible nodes, scoring (priorities) ranks the survivors. The scheduler uses a plugin framework so candidates name real plugins like NodeResourcesFit, PodTopologySpread, TaintToleration, and InterPodAffinity. For Pending pods they would check Events, describe the pod, inspect scheduler logs, and consider whether resource quotas, taints, or missing node selectors are blocking assignment.

Expect these follow-ups

• What is the extender mechanism, and when would you write a custom scheduler extender versus a custom scheduler plugin?
• How does descheduler complement the scheduler, and which eviction policies have you used?

As asked

Explain how Terraform implements state locking when using an S3 backend with DynamoDB. What happens if a lock is never released after a crash, how do you safely recover, and what design choices reduce the blast radius of a corrupted state file?

Sample answer outline

Candidates should describe that Terraform writes a lock entry to a DynamoDB table before any operation and deletes it when done. If the process crashes the lock remains; recovery requires terraform force-unlock with the lock ID. To reduce blast radius: separate state files per environment and per component so a corruption in one area does not affect others, enable S3 versioning to roll back, and use workspaces or directory-per-env patterns.

Expect these follow-ups

• What is the risk of running terraform apply with a partial state file, and how does the -refresh=false flag change the behavior?
• How do you move resources between state files without destroying and recreating them?

As asked

We have 15 product teams sharing a single cluster. Each team owns a set of namespaces. Developers should deploy workloads but not touch cluster-level resources. Platform engineers need cluster-wide read access. SREs need the ability to restart pods anywhere but not modify RBAC. Sketch the Role, ClusterRole, and binding strategy.

Sample answer outline

Strong answers distinguish Roles (namespaced) from ClusterRoles (cluster-wide) and show that ClusterRoles can be bound at namespace scope with RoleBindings. Developers get a ClusterRole with common verbs (get/list/watch/create/delete on pods, deployments, services, configmaps) bound per namespace via RoleBinding. Platform engineers get a ClusterRole with read-only verbs on all resources via ClusterRoleBinding. SREs get a ClusterRole with patch/delete on pods only, cluster-wide.

Expect these follow-ups

• How do you audit who has access to what across all namespaces? Which tools or commands help?
• What is impersonation in Kubernetes RBAC and what legitimate use case does it serve?

As asked

A team checks in a 2.4 GB Node.js Docker image. You need to get it below 300 MB and reduce CVE exposure. Walk through the specific changes you would make to the Dockerfile, base image choice, and build pipeline to achieve this.

Sample answer outline

Switch from node:18 to node:18-alpine or a distroless base. Use multi-stage builds to separate the build stage (with devDependencies) from the runtime stage (production deps only). Combine RUN commands to avoid layer bloat, use .dockerignore to exclude node_modules and test files from the build context, and add npm ci --omit=dev (the --only=production flag was deprecated in npm v7). For distroless, copy just the app files and node_modules into the final stage. Add Trivy or Grype in CI to gate on critical CVEs.

Expect these follow-ups

• What is the difference between a distroless image and a scratch image, and when is each appropriate?
• How do you handle the case where a production npm dependency pulls in a CVE-flagged transitive package?

As asked

Your tracing bill is too high, but engineers rely on traces to debug rare checkout failures. How would you change the sampling strategy?

Sample answer outline

Separate head sampling, which decides early and is cheap, from tail sampling, which can keep traces based on outcome after the request completes. Keep all error traces, slow traces, and traces for important business flows, while sampling routine successful traffic at a lower rate. Propagate sampling decisions consistently across services so one request does not produce broken partial traces. Use attributes with bounded cardinality to drive retention rules, and validate with engineers that rare failure modes remain findable. A weak answer simply lowers the global sampling percentage and loses the exact traces people need during incidents.

Expect these follow-ups

• What metadata must be propagated for tracing to work across services?
• When is tail sampling too expensive or too late?
• How would you prove the new sampling policy is not hiding production bugs?

As asked

A cluster starts throwing 'etcdserver: mvcc: database space exceeded' errors. Explain what caused this, what compaction and defragmentation do differently, and what steps you take to recover without downtime.

Sample answer outline

Candidates should explain etcd's MVCC model and how old revisions accumulate until compacted. Compaction removes old key revisions from the logical store but the on-disk bbolt file stays large until defragmentation physically reclaims space. Recovery involves running etcdctl compact, then etcdctl defrag on each member one at a time to avoid quorum loss, and setting --auto-compaction-retention to prevent recurrence.

Expect these follow-ups

• How do you set etcd quotas, and what is the default backend-bytes quota that often trips people up?
• What monitoring alerts would you add to catch etcd disk growth before it causes an outage?

As asked

We are running Cilium as our CNI. Walk me through the full packet path when pod A on node 1 sends a TCP packet to pod B on node 2. Where does eBPF intercept the packet, what is the role of the BPF map, and how does this differ from the path when using Calico with IPIP encapsulation?

Sample answer outline

A strong answer traces the packet from the veth pair inside the pod namespace, through the TC eBPF hook on the host-side veth, into the Cilium datapath which does an endpoint map lookup, then either routes directly via native routing or encapsulates with VXLAN/Geneve. Compared to Calico IPIP, Cilium avoids encap overhead with eBPF-based routing. Candidates should mention BPF_MAP_TYPE_HASH maps for policy enforcement and the CT (connection tracking) map.

Expect these follow-ups

• How do you troubleshoot a dropped packet with Cilium? Which CLI commands and eBPF debug options do you use?
• What is Cilium's Hubble, and how have you used it for network observability?

As asked

Our security team says every namespace must declare a LimitRange and ResourceQuota before any workload can be admitted. How would you implement this using OPA Gatekeeper? Describe the ConstraintTemplate structure, the Rego policy logic, and how you would test it before rolling it out to production clusters.

Sample answer outline

Candidates should describe a ConstraintTemplate with a Rego rule in spec.targets that receives the AdmissionReview object, checks for the existence of LimitRange and ResourceQuota objects in the namespace via the OPA data cache, and denies the request if they are absent. Testing involves the gator CLI for unit testing Rego and running against sample manifests before applying to a staging cluster. They should also mention constraint enforcement actions: deny vs warn vs dryrun.

Expect these follow-ups

• How do you handle the chicken-and-egg problem where the namespace itself needs to be created before the quota objects can exist?
• What is the difference between OPA Gatekeeper and Kyverno, and when would you pick each?

As asked

We have a queue-processing service and want HPA to scale based on the number of unprocessed messages in a RabbitMQ queue, exposed via Prometheus. Walk through the full pipeline: how the metric gets from RabbitMQ into Kubernetes HPA, what components are involved, and what happens when the metric server is unavailable.

Sample answer outline

Strong answers trace the path: RabbitMQ exposes metrics via its exporter, Prometheus scrapes them, KEDA or the Prometheus Adapter translates them into the external.metrics.k8s.io API, and HPA queries that API. With KEDA the candidate describes ScaledObject and TriggerAuthentication. When the metric server is unavailable, HPA stops scaling in both directions and sets a condition indicating metrics are unavailable. It does not fall back silently; it preserves the current replica count until metrics become available again, which is why availability of the adapter matters.

Expect these follow-ups

• What is KEDA's ScaledJob, and when would you use it instead of ScaledObject?
• How do you prevent HPA and cluster autoscaler from fighting each other during a scale-down event?

As asked

We use Flux v2 with GitRepository, Kustomization, and HelmRelease objects. Walk me through the reconciliation loop for a HelmRelease: what controllers are involved, what happens when a Helm chart value changes in Git, and how do you debug a HelmRelease stuck in a failed state?

Sample answer outline

Candidates describe the source-controller fetching the Git repo and producing an Artifact, then the helm-controller watching HelmRelease objects, fetching the chart from the Artifact, and running helm upgrade. On failure the HelmRelease status condition shows the error and the controller respects retryInterval. Debugging involves kubectl get helmrelease -o yaml to read status.conditions, flux logs --kind=HelmRelease, and looking at Helm release history with helm history.

Expect these follow-ups

• How do you handle secrets in a GitOps workflow without committing them to the repo? Walk through a Sealed Secrets or SOPS-based approach.
• What is the difference between Flux's remediation.retries and the suspend flag, and when would you set each?

As asked

Explain the scale-up and scale-down logic in cluster autoscaler. What triggers a scale-up? What safeguards prevent aggressive scale-down from evicting critical workloads? We have seen nodes stay at 60% utilization but never scale down. Walk through the likely causes.

Sample answer outline

Scale-up triggers when an unschedulable pod exists and CA simulates whether a new node would fit it. Scale-down fires when a node's requested resources are below the scale-down-utilization-threshold (default 50%) for a sustained period. Common reasons a node never scales down: pods without proper resource requests (CA uses requests not actual usage), pods with PodDisruptionBudgets that block eviction, pods with local storage emptyDir, or DaemonSet pods which CA ignores.

Expect these follow-ups

• What is the difference between cluster autoscaler and Karpenter? When would you choose Karpenter?
• How do you configure cluster autoscaler to respect pod priority during scale-down?

As asked

A service mesh migration introduced NetworkPolicies and now a frontend pod cannot reach a backend pod in the same namespace. Ingress and egress are enabled. How do you systematically isolate whether the issue is a NetworkPolicy, DNS, a service selector mismatch, or something at the CNI level?

Sample answer outline

Start by exec-ing into the frontend pod and trying curl against the backend's ClusterIP to isolate DNS vs connectivity. Check NetworkPolicy with kubectl get networkpolicy -n <ns> -o yaml and verify the podSelector and namespaceSelector labels match the actual pod labels. Use CNI-specific tools (cilium monitor or Calico calicoctl trace) to trace dropped packets. Verify the service selector matches pod labels with kubectl get endpoints.

Expect these follow-ups

• What does an empty podSelector on a NetworkPolicy ingress rule mean, and how is that different from omitting the ingress field entirely?
• How do you write a NetworkPolicy that allows DNS egress to kube-dns without opening all UDP traffic?

As asked

We use a pre-upgrade Helm hook to run database migrations before a new chart version deploys. Explain exactly when the hook job runs relative to the main manifests, what happens to the job after it completes, and what failure scenarios can leave a release stuck in a pending-upgrade state.

Sample answer outline

Pre-upgrade hooks run after Helm marks the release as pending-upgrade but before any non-hook resources are applied. The hook job is deleted based on its hook-delete-policy annotation (before-hook-creation by default, hook-succeeded, or hook-failed). If the job fails and the policy does not delete it, the next upgrade attempt fails because the job exists. Stuck releases need helm rollback or manual kubectl delete of the hook job.

Expect these follow-ups

• How does helm.sh/hook-weight control ordering when multiple hooks have the same phase?
• What is the risk of running schema migrations as a pre-upgrade hook, and how would you make them backward-compatible?

As asked

We deploy a stack that requires a Namespace, then a Secret from an external-secrets operator, then a Deployment that reads that secret. Using Argo CD sync waves, how do you guarantee this ordering? What are the limits of sync waves, and what do you use when waves are insufficient?

Sample answer outline

Sync waves use the argocd.argoproj.io/sync-wave annotation with integer values; Argo CD applies resources in wave order and waits for each wave to be healthy before proceeding. The Namespace gets wave 0, the ExternalSecret gets wave 1, the Deployment gets wave 2. Limits: waves do not wait for CRD-created resources to be ready if the health check is not configured. For finer control, sync hooks (PreSync, Sync, PostSync) or resource health checks with custom Lua scripts fill the gap.

Expect these follow-ups

• What is Argo CD's resource health customization, and how do you write a Lua script to declare an ExternalSecret healthy only when its status shows synced?
• How does Argo CD handle a situation where two Applications manage the same resource?

As asked

We have a Kafka cluster with 3 brokers running as a StatefulSet. Explain how you would configure PodDisruptionBudgets to allow cluster autoscaler to scale down nodes while keeping the Kafka cluster available during voluntary disruptions. What value do you choose for minAvailable or maxUnavailable, and why?

Sample answer outline

For a 3-broker Kafka cluster you need at least 2 in-sync replicas to maintain availability with default replication factor 3, so minAvailable: 2 is the right choice. This lets cluster autoscaler evict one pod at a time. The PDB selector must match the StatefulSet pods exactly. Candidates should note that PDB only covers voluntary disruptions and that anti-affinity rules on different nodes prevent the remaining 2 from co-locating, which would leave only 1 node that can be drained.

Expect these follow-ups

• How does a PDB interact with kubectl drain versus a node hardware failure?
• What happens if a PDB blocks a node drain indefinitely and the node needs urgent maintenance?

As asked

Describe two different patterns for getting a Vault-managed secret into a Kubernetes pod: the Vault Agent Injector sidecar approach and the External Secrets Operator approach. What are the operational tradeoffs, and which would you recommend for a new platform?

Sample answer outline

Vault Agent Injector uses a mutating webhook to inject a sidecar that authenticates via Kubernetes service account, fetches secrets, writes them to a shared in-memory volume, and keeps them rotated. External Secrets Operator creates Kubernetes Secret objects from Vault, which means secrets are persisted in etcd (encrypted at rest if configured) and native tooling like envFrom works without sidecar overhead. ESO is simpler operationally but trades the zero-persistence property of the sidecar approach.

Expect these follow-ups

• How does the Kubernetes Auth method in Vault work under the hood? What is the TokenReview API call?
• What happens to running pods if Vault goes down and they need secret rotation?

As asked

We have a Prometheus that is using 120 GB of memory and scrape intervals are timing out. You investigate and find that one team is exposing a metric with a user_id label that has 2 million unique values. Explain what high cardinality means, why it is so destructive, and what you do to fix it both short-term and long-term.

Sample answer outline

Each unique label combination creates a separate time series in the TSDB, so a 2M user_id label generates 2M series just for that one metric. Short-term: add a metric_relabel_config to drop the label or the metric entirely. Long-term: set cardinality limits at the scrape config level, use recording rules to pre-aggregate, move high-cardinality data to a log-based backend or Tempo/Loki, and implement metric review gates in the golden path CI pipeline. Mimir or Thanos with per-tenant cardinality limits also help.

Expect these follow-ups

• What is the difference between out-of-order samples and the duplicate sample error in Prometheus, and what causes each?
• How does Prometheus TSDB's two-hour block compaction model affect query performance over long time ranges?

As asked

We have 20 product teams on a shared cluster with a total of 500 CPU cores and 2 TB RAM. How do you implement ResourceQuotas, LimitRanges, and priority classes so that no single team can starve others, burst capacity is available for urgent workloads, and platform components (ingress, monitoring) are protected?

Sample answer outline

Set per-namespace ResourceQuotas sizing each team's baseline plus a burst allowance. LimitRanges enforce default request/limit ratios so pods without explicit resources still count against the quota. Define three PriorityClasses: system-cluster-critical for platform components (preempts others), high for urgent team workloads, normal for steady-state. The ResourceQuota can scope by priorityClass so bursting teams cannot consume the system-critical priority pool. Candidates should mention the Hierarchical Namespace Controller for nested quota inheritance.

Expect these follow-ups

• What happens when a team's namespace quota is reached but their deployment needs to scale due to high traffic? How do they request an increase?
• How do you model chargeback or showback per team using the quota and actual usage data?

As asked

A team wants to roll out a new version of their service to 5% of traffic, monitor error rate, and automatically promote or rollback based on a Prometheus metric. Walk through the Istio VirtualService and DestinationRule configuration, the automation layer you would build, and how Argo Rollouts fits into this.

Sample answer outline

A DestinationRule defines two subsets (stable and canary) using pod label selectors. A VirtualService splits traffic by weight (95/5) between subsets. Argo Rollouts replaces the standard Deployment with a Rollout object that manages the CanaryStrategy, integrating with Istio via the rollouts-traffic-routing annotation. It uses AnalysisTemplates to query Prometheus for the error rate metric and calls setWeight to increment canary traffic automatically on success or trigger rollback on failure.

Expect these follow-ups

• How do you ensure that a single user's session is consistently routed to the canary or stable version across multiple requests?
• What is the difference between a blue-green and a canary deployment from a database migration perspective?

As asked

We have a cluster with three node groups: on-demand general, spot general, and on-demand GPU. Describe how you use taints, tolerations, nodeSelectors, and nodeAffinity together to ensure GPU workloads run only on GPU nodes, spot-tolerant batch jobs prefer spot but fall back to on-demand, and stateful services never land on spot.

Sample answer outline

Taint GPU nodes with nvidia.com/gpu=present:NoSchedule so only pods with the matching toleration can land there. Taint spot nodes with spot=true:NoSchedule. Spot-tolerant batch jobs add the spot toleration and use preferredDuringSchedulingIgnoredDuringExecution with a weight to prefer spot nodes but not require them. Stateful services simply do not have the spot toleration, so the NoSchedule taint keeps them off spot. Node selectors or nodeAffinity with requiredDuringScheduling enforce GPU workloads to the correct pool.

Expect these follow-ups

• What happens to pods already running on a spot node when the taint is added after the fact?
• How do you handle the case where a spot node is terminated by AWS mid-deployment and the pod needs to reschedule?

As asked

We want all teams to use a standard CI pipeline that handles building, scanning, testing, and deploying container images. The pipeline must be opinionated enough to enforce security gates but flexible enough for teams with different tech stacks. How do you design this? What is your templating strategy, and how do you roll out updates to 50 pipelines without breaking everyone?

Sample answer outline

Use pipeline-as-code templates (GitHub Actions reusable workflows, GitLab CI includes, or Tekton pipelines via a catalog). Teams reference the template at a pinned version tag, consuming it like a library. The template encodes required stages (SAST via Semgrep, image build, Trivy scan, push, deploy) and exposes extension points via inputs. Updates follow a deprecation cycle: the new version is released, teams have a migration window, and automated PRs update their version pins. A renovate bot or internal tooling automates the pin upgrades.

Expect these follow-ups

• How do you enforce that teams cannot skip the security scan stage even if they override the template?
• What metrics do you collect from the golden-path pipeline to measure developer experience, and how do you alert on pipeline health?

As asked

A StatefulSet is stuck because its PVCs are in Pending state. Describe your debugging steps: how do you distinguish between a missing StorageClass, a CSI driver issue, a node without the volume plugin installed, and a capacity problem in the underlying storage backend?

Sample answer outline

Start with kubectl describe pvc to read the events: a missing StorageClass shows a 'no provisioner found' error, a CSI driver issue shows provisioner errors in the CSI controller pod logs, a node problem shows no-volume-plugin events and requires checking the DaemonSet for the CSI node driver. For EBS on AWS, check for zone mismatches between the PVC and available nodes. Use kubectl get csinode to verify CSI topology annotations. Storage capacity exhaustion on the backend appears as provisioning failures in the external-provisioner logs.

Expect these follow-ups

• What is the volumeBindingMode: WaitForFirstConsumer setting and when is it critical to use it?
• How do you safely expand a PVC that is attached to a running pod?

As asked

How do you run OPA policies against Kubernetes YAML in a CI pipeline before anything reaches the cluster? Describe the conftest setup, how you write Rego tests for the policies themselves, and how you handle policies that need to query cluster state (like checking whether an image tag already exists in the registry).

Sample answer outline

Conftest reads Kubernetes manifests and evaluates them against Rego policies in a policy/ directory, failing the CI job if any deny rule fires. Rego tests live alongside the policies using the testing.Rego framework with mock input fixtures. For policies that need cluster state (image existence, quota headroom), the CI job must call the relevant API (registry API for tag checks, Kubernetes API for quota) and inject the data as a conftest data document using --data. This keeps policies pure but requires CI to have appropriate read permissions.

Expect these follow-ups

• What is the difference between conftest test and conftest verify? Which do you run in CI?
• How do you structure a Rego policy that applies different rules to production namespaces versus development namespaces?

As asked

We are standardizing on OpenTelemetry and need all 50 services to send traces, metrics, and logs to a central backend. Describe how you deploy and configure the OTel Collector as a DaemonSet sidecar versus a deployment, what processors you include in the pipeline, and how you handle backpressure when the backend is slow.

Sample answer outline

A DaemonSet collector per node handles log collection efficiently and avoids inter-node traffic for traces and metrics. A gateway deployment in front of the backend handles batching, sampling, and routing to multiple backends. The pipeline includes the memory_limiter processor first to prevent OOM, then batch for efficiency, then an attributes processor to add cluster and environment metadata. Backpressure is handled via the receiver queue and the exporter retry/queue settings; the memory_limiter sheds data before the collector crashes.

Expect these follow-ups

• How do you implement tail-based sampling in the OTel Collector to keep only traces that contain errors or high latency spans?
• What is the difference between OTLP/gRPC and OTLP/HTTP and when does the choice matter?