Cloud engineer coding interview questions

As asked

You need to create one S3 bucket per environment from a list. Compare count and for_each. Which do you use and why? What breaks if you use count and then remove an item from the middle of the list?

Sample answer outline

Candidate must explain that count uses integer indices so removing an item from the middle shifts all subsequent indices, causing Terraform to plan destroy-and-recreate for every resource after the removed one. for_each uses map keys as stable identifiers so removing one key only destroys that resource. Always prefer for_each when the set of items may change.

Reference implementation (hcl)

variable "environments" {
  default = ["dev", "staging", "prod"]
}

# count approach - fragile
resource "aws_s3_bucket" "env_bucket_count" {
  count  = length(var.environments)
  bucket = "myapp-${var.environments[count.index]}"
}

# for_each approach - stable
resource "aws_s3_bucket" "env_bucket" {
  for_each = toset(var.environments)
  bucket   = "myapp-${each.key}"
}

Expect these follow-ups

• How do you convert a list to a set for use with for_each?
• What does toset() do and when is it needed?

As asked

A Lambda function processes SQS messages in batches of 10. One message in a batch fails. What happens by default and how do you fix it so only the failed message is retried?

Sample answer outline

By default the entire batch is retried when Lambda throws, meaning 9 already-succeeded messages are processed again (causing duplicates). The fix is enabling ReportBatchItemFailures and returning a SQS batch response object from the Lambda handler that lists only the failed messageId values. Candidate should write the response shape correctly.

Reference implementation (python)

import json

def handler(event, context):
    failures = []
    for record in event['Records']:
        try:
            process(record['body'])
        except Exception as e:
            failures.append({'itemIdentifier': record['messageId']})
    # Return only failed message IDs
    return {'batchItemFailures': failures}

def process(body):
    data = json.loads(body)
    # ... business logic ...

Expect these follow-ups

• What happens to a message that fails more times than the maxReceiveCount on the SQS queue?
• How do you prevent poison-pill messages from blocking the queue forever?

As asked

You need CloudFormation to create a resource type that AWS does not natively support. Show me how you would implement a custom resource backed by Lambda.

Sample answer outline

Candidate should show the Lambda sending a response to the pre-signed S3 URL provided in the event (cfnresponse module), handling Create/Update/Delete events, and always sending a response even on failure so CloudFormation does not time out. Key pitfalls: forgetting to send SUCCESS/FAILED, not handling the physical resource ID correctly on Update.

Reference implementation (python)

import json, urllib.request

def handler(event, context):
    response = {
        'Status': 'SUCCESS',
        'PhysicalResourceId': event.get('PhysicalResourceId', 'my-resource'),
        'StackId': event['StackId'],
        'RequestId': event['RequestId'],
        'LogicalResourceId': event['LogicalResourceId'],
        'Data': {}
    }
    try:
        if event['RequestType'] == 'Create':
            response['Data'] = {'Result': 'created'}
    except Exception as e:
        response['Status'] = 'FAILED'
        response['Reason'] = str(e)
    finally:
        body = json.dumps(response).encode()
        req = urllib.request.Request(event['ResponseURL'],
            data=body, method='PUT')
        urllib.request.urlopen(req)

Expect these follow-ups

• What happens if your Lambda throws an unhandled exception without sending a response?
• How long does CloudFormation wait for a custom resource response before timing out?

As asked

Write a Python function that returns a list of all object keys in an S3 bucket, handling pagination correctly. What breaks if you use list_objects_v2 without a paginator?

Sample answer outline

Without a paginator, list_objects_v2 returns at most 1000 objects. Candidate should use boto3 paginators or manually track the ContinuationToken. The paginator approach is cleaner. They should also handle the case where the bucket might be empty.

Reference implementation (python)

import boto3

def list_all_keys(bucket_name: str) -> list[str]:
    s3 = boto3.client('s3')
    paginator = s3.get_paginator('list_objects_v2')
    keys = []
    for page in paginator.paginate(Bucket=bucket_name):
        for obj in page.get('Contents', []):
            keys.append(obj['Key'])
    return keys

Expect these follow-ups

• How would you parallelise this across many prefixes to speed it up?
• What is the performance difference between list_objects and list_objects_v2?

As asked

Write a Bash script that finds all files in /data larger than 100 MB and uploads them to s3://my-bucket/large-files/ using the AWS CLI, preserving the relative path.

Sample answer outline

Candidate should use find with -size +100M, loop over results, strip the /data prefix to preserve relative path structure, and call aws s3 cp. A strong answer also handles spaces in filenames by quoting variables and uses --storage-class for cost if these are archival files.

Reference implementation (bash)

#!/usr/bin/env bash
set -euo pipefail

BASE=/data
BUCKET=s3://my-bucket/large-files

find "$BASE" -type f -size +100M | while IFS= read -r file; do
  relative="${file#$BASE/}"
  aws s3 cp "$file" "$BUCKET/$relative"
done

Expect these follow-ups

• How would you parallelize this with xargs or GNU parallel?
• What flag on aws s3 sync would achieve the same thing without a custom script?

As asked

You have a list of CIDR blocks that should all be allowed on port 443 for a security group. Show how you would write this in Terraform without repeating an ingress block for each CIDR.

Sample answer outline

Candidate should use a dynamic block with for_each iterating over the list of CIDRs, producing one ingress rule per CIDR. Alternative: pass all CIDRs as the cidr_blocks list in a single ingress block if AWS supports multiple CIDRs in one rule (it does). Both approaches should be shown and compared.

Reference implementation (hcl)

variable "allowed_cidrs" {
  type    = list(string)
  default = ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
}

resource "aws_security_group" "web" {
  name   = "web-sg"
  vpc_id = var.vpc_id

  dynamic "ingress" {
    for_each = var.allowed_cidrs
    content {
      from_port   = 443
      to_port     = 443
      protocol    = "tcp"
      cidr_blocks = [ingress.value]
    }
  }
}

Expect these follow-ups

• What is the difference between inline ingress blocks and aws_security_group_rule resources?
• Why might you prefer separate aws_security_group_rule resources over inline blocks?

As asked

You call aws s3.put_object in a tight loop and occasionally hit ThrottlingException. Write a utility that wraps an AWS API call with exponential backoff and jitter.

Sample answer outline

Candidate should implement a retry loop with sleep time doubling each attempt, capped at a maximum, plus random jitter to prevent thundering herd. They should catch the specific exception type from botocore.exceptions. Note that boto3 already has built-in retry with Config, so a strong answer also mentions that and explains when you need your own wrapper on top.

Reference implementation (python)

import random, time
from botocore.exceptions import ClientError

def with_backoff(func, max_attempts=5, base_delay=0.5, max_delay=30):
    for attempt in range(max_attempts):
        try:
            return func()
        except ClientError as e:
            code = e.response['Error']['Code']
            if code not in ('ThrottlingException', 'RequestLimitExceeded'):
                raise
            if attempt == max_attempts - 1:
                raise
            delay = min(base_delay * (2 ** attempt), max_delay)
            time.sleep(delay + random.uniform(0, delay * 0.1))

Expect these follow-ups

• What is the botocore Config parameter that controls built-in retries?
• What is the difference between standard and adaptive retry modes in boto3?

As asked

Write a CloudWatch Logs Insights query that finds Lambda invocations where duration exceeded 5000 ms in the last 24 hours, returning the requestId, duration, and billed duration, sorted by duration descending.

Sample answer outline

Candidate should use parse to extract fields from the REPORT log line, filter on Duration > 5000, and sort descending. They should know the REPORT line format and the difference between Duration and Billed Duration. A strong answer also extracts Max Memory Used to correlate memory pressure with latency.

Reference implementation (sql)

fields @timestamp, @requestId
| filter @message like /REPORT/
| parse @message 'Duration: * ms' as duration
| parse @message 'Billed Duration: * ms' as billedDuration
| parse @message 'Request ID: *\t' as requestId
| filter duration > 5000
| sort duration desc
| limit 50

Expect these follow-ups

• How would you save this query and run it on a schedule to generate a metric?
• What is the difference between filter and stats in Logs Insights?

As asked

Write a Go function that downloads 50 S3 objects concurrently, limits parallelism to 10, and returns all errors without cancelling successful downloads.

Sample answer outline

Candidate should use a semaphore (buffered channel of capacity 10) and a sync.WaitGroup, collecting errors in a slice protected by a sync.Mutex or using an errgroup. They should avoid closing the result channel prematurely. Strong answers also mention the AWS SDK v2 s3manager downloader for large objects.

Reference implementation (go)

package main

import (
    "context"
    "fmt"
    "sync"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

func downloadAll(ctx context.Context, client *s3.Client, bucket string, keys []string) []error {
    sem := make(chan struct{}, 10)
    var mu sync.Mutex
    var wg sync.WaitGroup
    var errs []error
    for _, key := range keys {
        wg.Add(1)
        sem <- struct{}{}
        go func(k string) {
            defer wg.Done()
            defer func() { <-sem }()
            _, err := client.GetObject(ctx, &s3.GetObjectInput{
                Bucket: &bucket, Key: &k,
            })
            if err != nil {
                mu.Lock()
                errs = append(errs, fmt.Errorf("%s: %w", k, err))
                mu.Unlock()
            }
        }(key)
    }
    wg.Wait()
    return errs
}

Expect these follow-ups

• What does the s3manager.Downloader do that a raw GetObject call does not?
• How would you handle partial downloads if your context is cancelled mid-flight?

As asked

A Node.js Lambda function's memory usage grows per invocation and eventually hits the memory limit, causing OOM crashes. How do you investigate and resolve this in a Lambda context?

Sample answer outline

Candidate should mention that Lambda reuses execution environments between warm invocations, so module-level state persists. Common causes: event listeners appended on every invocation, caches growing unboundedly, or unclosed DB connections. Investigation uses CloudWatch Metrics (memory used from REPORT line), X-Ray, or adding process.memoryUsage() logging. Fix involves capping caches, removing listeners on handler exit, and structuring singletons correctly at module scope.

Reference implementation (javascript)

// BAD: listener added on every warm invocation
exports.handler = async (event) => {
  someEmitter.on('data', handler); // grows unboundedly
  return process(event);
};

// GOOD: register at module scope, runs once per execution env
const handler = (data) => { /* ... */ };
someEmitter.on('data', handler);

exports.handler = async (event) => {
  return process(event);
};

Expect these follow-ups

• What is the execution environment lifecycle and how does it affect module-level variables?
• How does the Lambda runtime differ from a long-running Node.js server in terms of garbage collection opportunities?

As asked

You need to create one S3 bucket per environment from a list. Compare count and for_each. Which do you use and why? What breaks if you use count and then remove an item from the middle of the list?

Sample answer outline

Candidate must explain that count uses integer indices so removing an item from the middle shifts all subsequent indices, causing Terraform to plan destroy-and-recreate for every resource after the removed one. for_each uses map keys as stable identifiers so removing one key only destroys that resource. Always prefer for_each when the set of items may change.

Reference implementation (hcl)

variable "environments" {
  default = ["dev", "staging", "prod"]
}

# count approach - fragile
resource "aws_s3_bucket" "env_bucket_count" {
  count  = length(var.environments)
  bucket = "myapp-${var.environments[count.index]}"
}

# for_each approach - stable
resource "aws_s3_bucket" "env_bucket" {
  for_each = toset(var.environments)
  bucket   = "myapp-${each.key}"
}

Expect these follow-ups

• How do you convert a list to a set for use with for_each?
• What does toset() do and when is it needed?

As asked

A Lambda function processes SQS messages in batches of 10. One message in a batch fails. What happens by default and how do you fix it so only the failed message is retried?

Sample answer outline

By default the entire batch is retried when Lambda throws, meaning 9 already-succeeded messages are processed again (causing duplicates). The fix is enabling ReportBatchItemFailures and returning a SQS batch response object from the Lambda handler that lists only the failed messageId values. Candidate should write the response shape correctly.

Reference implementation (python)

import json

def handler(event, context):
    failures = []
    for record in event['Records']:
        try:
            process(record['body'])
        except Exception as e:
            failures.append({'itemIdentifier': record['messageId']})
    # Return only failed message IDs
    return {'batchItemFailures': failures}

def process(body):
    data = json.loads(body)
    # ... business logic ...

Expect these follow-ups

• What happens to a message that fails more times than the maxReceiveCount on the SQS queue?
• How do you prevent poison-pill messages from blocking the queue forever?

As asked

You need CloudFormation to create a resource type that AWS does not natively support. Show me how you would implement a custom resource backed by Lambda.

Sample answer outline

Candidate should show the Lambda sending a response to the pre-signed S3 URL provided in the event (cfnresponse module), handling Create/Update/Delete events, and always sending a response even on failure so CloudFormation does not time out. Key pitfalls: forgetting to send SUCCESS/FAILED, not handling the physical resource ID correctly on Update.

Reference implementation (python)

import json, urllib.request

def handler(event, context):
    response = {
        'Status': 'SUCCESS',
        'PhysicalResourceId': event.get('PhysicalResourceId', 'my-resource'),
        'StackId': event['StackId'],
        'RequestId': event['RequestId'],
        'LogicalResourceId': event['LogicalResourceId'],
        'Data': {}
    }
    try:
        if event['RequestType'] == 'Create':
            response['Data'] = {'Result': 'created'}
    except Exception as e:
        response['Status'] = 'FAILED'
        response['Reason'] = str(e)
    finally:
        body = json.dumps(response).encode()
        req = urllib.request.Request(event['ResponseURL'],
            data=body, method='PUT')
        urllib.request.urlopen(req)

Expect these follow-ups

• What happens if your Lambda throws an unhandled exception without sending a response?
• How long does CloudFormation wait for a custom resource response before timing out?

As asked

Write a Python function that returns a list of all object keys in an S3 bucket, handling pagination correctly. What breaks if you use list_objects_v2 without a paginator?

Sample answer outline

Without a paginator, list_objects_v2 returns at most 1000 objects. Candidate should use boto3 paginators or manually track the ContinuationToken. The paginator approach is cleaner. They should also handle the case where the bucket might be empty.

Reference implementation (python)

import boto3

def list_all_keys(bucket_name: str) -> list[str]:
    s3 = boto3.client('s3')
    paginator = s3.get_paginator('list_objects_v2')
    keys = []
    for page in paginator.paginate(Bucket=bucket_name):
        for obj in page.get('Contents', []):
            keys.append(obj['Key'])
    return keys

Expect these follow-ups

• How would you parallelise this across many prefixes to speed it up?
• What is the performance difference between list_objects and list_objects_v2?

As asked

Write a Bash script that finds all files in /data larger than 100 MB and uploads them to s3://my-bucket/large-files/ using the AWS CLI, preserving the relative path.

Sample answer outline

Candidate should use find with -size +100M, loop over results, strip the /data prefix to preserve relative path structure, and call aws s3 cp. A strong answer also handles spaces in filenames by quoting variables and uses --storage-class for cost if these are archival files.

Reference implementation (bash)

#!/usr/bin/env bash
set -euo pipefail

BASE=/data
BUCKET=s3://my-bucket/large-files

find "$BASE" -type f -size +100M | while IFS= read -r file; do
  relative="${file#$BASE/}"
  aws s3 cp "$file" "$BUCKET/$relative"
done

Expect these follow-ups

• How would you parallelize this with xargs or GNU parallel?
• What flag on aws s3 sync would achieve the same thing without a custom script?

As asked

You have a list of CIDR blocks that should all be allowed on port 443 for a security group. Show how you would write this in Terraform without repeating an ingress block for each CIDR.

Sample answer outline

Candidate should use a dynamic block with for_each iterating over the list of CIDRs, producing one ingress rule per CIDR. Alternative: pass all CIDRs as the cidr_blocks list in a single ingress block if AWS supports multiple CIDRs in one rule (it does). Both approaches should be shown and compared.

Reference implementation (hcl)

variable "allowed_cidrs" {
  type    = list(string)
  default = ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
}

resource "aws_security_group" "web" {
  name   = "web-sg"
  vpc_id = var.vpc_id

  dynamic "ingress" {
    for_each = var.allowed_cidrs
    content {
      from_port   = 443
      to_port     = 443
      protocol    = "tcp"
      cidr_blocks = [ingress.value]
    }
  }
}

Expect these follow-ups

• What is the difference between inline ingress blocks and aws_security_group_rule resources?
• Why might you prefer separate aws_security_group_rule resources over inline blocks?

As asked

You call aws s3.put_object in a tight loop and occasionally hit ThrottlingException. Write a utility that wraps an AWS API call with exponential backoff and jitter.

Sample answer outline

Candidate should implement a retry loop with sleep time doubling each attempt, capped at a maximum, plus random jitter to prevent thundering herd. They should catch the specific exception type from botocore.exceptions. Note that boto3 already has built-in retry with Config, so a strong answer also mentions that and explains when you need your own wrapper on top.

Reference implementation (python)

import random, time
from botocore.exceptions import ClientError

def with_backoff(func, max_attempts=5, base_delay=0.5, max_delay=30):
    for attempt in range(max_attempts):
        try:
            return func()
        except ClientError as e:
            code = e.response['Error']['Code']
            if code not in ('ThrottlingException', 'RequestLimitExceeded'):
                raise
            if attempt == max_attempts - 1:
                raise
            delay = min(base_delay * (2 ** attempt), max_delay)
            time.sleep(delay + random.uniform(0, delay * 0.1))

Expect these follow-ups

• What is the botocore Config parameter that controls built-in retries?
• What is the difference between standard and adaptive retry modes in boto3?

As asked

Write a CloudWatch Logs Insights query that finds Lambda invocations where duration exceeded 5000 ms in the last 24 hours, returning the requestId, duration, and billed duration, sorted by duration descending.

Sample answer outline

Candidate should use parse to extract fields from the REPORT log line, filter on Duration > 5000, and sort descending. They should know the REPORT line format and the difference between Duration and Billed Duration. A strong answer also extracts Max Memory Used to correlate memory pressure with latency.

Reference implementation (sql)

fields @timestamp, @requestId
| filter @message like /REPORT/
| parse @message 'Duration: * ms' as duration
| parse @message 'Billed Duration: * ms' as billedDuration
| parse @message 'Request ID: *\t' as requestId
| filter duration > 5000
| sort duration desc
| limit 50

Expect these follow-ups

• How would you save this query and run it on a schedule to generate a metric?
• What is the difference between filter and stats in Logs Insights?

As asked

Write a Go function that downloads 50 S3 objects concurrently, limits parallelism to 10, and returns all errors without cancelling successful downloads.

Sample answer outline

Candidate should use a semaphore (buffered channel of capacity 10) and a sync.WaitGroup, collecting errors in a slice protected by a sync.Mutex or using an errgroup. They should avoid closing the result channel prematurely. Strong answers also mention the AWS SDK v2 s3manager downloader for large objects.

Reference implementation (go)

package main

import (
    "context"
    "fmt"
    "sync"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

func downloadAll(ctx context.Context, client *s3.Client, bucket string, keys []string) []error {
    sem := make(chan struct{}, 10)
    var mu sync.Mutex
    var wg sync.WaitGroup
    var errs []error
    for _, key := range keys {
        wg.Add(1)
        sem <- struct{}{}
        go func(k string) {
            defer wg.Done()
            defer func() { <-sem }()
            _, err := client.GetObject(ctx, &s3.GetObjectInput{
                Bucket: &bucket, Key: &k,
            })
            if err != nil {
                mu.Lock()
                errs = append(errs, fmt.Errorf("%s: %w", k, err))
                mu.Unlock()
            }
        }(key)
    }
    wg.Wait()
    return errs
}

Expect these follow-ups

• What does the s3manager.Downloader do that a raw GetObject call does not?
• How would you handle partial downloads if your context is cancelled mid-flight?

As asked

A Node.js Lambda function's memory usage grows per invocation and eventually hits the memory limit, causing OOM crashes. How do you investigate and resolve this in a Lambda context?

Sample answer outline

Candidate should mention that Lambda reuses execution environments between warm invocations, so module-level state persists. Common causes: event listeners appended on every invocation, caches growing unboundedly, or unclosed DB connections. Investigation uses CloudWatch Metrics (memory used from REPORT line), X-Ray, or adding process.memoryUsage() logging. Fix involves capping caches, removing listeners on handler exit, and structuring singletons correctly at module scope.

Reference implementation (javascript)

// BAD: listener added on every warm invocation
exports.handler = async (event) => {
  someEmitter.on('data', handler); // grows unboundedly
  return process(event);
};

// GOOD: register at module scope, runs once per execution env
const handler = (data) => { /* ... */ };
someEmitter.on('data', handler);

exports.handler = async (event) => {
  return process(event);
};

Expect these follow-ups

• What is the execution environment lifecycle and how does it affect module-level variables?
• How does the Lambda runtime differ from a long-running Node.js server in terms of garbage collection opportunities?