Kafka engineer role-specific interview questions

As asked

Walk me through exactly what happens when a Kafka broker that holds a partition leader crashes. Who detects the failure, what mechanism picks the new leader, and what conditions can cause that election to fail or stall?

Sample answer outline

The controller broker monitors ZooKeeper (or KRaft metadata log) for broker liveness. On failure it selects the new leader from the in-sync replica set (ISR); if the ISR is empty and unclean.leader.election.enable is false, the partition goes offline. Strong answers distinguish ZooKeeper mode from KRaft and explain how controller epoch prevents split-brain.

Expect these follow-ups

• What happens if the ISR shrinks to zero replicas before the crash?
• How does KRaft change the controller election process compared to ZooKeeper mode?

As asked

A product manager says we need exactly-once delivery for our payment events. Explain exactly what Kafka's exactly-once semantics actually guarantee, the three moving pieces that make it work (idempotent producer, transactions, transactional consumer), and where it can still fail in practice.

Sample answer outline

Idempotent producers assign a producer ID (PID) and a per-partition sequence number so the broker deduplicates retries within a single producer session. Transactions (beginTransaction/commitTransaction) atomically write records across multiple partitions and then write a commit or abort control record to each involved partition so consumers know whether to include those records. Transaction state itself is persisted in the internal __transaction_state topic managed by the transaction coordinator. Consumers set isolation.level=read_committed to skip records belonging to aborted or in-flight transactions. Strong answers note that EOS does not cover external sinks and that the PID is lost on producer restart unless a transactional.id is configured, which allows the broker to fence any previous instance.

Expect these follow-ups

• What is the role of the transaction coordinator and __transaction_state topic?
• Why does EOS not automatically apply to Kafka Connect sinks?

As asked

When would you use auto-commit versus manual offset commits in a Kafka consumer, and what are the exact failure scenarios where auto-commit causes duplicate processing or data loss?

Sample answer outline

Auto-commit fires on a timer (auto.commit.interval.ms) and can commit offsets for records that have not finished processing, causing data loss if the consumer crashes. Manual commit with commitSync or commitAsync after processing guarantees at-least-once delivery. Strong answers describe the commit-after-poll vs commit-after-process distinction and using commitSync on shutdown for clean finalization.

Expect these follow-ups

• How do you implement exactly-once semantics using manual commits plus an idempotent sink?
• What is the danger of calling commitAsync without an error callback?

As asked

Your monitoring shows a Kafka consumer group with 50 million messages of lag on a critical topic and the lag is growing. Walk me through your diagnostic process: what metrics you check, what tooling you use, and what the three most likely root causes are.

Sample answer outline

Start with kafka-consumer-groups.sh --describe to see per-partition lag and which consumer is assigned each partition. Check consumer throughput (records-consumed-rate), broker throughput (bytes-in, bytes-out), and GC pause times. The three most common causes are slow message processing (CPU-bound or blocking I/O), under-partitioning (cannot scale horizontally), and a skewed partition causing one consumer to bottleneck. Strong answers also mention checking for rebalance storms via rebalance-rate metric.

Expect these follow-ups

• How do you safely add partitions to a topic that already has data?
• What tools would you use to replay messages from a specific offset after fixing the slow consumer?

As asked

A teammate says 'Kafka guarantees message order.' Correct or qualify that statement. Under what producer and topic configurations is ordering guaranteed, and name three common scenarios where ordering silently breaks.

Sample answer outline

Kafka guarantees order within a partition only. Ordering breaks when: messages with the same logical key are routed to different partitions (custom or null key), when a producer retries without idempotence (max.in.flight.requests.per.connection greater than 1 can cause reordering), and when a consumer reads from multiple partitions and merges them. Strong answers mention that idempotent producers enforce ordered delivery even with retries.

Expect these follow-ups

• How would you design a system that needs global ordering across all keys?
• What does max.in.flight.requests.per.connection=1 cost you in throughput?

As asked

We need to maximize producer throughput to a Kafka cluster without sacrificing durability. Walk me through the key producer configuration knobs: batch.size, linger.ms, compression.type, buffer.memory, and acks, and explain the tradeoffs each one makes.

Sample answer outline

Larger batch.size and higher linger.ms allow more messages per request, improving throughput at the cost of added latency. Compression (lz4 or snappy) reduces network bandwidth and can improve throughput when CPU is not the bottleneck. buffer.memory controls how much the producer buffers before blocking; insufficient buffer causes send to block or throw BufferExhaustedException. acks=all provides durability but adds round-trip latency; acks=1 halves the wait. Strong answers quantify typical linger.ms values (0 to 100ms) and note that lz4 is generally preferred for speed.

Expect these follow-ups

• How would you profile whether your producer is CPU-bound or network-bound?
• What does a RecordTooLargeException indicate about batch configuration?

As asked

A new high-traffic topic is being designed. Walk me through the factors you consider when choosing partition count: consumer parallelism, throughput per partition, ordering requirements, and the cost of changing it later.

Sample answer outline

Partition count determines maximum consumer parallelism (one partition per consumer thread in a group). Estimate target throughput divided by single-broker throughput per partition (typically 10 to 100 MB/s depending on workload). Consider ordering: if you need per-key ordering, all messages for a key must land in the same partition. Adding partitions later is disruptive (key routing changes, no repartitioning of existing data). Strong answers suggest starting conservatively (10 to 50 partitions for most topics) and benchmarking.

Expect these follow-ups

• What happens to messages already in a topic when you increase partition count?
• How do you model the relationship between partitions and consumer threads in a Streams application?

As asked

If you had to pick the ten most critical Kafka metrics to alert on for a production cluster, what would they be and why? Cover broker, producer, and consumer sides.

Sample answer outline

Broker: UnderReplicatedPartitions (replication health), OfflinePartitionsCount (availability), RequestHandlerAvgIdlePercent (I/O thread saturation), NetworkProcessorAvgIdlePercent (network thread saturation), BytesInPerSec/BytesOutPerSec (throughput). Consumer: consumer-fetch-manager-metrics records-lag-max (lag), records-consumed-rate. Producer: record-error-rate, request-latency-avg. JVM: GC pause time. Strong answers mention which expose via JMX vs Prometheus and how to set meaningful thresholds.

Expect these follow-ups

• How would you set up alerting for consumer lag using Prometheus and Grafana?
• What does a spike in UnderReplicatedPartitions indicate and how do you investigate?

As asked

You find an unsafe block around a C library call in a production Rust service. What invariants do you check before approving the change?

Sample answer outline

The answer should make the safety contract explicit: pointer validity, alignment, ownership, aliasing, lifetime, thread-safety, error handling and cleanup responsibilities. Unsafe should be isolated behind a small safe abstraction whose public API prevents callers from violating those invariants. FFI return codes and null pointers need to become Rust Result or Option values immediately. The candidate should mention tests that exercise boundary cases and tooling such as Miri, sanitizers or fuzzing where practical. A weak answer treats unsafe as a local syntax escape hatch rather than a module-level proof obligation.

Expect these follow-ups

• How do you document a safety invariant without making every caller read C headers?
• What is the difference between unsafe to call and unsafe inside?
• How would you fuzz this boundary?

As asked

Kafka 2.4 introduced cooperative sticky rebalancing. Walk me through the difference between eager and cooperative rebalance protocols, specifically what happens to partition assignments during a rebalance, and when you would choose one over the other.

Sample answer outline

Eager rebalance revokes all partitions before reassigning, causing a full stop-the-world pause. Cooperative sticky rebalancing revokes only the partitions that need to move, in two phases, so consumers keep unaffected partitions. Strong answers mention CooperativeStickyAssignor, the two-phase join-sync cycle, and the impact on processing latency during rolling restarts.

Expect these follow-ups

• What is the incremental cooperative rebalancing algorithm?
• How does static group membership reduce rebalance frequency?

As asked

We have a changelog topic where we want to keep only the latest value per key. Explain how Kafka log compaction works: what the cleaner thread actually does, what a dirty ratio is, and what guarantees compaction does and does not give you.

Sample answer outline

The cleaner thread reads the dirty portion of the log (writes since last clean) and builds an offset map of the latest offset per key, then copies segments into cleaned segments dropping older offsets. The dirty ratio is dirty_bytes / total_log_bytes. Compaction does not guarantee deduplication within a retention window; tombstones (null values) are retained for delete.retention.ms before removal.

Expect these follow-ups

• What happens to a tombstone record and how long is it kept?
• Can compacted topics still have retention time set?

As asked

Define in-sync replicas, high watermark, and log-end offset. Then explain what happens to these three values when a follower replica falls behind and when it catches up again.

Sample answer outline

LEO is the next offset to be written on a given replica. HWM is the offset up to which all ISR replicas have confirmed receipt; consumers can only read up to HWM. The ISR is the set of replicas within replica.lag.time.max.ms of the leader. When a follower falls behind it is removed from ISR, HWM does not advance unless min.insync.replicas is still met, and the follower is re-added once it catches up to leader LEO.

Expect these follow-ups

• What is the risk of setting min.insync.replicas=1 with replication-factor=3?
• How does acks=all interact with min.insync.replicas?

As asked

We use Confluent Schema Registry. Explain the difference between BACKWARD, FORWARD, FULL, and TRANSITIVE compatibility modes, and give a concrete example of a schema change that is BACKWARD compatible but not FORWARD compatible.

Sample answer outline

BACKWARD means new schema can read old data (new consumers handle old messages). FORWARD means old schema can read new data (old consumers handle new messages). FULL requires both. In Avro, adding a field with a default value is FULL compatible: new consumers can read old messages because the new schema supplies the default when the field is absent (BACKWARD), and old consumers can read new messages because Avro schema resolution ignores writer fields that are absent from the reader schema (FORWARD). To get a concrete example of BACKWARD-only, you would need to add a field without a default: new consumers can read old data by using the new schema default, but old consumers cannot read new data that contains an unfamiliar required field. TRANSITIVE extends the compatibility check across all previous schema versions, not just the latest.

Expect these follow-ups

• How does Avro handle schema evolution differently from Protobuf in this context?
• What happens if a producer tries to register an incompatible schema?

As asked

Describe how Kafka Streams manages local state stores: what RocksDB has to do with it, how changelogs make state fault-tolerant, and what happens to state when a Streams instance dies and another takes over.

Sample answer outline

Kafka Streams uses RocksDB (or in-memory) for local state. Every write to a state store is also written to a compacted changelog topic on Kafka. When an instance dies its partitions are reassigned in the next rebalance; the new instance restores state by replaying the changelog from the beginning (or from a standby replica if standby replicas are configured). Strong answers mention standby replicas (num.standby.replicas) for faster failover and the trade-off with storage cost.

Expect these follow-ups

• How do standby replicas speed up failover and what do they cost?
• What is the difference between a persistent and an in-memory state store in Streams?

As asked

We need to aggregate click events by user in real time. Explain the difference between tumbling, hopping, and session windows in Kafka Streams, and describe a use case where each is the right choice.

Sample answer outline

Tumbling windows are fixed-size non-overlapping: one window per interval, good for hourly totals. Hopping windows advance by a step smaller than the window size, so each event may appear in multiple windows; useful for rolling 5-minute stats refreshed every minute. Session windows group events separated by less than an inactivity gap; ideal for user sessions where activity bursts have variable length. Strong answers mention late arrivals and grace periods.

Expect these follow-ups

• How does Kafka Streams handle out-of-order messages within a window?
• What is a grace period and how does it affect result correctness?

As asked

A new topic will receive 500 MB per second total (across all producers), with 7-day retention and replication factor of 3. Walk me through how you calculate disk requirements per broker for a 6-broker cluster.

Sample answer outline

Total bytes written per day = 500 MB/s times 86400s = 43.2 TB per day. Over 7 days that is 302.4 TB. With replication factor 3, total raw data = 907.2 TB. Distributed across 6 brokers = 151.2 TB per broker. Add 20 to 30 percent headroom for compaction overhead and peak bursts. Strong answers also mention segment overhead (index files, time-based vs size-based log rolling) and the impact of compression on actual disk footprint.

Expect these follow-ups

• How would compression change this calculation, and which codec would you choose?
• What happens to storage if a broker goes down and the cluster cannot replicate?

As asked

Kafka 3.x deprecated ZooKeeper in favor of KRaft. Explain what ZooKeeper was doing in the old architecture, what the controller quorum does in KRaft, and what operational benefits and trade-offs the migration introduces.

Sample answer outline

ZooKeeper stored cluster metadata (broker registration, topic configs, partition leader epochs) and ran controller election. KRaft replaces this with an internal Raft log maintained by a quorum of controller nodes (which can be colocated with brokers or separate). Benefits include fewer moving parts, faster controller failover (milliseconds vs seconds), and support for millions of partitions. Trade-offs are that the migration requires a manual offline process or the 3.7+ online migration path, and the metadata topic becomes a hard dependency.

Expect these follow-ups

• How does KRaft handle controller leader election?
• What is the metadata log and how does it differ from a regular Kafka topic?

As asked

A downstream service occasionally produces messages that our consumer cannot parse. Describe how you would implement a dead letter queue (DLQ) in Kafka: where the DLQ topic lives, what metadata to include, how retries work, and how you handle poison pills.

Sample answer outline

On deserialization or processing failure the consumer catches the exception, wraps the raw bytes along with headers (original topic, partition, offset, error message, timestamp) and produces to a dedicated DLQ topic. The main consumer continues past the bad record. A separate DLQ consumer can replay fixed messages back to the original topic. Poison pills (messages that always fail) require a maximum retry count stored in the header to prevent infinite loops. Strong answers also mention Kafka Connect's DLQ configuration.

Expect these follow-ups

• How would you implement automatic retry with exponential backoff before DLQ?
• How does the Spring Kafka DefaultErrorHandler differ from the old SeekToCurrentErrorHandler?

As asked

Describe how you would configure mutual TLS authentication and ACL-based authorization on a Kafka cluster. What files need to change on the broker and client side, and how do ACLs map to Kafka operations?

Sample answer outline

On the broker, configure ssl.keystore.location, ssl.truststore.location, and listeners=SSL://. Set security.inter.broker.protocol=SSL. On clients, configure ssl.keystore and ssl.truststore plus ssl.client.auth=required. ACLs map to operations (Read, Write, Create, Describe, Delete) on resources (Topic, Group, Cluster, TransactionalId). Use kafka-acls.sh --add to grant a principal (CN from the cert) specific operations. Strong answers mention that SASL/SCRAM or OAuth are alternatives to mTLS for authentication.

Expect these follow-ups

• How would you rotate TLS certificates with zero downtime?
• What is the difference between SASL/GSSAPI and SASL/SCRAM in Kafka?

As asked

You are modeling user profile updates and user click events in Kafka Streams. Which would you model as a KTable and which as a KStream, and why? Describe what happens internally when you join the two.

Sample answer outline

KStream represents an append-only event log (each record is an independent event). KTable represents a compacted changelog where each record is an upsert keyed by some identifier. User profiles are a KTable (latest state per user ID). Click events are a KStream (each click is a discrete event). A KStream-KTable join looks up the current table value for the stream key at processing time, which is a foreign-key enrichment. Strong answers explain that the join is non-windowed and that the KTable is local state backed by a changelog topic.

Expect these follow-ups

• What is the difference between a KTable and a GlobalKTable?
• How do you handle a KStream-KTable join when data arrives out of order?

As asked

The Kafka documentation says KafkaConsumer is not thread-safe. Explain exactly what breaks if you share a consumer across threads, and describe two common multi-threaded consumer patterns that work correctly.

Sample answer outline

KafkaConsumer uses internal state (fetch metadata, assignment map, offset tracking) that is not protected by locks. Concurrent poll() or commitSync() calls from multiple threads corrupt this state and produce undefined behavior. Two safe patterns: (1) one consumer per thread with each thread calling poll(), which scales horizontally up to partition count; (2) one consumer thread for polling that dispatches records to a worker thread pool, with the consumer thread handling commits after workers report completion. Strong answers note the coordination complexity of pattern 2.

Expect these follow-ups

• How do you handle pausing consumption while a worker thread processes a batch?
• What is the wakeup() method and when do you call it?

As asked

Using the Kafka Java producer API, show me the correct sequence of calls to implement a transactional read-process-write where you consume from topic A, transform, and write to topic B, committing the source offset atomically with the output write.

Sample answer outline

Initialize producer with transactional.id, call initTransactions(), then in a loop: beginTransaction(), consume from A, process, produce to B, call sendOffsetsToTransaction(offsets, consumerGroupMetadata), then commitTransaction(). On failure call abortTransaction(). The consumer must use isolation.level=read_committed and assign partitions manually (not subscribe) to avoid rebalance complications. Strong answers mention that groupMetadata is used instead of groupId in newer Kafka versions for stronger EOS guarantees.

Expect these follow-ups

• What is the risk of calling commitTransaction without sendOffsetsToTransaction?
• How does the transaction coordinator decide when to abort a stalled transaction?

As asked

Describe the role of the group coordinator broker in a Kafka consumer group lifecycle: how a consumer finds its coordinator, what the group goes through from startup to steady state, and what causes the coordinator to trigger a rebalance.

Sample answer outline

The group coordinator is a specific broker, selected by hashing group.id against the __consumer_offsets partition count. On startup, consumers send FindCoordinator, then JoinGroup (advertising their subscriptions and supported assignors). The coordinator elects one consumer as group leader who computes the partition assignment and returns it via SyncGroup. A rebalance is triggered by member join/leave, heartbeat timeout (session.timeout.ms), or a metadata change (new partitions). Strong answers describe the heartbeat thread and the distinction between session timeout and max.poll.interval.ms.

Expect these follow-ups

• What happens if a consumer exceeds max.poll.interval.ms between polls?
• How does the heartbeat thread differ from the polling thread?

As asked

You get paged: a Kafka broker's disk is at 95% utilization and climbing. The cluster has 6 brokers and no data has been lost yet. Walk me through your immediate response and the decisions you need to make in the next 30 minutes.

Sample answer outline

Immediate: pause or throttle producers writing to topics hosted on that broker (using quota override or kafka-configs.sh). Identify which topics consume the most space on that broker using log size metrics. Options in order of risk: (1) temporarily reduce log.retention.bytes or log.retention.ms for non-critical topics; (2) trigger a partition reassignment to move leaders off the broker; (3) add disk capacity. Avoid deleting log files manually, which corrupts Kafka state. Alert on at 80% going forward. Strong answers show calm prioritization and awareness of broker fence risk if disk fills completely.

Expect these follow-ups

• What is the risk of reducing retention on a compacted topic?
• How would you do a live partition reassignment without dropping messages?

As asked

Your Kafka producer application starts throwing org.apache.kafka.common.errors.BufferExhaustedException in production. Describe your diagnostic steps, the possible root causes, and what configuration changes you would make.

Sample answer outline

BufferExhaustedException means the producer's send buffer (buffer.memory, default 32 MB) is full and max.block.ms has elapsed without space becoming available. Root causes: broker is slow to acknowledge (acks=all with slow ISR), batch accumulation is too slow to free buffers, or throughput spike. Diagnostics: check broker latency, network metrics, and producer metrics (buffer-available-bytes, batch-size-avg). Fixes: increase buffer.memory, reduce max.block.ms to fail fast, increase broker throughput by adding brokers, or add backpressure at the producer. Strong answers distinguish this from a RecordTooLargeException.

Expect these follow-ups

• How would you add back-pressure to the application layer when the producer buffer is full?
• What does max.block.ms control and what happens when it expires?