Kafka engineer technical fundamentals interview questions

As asked

Explain what Apache Kafka is, the core problem it was designed to solve at LinkedIn, and the three primary use cases it is used for today.

Sample answer outline

Kafka is a distributed event streaming platform originally designed to move log and activity data at LinkedIn's scale. It solves the problem of building point-to-point integrations between systems by acting as a central, durable, replay-able event bus. The three main use cases are: (1) messaging (decoupling services), (2) event streaming and real-time analytics, and (3) data integration (moving data between systems via Connect). Strong answers mention that Kafka's key design decision is treating the log as a first-class abstraction.

Expect these follow-ups

• What makes Kafka's persistence model different from a traditional message queue like RabbitMQ?
• How does Kafka's consumer offset model enable replay of historical data?

As asked

Explain the hierarchy: what is a Kafka topic, what is a partition, and what is a segment? Where does data actually live on disk?

Sample answer outline

A topic is a logical category of events named by the user. A topic is split into N partitions, each of which is an ordered, immutable append-only log. Each partition is stored as a set of segment files on broker disk. Each segment consists of a .log file (the actual record bytes) and index files (.index for offset-to-position, .timeindex for timestamp-to-offset lookup). Segment files roll over when they reach log.segment.bytes or log.roll.hours. Strong answers draw the hierarchy clearly and note that each partition's directory is named topic-partition_number.

Expect these follow-ups

• What is the role of the .index file and how does it make random offset reads efficient?
• How does Kafka use memory-mapped files to avoid redundant copying?

As asked

Explain what a consumer group is, how partition assignment works within a group, and what happens when you have more consumers than partitions in a group.

Sample answer outline

A consumer group is a set of consumers identified by the same group.id that collectively consume a topic. Each partition is assigned to exactly one consumer in the group at a time, providing horizontal parallelism up to the number of partitions. If there are more consumers than partitions, the excess consumers sit idle. Multiple groups can consume the same topic independently, each tracking their own offsets, which enables pub-sub semantics (all groups receive all messages). Strong answers mention that offsets are stored in __consumer_offsets.

Expect these follow-ups

• If a topic has 10 partitions and a consumer group has 3 members, how are partitions distributed?
• How does adding a second consumer group differ from adding a member to the existing group?

As asked

Name and define the three message delivery guarantees (at-most-once, at-least-once, exactly-once), and for each one state which Kafka producer and consumer configurations produce that behavior.

Sample answer outline

At-most-once: acks=0 or auto-commit before processing; messages may be lost but never duplicated. At-least-once: acks=1 or acks=all with producer retries, and manual commit after processing; messages may be duplicated but never lost. Exactly-once: idempotent producer plus transactions on the producer side, isolation.level=read_committed on the consumer side; each message is delivered exactly once within the Kafka-to-Kafka path. Strong answers note that EOS requires transactional.id and does not automatically extend to external systems.

Expect these follow-ups

• In practice which delivery guarantee do most Kafka pipelines use, and why?
• What is the cheapest way to implement at-least-once for a stateless consumer?

As asked

Define replication factor in Kafka. Explain the relationship between replication factor and the number of broker failures you can tolerate, and walk me through how you would choose replication factor for a critical production topic.

Sample answer outline

Replication factor is the number of copies of each partition maintained across brokers. With replication factor R, you can tolerate R-1 broker failures without data loss. For critical production topics, replication factor of 3 is standard, tolerating one broker failure. Combined with min.insync.replicas=2 and acks=all, writes require at least 2 replicas to acknowledge, which prevents data loss if one broker fails mid-write. Strong answers note that RF must not exceed the number of brokers and that RF greater than 3 is rare because the additional cost exceeds the reliability benefit.

Expect these follow-ups

• What is the minimum cluster size to run RF=3 without risk?
• What is the write latency impact of increasing RF from 2 to 3?

As asked

A consumer reads from Kafka, writes to a database and then commits offsets. Explain at-least-once, at-most-once and exactly-once behaviour in this flow.

Sample answer outline

If the consumer commits before the database write, a crash can lose the message, which is at-most-once. If it writes first and commits after, a crash between those operations can replay the message, which is at-least-once. Exactly-once for external side effects requires idempotent writes, transactions that include both offset and output, or a design where replay is harmless. Kafka transactions help when consuming and producing back to Kafka, but they do not magically make an arbitrary database update exactly once. Strong candidates discuss idempotency keys and deduplication tables.

Expect these follow-ups

• Where would you store an idempotency key?
• How do Kafka transactions work when the sink is another Kafka topic?
• What is the operational cost of exactly-once processing?

As asked

A GenServer that batches analytics events starts building a mailbox with millions of messages during a traffic spike. How do you debug it and redesign it?

Sample answer outline

The diagnosis should inspect process info, message queue length, reductions, memory, and the shape of incoming messages before guessing. The redesign should avoid treating a GenServer as an unbounded queue: use bounded demand, partition work across multiple processes, and push durable backlog into Broadway, Oban, RabbitMQ, or Kafka depending on delivery requirements. Batching should be time and size bounded, with explicit shedding or backpressure when downstream is slow. A senior answer distinguishes CPU saturation from downstream latency and from a single hot key. The common mistake is adding more callers while leaving one serial bottleneck in place.

Expect these follow-ups

• How would you expose mailbox length as an operational metric?
• When is dropping events acceptable?
• How would Broadway change the design?

As asked

Where are consumer group offsets stored, how are they written, and what happens to them if the __consumer_offsets topic is corrupted or unavailable?

Sample answer outline

__consumer_offsets is an internal compacted topic with 50 partitions (by default). Each committed offset is a keyed message where the key is (group, topic, partition) and the value is the committed offset plus metadata. The group coordinator broker handles writes to this topic. If __consumer_offsets is unavailable, consumers cannot commit or fetch their committed offset and new group startups fail; existing consumers that cached their offset can continue consuming but cannot commit. Strong answers mention that the latest committed offset can be reset using kafka-consumer-groups.sh.

Expect these follow-ups

• Why is __consumer_offsets compacted and not simply retained by time?
• How would you migrate consumer group offsets from one cluster to another?

As asked

Trace the journey of a message from the moment the application calls producer.send() to the moment it is acknowledged by the broker. Name each internal component the message passes through.

Sample answer outline

send() serializes the key and value using the configured serializers, then invokes the partitioner to determine the target partition. The record is placed into the RecordAccumulator batch for that TopicPartition. A background Sender thread checks each batch: if batch.size is full or linger.ms has elapsed, it compresses the batch (if enabled) and sends a ProduceRequest to the leader broker. The broker writes to its log and replicates to followers (if acks=all). The broker sends back a ProduceResponse; the Sender thread invokes the callback or completes the Future. Strong answers name RecordAccumulator, Sender, and ProduceRequest.

Expect these follow-ups

• At what point can the producer retry on a retriable error?
• What is the role of the metadata cache in the producer and how does it get refreshed?

As asked

Explain why the Kafka consumer must call poll() on a regular cadence and what happens if it does not. How does this relate to max.poll.interval.ms and the heartbeat thread?

Sample answer outline

The poll() call serves two main purposes: it sends fetch requests to the broker and returns available records. Coordination and heartbeat logic runs on a separate background heartbeat thread (introduced in Kafka 0.10.1), which sends heartbeats to the group coordinator every heartbeat.interval.ms independently of poll(). If poll() is not called within max.poll.interval.ms, the consumer client itself signals to the coordinator that it is no longer making progress, and the coordinator triggers a rebalance to reassign its partitions. This is distinct from session.timeout.ms, which governs when the coordinator decides a member has stopped sending heartbeats entirely (i.e., the process crashed or the network is down).

Expect these follow-ups

• What is the difference between session.timeout.ms and max.poll.interval.ms?
• Under what conditions would a consumer fail to call poll() within max.poll.interval.ms?

As asked

Compare Avro, Protobuf, and JSON as Kafka message serialization formats. What are the concrete advantages of schema-based formats for Kafka pipelines at scale?

Sample answer outline

JSON is human-readable but schema-less (each message carries field names, which wastes space) and has no enforcement of field types or presence. Avro and Protobuf embed only the schema ID (4 bytes) in each message and store the schema centrally in Schema Registry, reducing message size by 30 to 80 percent. Schema-based formats enforce compatibility rules, catch breaking changes at registration time rather than at consumer runtime, and enable automatic code generation. Avro is most common in the Confluent ecosystem; Protobuf is preferred in polyglot environments due to broad language support.

Expect these follow-ups

• What is the wire format of an Avro-encoded Kafka message?
• How does Schema Registry embed the schema ID in the message bytes?

As asked

Explain the Kafka Connect distributed mode architecture. What is the difference between a connector, a task, and a worker? How does Kafka Connect achieve fault tolerance and horizontal scalability?

Sample answer outline

A connector is a logical configuration describing what data to move and where (e.g., JDBC Sink). Tasks are the actual units of parallelism; a connector splits its work into tasks.max tasks. Workers are JVM processes that run tasks. In distributed mode, multiple workers form a group (identified by group.id) and tasks are distributed across them by the Connect rebalancing protocol. Fault tolerance: if a worker dies its tasks are redistributed to remaining workers. State (offsets, configs, status) is stored in Kafka topics. Strong answers name the three internal topics (offsets, config, status).

Expect these follow-ups

• What are the three Kafka topics that Connect uses internally and what is stored in each?
• What is the difference between standalone and distributed Connect mode?