Kafka engineer system design interview questions

As asked

You are designing an orders topic with 50,000 events per second. How do you choose the partition key, and what can go wrong?

Sample answer outline

The partition key controls ordering, load distribution and consumer parallelism, so it should match the ordering requirement first. If all events for an order must be processed in sequence, orderId is a natural key; if customer-level ordering matters, customerId may be required but can create hot partitions. The number of partitions should allow enough consumer parallelism without creating unnecessary broker overhead. The answer should discuss skew, key cardinality, compaction needs and future repartitioning pain. Candidates often say random keys for balance while missing that this destroys per-entity ordering.

Expect these follow-ups

• How would you detect a hot partition in production?
• What happens to ordering when you increase partition count?
• When would you split one logical stream across multiple topics?

As asked

Design an order management system where Kafka is the source of truth (event sourcing). Orders go through states: created, paid, shipped, delivered. Cover topic design, event schema, how read models are built, and how you handle compensating transactions for a payment failure.

Sample answer outline

Use a single compacted 'orders' topic keyed by order_id so the latest event per order is retained. Events are versioned Avro records with an event_type discriminator. Read models (CQRS projections) are built by consumer groups that project into a Postgres or Redis read store. For payment failure, produce a PaymentFailed event; a saga orchestrator (or choreography via consumers) issues a CompensatingCancelOrder event. Strong answers discuss idempotent event handlers and the challenge of rebuilding projections from the beginning.

Expect these follow-ups

• How would you rebuild a read model from scratch after a schema migration?
• How do you handle exactly-once delivery from the Kafka event log to Postgres?

As asked

Multiple teams produce and consume the same Kafka topic. How would you manage schema evolution without breaking consumers?

Sample answer outline

Use a schema registry with a compatibility mode that matches the organisation's deployment reality, often backward or full transitive compatibility. Producers should only add optional fields or fields with defaults unless a versioned topic or migration plan is in place. Consumers should ignore unknown fields and tolerate missing optional values. The answer should cover contract tests, ownership of event definitions and a deprecation process for fields that are no longer used. Candidates often miss that event schemas are public APIs with slower migration cycles than service internals.

Expect these follow-ups

• When would you create a new topic instead of evolving the schema?
• How do Protobuf and Avro differ for compatibility?
• What breaks if a producer reuses a field name with different meaning?

As asked

A Scala service ingests events, enriches them over HTTP and writes to Kafka. How would you use Akka Streams or a similar streaming library to control backpressure?

Sample answer outline

The stream should model bounded stages explicitly rather than spawning untracked futures. mapAsync with a fixed parallelism caps HTTP concurrency, buffers should be sized intentionally, and supervision should define whether a bad record is skipped, retried or sent to a dead-letter path. Kafka commits should be tied to successful processing so replay semantics are clear. The answer should mention materialised values, kill switches and graceful shutdown. Candidates often add buffers everywhere, which hides overload until latency and memory fail together.

Expect these follow-ups

• Where would you put retries and how would you avoid retry storms?
• How do you choose mapAsync parallelism?
• What happens during shutdown with in-flight messages?

As asked

Design a system that processes 500,000 transactions per second, detects potentially fraudulent ones in under 100ms, and routes suspicious transactions to a human review queue. Use Kafka as the backbone and describe topic partitioning, stream processing, and how you handle the ML model lookup.

Sample answer outline

Partition the 'transactions' topic by account_id to ensure per-account ordering. A Kafka Streams application or Flink job enriches each transaction with account history from a local RocksDB state store (populated from a KTable of account profiles). The fraud scoring calls an in-process ML model or a sidecar to stay within the 100ms SLA. Suspicious transactions are produced to a 'suspicious-transactions' topic consumed by a review dashboard. Clean transactions go to 'cleared-transactions'. Strong designs discuss handling model updates without restarts and circuit-breaking the ML call.

Expect these follow-ups

• How would you update the fraud model without causing downtime or increased lag?
• What happens if the account history enrichment state store falls behind during a rebalance?

As asked

Design a change data capture pipeline that replicates every row-level change in a Postgres database to downstream services via Kafka. Cover Debezium connector configuration, WAL slot management, schema evolution handling, and how you handle a connector restart after prolonged downtime.

Sample answer outline

Debezium Postgres connector uses logical decoding (pgoutput plugin) and a replication slot to stream WAL changes. Each table gets a topic (server.schema.table). Schema changes are captured in a schema change topic. On connector restart after extended downtime, the WAL slot retains all changes (controlled by wal_keep_size), but long outages can cause Postgres disk fill. Strong designs include monitoring slot lag, having a replication slot age alert, and a snapshot mode (initial or when_needed) for recovery after slot expiry.

Expect these follow-ups

• What happens if the Debezium replication slot falls too far behind and Postgres must trim the WAL?
• How do you handle a large table initial snapshot without overloading the Kafka cluster?

As asked

Ten million IoT devices send sensor readings every 10 seconds. Design the Kafka-based ingestion layer: how you structure topics, how devices authenticate, how you handle backpressure when Kafka is overwhelmed, and how downstream analytics consumes the data.

Sample answer outline

Devices connect via an MQTT broker or HTTP gateway that forwards to Kafka producers (not direct Kafka connections from devices). Partition by device_id or device_group for locality. Use mTLS certificates per device class or OAuth tokens for authentication. On backpressure, the gateway applies rate limiting per device and drops or buffers low-priority telemetry. Downstream: a Kafka Streams or Flink job aggregates readings into time buckets and writes to a time-series database (InfluxDB or TimescaleDB). Cold storage uses a Kafka S3 sink connector.

Expect these follow-ups

• How would you handle 10 million simultaneous reconnects after a network partition?
• What connector would you use to archive raw telemetry to S3 cost-effectively?

As asked

A microservice must write a database row and publish a Kafka event atomically without using distributed transactions. Design the transactional outbox pattern: the outbox table schema, how events get from Postgres to Kafka, and how you ensure exactly-once delivery to consumers.

Sample answer outline

The outbox table has columns: id, aggregate_type, aggregate_id, event_type, payload, created_at, published_at. The service writes the business row and the outbox row in one Postgres transaction. A Debezium connector (or a polling relay) reads new outbox rows and produces to Kafka. The Debezium approach uses CDC for low latency. The outbox row's aggregate_id is used as the Kafka message key for ordering. Consumers use idempotent handlers and the Kafka message key/event_id to deduplicate. The outbox pattern itself guarantees at-least-once; consumer idempotence closes the gap to effectively-once.

Expect these follow-ups

• How does Debezium's outbox event router differ from a simple Debezium connector?
• What is the polling relay alternative and when would you choose it over Debezium?

As asked

A Rust service reads from Kafka, calls an HTTP dependency, and writes results to Postgres. How would you design backpressure so it fails predictably under load?

Sample answer outline

A good design bounds every queue and makes overload visible before memory grows without limit. Kafka polling should be tied to downstream capacity, with pause and resume or controlled commit lag rather than unbounded task spawning. HTTP concurrency should be capped with a semaphore and short deadlines, while Postgres writes should use bounded batches and connection pool limits. The service should expose lag, queue depth, retry counts and dropped or dead-lettered messages. Candidates often miss cancellation and shutdown semantics, especially making sure in-flight work is either committed or replayable.

Expect these follow-ups

• Where do you commit offsets in this pipeline?
• How do you distinguish a slow dependency from a poison message?
• What metrics would page you before users notice?

As asked

Your company runs a SaaS platform and needs to serve 200 tenant organizations on a shared Kafka cluster without one tenant impacting another. Design the isolation strategy: topic naming, quota configuration, ACLs, schema Registry namespacing, and when you would choose cluster-per-tenant instead.

Sample answer outline

Use a tenant prefix for topics (tenant_id.domain.event) and assign each tenant a service account with ACLs scoped to their prefix. Kafka quotas (client and user quotas) cap produce/consume bytes per second and connection rate per tenant. Schema Registry subjects follow the topic naming convention for automatic subject isolation. Cluster-per-tenant is preferable when tenants have regulatory isolation requirements (GDPR, HIPAA data residency) or when a single tenant workload would dominate the cluster. Strong answers mention Confluent's tenant-aware partition assignment.

Expect these follow-ups

• How would you enforce topic creation policies to prevent misconfiguration by tenants?
• What is the risk of sharing a broker network between tenants and how do quotas mitigate it?

As asked

Design a Kafka deployment across three geographic regions that allows producers and consumers in each region to use their local cluster while keeping all regions eventually consistent. Address data locality, cross-region replication lag, conflict resolution, and failover.

Sample answer outline

Deploy independent Kafka clusters per region. Use MirrorMaker 2 in active-active mode with cycle detection to replicate between regions. Topic prefixes (region.topic) prevent replication loops. Consumers in each region read from local topics; producers write locally. Conflict resolution relies on application-level logic (last-write-wins by timestamp or CRDTs). MirrorMaker lag (typically 1 to 10 seconds across regions) means consumers may see stale data. Failover requires remapping the consumer group to the target region's topic using MM2 checkpoint translation.

Expect these follow-ups

• How does MirrorMaker 2 prevent infinite replication loops between regions?
• What SLA can you realistically promise for cross-region consistency given WAN latency?

As asked

A Kafka Streams application needs to enrich 500,000 events per second with user profile data from a KTable. The user profiles topic has 50 million keys. Walk through the design decisions around GlobalKTable vs KTable, data locality, co-partitioning, and performance tuning for this scale.

Sample answer outline

GlobalKTable replicates all data to every Streams instance, which is impractical at 50M keys (memory and startup time). Use a regular KTable co-partitioned with the events topic (same number of partitions, same key serializer). Both topics must be partitioned by user_id. Streams instances only load their assigned partitions, keeping memory proportional to partition count. For performance: tune RocksDB (block cache size, block size, compaction), use standby replicas to avoid cold start lag, and set large fetch buffer sizes. At 500K/s, watch for state store I/O becoming the bottleneck.

Expect these follow-ups

• How do you co-partition two topics that were originally created with different partition counts?
• What happens to join correctness if the events and user_profiles topics are not co-partitioned?

As asked

Your platform has 300 topics and 50 teams producing and consuming data. Design a schema governance strategy that prevents breaking changes, enables schema evolution, and provides discoverability. Cover Schema Registry configuration, CI/CD integration, naming conventions, and ownership.

Sample answer outline

Use Confluent Schema Registry in FULL_TRANSITIVE mode for strict evolution control. Integrate schema registration into CI/CD: producers must commit Avro/Protobuf schemas to a Git repo, a pipeline validates compatibility against the registry before merge, and deployment registers the schema automatically. Topic naming follows team.domain.event convention; Schema Registry subjects use TopicNameStrategy. A data catalog (Confluent Control Center or custom) maps topics to owners. ACLs on the registry prevent teams from modifying each other's subjects. Strong answers discuss the trade-off between BACKWARD and FULL_TRANSITIVE strictness.

Expect these follow-ups

• How do you handle schema governance for teams that use JSON without a formal schema?
• What is the difference between TopicNameStrategy and RecordNameStrategy in Schema Registry?