Scala engineer system design interview questions

As asked

Design a daily ETL pipeline that reads raw JSON events from S3, performs joins with reference data, aggregates by user and day, and writes Parquet to a data warehouse. Address schema evolution, partitioning strategy, and failure recovery.

Sample answer outline

Use Spark on EMR or Databricks. Partition input S3 data by date for predicate pushdown. Broadcast join for small reference tables. Schema evolution: use Delta Lake or Iceberg for ACID writes and schema merging. Partition output by (year, month, day) for efficient downstream queries. Failure recovery: write to a staging location first, then atomic move; use job checkpointing or idempotent job IDs. A strong answer covers data quality checks (Great Expectations or dbt tests) and monitoring via Spark metrics.

Expect these follow-ups

• How does Delta Lake handle schema evolution differently from plain Parquet on S3?
• What happens if the job fails halfway through writing output partitions?

As asked

A Scala service ingests events, enriches them over HTTP and writes to Kafka. How would you use Akka Streams or a similar streaming library to control backpressure?

Sample answer outline

The stream should model bounded stages explicitly rather than spawning untracked futures. mapAsync with a fixed parallelism caps HTTP concurrency, buffers should be sized intentionally, and supervision should define whether a bad record is skipped, retried or sent to a dead-letter path. Kafka commits should be tied to successful processing so replay semantics are clear. The answer should mention materialised values, kill switches and graceful shutdown. Candidates often add buffers everywhere, which hides overload until latency and memory fail together.

Expect these follow-ups

• Where would you put retries and how would you avoid retry storms?
• How do you choose mapAsync parallelism?
• What happens during shutdown with in-flight messages?

As asked

How would you design a Kafka consumer in Scala (using fs2-kafka or Alpakka Kafka) that processes messages exactly once and writes results to PostgreSQL? What are the specific failure modes you guard against?

Sample answer outline

Exactly-once requires transactional writes: commit the Kafka offset in the same database transaction as the write. Use offset storage in PostgreSQL instead of Kafka's __consumer_offsets. On restart, read the last committed offset from the DB and seek to it. Failure modes: consumer crash after DB write but before Kafka commit (handled by idempotent writes keyed on Kafka offset), network partition, and duplicate processing from rebalance. Strong candidates mention using Alpakka Kafka's transactional source or fs2-kafka's commitBatch.

Expect these follow-ups

• How do you handle a poison pill message that always fails processing?
• What is the role of Kafka consumer groups in your design and how does rebalancing affect your offset strategy?

As asked

Design a CQRS system where the write side uses Akka Persistence and the read side uses a separate projection into a PostgreSQL read model. How do you keep the read model eventually consistent and handle projection failures?

Sample answer outline

Write side: PersistentActor or EventSourcedBehavior processes commands and persists events. Read side: Akka Projections reads from the event journal and updates PostgreSQL. The projection tracks an offset in the DB; on restart it resumes from the last committed offset. Failure handling: use at-least-once delivery, make projections idempotent on event ID. Strong candidates distinguish between same-process projections and separate consumer processes reading from Kafka as the event backbone.

Expect these follow-ups

• How do you handle a projection that is hours behind the write side due to a long outage?
• How would you add a second read model (Elasticsearch) without changing the write side?

As asked

Design a distributed rate limiter that limits each user to 1000 API calls per minute across a fleet of 50 Scala microservice instances. Walk me through the architecture, the data store choice, and the failure mode when Redis is down.

Sample answer outline

Use Redis with sliding window counters or token bucket implemented as a Lua script for atomic incr+expire. Each service calls Redis before serving the request. When Redis is down, choose between fail-open (allow all, risk abuse) or fail-closed (reject all, risk availability). A strong candidate discusses the sliding window log versus fixed window trade-off, Redis cluster for HA, and using Cats Effect or ZIO to wrap the Redis calls with timeout and fallback. Also mention local token bucket as a two-tier approach.

Expect these follow-ups

• How does a sliding window algorithm differ from a fixed window in terms of traffic spike behaviour?
• How would you implement the Redis Lua script for an atomic sliding window check?

As asked

Design a system that aggregates click events from Kafka in near real time (5-second windows) and writes metrics to a time-series database. Address throughput of 500k events per second and late-arriving events.

Sample answer outline

Use Spark Structured Streaming or Flink consuming from Kafka. 5-second tumbling windows with a 30-second watermark for late events. At 500k events/sec, estimate partition count: aim for each partition at ~10k events/sec, so 50 Kafka partitions, matched to Spark parallelism. Output to TimescaleDB or InfluxDB using foreachBatch with batched upserts on the window key. A strong candidate discusses the append versus update output mode and how late events after the watermark are dropped (and why that is usually acceptable for metrics).

Expect these follow-ups

• How would you handle a Kafka partition that is significantly behind the others?
• What changes if you need exactly-once guarantees instead of at-least-once for this metrics pipeline?

As asked

You have a multi-step order flow: reserve inventory, charge payment, and notify shipping, each in a different microservice. Design a saga that handles partial failures and compensates correctly using Akka actors.

Sample answer outline

Use choreography (event-driven) or orchestration (saga actor). The orchestrating actor tracks the saga state and sends commands step by step. On failure at any step, it issues compensating commands in reverse (release inventory, refund payment). Persist saga state using Akka Persistence so the orchestrator survives crashes. Timeout handling: if a response does not arrive, resend with idempotency key or move to a compensation state. Strong candidates discuss idempotent compensating actions and the at-least-once delivery contract.

Expect these follow-ups

• How do you handle a compensation that itself fails?
• What is the difference between choreography and orchestration sagas and when do you choose each?

As asked

How would you design a type-safe, validated configuration loading system for a Scala microservice that reads from Typesafe Config (HOCON) and fails fast at startup if any required value is missing or invalid?

Sample answer outline

Use Ciris or PureConfig to map HOCON to case classes. Define a Config case class and use pureconfig.loadConfigOrThrow or ciris.env to load and validate. Compose with Validated to accumulate all errors before starting. The service should not start if any config is invalid; print all errors at once rather than stopping on the first. A strong answer covers how to handle secrets from environment variables separately from HOCON and how to test configuration loading in unit tests.

Expect these follow-ups

• How do you test configuration loading without reading real environment variables?
• How would you add a config hot-reload capability without restarting the JVM?

As asked

Design a distributed in-memory cache using Akka Cluster where each node holds a shard of the data. How do you route reads and writes, handle node failures, and decide between replication and sharding?

Sample answer outline

Use Akka Cluster Sharding to distribute cache entries. A consistent hash routes each key to a shard actor. On write, the shard actor updates its in-memory map. On node failure, shards rebalance and state is either lost (pure cache, re-populate lazily) or recovered from the backing store. Replication means copying each write to N shard replicas for read availability during failure; sharding-only means simpler code but a cache miss on rebalance. Strong candidates compare this with off-the-shelf solutions like Redis and justify when a custom solution is warranted.

Expect these follow-ups

• How do you prevent a hot shard when many keys hash to the same node?
• What is the trade-off between synchronous replication and eventual consistency for this cache?

As asked

Design an API gateway in Scala using Akka HTTP that handles authentication (JWT), routes to downstream microservices, applies per-service circuit breakers, and enforces rate limits per user.

Sample answer outline

JWT validation middleware as an Akka HTTP directive before routing. Route table maps path prefixes to downstream hosts. Circuit breaker using Akka's CircuitBreaker with failure threshold and reset timeout. Rate limiting with a Redis sliding window (see distributed rate limiter). All HTTP calls to downstream are Akka HTTP client requests using connection pools for keep-alive. Metrics per route using Prometheus. A strong candidate discusses timeout layering: global request timeout plus per-downstream timeout.

Expect these follow-ups

• How does the Akka CircuitBreaker differ from a Hystrix-style circuit breaker?
• How would you do canary routing where 5% of traffic goes to a new service version?