Elixir engineer system design interview questions

As asked

Design a group chat application that supports 100,000 concurrent users across 1,000 chat rooms. Walk me through the data model, how you handle WebSocket connections at scale, message persistence, and delivery guarantees.

Sample answer outline

Use Phoenix Channels with PubSub per room topic. Scale horizontally with a Redis PubSub adapter (or pg adapter within a BEAM cluster). Messages persist to PostgreSQL with an append-only messages table. Delivery is at-most-once via PubSub; for guaranteed delivery, store message ID in the client and re-fetch since-last-seen on reconnect. Presence for online indicators. Consider Oban for asynchronous processing (notifications, indexing).

Expect these follow-ups

• How would you handle a user's unread message count across devices?
• What happens to in-flight messages during a node rolling restart?

As asked

Design a distributed background job system that supports priority queues, retries with exponential backoff, and exactly-once processing. Assume a multi-node Elixir cluster with PostgreSQL.

Sample answer outline

Oban is the idiomatic answer: jobs in a Postgres table, advisory locks for exactly-once dequeue per node, configurable retry with backoff. A strong answer goes beyond Oban to discuss the schema design (priority column with index, state machine for job states), multi-queue routing, and global uniqueness constraints. Also covers what 'exactly-once' means in practice (at-least-once with idempotent workers).

Expect these follow-ups

• How would you implement dead-letter queues for jobs that exceed max retries?
• How does advisory locking in PostgreSQL prevent two nodes from running the same job?

As asked

Design a rate limiting system for a Phoenix REST API that enforces per-user limits of 1,000 requests per minute across a 10-node cluster, with sub-millisecond overhead per request.

Sample answer outline

Options: centralized Redis INCR with TTL (simplest, single point of failure), partitioned counters (shard users to specific nodes, route via consistent hashing), or approximate local counters with periodic cross-node reconciliation (highest throughput, allows brief overages). A strong answer compares the consistency-performance tradeoff, discusses token bucket vs sliding window algorithms, and notes that Hammer (Elixir rate-limiting library) uses ETS locally or Redis for distributed scenarios.

Expect these follow-ups

• How does Hammer implement the sliding window algorithm, and what is its time complexity per check?
• How would you implement graceful degradation if the Redis rate limit backend is unavailable?

As asked

Design a multi-tenant SaaS application where each tenant's data must be isolated. Compare row-level isolation, schema-per-tenant, and database-per-tenant, and explain which you would choose for an Elixir app expected to reach 10,000 tenants.

Sample answer outline

Row-level isolation (tenant_id column + Row Level Security) scales easily and Ecto supports it, but a bug can leak data. Schema-per-tenant (Triplex library) gives strong isolation and easy migrations per tenant, but migration fan-out is slow at 10,000 tenants. Database-per-tenant is strongest but operationally prohibitive at scale. A strong answer picks row-level with Postgres RLS, explains how to configure RLS in Ecto's repository, and covers migration strategy.

Expect these follow-ups

• How does the Triplex library manage schema migrations across many PostgreSQL schemas?
• How would you enforce tenant isolation in Ecto queries without relying on every developer remembering to add a where clause?

As asked

A Phoenix API endpoint serves product catalog data to 50,000 RPS but the data changes at most once per minute. Design a multi-layer caching strategy using in-process caches, ETS, and Redis, including cache invalidation.

Sample answer outline

Layer 1: :persistent_term or ETS for in-process, per-node caching with sub-microsecond reads. Layer 2: Nebulex distributed cache backed by Redis for cluster-coherent caching. On data change, publish an invalidation event via PubSub so all nodes flush ETS. Use a TTL as a safety net. A strong answer also covers cache stampede protection (mutex/locking on cache miss), cache warming on startup, and monitoring cache hit rates via Telemetry.

Expect these follow-ups

• How do you prevent a thundering herd when a cache entry expires under high load?
• How would you version cached content to support incremental invalidation without purging everything?

As asked

Design a webhook delivery system that sends HTTP callbacks to customer endpoints with retries, at-least-once delivery, per-endpoint rate limiting, and observability into delivery failures.

Sample answer outline

Oban jobs per webhook event, each worker making the HTTP call with Finch. Retry with exponential backoff on non-2xx or timeout. Per-endpoint rate limiting via Oban queues with rate_limit option or Hammer. Dead-letter storage for permanently failed deliveries. Telemetry events for each attempt (status, latency) feeding into Prometheus/Grafana. A strong answer also covers HMAC request signing for security and ordering guarantees (optional, discuss tradeoffs).

Expect these follow-ups

• How do you detect and disable a customer endpoint that is consistently failing to reduce load?
• How would you implement ordered delivery of webhooks for a single customer endpoint?

As asked

Design a system to ingest 500,000 sensor readings per second from IoT devices, process them in near-real-time for anomaly detection, and persist them for historical queries.

Sample answer outline

Ingest via a pool of Phoenix or raw TCP connections, use GenStage or Broadway for back-pressured processing. Fan out to anomaly detection (stateful windowed aggregations via Flow) and persistence (TimescaleDB or ClickHouse for time-series, bulk inserts via Repo.insert_all). A strong answer discusses partitioning by device ID for ordered processing, how to handle out-of-order events with watermarks, and operational concerns like backpressure signaling to devices.

Expect these follow-ups

• How does TimescaleDB's hypertable partitioning align with your ingestion model?
• How do you manage state across Flow windows when a node crashes mid-window?

As asked

Design an order management system using event sourcing. What are the core building blocks (commands, events, aggregates, projections), and how does the Commanded framework map to them in Elixir?

Sample answer outline

Commands are structs dispatched to an aggregate via Commanded.Application. Aggregates are functional modules (not processes) that validate commands and return events. Commanded replays events to rebuild state. Projections (event handlers or read-model projectors) build query-optimized views. An event store (EventStoreDB or Postgres-backed) persists events. A strong answer discusses eventual consistency of projections, snapshot strategies for long event streams, and when event sourcing adds more complexity than it solves.

Expect these follow-ups

• How do you handle schema evolution when an event's payload format changes?
• What is the difference between a process manager and a saga in Commanded?

As asked

Design the backend for a real-time turn-based card game with 10 million registered users and up to 500,000 concurrent game sessions. Focus on how BEAM's process model maps to the domain.

Sample answer outline

Each game session is a GenServer holding game state, supervised under a DynamicSupervisor, registered in a global Registry (via :global or Horde for cluster-wide lookup). Players connect via Phoenix Channels, which proxy events to the game GenServer. State recovery on crash is from the event log (PostgreSQL). A strong answer covers game state serialization, handling player disconnects with a reconnect window, and sharding game sessions across cluster nodes using consistent hashing.

Expect these follow-ups

• How does Horde differ from the built-in Elixir Registry for cluster-wide process registration?
• How would you gracefully migrate game sessions from a node being drained for deployment?