Design a Rate Limiter: The System Design Interview Answer

The one-line answer, then the tradeoff

If an interviewer asks you to design a rate limiter, the winning first sentence is a policy, not an algorithm: "I want to cap each caller at a fixed rate, allow a small burst above it, and reject the overflow cheaply and consistently across every server." Only then do you reach for token bucket, sliding window, or fixed window, because each of those is just a different way to enforce that one sentence.

Short answer: For most APIs, pick a token bucket when short bursts are acceptable and a sliding window counter when you want a smooth ceiling with tiny memory. Keep the counter in a shared store like Redis so the limit holds across your whole fleet, make the read-and-write atomic so two requests cannot both slip through, and return a 429 Too Many Requests with a Retry-After header so honest clients back off instead of hammering you.

The reason this question keeps appearing in loops is that it is small enough to finish in 40 minutes yet it touches distributed counters, race conditions, memory sizing, and failure policy. Interviewers use it to see whether you can move from a vague requirement to numbers, and then defend the numbers. This guide walks that path.

Frame it as a policy before an algorithm

Before any diagram, settle four questions out loud. They shape every later decision, and skipping them is the most common way strong engineers still lose points here.

• What is the unit of limiting? Per IP, per user account, per API key, or per tenant. IP is easy but blunt (one office NATs behind a single address); API key is the usual answer for a paid API. Say which and why.
• What is the limit, and is a burst allowed? "100 requests per second" and "100 per second but tolerate a 3 second spike" are different systems. The second needs a bucket that stores headroom.
• What happens on rejection? Return 429 with Retry-After, or silently drop, or queue and delay. For a public API the honest answer is a 429 plus rate headers so clients self-regulate.
• Where does it sit? At the edge (API gateway or reverse proxy), as a shared middleware, or inside each service. Closer to the edge protects more of the stack; inside a service gives finer per-endpoint control.

Stripe's production write-up is a useful anchor here because it shows that "a rate limiter" in a real system is usually several limiters with different jobs. Their engineering post on rate limiters describes four: a request rate limiter (N per second per user), a concurrent requests limiter (only so many in flight at once), and two load shedders that reserve capacity for critical traffic. Naming that you may need more than one limiter, each on a different signal, is a senior move.

The four algorithms, and what each one is really buying you

There are four algorithms worth knowing cold, plus leaky bucket as a close cousin of token bucket. The differences come down to how they handle a burst and how much memory they cost per caller.

Fixed window counter. Count requests in the current clock window (say each minute) and reset at the boundary. It is one integer per caller and trivial to build. Its flaw is the boundary: a caller can fire the full limit at the end of one window and the full limit at the start of the next, so a "100 per minute" rule can pass 200 requests inside a 2 second span. Figma's rate limiting write-up gives the same example with a limit of five: five requests at 11:00:59 and five more at 11:01:00 is ten in under a minute.

Sliding window log. Store a timestamp for every request and, on each new request, drop timestamps older than the window and count what is left. It is exact, but the memory grows with traffic: Figma calculated that logging five million request timestamps for ten thousand users could cost around 20 MB in Redis. Use it only for low-volume, high-value endpoints where exactness matters more than cost.

Sliding window counter. The practical middle ground. Keep two counters, the previous window and the current one, and estimate the rate by weighting the previous window by how much of it still overlaps the sliding window. Cloudflare runs this across its network and reported, over 400 million requests from 270,000 sources, that only 0.003% were wrongly allowed or blocked, with an average difference of about 6% between the real rate and the approximation. Two integers per caller buys near-exact behavior. Its formula appears in the next section.

Token bucket. A bucket holds up to capacity tokens and refills at a fixed rate. Each request takes a token; an empty bucket means reject. Because the bucket can be full, it naturally allows a burst up to its capacity and then settles to the refill rate. This is what Stripe uses, implemented on Redis. It is the default answer whenever the product needs to tolerate short spikes, like a flash sale.

Leaky bucket. Requests enter a queue that drains at a constant rate; overflow is dropped. It is token bucket's mirror image: instead of allowing bursts, it smooths output to a steady stream. Reach for it when a downstream system needs a constant, predictable inflow rather than a bursty one.

The algorithm is the easy half. The interview is won on where the counter lives and what happens when it is briefly wrong.

Size it with real numbers: the payments-gateway example

Interviewers reward a candidate who puts numbers on the table. Take a concrete brief: an API gateway in front of a payments platform, 2 million merchants, each capped at 100 requests per second sustained, but tolerant of a 3 second burst so a checkout page can fire a batch of calls without being throttled. Work it through.

Token bucket parameters. The sustained rate is the refill rate: 100 tokens per second. The burst tolerance is the capacity: 100 per second for 3 seconds is 300 tokens. So each merchant gets a bucket of capacity 300 that refills at 100 per second. A merchant can spend up to 300 tokens instantly, then is held to 100 per second as the bucket refills.

Here is that bucket in code. It refills lazily by elapsed time rather than on a timer, which is how you would actually build it so you do not run a background job per merchant. The final line demonstrates it: hammer the bucket 305 times at the same instant and only the 300 in the bucket get through.

function tokenBucket(capacity, refillPerSec) {
  let tokens = capacity;
  let last = 0; // seconds of the last refill
  return (nowSec, cost = 1) => {
    tokens = Math.min(capacity, tokens + (nowSec - last) * refillPerSec);
    last = nowSec;
    if (tokens >= cost) {
      tokens -= cost;
      return true;
    }
    return false;
  };
}

const allow = tokenBucket(300, 100); // 300 burst, 100/sec sustained
let granted = 0;
for (let i = 0; i < 305; i++) {
  if (allow(0)) granted++; // all fired at t = 0
}
console.log(granted); // 300 -> the burst ceiling, not 305

Memory cost, and why the algorithm choice matters at scale. Now size the store across all 2 million merchants. A sliding window log during a full burst could hold up to 300 timestamps per merchant. At 8 bytes each that is 2,400 bytes per merchant, so 2,000,000 x 2,400 bytes is about 4.8 GB before Redis overhead. Switch to a sliding window counter or a token bucket and each merchant needs roughly two numbers, on the order of 16 bytes of payload, so about 32 MB before overhead. That is the same tradeoff Figma measured when they moved from a log to counters and cut memory from around 20 MB to 2.4 MB for their test load. At 2 million keys, the algorithm you pick is a two-orders-of-magnitude memory decision, not a style preference.

The fixed window trap, in numbers. If you had chosen a plain fixed window of 100 per second for this gateway, a merchant could send 100 requests in the last 10 milliseconds of one second and 100 in the first 10 milliseconds of the next: 200 requests in a 20 millisecond span while never breaking the stated "100 per second" rule. For a payments path feeding a fraud or ledger system downstream, that doubling is exactly the spike you were trying to prevent. Stating this out loud is what justifies the extra complexity of a sliding or token approach.

Where the counter actually lives

A limiter that only works on one server is not a limiter; a caller just spreads requests across your fleet and beats it. The counter has to be shared, and the classic answer is Redis, because two commands do almost all the work. Redis's own rate limiting guide shows the fixed window pattern: build a key per caller per minute, increment it, and let it expire so old windows clean themselves up.

MULTI
  INCR  {api-key}:{current-minute}
  EXPIRE {api-key}:{current-minute} 59
EXEC
# if INCR result > limit -> reject with 429

The MULTI ... EXEC matters. It makes the increment and the expiry a single atomic step, so two concurrent requests cannot both read a stale count and both decide they are under the limit. That read-then-write race is the exact weakness Figma flagged in a naive token bucket: without atomicity the limiter is occasionally too lenient under contention. In an interview, name the race before you are asked, then close it with an atomic operation (a MULTI transaction, or a small Lua script that reads, decides, and writes in one round trip).

Two more distribution points earn credit. First, a single Redis node is a bottleneck and a single point of failure at high request rates, so you either shard callers across a Redis cluster by key or accept a small approximation by limiting per node against a fraction of the global quota. Second, decide your failure policy explicitly: if the limiter store is unreachable, do you fail open (allow traffic and risk overload) or fail closed (reject and risk an outage from your own defense)? For a login endpoint you might fail closed; for a read API you often fail open. There is no universally right answer, which is why interviewers ask.

The follow-ups interviewers push on

Once the core design is on the board, the question turns into a series of "what about" probes. These are where the grade is actually set, so rehearse crisp answers.

• Clock skew and time source. If each server trusts its own clock, windows drift apart across the fleet. Use a shared time source or, better, let Redis own the time via its own commands so every server reads one clock.
• Response contract. Return 429 Too Many Requests with Retry-After telling the client when to try again, and ideally rate headers reporting the limit, remaining quota, and reset time. A limiter that just drops connections trains clients to retry aggressively, which makes the overload worse.
• Multi-tenant fairness. A global limit lets one noisy tenant starve the rest. Per-tenant buckets, plus a separate global load shedder that reserves capacity for critical calls (Stripe's fleet load shedder is exactly this), keep one bad actor from taking down shared infrastructure.
• Cost of the check. The limiter runs on every request, so its own latency and its load on Redis are part of the design. This is why two-integer algorithms win at scale: cheaper reads, smaller memory, one round trip.
• Configuration and rollout. Limits change. Store them as data, not code, so you can raise a customer's ceiling or dial a limit down during an incident without a deploy.

To show the sliding window counter concretely when they push on accuracy, this is Cloudflare's weighting formula, with their own worked value:

function slidingWindowRate(prevCount, currCount, elapsedSec, windowSec) {
  const weight = (windowSec - elapsedSec) / windowSec;
  return prevCount * weight + currCount;
}

// 42 requests last minute, 18 so far this minute, 15s into the window:
console.log(slidingWindowRate(42, 18, 15, 60)); // 49.5

If that estimate is above the limit you reject, otherwise you allow and increment the current counter. Two integers, one multiply, and you are within a few percent of an exact sliding window at a fraction of the memory.

FAQ

What is the difference between rate limiting and throttling? They overlap in casual use, but a clean distinction is that rate limiting rejects requests over a hard ceiling (a 429), while throttling deliberately slows requests down, for example by queuing and delaying them so the caller experiences a lower steady rate rather than an outright refusal. Leaky bucket is a throttling shape; a fixed window counter is a limiting shape.

Should the rate limiter fail open or fail closed? It depends on what the endpoint protects. Fail open (allow traffic when the limiter store is down) suits read paths where availability beats strict enforcement. Fail closed (reject) suits sensitive paths like login or payment initiation where letting unlimited traffic through is worse than a brief outage. State the choice and its risk rather than treating it as an implementation detail.

Where should a rate limiter sit in the architecture? As close to the edge as gives you the signal you need. An API gateway or reverse proxy protects the whole stack and is the common home for coarse per-key limits. Finer per-endpoint or per-operation limits often live in a shared middleware inside the service so they can see application context the edge cannot.

How do you rate limit across many servers? Keep the counter in a shared store (commonly Redis) rather than in each server's memory, and make the check atomic so concurrent requests cannot both pass. At very high rates, shard callers across a Redis cluster by key, or limit per node against a fraction of the global quota and accept a small approximation.

What HTTP status code should a rate limiter return? 429 Too Many Requests, with a Retry-After header giving the client a concrete time or delay to wait. Adding headers that report the limit, the remaining quota, and the reset time lets well-behaved clients pace themselves instead of retrying blindly.

Token bucket or leaky bucket, which should I pick? Token bucket if the product should tolerate short bursts above the sustained rate (checkout spikes, batch calls); the full bucket absorbs the burst, then the refill rate takes over. Leaky bucket if a fragile downstream needs a smooth, constant inflow; it trades away burst tolerance for a steady drain.

How do you handle a caller sitting right at the limit (the boundary problem)? Avoid a plain fixed window, which lets a caller double the intended rate across a window edge. A sliding window counter weights the previous window so the boundary is smoothed, and a token bucket has no window edge at all because it refills continuously. Both remove the doubling that the fixed window allows.

Sources

• Scaling your API with rate limiters, Stripe - four production limiter types and a token bucket on Redis.
• How we built rate limiting capable of scaling to millions of domains, Cloudflare - the sliding window counter formula and its measured accuracy across 400 million requests.
• An alternative approach to rate limiting, Figma - the fixed window doubling problem and the memory cost of a sliding window log versus counters.
• Rate limiting, Redis - the fixed window pattern with INCR and EXPIRE in an atomic transaction.

Where to take this next

• System design interview prep for senior roles for how a limiter fits a larger design conversation.
• Backend system design deep dive for the storage and caching decisions the counter sits alongside.
• System design walkthrough: Twitter clone for another end-to-end design where rate limits appear as one component.