End-to-End Design

Design a service that decides, for each incoming API request, whether to allow it or reject it because the caller has exceeded a configured rate. The limiter sits in front of an API gateway or an internal RPC fabric, must answer in single-digit milliseconds, and must keep working when individual nodes — or an entire datacenter — fail.

The naive single-process counter falls over once two app servers sit behind a load balancer. The rest of the design is about what to do next. Adjacent problems — quotas, billing, and DDoS defense — are separate systems and out of scope here.