The Four Core Challenges of a Large System
A system with a few hundred users needs one server and one database. Every request goes to the same machine, and every read returns the latest data. With few users, system design reduces to ordinary programming.
Scale changes this. As the number of users grows, four challenges appear. Three of them come from the load itself: too many requests, too much data, and slow responses. The fourth, stale reads, comes from the fixes for the first three. Almost every technique in this course is a response to one of these four challenges.
Grow the app below. It starts with a few thousand users, one app server, and one database. Watch which challenge appears first, then turn on its fix. The diagram adds the components that each fix needs.
The order in the widget depends on its illustrative numbers. In a real system, the order depends on the product. A video site runs out of storage early. A chat app runs out of request capacity early. The four challenges and their fixes stay the same.
Challenge 1: Too Many Requests
One machine can handle only a limited number of requests per second. Past that limit, requests wait in a queue, response times grow, and requests start to time out.
The fix is to replicate: run several identical copies and spread the requests among them. For the application code, a load balancer sends each request to one of several identical servers. For the data, read replicas keep copies of the database, and each copy can answer reads.
Replication works for application servers because they are stateless: any server can handle any request. The Load Balancer and Database Replication lessons explain both in detail.
Challenge 2: Too Much Data
Replication copies all of the data to each machine. That does not help when the data itself no longer fits on one machine. All the posts on a social network, or all the videos on a video site, are far larger than one disk.
The fix is to shard: split the data into parts, and put each part on a different machine. A shard key decides which part each record goes to. For example, if posts are sharded by user_id, all posts of one user are on the same shard, so a query for one user's posts reads one machine.
The shard key matters. A key that spreads records evenly keeps each shard at a similar size and load. A key such as the creation date sends all new writes to one shard. The Database Partitioning lesson covers shard keys and their trade-offs.
Challenge 3: Slow Responses
Users expect a fast response. A read is usually fast, because it fetches data that already exists. A write is often slow, because it can update several tables, resize an image, or notify other users. As the product grows, each write does more of this work, such as updating the feeds of more followers.
The fix is to process slow work asynchronously. The server puts the write on a message queue and answers the user at once. Separate workers take items from the queue and do the slow work a moment later. The user sees a fast response, and the work still completes.
A queue also absorbs bursts. If writes arrive faster than the workers process them, the queue grows for a while instead of the servers failing. The Message Queue lesson explains how.
Challenge 4: Stale Reads
The fixes for challenges 1 and 3 create the fourth challenge. With replicas, a change reaches the replicas a moment after it reaches the primary. With a queue, a write is complete only after a worker processes it. In both cases, a read can return old data for a short time.
Stale here means old, not wrong. The read returns an earlier version, and the copies catch up soon. A system that accepts this short delay is eventually consistent. Most products accept it: a like count or a new post can appear a second late.
Some operations cannot accept stale data. A bank transfer or a seat reservation must read the current value, or two users can spend the same money or book the same seat. Those operations read from the primary and wait for the write to finish, at the cost of a slower response. The Consistency Models lesson explains the options between these two ends.
How the Challenges Connect
Each fix answers one challenge, and the fixes add new trade-offs:
| Challenge | Fix | Cost of the fix |
|---|---|---|
| Too many requests | Replicate servers and data | Copies can be stale |
| Too much data | Shard the data | Queries across shards are harder |
| Slow responses | Process writes asynchronously | Results appear later; work can fail after the response |
| Stale reads | Accept eventual consistency, or read from the primary where it matters | Slower responses for strict reads |
A large system replicates its servers and data, shards data that is too large for one machine, and moves slow work to queues. The result can serve many users and a large amount of data, and it adds machines as it grows. The next lesson combines these fixes into one template that fits most problems.
Key idea. Large systems face four challenges: too many requests, too much data, slow responses, and stale reads. Replication, sharding, and asynchronous processing fix the first three, and they cause the fourth. Choosing where stale reads are acceptable is the central trade-off of system design.