Dataflow Overview

Bob posts a photo on a social app. Alice follows Bob, so the photo should appear in her home feed. The post service saves the photo in its own database on one server. Alice's feed request reaches a feed service on another server, perhaps hours later. Between the write and the read, the post must travel from one process to another and wait until someone requests it.

The path data takes from the writing process to the reading processes is called dataflow. Every design in this section answers two questions about that path: which channel carries the data, and whether work happens on write or on read. Those answers raise two further questions covered later in this section.

Three Channels Between Processes

Two processes on different servers share no memory, so data moves between them in one of three ways:

• Through a database. One process writes rows, and another reads them later. The feed service could read Bob's post straight from the posts table.
• Through a service call. One process requests data from another and waits for the response. The feed service could call GET /users/bob/posts on the post service.
• Through a message. One process sends a message to a queue or event log, and other processes read it when ready. The post service could publish a PostCreated event for the feed service to consume.

The three channels differ in what must run at the same time and how tightly the two sides depend on each other:

A shared database requires the least initial setup, but both services depend on the same table layout, so neither team can change it alone. A service call returns an answer right away, but the caller fails when the callee is down or slow. A message lets the writer continue when the reader is down, though the reader receives the data later. The Microservice Communications lesson compares calls and messages in depth.

Work on Write or Work on Read

The channel determines how data travels. A second decision matters as much for performance: where the system turns raw data into an answer. A product page shows an average star rating. The raw data is the list of reviews. That average can be computed in two places:

• Compute on read. Each page view reads all of the product's reviews and averages them.
• Precompute on write. Each new review updates a summary row for its product that holds the review count and star sum. Each page view reads only that summary row and divides. This is the lighter form of CQRS (Command Query Responsibility Segregation), which keeps a separate read model, here the summary row, and updates it in the same transaction as the write.

Consider a popular product page with 2,000 views per second, 5 new reviews per second, and 500 reviews per product. Computing on read scans 2,000 × 500 = 1,000,000 review rows per second, plus 5 inserts, totaling 1,000,005 rows. Precomputing performs 5 inserts, 5 summary updates, and 2,000 summary reads, totaling 2,010 rows per second. Between two new reviews, users view the page about 400 times (2,000 ÷ 5). The system updates the summary once and reads it about 400 times. The rows touched drop from about 1,000,000 to about 2,010 per second, about 500 times fewer.

Read-to-write ratio. The number of reads for each write of the same data. A product page read 2,000 times per second with 5 new reviews per second has a 400:1 ratio. Most consumer products are read-heavy, whereas logging, metrics, and sensor data are often write-heavy.

The widget computes both designs from three rates. Start with the product page. Move reviews per product from 500 down to 1, then press "Sensor dashboard (write-heavy)".

With one review per product, a read scans only one row, so both designs touch about 2,000 rows per second and precomputing saves nothing. In the write-heavy case, sensors write 2,000 readings per second, a dashboard loads once per second, and each view reads the last 3 readings. Computing on read touches 2,000 + 3 = 2,003 rows per second. Precomputing performs 2,000 inserts, 2,000 summary updates, and 1 read, totaling 4,001 rows, or about twice as many. That extra work happens because the system updates a summary for each of the 2,000 writes that almost nobody reads. Precomputing pays off when data is read many more times than it is written and when each read would scan a lot of data.

Limitations of Each Approach

Both approaches carry costs beyond row counts.

Computing on read repeats the same work for every reader, and its latency grows with the dataset size. A product with 100,000 reviews forces every page view to scan 100,000 rows. Caching the computed answer helps, though a cache is a form of precomputing on the first read and introduces its own freshness problems.

Precomputing on write shifts the cost to the writer, which multiplies when one write affects many answers. A single post by an account with 10 million followers can require 10 million feed updates. One write spreading to many readers is called fan-out. Every path that modifies the raw data must also update the precomputed answer. An edited or deleted review must adjust the summary, or the average drifts from the truth. When the update runs asynchronously through a queue, readers see the old answer until the update finishes.

Advanced Approaches

Many production systems combine both approaches. A feed can precompute for most accounts and compute on read for the few accounts with millions of followers. A system can also precompute only for active users and compute on read when a user returns after months away.

Two further questions follow from the feed example. The first is whether the server pushes new data to the reader or the reader pulls it by asking. The second is how a single write reaches many readers through fan-out.

The following lessons cover each question in depth. Push vs Pull compares polling, long polling, and server push for delivering updates to clients. Push vs Pull in Twitter Timeline applies work on write and work on read to a home timeline, combining fan-out on write and fan-out on read in a hybrid design.

Key idea. Data moves between processes through a database, a service call, or a message, and the channel determines what must run at the same time. Separately, a system can compute a derived answer on read or precompute it on write. Precomputing pays off when the read-to-write ratio is high and each read would scan a lot of data, such as 2,000 views per second against 5 reviews per second. However, it adds work on every write and multiplies when one write feeds many answers.