Blob/Object Storage
A video app receives a 5 GB file for each new upload. Its title, owner, and upload time fit in a database row of a few hundred bytes, while the 5 GB of video bytes do not. Object storage is built for such files. It stores each file as one object (also called a blob, a binary large object) under a key, spreads objects across many machines, and serves them over HTTP. The introduction to storage showed the usual split: the bytes go to object storage, while a database row holds the object's key.
Why Large Files Need Their Own Store
A large file can live in three places. First, a binary column in a database row keeps the file next to its metadata. The cost is that every backup and every replica then copies 5 GB per video, and the database's memory cache fills with bytes that no query filters on. Some databases also limit the size of one value; DynamoDB rejects any item over 400 KB (DynamoDB constraints).
Second, the file can live on the local disk of one server. Capacity is limited to that one disk, and the files are lost when that disk fails. No other app server behind the same load balancer can read them, and the first app server must stream every download itself.
Third, the file can live in object storage. Every app server reaches the same objects by key over HTTP. The store keeps copies across many machines, and a client can download an object without passing through an app server.
The Object Model
Object storage has four parts: buckets, keys, bytes, and metadata. A bucket is a named container for objects, such as video-uploads. Each key is the name of one object inside the bucket, such as videos/2026/v81.mp4. Together, the bucket and key identify one object. Each object holds its bytes and a small set of metadata, which are name-value pairs stored with the object. The store sets some metadata itself, such as the size and last-modified time. The client sets other metadata at upload, such as the Content-Type header or a custom owner-id (S3 objects).
PUT /videos/2026/v81.mp4 HTTP/1.1
Host: video-uploads.storage.example.com
Content-Type: video/mp4
Content-Length: 5000000000
x-amz-meta-owner-id: 42
<5 GB of video bytes>A GET on the same path returns the bytes and metadata. The same GET with a Range: bytes=0-1048575 header returns only the first 1 MiB, letting a video player start or seek without downloading the whole file. Finally, a DELETE removes the object.
A Flat Namespace
The key videos/2026/v81.mp4 looks like a file path, but the store has no folders; the / characters are part of a single flat string. Whereas a file system walks and updates a directory tree for each file, object storage keeps one flat, sorted map from key to object. Without a tree, the store can split the key range across many servers and look up any object by key directly.
To show folders, the store lists keys by prefix. The prefix is the leading characters of the key. A list request with prefix videos/2026/ and delimiter / returns keys that start with that prefix, grouping longer keys into common prefixes that act like subfolders. One Amazon S3 list call returns up to 1,000 keys. When more keys match, the response includes a continuation token that the next call passes to fetch the next page (ListObjectsV2).
Because a list call reads keys in sorted order from the start of the prefix, it costs one request per page. Reading 10 million keys under one prefix takes 10,000 list calls. Queries like "all videos of user 42" therefore belong in a database index, with the object keys stored in the rows.
Prefixes also affect request rates. Amazon S3 supports at least 3,500 writes and 5,500 reads per second for each partitioned prefix, adding partitions as the rate grows (S3 performance). Here, a partition is one slice of the sorted key range served by its own servers. Keys sharing a prefix sort together, so a burst of uploads under videos/2026/ lands on the same slice at first. Starting keys with a short hash, such as videos/a1/ and videos/b7/, places neighboring uploads in different slices and spreads the load across more partitions.
Whole-Object Writes and Versioning
Objects are written as a whole; object storage has no call to change byte 1,000 of an existing object in place. A PUT to an existing key replaces the entire object, so changing one second of a 5 GB video means uploading 5 GB again. Data that changes in small pieces, such as a log that grows line by line, fits better as many small objects or in a database.
Writes to a single key are atomic. During an overwrite, a reader gets either the old object or the new one, and never a mix of the two (S3 consistency model). If two clients write the same key at the same time, the later write wins and overwrites the earlier one without an error. A conditional write prevents this. Each object has an ETag, an ID that the store computes from the object's current bytes. A PUT with If-Match: <ETag> succeeds only if the object still has the ETag the client read, returning an error to a second writer instead of overwriting silently. Similarly, a PUT with If-None-Match: * succeeds only if the key holds no object (S3 conditional writes).
Versioning
When versioning is turned on, a bucket keeps every version of an object. Each PUT to a key creates a new version with its own version ID while keeping older versions stored. A DELETE without a version ID removes no data; instead, it adds a delete marker, an empty version that makes a plain GET return 404 Not Found. Deleting the marker undoes the delete, and older versions stay readable by their version IDs (S3 delete markers).
Versioning protects against bugs that overwrite or delete objects. Because each old version stays stored and billed, versioned buckets usually include a rule that deletes old versions after some days, as covered in the storage-class section.
Consistency
Consistency here means whether a read sees the latest write. Amazon S3 used to return an older object briefly after some overwrites and deletes. Since December 2020, it gives strong read-after-write consistency (AWS announcement). Once a PUT or DELETE returns success, every subsequent GET and LIST sees the result, including from other clients (S3 consistency model). Because other object stores define their own consistency models, a design should check the store it uses.
Strong consistency applies to one key at a time. The store has no transaction that writes two objects together, or an object and a database row together, so the application orders the two writes. One option is to upload the object first and only then write the database row, ensuring a row does not point to an object that is still missing. Another option writes the row first with status = pending and updates it to ready after the upload. Reads return only ready rows; the pre-signed URL flow below uses this order. In either case, a crash can leave an object that no ready row references. A background cleanup job or a lifecycle rule (a bucket setting that deletes or moves objects by age, covered below) removes these objects later.
How Object Storage Keeps Data Durable
Durability is the chance that a stored object is not lost. Disks, servers, racks, and whole data centers can fail, and each forms a failure domain: a group of parts that can fail together. A durable store places enough redundant data across separate failure domains so it can rebuild an object after some of them fail. The two common methods are keeping full copies and erasure coding.
Copies
The direct method stores full copies of each object, such as 3 copies across 3 data centers. Any 1 copy can serve a read, and the object survives the loss of 2 copies. This costs 3 bytes of disk per byte of data, or 3x storage. When a disk fails, the store copies the lost data from a healthy copy to restore the count to 3.
Erasure Coding
Erasure coding achieves the same protection with less extra data. It splits an object into data pieces called shards and computes additional parity shards from them. Parity is extra data calculated so that a lost shard can be reconstructed from the remaining shards.
The simplest example uses XOR (exclusive or), which compares two bits and outputs 1 when they differ. Given two data shards A = 0110 and B = 1010, the parity shard is P = A XOR B = 1100. If the disk holding A fails, the store reconstructs A = P XOR B = 1100 XOR 1010 = 0110. Because three shards store two shards worth of data, the storage cost is 1.5x, and the system can lose any 1 shard.
Codes like Reed-Solomon extend this to multiple parity shards. Consider a 6+3 configuration with 6 data shards and 3 parity shards. A 600 MB object splits into 6 data shards of 100 MB and 3 parity shards of 100 MB, so the store writes 900 MB in all across 9 failure domains. Any 6 of the 9 shards rebuild the object, so it survives losing any 3 shards.
For a 600 MB object, 3 copies consume 1,800 MB and survive 2 losses, whereas 6+3 erasure coding consumes 900 MB and survives 3 losses. The trade-off shifts the cost to reads and repairs: reading the full object fetches 6 shards from 6 machines, and rebuilding 1 lost disk reads 6 shards to compute 1. Consequently, stores often keep full copies for small or frequently read objects and use erasure coding for large or less frequently read ones. Facebook described this trade-off in its f4 store for less-read photos. Erasure coding there reduced the effective replication factor—the bytes stored per byte of data across all data centers—from 3.6 to 2.8 or 2.1 (f4, OSDI 2014).
Cloud providers publish the durability their designs target. Amazon S3 Standard is designed for 99.999999999% durability of objects over a year. It stores data across multiple devices in at least three Availability Zones (isolated groups of data centers within a region) and verifies integrity with checksums (S3 data protection). Note that durability differs from availability: an object can remain safe on disk while a brief outage prevents the store from serving it for a few minutes.
Uploading Large Files
Uploading a 5 GB video in a single PUT has two drawbacks. If the connection drops at 4 GB, the store keeps nothing and the upload restarts from byte 0. Furthermore, a single connection can be slower than the underlying network link, so one stream may not use all the bandwidth. Amazon S3 also limits a single PUT to 5 GB (S3 uploads).
Multipart Upload
A multipart upload splits the file into parts and uploads each part in a separate request. It has three steps (S3 multipart upload):
- The client initiates the upload, and the store returns an upload ID.
- The client uploads each part with the upload ID and a part number, receiving an ETag for each stored part. Parts can be uploaded in any order and in parallel, and a failed part is retried on its own.
- The client sends a complete request listing the part numbers and ETags, and the store assembles the parts in order into one object.
In Amazon S3, a part is 5 MiB to 5 GiB (the last part can be smaller), and an upload has at most 10,000 parts. AWS suggests multipart uploads for objects of 100 MB or larger (S3 multipart limits). A 5 GB video split into 100 MB parts has 50 parts; adding 1 start call and 1 complete call gives 52 requests when no part fails.
The widget runs both uploads on the same clock, with the same illustrative inputs. Each connection carries 25 MB/s, the link carries up to 4 connections, and the connection drops at 45 s and at 170 s. Press Play. The widget pauses at each drop, so press Play again to continue until both uploads finish. Then set "Parallel streams" to 4 and press Play again. Last, try a part size of 500 MB with 4 streams.
At 25 MB/s, a single PUT needs 200 s to send 5,000 MB. Drops at 45 s and 170 s discard 1,125 MB and 3,125 MB, so the upload finishes at 370 s on its third attempt after wasting 4,250 MB.
With 1 stream, a multipart upload loses only the 25 MB of the single part in flight at each drop. It finishes at 203 s across 54 requests, including 2 retried parts.
With 4 parallel streams, the client sends 100 MB/s, stores all 50 parts by 53 s, and finishes before the second drop occurs.
Part size trades wasted bytes on a drop against request count. With 4 streams, 500 MB parts lose 250 MB on the first drop, compared to 100 MB with 100 MB parts. With 1 stream, 50 MB parts take 104 requests, whereas 500 MB parts take 14.
If a client starts an upload and never completes it, the uploaded parts remain stored and billed. Amazon S3 deletes them when the client aborts the upload, or via a lifecycle rule that aborts incomplete uploads after a set number of days (S3 multipart upload).
Pre-Signed URLs
Routing a file upload through an app server consumes memory, bandwidth, and an open connection for minutes. Instead, a pre-signed URL lets the client upload bytes directly to object storage. The app server signs a URL using its storage credentials, granting one specific operation—such as a PUT of one part to one key—until an expiry time. The signature is a hash of the HTTP method, key, and expiry computed with the app server's secret key. When the client uses the URL, the store recomputes the hash with the same secret and compares the two. If the client alters the key or expiry, the hashes do not match and the store rejects the request. The client never holds any storage credentials (S3 pre-signed URLs).
Here, the app server exchanges a few small control messages while the 5 GB of video bytes flow straight from the client to the store. Downloads of private files work the same way: the app server checks permissions and returns a short-lived pre-signed GET URL. The handling large files lesson covers this upload flow in detail, including the status field and validation checks.
Serving Reads Through a CDN
Public files like product images or popular videos are read far more often than they are written, and by users far from the storage region. A CDN (content delivery network) caches copies of objects on edge servers close to users. On the first request for an object, an edge server fetches the object from the bucket—called the origin—and serves subsequent requests in that location from its local copy.
With a CDN in front, the bucket serves about one request per object per edge until the cached copy expires, rather than one request per user. The CDN lesson covers caching rules and invalidation.
Storage Classes and Lifecycle Rules
Many objects, such as logs and older uploads, are read often in their first few days and rarely afterward. A storage class sets an object's storage price and access speed. Cheaper storage classes charge more per read, enforce a minimum billed duration, or require a restore step before reading. Amazon S3's classes illustrate the spectrum (S3 storage classes):
| Class | Read access | Minimum billed duration | Retrieval fee |
|---|---|---|---|
| S3 Standard | Milliseconds | None | None |
| S3 Standard-IA (infrequent access) | Milliseconds | 30 days | Per GB |
| S3 Glacier Instant Retrieval | Milliseconds | 90 days | Per GB |
| S3 Glacier Flexible Retrieval | Restore first, minutes to hours | 90 days | Per GB (bulk restores free) |
| S3 Glacier Deep Archive | Restore first, hours | 180 days | Per GB |
A lifecycle rule automatically transitions objects between classes or deletes them based on age and key prefix (S3 Lifecycle). For example, a rule for application logs might look like this:
A lifecycle configuration can also delete old versions in a versioned bucket and abort multipart uploads left incomplete for 7 days. Because each transition to another class is a billed request, lifecycle rules save the most money on large objects stored for months, and less on many small objects.
Limitations
Object storage supports a narrow API: put, get, and delete an object by key, and list keys by prefix. Four limitations follow from this design:
- Latency per request. Each operation is an HTTP call to a remote service. AWS notes small-object latencies of roughly 100–200 ms for Amazon S3 Standard (S3 performance). Loading 50 small records one by one from object storage means waiting on 50 such calls.
- No queries on contents. The store cannot filter for "videos longer than 10 minutes" or "invoices of user 42". Those queries require a database index whose rows reference the object keys.
- Cost per request and per byte out. Providers bill per request and per byte transferred out of the region, on top of storage capacity (S3 pricing). Writing one million 1 KB objects costs 1 million requests, whereas writing one 1 GB object in 100 MB parts costs 12.
- No partial updates or multi-object transactions. Updating an object rewrites it entirely, and two objects cannot be updated together atomically.
When to Use Object Storage
Object storage fits data that is written once or rarely and read as a whole file or byte range. It does not fit small records that change often or require query filtering.
| Use object storage for | Use something else for |
|---|---|
| Images, video, audio, and documents that users upload | Records that requests filter or join on: a database |
| Database backups and snapshots | Small values read many times per second: a cache or a key-value store |
| Logs and event files for later analysis | Text search over contents: a search engine |
| Static website files and build artifacts | Files that programs edit in place or lock: a file system or block storage |
In practice, object storage usually sits alongside a database: the database row holds the key, owner, status, and searchable fields, while the object holds the raw bytes.
Key idea. Object storage keeps whole objects under a bucket and key in a flat namespace. It looks them up only by key or prefix and replaces the object on each write. Durability comes from redundancy across failure domains: 3 copies cost 3x storage and survive 2 losses, while a 6+3 erasure code costs 1.5x and survives 3. Large files upload in parts via pre-signed URLs, so a dropped connection retries one part and the bytes bypass the app servers.
The next lesson, SQL vs NoSQL, compares the two main database choices for the records that sit alongside these objects.
Further Reading
- What is Amazon S3? — Amazon S3 docs — buckets, keys, versions, and the strong read-after-write consistency model.
- Amazon S3 strong read-after-write consistency — AWS announcement, December 2020 — when the consistency model changed.
- ListObjectsV2 — Amazon S3 API reference — prefix, delimiter, the 1,000-key page, and continuation tokens.
- Multipart upload — Amazon S3 docs and multipart limits — the three steps, part retries, billing of incomplete uploads, and part-size limits.
- Sharing objects with pre-signed URLs — Amazon S3 docs — how a signed, time-limited URL grants one operation.
- Data protection — Amazon S3 docs — the durability design target and storage across Availability Zones.
- Storage classes and lifecycle rules — Amazon S3 docs — access speed, minimum durations, and moving objects by age.
- f4: Facebook's Warm BLOB Storage System — OSDI 2014 — erasure coding for less-read photos and its effect on storage overhead.