Design ChatGPT

Problem statement

Design a conversational assistant like ChatGPT. A user sends a message in an ongoing conversation. A reply streams back word by word, generated by a large language model (LLM): a neural network that predicts text one small piece at a time.

The design has three parts. The first is the conversation surface: sessions, persistent history, and streaming delivery to the user's devices. It works like a chat product, except the other party is a model, so the reply is generated rather than relayed. The second is serving the model, which runs on graphics processing units (GPUs), the accelerators that execute the model's math. The third is orchestrating the context: assembling what the model sees from conversation history, retrieved documents, and tool results.

In scope: sending a message and streaming the reply, conversation history across devices, context assembly, retrieval over documents, and tool calls. Out of scope: training and fine-tuning the model, image and audio input, and accounts and billing.

Clarifying questions

Each question below, with the assumption its answer establishes:

• Just the model, or the whole product? The whole product. The conversation surface is in scope alongside serving and orchestration.
• Inference only, or training too? Serve a given, already-trained model. Inference means running a trained model to produce output. Training and fine-tuning are a separate offline system and are deferred.
• Streaming or wait-for-complete? Streaming, token by token. This one choice defines the latency model and the transport.
• One model or many? One primary model class. Routing easy turns to a cheaper model is a variant.
• How long are conversations, and how big is the context window? The context window is the maximum amount of text the model can read in one request. Long conversations exceed it, which forces a decision about summarizing or truncating old turns.
• Are retrieval and tools in scope? Yes. Answering from documents and calling tools are first-order requirements, so context assembly and an agent loop sit in the core design.
• What is the scale unit? Concurrent active generations, not requests per second. Each generation holds scarce GPU time for seconds.

What makes this problem distinctive

Most product designs spend their effort on storing and fetching data. Here the stored data is small text. The expensive work is the reply itself, which the system computes on demand and cannot look up.

That computation has an awkward shape. The model produces the reply one token at a time, and each token depends on all the tokens before it. So the work is slow to finish but can start showing results early. The user judges the product by how fast the first word appears, then by how smoothly the rest flows.

The resource the computation uses is also unusual. GPUs are scarce and expensive, and a single GPU can serve many users at once only if their work is packed together. Packing more users together raises the work done per GPU, but it slows each user's stream. The two goals pull in opposite directions.

Token. The unit of text a language model reads and writes. A token is usually a word or a piece of a word: "streaming" may be one token, and "unbelievable" may split into "un", "believ", "able". Model costs, context limits, and generation speed are all measured in tokens.

Key idea. The reply is a computation, not a lookup. Per-user speed and GPU efficiency pull against each other, and the model needs context it cannot hold itself.

Key concepts

This section covers the concepts needed to solve this problem — prerequisites for the design work that follows.

Autoregressive generation

An LLM generates text autoregressively: it reads the whole prompt, predicts one next token, appends that token to the input, and predicts again. The loop stops when the model emits an end-of-sequence token or reaches a length limit.

For example, given the prompt "The capital of France is", the model predicts "Paris". The input becomes "The capital of France is Paris", and the model predicts ".". A 500-token reply needs 500 passes through the model, one after another. The passes cannot run in parallel, because each one needs the token from the pass before.

Prefill, decode, and time-to-first-token

Generation has two phases with different costs. Prefill processes the entire prompt in one large pass, because all prompt tokens are known in advance. Decode then produces output tokens one at a time. Prefill is limited by compute, since it does a lot of math at once. Decode is limited by memory bandwidth, since each step reads the model's weights to produce one token.

These two phases give the two latency metrics. Time-to-first-token (TTFT) is the time from "user presses enter" to the first visible word: queue wait plus prefill plus the first decode step. Inter-token latency is the gap between later tokens, which sets how smoothly the text flows.

The widget below runs one request. Press Send message to watch it wait in the queue, prefill the prompt, and then decode word by word. Drag the conversation length to see prefill grow. Switch to "Buffer whole reply" to see why streaming matters: the total time is the same, but the first word appears only at the end.

The prefix cache toggle previews an optimization covered later. When the earlier conversation was already processed on a previous turn, its work can be reused, so prefill only covers the new message.

The KV cache

Inside the model, each token produces two vectors per layer, called the key and the value. The attention mechanism compares the newest token against the keys and values of every earlier token. Recomputing them for all earlier tokens on every step would repeat the same work hundreds of times.

The KV cache stores each token's keys and values in GPU memory the first time they are computed. Each decode step then computes keys and values only for the newest token and reads the rest from the cache. The cache grows by one entry per token for every request in flight. It lives in the same GPU memory as the model's weights, so it competes with them for space.

Batching on a GPU

Each decode step must read all of the model's weights from GPU memory. If the GPU serves one request, it reads every weight to produce a single token. If it serves 16 requests in the same step, it reads the weights once and produces 16 tokens. Batching is running several requests through the same step. It raises the number of tokens produced per GPU-second, which is the quantity that sets the bill.

The batch has two limits. More requests per step means each step takes longer, so every user's next token arrives later. And each request in the batch keeps its KV cache in GPU memory, so memory caps how many requests fit.

Embeddings and vector search

An embedding is a list of numbers that represents the meaning of a piece of text. An embedding model maps similar meanings to nearby points. "How do I reset my password?" and "I forgot my login" land close together even though they share no words.

A vector store indexes embeddings for nearest-neighbor search: given a query embedding, return the stored items closest to it. Exact search compares the query against every item, so large stores use approximate nearest-neighbor (ANN) indexes that check a small, well-chosen subset and accept rare misses.

Tool calls

A tool call is a structured request the model emits instead of text, such as get_calendar(date="2026-10-28"). The model does not run the tool. The surrounding system runs it, gives the result back to the model as new input, and lets the model continue. A model that can call tools can look up facts and take actions it cannot produce from its weights alone.

Key idea. A reply is a token-by-token loop on a GPU. Prefill sets the first-token delay, batching sets the cost, and the KV cache sets how big a batch fits.

1. Requirements

1.1 Functional requirements

The functional requirements come from the verbs in the problem statement:

• Send and stream. A user sends a message in a conversation and receives a streamed reply generated by the model.
• Manage conversations. Create, list, and title conversations. The full history persists and is readable from any of the user's devices.
• Carry context. The model sees the relevant earlier turns of the conversation.
• Ground and act. The model answers from retrieved documents, a technique called retrieval-augmented generation (RAG), and calls tools when the task needs them.

Training, image and audio input, and billing are out of scope.

1.2 Non-functional requirements

The non-functional requirements come from the product's constraints:

• Low time-to-first-token. A low p99 TTFT is the headline latency. p99 is the value that 99% of requests beat. A chat that stalls before the first word feels broken.
• Smooth streaming. After the first token, inter-token latency stays steady so the text flows.
• Throughput per GPU. Maximize tokens per GPU-second, because GPU capacity is the dominant cost and the scarce resource.
• Graceful degradation. When GPUs saturate, the system queues, sheds load, or falls back to a smaller model rather than collapsing.
• Reliable delivery and session continuity. The reply reaches the user's connected devices. A dropped connection resumes the stream, and history stays consistent across devices.
• Durable history. Conversation history must not be lost. This is a modest requirement, because the data is small text in an ordinary database.

1.3 The constraint versus the property

The property the design refuses to compromise is per-user responsiveness: a fast first token and a smooth stream. A slow assistant is not a usable product.

The constraint that shapes the architecture is GPU economics. Generation is GPU-bound, and GPUs are scarce and expensive. So most decisions in the design exist to raise tokens per GPU-second: batching, KV cache management, reusing shared prompt prefixes, admission control, and model sizing.

The two collide at one place. Putting more requests into each GPU step raises throughput but slows each user's tokens. That is why the batch scheduler is the center of the system, and why a latency budget, not only utilization, decides how large a batch can grow.

Key idea. Responsiveness is the property; tokens per GPU-second is the constraint. They meet at the batch scheduler.

2. Back-of-the-envelope estimation

Size the system from usage assumptions, and count tokens rather than requests. All figures below are illustrative anchors, not measurements. (For the method, see Back-of-the-envelope Calculation.)

2.1 The model does not fit on one GPU

Consider a model with roughly 70 billion parameters stored as 16-bit numbers, which take 2 bytes each. The weights alone need 70 × 2 = 140 GB. A single accelerator holds on the order of tens of gigabytes, so the model must be split across several GPUs. One request therefore runs on a group of GPUs, not one.

2.2 GPU-seconds are the budget

Assume roughly 100 million daily active users, each sending about 10 messages per day, with replies of about 500 output tokens. That gives 100M × 10 × 500 = 500 billion output tokens per day, or about 5.8 million tokens per second on average.

Assume a GPU produces on the order of 100 million tokens per day at full load (illustrative). The fleet then needs 500B ÷ 100M = 5,000 GPUs running flat out. At a realistic utilization near 50%, that becomes about 10,000 GPUs. The exact magnitude matters less than the method: divide tokens per day by tokens per GPU-day. The conclusion is "many thousands of GPUs."

2.3 The KV cache competes with the weights

Each in-flight token keeps its keys and values in GPU memory. The cache grows with context length times batch size, and it shares memory with the 140 GB of weights. So memory, not compute, usually limits how many requests a GPU group can batch. Deep dive 6.1 returns to this.

2.4 Conversation storage is small

Each message is small text. If a stored message takes about 2 KB, 100M users × 10 messages × 2 messages per exchange × 2 KB is about 4 TB per day. That is a normal sharded database workload. Next to a fleet of thousands of GPUs, the conversation store is a rounding error. This problem inverts the usual shape: compute dominates and storage is cheap.

Key idea. Size the fleet by tokens per day divided by tokens per GPU-day. Thousands of GPUs, not the database, set the cost.

3. API design

The API has one job a normal request/response API does not: return a stream that starts before the answer is known. The endpoints follow the data. First a conversation exists, then a turn is submitted and streamed, then a generation in flight can be controlled.

3.1 Create and list conversations

The owner comes from the authenticated session, not the request body. Listing and reading history are what another device calls to load a conversation.

3.2 Send a message and stream the reply

The reply is a stream of Server-Sent Events (SSE): a one-way server-to-client stream of events over a held HTTP connection. (See Real-time Transport.) The first token event marks TTFT, and the client renders tokens as they arrive.

The client sends only the conversation id and the new message. The server holds the conversation and assembles the context: history, summary, and retrieved documents. The client never re-uploads the whole history, and the context policy stays on the server. A stateless developer-style API that accepts the full messages array also exists; it moves context cost to the client and is a variant.

The client_message_id lets the server ignore a retried submit, so a network retry does not start a second generation. The tool_call event makes the agent loop part of the protocol: the client can show "searching…" while the orchestrator runs the tool and resumes generation.

3.3 Cancel a generation

Cancellation is a first-class endpoint. An abandoned tab that keeps generating wastes GPU time. Cancel propagates to the GPU worker and frees the request's slot in the batch. At GPU prices, reclaiming abandoned work is a real cost lever.

3.4 Resume a stream

A reconnecting client asks for tokens after the last one it received. Deep dive 6.5 explains how the server answers this without generating the reply again.

Key idea. The send call returns a token stream, and cancel is a cost control. The server owns the conversation, so the client sends only the new message.

4. Data model

Start with one entity, and let each thing it cannot represent introduce the next.

4.1 The message

One turn in a conversation: who said it, what was said, and when. The role field distinguishes the speaker: user, assistant, system, or tool. That one field does more work later than it appears to now.

The seq field orders messages within a conversation. The status field marks an assistant message as streaming, complete, or cut_off, because the reply is written while it streams.

4.2 The conversation

Messages belong to an ordered thread. The system loads a whole thread to rebuild what the model should see, and a thread has its own properties: which model, a title, and an owner. That is a one-to-many relationship, so the conversation is its own entity.

4.3 The rolling summary

The model cannot read an unbounded thread, because the context window is finite. A long conversation does not fit, so the system cannot concatenate every message. The thread needs a compressed form: a rolling summary of old turns, kept alongside the recent turns verbatim. Context assembly then becomes system prompt + summary + last K turns.

The covers_through_seq field records which messages the summary already includes. Messages after it are sent verbatim.

4.4 The document chunk

Grounded answers need facts that are not in the thread at all. Retrieval pulls relevant documents in at request time, so the system needs a corpus searchable by meaning: documents split into chunks, each with an embedding, queried by nearest-neighbor search.

4.5 Tool results need no new entity

A tool result is another message with role = tool, appended to the thread the model reads next. The role field from the first entity already absorbs the agent loop.

A conversation has many messages and at most one current summary. The many-to-many link between messages and chunks records which chunks grounded a reply, so the client can cite them.

4.6 Where each entity lives

• Messages, conversations, and summaries live in a database partitioned by conversation_id. They are small structured text, read in full to assemble context. (See Database Partitioning.)
• Chunks and embeddings live in a vector store with an ANN index. They are separate because the access pattern is similarity search, not key lookup.
• The KV cache is not in the data model. It is per-request attention state in GPU memory during generation. It can be rebuilt from the messages, which are the source of truth, so it is never persisted.

Key idea. Four entities, each forced by what the one before it could not hold. Tool calls reuse role, and the KV cache is derived state, not data.

5. High-level design

Rather than present the final system, build it from the simplest design that works and let each failure introduce the next component.

Reading the diagrams. Each step marks the components newly added at that step with a dashed outline and a NEW badge, so you can see what changed from the step before.

5.1 One server with the model inside

The simplest design puts the model in the app server. The server takes the message, runs the model in-process, and returns the text.

Five things break once this is a real LLM service:

• The model is large and GPU-bound. It cannot sit in a stateless app server, and one generation holds the GPU for seconds.
• The reply is produced token by token over seconds. One JSON body means the user watches a spinner the whole time.
• GPUs are scarce. One request per GPU at a time wastes most of each GPU.
• The model knows only its training data and the current prompt. It needs history beyond the window, fresh facts, and a way to act.
• The reply must reach the user's devices reliably, survive a dropped connection, and stay consistent across devices.

The next steps fix them one at a time.

5.2 Fix 1: split the GPU fleet from the stateless tier

Move the model out of the app server. A stateless API tier handles authentication, conversation storage, and context assembly. It scales horizontally on cheap machines. It hands each generation request to a pool of GPU workers through an inference queue. Each worker is a group of GPUs holding one copy of the model, split across them because the weights do not fit on one accelerator (section 2.1).

The queue decouples bursty user traffic from fixed GPU capacity. (See Message Queue.) But the API tier still waits for the full reply before it responds.

5.3 Fix 2: stream tokens as they are generated

The worker emits each token onto an internal stream as soon as it is decoded. The API tier relays each token to the client over SSE. TTFT is now queue wait + prefill + first decode step. After that, the user sees a steady flow instead of a spinner.

Streaming fixes the perceived wait, but each GPU group still serves one request at a time, so most of its capacity sits idle during decode.

5.4 Fix 3: batch on the workers and admit carefully

Add a batch scheduler on each worker. It packs many active generations into each decode step and adds or removes requests as they start and finish, rather than waiting to fill a fixed batch. The KV cache holds each request's attention state, and allocating it in blocks lets more requests share the memory (deep dive 6.1).

Batching creates a new failure. When load exceeds capacity, a scheduler that admits every request makes every stream slow. An admission controller in front of the queue protects TTFT and inter-token latency. When GPUs saturate, new requests wait, and the system autoscales, sheds load, or falls back to a smaller model rather than slowing everyone.

The GPUs are now efficient, but the model still sees only what the API tier sends it, which so far is the latest message.

5.5 Fix 4: an orchestrator for context and tools

Add an orchestrator between the API tier and the inference queue. It builds the prompt from the system prompt, the rolling summary, the last K turns, and chunks retrieved from the vector store. When the model emits a tool call, the orchestrator runs the tool, appends the result as a role = tool message, and enqueues generation again. It loops until the model produces a final answer.

The model now has context and can act. The last failure is delivery: one held connection from one device is fragile, and the user may switch devices mid-reply.

5.6 Fix 5: the conversation surface

The stateless tier is also the product shell. It persists the user's message before generation starts and persists the reply as it streams. Each device holds a connection to a gateway tier that relays the token stream. A connection registry records which gateway holds each device's connection. This is the delivery layer of a chat product, described in Design a Chat App.

Because the conversation lives on the server, any device can load history and resume a dropped stream from the last delivered token. A flaky network does not lose a reply.

5.7 The send-message flow

The sequence below follows one message from submit to the last token. It assumes no tool call.

5.8 The composed design

The boxes are the data model's homes made concrete. Messages, conversations, and summaries live in the conversations database. Chunks live in the vector store. The KV cache is transient GPU memory. The assembled context is not stored at all: the orchestrator builds it per request from database rows, the summary, and retrieval, then discards it.

Strong-answer criteria. Each box is justified by a constraint: the model is split across GPUs because the weights do not fit on one; the batch scheduler exists because idle GPUs are the cost; the orchestrator exists because the model needs context and actions it cannot hold; and admission control trades a longer queue for a protected TTFT under load.

Key idea. Five failures pull in five components: a separate GPU fleet, token streaming, batching with admission control, an orchestrator, and a gateway-based conversation surface.

6. Deep dives

Five topics carry the design. On the serving side: the batch scheduler with its KV cache, and the TTFT budget. On the orchestration side: retrieval and the agent loop. Around both: the conversation surface.

6.1 Continuous batching and the KV cache

Before reading on: generations finish at different, unpredictable times. One user asks for a word, another for an essay. If the scheduler waits to assemble a fixed batch before running the GPU, what goes wrong? And what limits how many requests run at once?

From fixed to continuous batching

Fixed batching wastes the GPU in two ways. Waiting to fill a batch adds queue delay. And a batch finishes only when its longest request does, so short generations hold idle slots behind long ones.

Continuous batching, also called iteration-level batching, fixes both. The scheduler rebuilds the batch at every decode step. It removes finished requests and admits waiting ones immediately, so a freed slot is reused on the next step. The widget runs the same twelve requests through both schedulers. Watch the dashed cells in the fixed lane: those are slots held by finished requests.

Memory caps the batch

Each request's KV cache grows by one token per step and must stay in GPU memory. The caches of all in-flight requests share memory with the model's weights. A naive allocator reserves one contiguous region per request, sized for the maximum context, because it cannot know the final length. Most requests use a fraction of that region, so most of the reserved memory holds nothing.

A paged KV cache borrows the idea of operating-system virtual memory. It divides KV memory into fixed-size blocks, for example 16 tokens per block. Each request has a block table that maps its logical token positions to physical blocks anywhere in memory. A request takes a new block only when its last one fills, and returns all its blocks when it finishes. Waste is limited to the unfilled part of each request's last block.

The widget runs the same request stream through both allocators in a toy memory of 24 blocks. The contiguous allocator fits three requests at a time. The paged allocator fits more, because memory is claimed only as tokens arrive.

When memory still runs out, the scheduler can preempt a request: free its blocks, and later recompute its cache from its messages.

A latency budget caps the batch

A larger batch raises throughput, but each step takes longer, so every user's next token arrives later. The scheduler therefore caps the batch by an inter-token latency budget rather than filling memory blindly. It reserves headroom so streams stay smooth.

Prefill causes a second interference. A 50-page prompt needs a long, compute-heavy prefill. If it runs as one step, every decoding request in the batch stalls behind it. Chunked prefill splits a long prompt into pieces and runs one piece per step alongside the decode work. Some systems go further and run prefill and decode on separate GPU pools, because one phase is compute-bound and the other is memory-bandwidth-bound.

Failure and monitoring

A GPU worker that dies mid-generation loses its requests' KV caches. The system retries on another worker from the persisted messages, which are the source of truth. Generation is replayable, although sampling may produce different text. The output so far is already persisted, so the retry continues from it.

The signals to watch are GPU utilization, batch occupancy, p99 TTFT, tokens per second, queue depth, and preemption rate. A growing queue with low batch occupancy means the scheduler or KV memory is the bottleneck, not raw compute.

6.2 Streaming and the time-to-first-token budget

Before reading on: a user waits longest for the first token, and then the rest flow. Where does that first-token time go, and what can be computed ahead of time to shrink it?

Where TTFT goes

TTFT has three parts: queue wait, prefill, and the first decode step. Under heavy load, queue wait dominates, and admission control manages it. For long contexts, prefill dominates, because the model must process the entire prompt before producing a new token.

Prefix caching

Across turns of one conversation, the system prompt and the earlier turns are the same tokens in the same order. Prefix caching stores the computed KV state for a prompt prefix, keyed by a hash of its tokens. On the next turn, the worker finds the cached prefix and prefills only the new tokens. The same applies across users who share a system prompt. A high prefix-cache hit rate does for TTFT what a high CDN hit rate does for video delivery in Design YouTube: it removes repeated work from the expensive path. (For the general pattern, see Caching Read Patterns.)

The prefix cache lives in GPU memory on a specific worker. So the queue should route a conversation's next turn to the worker that likely holds its prefix, or the cache misses.

Transport and cancellation

The API tier keeps the client connection open and relays worker tokens as SSE. The worker-to-API hop is an internal stream. Cancellation must travel the whole path back: a closed tab closes the SSE connection, the API tier sends cancel, and the scheduler frees the request's slot and KV blocks on the next step.

Degrading from the user's point of view

Under load, a slightly late start is better than a reply that freezes halfway. So the system lets queue wait, and therefore TTFT, rise while it protects inter-token latency for requests already streaming. Admission control enforces this: it holds new requests in the queue rather than admitting them into a batch that would slow every active stream.

The signals to watch are p50 and p99 TTFT, the inter-token latency distribution, the prefix-cache hit rate, and the stream-abandonment rate, which indicates replies too slow to start.

6.3 Retrieval-augmented generation

Before reading on: the model's training is frozen and its context window is finite. How does the system answer a question about a document the model never saw, without retraining?

Retrieve, then generate

At request time, the orchestrator embeds the user's query, searches the vector store for the nearest chunks, and splices the top matches into the prompt as grounding context. The model reads them as part of its input. Documents are prepared ahead of time: split into chunks, embedded, and indexed.

Retrieval changes the latency and cost math

Every retrieved chunk lengthens the prompt, and prefill cost grows with prompt length. So RAG directly raises TTFT. The rule is to retrieve a few relevant chunks, not many. This is where the orchestration half pushes back on the serving half: more grounding means more prefill, which means a slower first token.

Grounding and correctness

Retrieval reduces hallucination (fluent but false output) by giving the model sources. But stale or wrong chunks ground the model in stale or wrong facts. The freshness of the corpus and the relevance of the retriever become part of the answer's correctness. The mitigations are re-embedding a document when it changes and citing which chunks the reply used, so the user can verify.

The signals to watch are retrieval relevance (are the top chunks on topic?), added prompt tokens per request (the TTFT cost of retrieval), and the rate of answers with citations.

6.4 Tool calling and the agent loop

Before reading on: the user asks "what is on my calendar tomorrow?" The model cannot know, but it can ask. How does a one-shot generator become something that takes an action and continues?

The loop

Some answers need an action the model cannot produce from text. The model emits a structured tool call instead of a final answer. The orchestrator runs the tool (a search, a calculator, an internal API), appends the result as a role = tool message, and enqueues generation again. The model continues with the result in its context. The loop repeats until the model emits a final answer. That loop is the agent.

Bounding the loop

The loop uses expensive resources in bursts and can run away. Each hop is a new generation, which costs more GPU time, plus a tool wait during which the GPU slot should be released rather than held idle. The bounds are a maximum step count, a timeout per tool, and releasing the GPU between hops so a slow tool does not pin an accelerator. The orchestrator can checkpoint the conversation between steps, so a failure resumes from the last completed step rather than restarting. This is the same step durability as a saga. (See Saga Pattern.)

Failure modes and blast radius

A tool that errors or times out returns the error as a tool message, so the model can recover or explain, rather than failing the whole turn. A model that loops without end hits the step cap. A tool with side effects, such as sending email or making a purchase, needs user confirmation and idempotency: a retried call must not repeat the action. The orchestrator passes an idempotency key per tool call. (See Idempotency.) The blast radius of a wrong tool call is a real-world action.

The signals to watch are the distribution of steps per turn, tool error and timeout rates, GPU idle time held during tool runs, and step-cap trips.

6.5 The conversation surface: history, delivery, and multiple devices

Before reading on: the reply is streaming when the user's phone drops off Wi-Fi, and the user opens the app on a laptop. What must already be persisted, and what does the laptop need to show the finished answer?

Persist first, then stream

The user's message is written to the conversations database before generation starts. The assistant's reply is persisted as it streams, in small appended batches of tokens, and a reply that is cut off keeps its partial text with status = cut_off. History is durable, so a crash or a device switch loses nothing. This is the persist-then-deliver discipline of a chat product, where the live push is an optimization on top of the durable write.

Delivery through the gateway tier

Each device holds a connection to a gateway, and the connection registry records which gateway holds it. The difference from person-to-person chat is the sender. Here the sender is the GPU fleet, so one generation fans out to the user's own devices rather than to other people.

Resume, do not regenerate

Because the reply is persisted as it is produced, a reconnecting client reads the conversation up to its last received token and subscribes to the live tail. The GPU work is not repeated. If generation is still running, the new connection attaches to the same token stream.

Server-authoritative across devices

The conversation lives on the server, so every device loads the same history and subscribes to the same generation. A reply started on the phone appears on the laptop. Conversation-list changes, such as a new chat, a rename, or a delete, sync the same way: small writes that other devices read. This is the metadata sync of Design Dropbox at a coarser grain.

A gateway that dies drops its connections. Clients reconnect to another gateway and resume from the persisted tail, with a brief interruption and no lost reply. The signals to watch are reconnect rate, stream-resume success rate, and cross-device history lag.

Key idea. Batching sets the cost, prefix caching sets TTFT, retrieval and tools set what the model can know and do, and persist-as-you-stream makes every reply survive a dropped connection.

7. Variants

7.1 Ten times the scale

More concurrent generations need a fleet that grows roughly in proportion to GPUs. Unlike static media, most generations are unique, so caching whole responses helps only for the most repeated prompts. The levers are efficiency rather than offload:

• Quantization: store weights in lower precision, which frees memory for larger batches and raises throughput at a small quality cost.
• Speculative decoding: a small draft model proposes several tokens, and the large model verifies them in one step, raising tokens per second.
• Larger batches within the latency budget, and a higher prefix-cache hit rate.

7.2 Multi-model routing

Route by difficulty. A small, cheap model answers easy turns, and the large model handles hard ones. The router is a fast classifier. This trades a little accuracy on easy queries for a large drop in GPU-seconds. It is the same right-sizing idea as tiering storage by popularity in a video platform.

7.3 Tighter latency or longer context

A real-time voice assistant tightens TTFT and inter-token latency. That pushes toward smaller models, speculative decoding, and aggressive prefix caching. The opposite extreme, very long contexts, makes prefill dominate. The design then leans harder on prefix reuse and chunked prefill, and retrieval becomes more selective to keep prompts short.

Key idea. At larger scale, caching cannot offload unique generations, so the levers become efficiency per GPU and matching model size to the request.

8. The transferable pattern

ChatGPT is a chat product, conversation surface included, where the reply is a GPU-bound, streamed, autoregressive computation. That turns the system into an inference-serving and scheduling problem, in the same way a video platform is a delivery problem.

When the unit of work is expensive and produced incrementally on a scarce accelerator, the shape repeats. Separate the expensive compute fleet from the stateless tier. Stream the result and optimize time-to-first-output. Batch to maximize work per accelerator-second, with a latency budget capping the batch. Cache the reusable prefix of the computation. Wrap the generator in an orchestration loop that feeds it the context and actions it cannot hold. The same shape appears in live transcription, image generation, and search ranking.

Key idea. Expensive incremental compute plus interactive latency gives the same design: a separate accelerator fleet, streaming, a latency-capped batch scheduler, prefix reuse, and an orchestration loop.

Review: the 30-second answer

• The reply is a GPU computation. The model generates one token at a time on a GPU, and tokens stream to the user as they are produced, so TTFT is the headline latency.
• Separate the GPU fleet from the stateless tier. The stateless tier handles auth, history, and context assembly. It hands generation to GPU workers that hold the sharded model.
• Keep the GPUs full with continuous batching. A scheduler packs many generations into each step. The KV cache, paged into blocks, is the memory that limits how many fit, and a latency budget caps the batch.
• Assemble the context the model cannot hold. Old history is summarized, fresh facts come from retrieval, and actions come from tool calls in a bounded orchestration loop.
• Persist as you stream. The server-authoritative conversation lets any device resume a reply without repeating GPU work.
• GPU capacity is the bill. Most decisions exist to raise tokens per GPU-second while the latency budget protects each user.

Quiz

Sources and further reading

• Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., SOSP 2023 — the paged KV cache and the block-table design behind deep dive 6.1.
• Orca: A Distributed Serving System for Transformer-Based Generative Models — Yu et al., OSDI 2022 — iteration-level (continuous) batching, the scheduling idea that keeps GPUs full.
• Attention Is All You Need — Vaswani et al., 2017 — the transformer and the attention keys and values that the KV cache stores.
• Fast Inference from Transformers via Speculative Decoding — Leviathan et al., 2023 — the draft-and-verify method behind the speculative-decoding lever in the variants.