Capacity Planning

Design a conversational assistant like ChatGPT. A user sends a message in an ongoing conversation. A reply streams back word by word, generated by a large language model (LLM): a neural network that predicts text one small piece at a time.

The design has three parts. The first is the conversation surface: sessions, persistent history, and streaming delivery to the user's devices. It works like a chat product, except the other party is a model, so the reply is generated rather than relayed. The second is serving the model, which runs on graphics processing units (GPUs), the accelerators that execute the model's math. The third is orchestrating the context: assembling what the model sees from conversation history, retrieved documents, and tool results.

In scope: sending a message and streaming the reply, conversation history across devices, context assembly, retrieval over documents, and tool calls. Out of scope: training and fine-tuning the model, image and audio input, and accounts and billing.