High-Level Design

Design a retrieval-augmented generation (RAG) system for "chat with your data." A user asks a question in natural language. The system finds the passages in a private corpus that answer it and hands them, with their sources, to a language model that writes the answer. The corpus belongs to many tenants (customer organizations), holds on the order of 100 million documents, and changes all the time.

The scope is retrieval and grounding. The system ingests documents, keeps them searchable, and returns the best passages for each query, scoped to what the asker may see. Scope ends where the prompt is assembled. Serving the language model that writes the answer, training the embedding model, and multi-step agent loops are out of scope.