High-Level Design

Design the serving layer that lets one LLM assistant answer at different reasoning-effort levels: low, medium, high, and max. Effort is the number of internal thinking tokens the model spends before it writes the final answer. A caller picks a level per request, and the system honors it.

The scope is effort as a routing, scheduling, and budgeting dimension on top of an existing inference fleet. The base fleet mechanics are out of scope and treated as an existing dependency: continuous batching, the KV cache, and GPU economics. Model training and the model's internal reasoning method are also out of scope.