Your API response is eating the agent's context window
Design bounded responses that preserve useful facts. Filter, select and aggregate before large intermediate payloads enter an agent's context.
An agent needs the total for overdue invoices. The API returns thousands of records, including nested customer profiles and audit histories. Most of that response contributes nothing to the calculation, but the model still has to process it if the entire payload enters its context.
Return the records and fields needed for the task, with enough context to check the calculation. Response size needs its own review even when the agent has only a few tools.
Put selection near the data
If the API supports server-side filtering, use it to select the intended records before transmission. If field selection is available, request the identifiers, amounts, dates and statuses the task needs. Apply tenant authorization before returning either details or aggregates.
Where suitable, expose an aggregate operation with explicit semantics. A total should say which records and time range it includes, what currency or unit applies, and whether the result is complete. A concise number without those boundaries can be more misleading than a verbose response.
Do not silently truncate. A capped result should advertise its limit and provide the documented continuation mechanism or a clear indication that more results exist. An agent cannot infer completeness reliably from the length of an array.
Preserve evidence and meaning
Keep stable resource identifiers and a supported way to fetch details. For a summary, retain the filters and observation time needed to reproduce it. If the data is eventually consistent, explain what freshness means for that endpoint.
Avoid dropping fields solely because they look repetitive. A currency code, timezone, null value or warning can change the interpretation of a result. Make projection rules explicit and test summaries against full records using synthetic fixtures.
An agent asking for a customer display name may not need the full customer object. It still needs a reliable association between that name and the resource it is discussing. Saving tokens must not turn distinct records into ambiguous prose.
Process intermediate results outside model context
Some workflows need large raw datasets for computation but only a small result for reasoning. A controlled execution layer can join, filter or aggregate records before passing the final output to the model.
Anthropic's programmatic tool-use engineering article describes one implementation of this pattern. Its measured savings belong to that implementation and workload. The general principle is useful independently: intermediate bytes need not all become model input.
Such a layer still needs ordinary controls: scoped credentials, bounded execution, output limits and safe handling of customer data. Moving processing into code changes where work happens; it does not remove authorization or privacy obligations.
Measure at the task boundary
Compare variants with the same records, question and correctness checks. Record response bytes, model input usage when available, elapsed time, call count and task success. A smaller response that forces repeated detail requests may be slower or more expensive overall.
Include difficult cases: empty matches, multiple currencies, missing optional fields and more results than the default cap. Check whether the agent recognizes an incomplete result instead of confidently reporting a partial total.
