Can an AI agent actually integrate your API? Build an eval to find out

An agent writes an integration, reports success, and leaves behind code that has never run. The import resolves, but the method name belongs to an older SDK. Or the code reads one page of results and misses the rest. An evaluation needs to catch both.

Run a customer task against an isolated API environment, using the documentation and tools customers receive. Check the resulting records through a separate path. Save the request trace when the result fails an assertion.

Choose a task you can grade

Choose a workflow your customers need: create a customer, attach a test resource, then retrieve the resulting record. State the starting conditions and expected result. Use synthetic identifiers and a sandbox account that cannot affect production.

The grader should inspect persisted application state through an independent path. A transcript saying “created successfully” is evidence of what the agent believes. A matching record with the expected fields is evidence of what happened. If the assignment includes writing code, execute it in a clean environment.

Record the API and SDK versions, documentation snapshot, allowed tools, initial fixtures, permitted operations, success assertions, and per-attempt budgets. Also record how resources are cleaned up. Without them, a failed run may be impossible to reproduce.

Keep the harness from becoming the problem

Each attempt needs fresh state or a reliable reset. Otherwise a previous successful run can make a later broken integration appear correct. Use a distinct fixture namespace and check cleanup explicitly. If the environment is unavailable, record a harness failure rather than silently counting it as a model failure.

Fix the agent configuration when comparing documentation variants. Record the model identifier, instructions, tool configuration and package lockfile. Responses can vary between attempts, so repeat tasks and report the number of trials. One successful demonstration is insufficient for a reliability claim.

Count failed tasks before comparing speed

First ask whether the requested outcome occurred within the allowed permissions. Then measure elapsed time, tool calls, request errors, tokens where available, and unnecessary reads or writes. A fast failure should not outrank a slower correct integration.

Classify failures by transcript evidence. An agent might select the wrong operation, misunderstand a field, stop before processing all results, mishandle authentication, or report completion before an asynchronous job finishes. A missing prerequisite belongs in the workflow guide. A confused parameter needs a clearer schema or example. Keep the failed assertion so the next run checks whether the edit helped.

Keep regression tasks separate from harder tasks that reveal opportunities to improve. Anthropic's guidance on agent evals describes combining automated evaluation with transcript review and production feedback. A passing suite measures its own cases, not every future request.

Save the failed run

For each failure, save the task, relevant calls, sanitized responses and failed assertion. Convert reproducible documentation problems into a specific edit and rerun the task. Publish execution conditions alongside any measured percentages.