MCP or CLI? Run the same API workflow through both

Suppose an agent needs to find an open test invoice, retrieve its line items and calculate a total. You can give it MCP tools or a CLI that calls the same API. To compare them, keep the invoice data, credentials and expected answer the same.

Give both interfaces the operations the task requires. A CLI with a purpose-built summary command and an MCP server returning raw records are testing different output designs as well as different interfaces.

Control the conditions

Record the agent model, instructions, API version, tool versions and fixture snapshot. State whether the agent begins with knowledge of the commands or has to discover them. Include help output, schema discovery and documentation retrieval when they form part of the real workflow.

Separate cold-start measurements from repeated use. An already running MCP connection and a newly launched CLI process have different setup costs; reversing those conditions can also mislead. Measure the actual process model rather than assuming every CLI starts slowly or every MCP connection remains open.

Tool loading matters. A client that loads every definition up front differs from one that discovers tools on demand. Anthropic's advanced tool-use article describes why discovery and programmatic processing change context consumption. Its published results apply to its stated experiments, not to every MCP or CLI deployment.

Measure complete work

Collect task success, elapsed time, request count, recoverable errors and model token usage when available. Save the sequence of calls. If a command returns a summary while a tool returns full objects, make outputs comparable or identify output design as the source of the difference.

Verify that results include every required page and use the intended account. A lower token count achieved by dropping records is a correctness failure. Repeat trials and report their number.

For latency, distinguish discovery, execution and model reasoning when instrumentation permits. A large network request can dominate both interfaces. Comparing end-to-end time alongside those components keeps an API bottleneck from being mistaken for a protocol property.

Authentication belongs to the implementation

A CLI can support per-user authentication, structured logs and constrained credentials. An MCP deployment still needs those controls to be implemented correctly. Neither a shell command nor a protocol connection automatically establishes tenant isolation.

For a local development agent, consider process isolation, shell access and command composition. For a shared service, consider user identity, credential storage, revocation and auditability. Structured parameters can simplify validation, while command pipelines can make transformations convenient. Both require authorization at the API boundary.

Also document the agent's surrounding permissions. Giving one variant unrestricted shell access and the other a narrow tool list changes the available capabilities, even if both eventually call the same HTTP endpoint.

Record results by workload

Record separate results for local inspection, bulk reports and shared-account operations. State the task fixtures and execution conditions with each result. A result for one task does not establish which interface will suit another.