Octri

mcp test

Has real models work through your project's sample tasks using your MCP tools, and prints each model's pass rate. It's a dry run: every tool call is answered from your spec's examples, and your API is never called. It spends AI credits, and transcripts are in MCP Studio.

Without --fail-under, it exits with an error when any model fails any task.

bash
octri mcp test [--models <models>] [--repeat <n>] [--fail-under <rate>]

Flags

--models <models>list

Which models to run.

--repeat <n>number

How many times to run each task.

--task <ids>list

Only these tasks.

--tasks <file>path

Replaces your saved sample tasks with the file's first.

--draft <file>path

Tests an unpublished draft of your tools instead of what's live.

--fail-under <rate>number

Exits with an error when a model's pass rate is below this, such as 0.9.

--no-waitboolean

Queues the run and returns.

Examples

Gate a release on pass rate

bash
octri mcp test --repeat 3 --fail-under 0.9