Quality and test runs
Tool names, descriptions and schemas are read by models, so MCP Studio checks them the way a model will, before anything reaches an agent. Every change here is staged with your other MCP edits, and goes live when you publish.
There are two checks. The quality score is instant and reads your definitions. A test run is slower and costs credits, but shows you how real models actually behave.
Quality score
Tool quality scores each tool out of 10, and the server as a whole. It reads what an agent is served: the listing after your customizations, including chained tools and approval notes. As you edit, it shows the score your staged changes would get once published.
What it checks
| Area | Check | Weight |
|---|---|---|
| Name | Uses only letters, digits, _ and - | 3 |
| Is distinct from every other tool | 3 | |
| Fits after the client's prefix | 2 | |
| Starts with a verb | 2 | |
| Description | Is written | 4 |
| Says more than the name | 3 | |
| Fits the length budget | 2 | |
| Isn't just the method and path | 2 | |
| Says when to use it | 2 | |
| Leads with its first sentence | 1 | |
| Is plain text | 1 | |
| Inputs | Are described | 4 |
| Are typed | 2 | |
| Mark which are required | 2 | |
| Stay shallow | 1 | |
| Behavior | Annotations agree with the description | 1 |
| Size | The tool's definition is compact | 1 |
| Whole server | Names follow one convention | 1 |
| Tools are easy to tell apart | 1 | |
| The tool count suits the listing mode | 1 | |
| The whole listing fits in context | 1 |
A server's score favors its weakest tools as well as its average, so one very poor tool shows up in the number.
Fixing what it finds
Low scorers come with a suggested fix. Accept one and it's staged as an override on that tool, so the endpoint's docs stay as they are unless you also draft the change in API Studio. You can also customize a tool by hand.
In CI
octri mcp score --fail-under 7 fails a CI job when the score drops below 7. If a finding doesn't apply to your API, octri mcp score ignore <key> dismisses it, using the key that octri mcp score --tool <name> prints. Only the score changes, and restore brings the finding back.
Test runs
Agent tests give real models a set of sample tasks, and let them work through your draft tools. You see where an agent picks the wrong tool or sends the wrong arguments before anyone connects.
- Write the tasks
Write your own, or start from the suggested ones. A good task reads like something a person would really ask.
- Pick the models
Choose under Models to test: Claude Opus, Sonnet and Haiku, and GPT. Repeat each task to catch flaky behavior.
- Run it
Each attempt is listed with the calls in order and how it ended, under Transcripts. Each model gets a verdict of pass, flaky or fail.
If you edit after a run, it's marked Edited since this run, so you know the result is out of date. Publishing with failing tasks asks you to confirm first.
Dry by default
Every request is built and checked, and none is sent to your API. On Business and Enterprise, a run can call your real API, if you point it at a test environment.
What you can run
| Plan | Models | Tasks per run | Repeats |
|---|---|---|---|
| Free | Haiku and GPT only | 10 | 1 |
| Starter | All | 50 | 5 |
| Growth | All | 100 | 5 |
| Business | All | 50 | 5 |
| Enterprise | All | 100 | 5 |
Running a test needs edit access to SDK Studio.
What it costs
Test runs spend AI credits, charged on what the models actually use. One run can hold at most your plan's monthly credit allowance. Cancelling a run keeps what finished.
In CI
octri mcp test --fail-under 0.9 fails a CI job when a model's pass rate drops below 90%. It can also test a draft, or replace your sample tasks with a file.
Which to use
| Question | Use |
|---|---|
| Are my tools well described? | Quality score. It's instant and free |
| Will agents actually pick the right tool? | A test run |
| Did this change make things worse? | Both, in CI |
After you publish
What an agent is told when a call fails, when your API is down, or when your tools have changed has its own page: errors, outages and changes.