How to test if your MCP server is spec compliant

What it actually does

It is not a linter reading your source. It connects to the URL you give it and speaks the protocol, so what gets graded is the behaviour a real client would meet — including anything your framework, proxy or gateway changes on the way out.

Three things are worth knowing before pointing it at production:

• Every probe is read-only. No tool is called and no resource is read. Requests needing a tools/call shape use a reserved sentinel name that cannot collide with one of yours.
• The URL is admitted before anything is fetched. Private ranges, cloud metadata endpoints and localhost are refused, and the connection is pinned to the address that was validated.
• robots.txt is respected. If your host disallows us, the run stops and says so rather than probing anyway.

Give it the MCP endpoint itself, not the marketing page — the path a client would POST to. If you are not sure which that is, it is the URL in your server.json under remotes[].url.

https://mcp.example.com/mcp
https://example.com/api/mcp

A server behind authorization is fine. The validator has no credentials, so it grades the authorization rules — whether you return a 401 with a discoverable metadata document — and skips the ones that need a token.

Reports get a permanent URL. The default is public and indexable, which is what makes the public compliance corpus useful — but it is a real choice, so make it deliberately:

• Public — permalinked and indexable. The default.
• Keep out of search engines — still a shareable URL, carries noindex, never enters the sitemap.
• Only I can see this — the report 404s for everyone else. Needs you to be signed in so the report has an owner; otherwise it falls back to the option above, and the result tells you it did.

Each finding is one rule, and there are four outcomes:

• pass — the rule is satisfied.
• warn — a SHOULD-level requirement is violated. Worth fixing; it will not break a client.
• fail — a MUST-level requirement is violated. A conformant client can legitimately refuse to work with your server.
• skip — the rule could not be evaluated, usually because the server did not answer the request it depends on.

Every rule links to the exact sentence of the specification it comes from, so a finding you disagree with can be checked against the source rather than against our opinion.

How the grade is calculated

The rubric is published and versioned rather than a judgement call. In order — the first matching line wins:

coverage < 70%      ->  unrated
3 or more fails     ->  F
1 or more fails     ->  D
more than 3 warns   ->  C
1 or more warns     ->  B
coverage >= 80%     ->  A
otherwise           ->  B

Two adjustments sit on top, and both exist to avoid misleading you:

• Any security-relevant failure caps the grade at D. A server that is otherwise clean but accepts an unrecognised Origin is not a B.
• A failure detected only by inference improves the grade one step. Where the evidence is heuristic rather than certain, we do not hand out a failing grade on it.

Why unrated is the most common result

Unrated does not mean nothing was graded. Where a whole rule category was covered past the same 70% floor, that category gets its own letter — a server behind authorization returns 401, which stops the protocol groups but leaves the authorization group fully assessable, so it is graded and labelled as what it is. Those scoped grades are not a grade for the server, and the report says so on every one of them.

The floor is not lowered to produce a server grade, and the reason is an incentive one: a server that answers nothing would collect an easy B for having no failures, while an open server exposing all 79 rules and tripping three of them gets an F. That would grade opacity above compliance.

When we probed 16 real public MCP servers, every one came back unrated, at a median coverage of 2.5%. That is not a quirk of the tool; it is the state of the ecosystem two weeks after a breaking revision. If your server returns unrated, the cause is almost always the same one — a session header the revision removed.

Running the same checks in CI

The command line runs the identical ruleset, so a green build and a green report cannot disagree.

npx createmcps validate https://mcp.example.com/mcp

# exit 0  no failures
# exit 1  failures found — stop the build
# exit 2  unreachable, or too little covered to judge
# exit 3  the validator itself could not run

Codes 2 and 3 are deliberately not 1. “Your server is broken”, “we could not reach it” and “our checker fell over” are three different messages, and collapsing them is how a check loses trust the first time a staging box is down. Add --format sarif to get findings as annotations on the pull request that introduced them.

FAQ

Related