Conformance
Conformance suite
Each case is { "description", "submission", "expect_error_path"? }. A conforming validator accepts every submission in valid and rejects every submission in invalid with an error whose path starts with expect_error_path.
Valid (6)
| Case | What it checks |
|---|---|
full.json | Every field populated |
mcp-tool-endpoint.json | Endpoint given as an MCP tool name |
minimal.json | Only the required fields |
performance.json | Performance report with a status code |
readme-example.json | The example from the project README |
unicode.json | Non-ASCII text |
Invalid (16)
| Case | What it checks | Error path |
|---|---|---|
agent-unknown-field.json | agent does not allow unknown fields | agent.api_key |
bad-method.json | method must be an HTTP method | method |
bad-outcome.json | outcome must be a known value | outcome |
bad-spec-version.json | spec_version must be 0.1 | spec_version |
empty-goal.json | goal must not be empty | goal |
goal-too-long.json | goal is limited to 1000 characters | goal |
metadata-array.json | metadata must be an object | metadata |
missing-goal.json | goal is required | goal |
missing-message.json | message is required | message |
missing-type.json | type is required | type |
not-an-object.json | the submission must be an object | (root) |
status-code-float.json | evidence.status_code must be an integer | evidence.status_code |
status-code-out-of-range.json | evidence.status_code must be 100-599 | evidence.status_code |
unknown-field.json | unknown top-level fields are rejected | severity |
unknown-type.json | type must be a known value | type |
workaround-string.json | workaround must be a boolean | workaround |