Skip to content

Schemathesis conformance gate #115

Description

@ExtraToast

What to build

A conformance gate that checks the running API actually behaves the way its spec now claims.

The existing export drift gate proves the committed spec matches springdoc output, and oasdiff proves changes are non-breaking — but nothing checks that a malformed body really returns 400, that an unknown id really returns 404, or that no undeclared status codes escape. Once the specs declare their error contract, that becomes checkable.

Read-only operations are fuzzed on pull requests, so the gate is fast and deterministic and never writes. The full suite, including mutations, runs on a schedule and files issues rather than blocking a merge — property-based fuzzing of write paths against a real database is where flakiness comes from, and a flaky gate gets ignored.

Acceptance criteria

  • Each service is fuzzed against its own application, not against a shared stack
  • agents-api write fuzzing never runs against a live cluster
  • No Python toolchain is added to the JVM repositories
  • The gate is shown to fail against a deliberately non-conforming build before it is trusted

Blocked by

  • The three service error-contract trackers

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: toolingReusable workflows, Gradle, templates, Renovate, and API tooling.component: testsUnit, integration, system, or contract test concern.priority: P2Medium; normal planned work.type: testTest coverage, fixtures, or validation changes.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions