Skip to content

feat(protocol): support AdCP 3.1.1 schemas#965

Merged
bokelley merged 4 commits into
mainfrom
upgrade-to-3-1-1
Jul 1, 2026
Merged

feat(protocol): support AdCP 3.1.1 schemas#965
bokelley merged 4 commits into
mainfrom
upgrade-to-3-1-1

Conversation

@bokelley

@bokelley bokelley commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Summary

  • upgrade the SDK schema cache and generated models to AdCP 3.1.1
  • move experimental TMP models/cache entries to the 3.1.1 Trusted Match namespace
  • expose the new Trusted Match collision aliases through the public API snapshot
  • add post-generation validators for Trusted Match HTTPS endpoints, identity dimensions, and tmpx_providers provider IDs
  • update README version support text to 3.1.1

Expert review notes

  • Python packaging/export review found the new Trusted Match files must be committed with the old TMP deletions; included here.
  • Protocol review found no blocking schema-cache issues.
  • Code review flagged Trusted Match runtime validation gaps; covered by tests/test_trusted_match_validation.py and scripts/post_generate_fixes.py.

Validation

  • make lint
  • make typecheck-all
  • make validate-generated
  • git diff --check
  • PYTHONPATH=src pytest tests/test_schemas_version_pin.py tests/test_validation_version.py tests/test_collision_guard.py tests/test_collision_aliases.py tests/test_public_api.py::test_public_api_surface_matches_snapshot tests/test_trusted_match_validation.py tests/test_client.py tests/test_mcp_schema_drift.py tests/test_lazy_types.py -q

Note: make lint-all currently fails on unrelated pre-existing test lint issues outside this change.

aao-ipr-bot[bot]
aao-ipr-bot Bot previously approved these changes Jul 1, 2026

@aao-ipr-bot aao-ipr-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean 3.1.0→3.1.1 regen. The tmptrusted_match rename stays entirely on the internal generated_poc module paths — no public name churns on the wire.

Things I checked

  • No public-surface removals. The rename rewrites only _generated.py and client.py import paths; client.py:303-306 swaps generated_poc.tmp.*generated_poc.trusted_match.*. Public export names (IdentityMatchRequest/Response, ContextMatch*, TmpError) are preserved. Zero - lines in any __all__ or in tests/fixtures/public_api_snapshot.json. feat(protocol) is the right semver signal — this is purely additive.
  • Collision-alias plumbing is consistent across all six surfaces. IdentityMatchTmpxMacro / ProviderRegistrationTmpxMacro land in lockstep in consolidate_exports.py, aliases.py, _generated.py, _eager.py, types/__init__.py, __init__.py, and the snapshot. Freshness and TmpxProviders are correctly scoped to _generated.py only (internal), so they never leak into _eager's named-import namespace or trip the lazy-surface test. code-reviewer: no blockers, no issues.
  • TmpxMacro dual-shape split is the right shape. Two genuinely distinct wire types collide on local class name — IdentityMatchTmpxMacro is the emitted {name, value} object, ProviderRegistrationTmpxMacro is the RootModel[str] registered name. Splitting them into distinct aliases follows the existing ProvenanceDeclaredBy/SiSponsoredContextDeclaredBy convention. tests/test_collision_aliases.py:208-211 asserts the field-shape divergence.
  • Injected validators are structurally sound. fix_trusted_match_runtime_validators string-injects field_validator/model_validator into both TmpProviderRegistration1/2 and IdentityMatchResponse; every referenced field exists, forward-ref return annotations resolve under from __future__ import annotations, the fixer is idempotent (sentinel guards) and fails closed (source = \"\" + if source:). The _require_identity_match_dimensions check exactly mirrors the schema if {identity_match: const true} then {required: [countries, uid_types]}, and the _validate_tmpx_provider_ids regex ^[A-Za-z0-9_]{1,64}$ is equivalent to the schema's ^[A-Za-z0-9_]+$ + min/max. code-reviewer and ad-tech-protocol-expert both verified against the on-branch generated files.
  • No forward-compat regression. No discriminator arm dropped, no additionalProperties narrowed on a published variant, no enum member removed. IdentityMatchResponse deprecates rather than removes tmpx, keeps additionalProperties: true. manifest_schema swaps the tmptrusted-match enum value cleanly.

Follow-ups (non-blocking — file as issues)

  • _require_https_endpoint is stricter than the 3.1.1 wire schema. ad-tech-protocol-expert: sound-with-caveats. The normative endpoint (schemas/cache/3.1/trusted-match/provider-registration.json) is format: uri with prose "MUST be HTTPS in production" — a deployment posture, not a schema invariant. The injected validator (scripts/post_generate_fixes.py, fix_trusted_match_runtime_validators) raises ValueError on any scheme != 'https', so a wire-legal http://localhost / staging registration parses per spec but is rejected by the SDK. tests/test_trusted_match_validation.py::test_provider_registration_rejects_non_https_endpoint bakes it in as intended. Not a block — happy path (production https) is unchanged, and Trusted Match is x-status: experimental — but consider gating the scheme check on a non-local host, or downgrading to a warning, so TMP conformance testbeds aren't locked out.
  • RegistryFeedResponse.freshness is now required. Upstream 3.1.1 added freshness to the schema's required list with Freshness(extra='forbid') — the SDK mirrors it faithfully (not SDK-side narrowing). But feed consumers deserializing pre-3.1.1 payloads (no freshness) will now ValidationError. Worth a CHANGELOG callout so adopters know the read path tightened.

Minor nits (non-blocking)

  1. SCHEMA_DELTAS.md removals read as churn, not loss. The regenerated deltas file drops 106 lines of prior cumulative-baseline content and keeps only the 3.1.0→3.1.1 window. Consistent with the codegen guidance — a reviewer skimming the - block should not read it as dropped types. Interesting choice to regenerate the whole file rather than append, but harmless.

LGTM. Follow-ups noted below.

aao-ipr-bot[bot]
aao-ipr-bot Bot previously approved these changes Jul 1, 2026

@aao-ipr-bot aao-ipr-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean regeneration to AdCP 3.1.1. Right shape: the tmp/trusted_match/ namespace move is internal (generated_poc/), the public adcp.* surface only gains names, and the tmpx_providers shape break is sanctioned under x-status: experimental — so feat: is the correct semver signal, not feat!:.

Things I checked

  • Additive public surface, no breaking removal. New exports IdentityMatchTmpxMacro, ProviderRegistrationTmpxMacro, Freshness, TmpxProviders land consistently across _generated.py, _eager.py, types/__init__.py, adcp/__init__.py, aliases.py, and the snapshot (tests/fixtures/public_api_snapshot.json). No existing name removed or retyped. feat: holds.
  • Regenerated, not hand-edited. ADCP_VERSION 3.1.0→3.1.1, generated timestamps all move to 2026-07-01T04:30:58Z, schema cache $schema/generatedAt bumped in lockstep. The TmpxMacro collision is registered in consolidate_exports.py:100 and disambiguated in aliases.py — the alias layer covers the two distinct wire shapes ({name,value} vs root string).
  • tmp/ fully removed, not orphaned. tmp/identity_match_response.py is deleted (not left behind); the other eight modules are git-renamed into trusted_match/. No stale generated_poc.tmp import survives — client.py:303-306 and _generated.py both repoint.
  • The RootModel access in the new test is safe. TmpProviderRegistration(RootModel[...]) carries the injected __getattr__ proxy (post_generate_fixes.py:834), so registration.provider_id resolves through self.root. test_provider_registration_accepts_valid_identity_registration will pass. (code-reviewer confirmed by reading the proxy; tests not run in-sandbox — see Follow-up.)
  • Injected validators reference real fields. fix_trusted_match_runtime_validators (post_generate_fixes.py:1857-1971) targets endpoint, identity_match, countries, uid_types on both TmpProviderRegistration1/2; anchors are unique, count=1. The self.identity_match is True guard is correct, not a truthy bug — the field is bool | None and Pydantic coerces before the mode='after' check fires.
  • No credential-echo regression. security-reviewer: IdentityMatchResponse is extra='allow', but it's a response shape — nothing routes it into RequestContext.metadata / _build_request_context, and the response-side scrubber is gated to credential-bearing account methods. tmpx* tokens are opaque HPKE pass-through, designed to round-trip. No High.
  • HTTPS validator is fail-closed. _require_https_endpoint checks value.scheme != 'https' on an AnyUrl; Pydantic lowercases the scheme at parse and a schemeless string raises before the validator runs. No case/normalization bypass.

Follow-ups (non-blocking — file as issues)

  • HTTPS validator is stricter than the spec text. ad-tech-protocol-expert: provider_registration.endpoint is format: uri in the source schema with a description of "MUST be HTTPS in production." The injected validator rejects all non-https unconditionally, so http://localhost dev/staging registrations that the "in production" qualifier tolerates won't validate through this SDK. Consider gating on an env/x- flag or downgrading to a warning.
  • tmpx_providers key pattern is a client-side tightening. The _validate_tmpx_provider_ids regex ^[A-Za-z0-9_]{1,64}$ enforces a propertyNames constraint the response schema only states in prose. Semantically justified (mirrors provider_id), but a schema-compliant-but-unusual key would be rejected by this SDK and accepted by the wire. If it's meant to be normative it belongs upstream in the response schema.
  • make lint-all reported red on merge. PR body notes it fails on pre-existing, unrelated test-lint. The core gate (ruff check src/, mypy, the pinned pytest set) is what matters here — confirm CI's core matrix is green before merge, since the validators were verified by reading, not by a sandbox run.

Minor nits (non-blocking)

  1. Un-abbreviated the namespace, kept the abbreviation on the types. tmp/ became trusted_match/, but TmpError, TmpProviderRegistration, and the Tmpx* prefixes ride along unchanged. Consistent with codegen output — noting it, not asking for a rename.

LGTM. Follow-ups noted below.

@aao-ipr-bot aao-ipr-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3.1.0 → 3.1.1 regeneration, additive public surface, tmp/trusted_match/ on the internal generated layer. Approving — the wire-shaped changes track upstream 3.1.1 and the two new exports are consistent everywhere they need to be.

Things I checked

  • Public surface is additive. Two new collision aliases IdentityMatchTmpxMacro / ProviderRegistrationTmpxMacro land consistently across aliases.py (import + __all__), _eager.py, types/__init__.py, adcp/__init__.py (both __all__ blocks + TYPE_CHECKING), and tests/fixtures/public_api_snapshot.json. consolidate_exports.py registers the TmpxMacro collision correctly. feat(protocol): is the right semver signal — nothing stable was removed or narrowed.
  • No discriminated-union regression. identity_match_response.py keeps extra='allow', the type: Literal['identity_match_response'] discriminator, and the deprecated tmpx field (nullable, "Removed in 4.0") alongside the new tmpx_providers: dict[str, TmpxProviders] shape. core/format.py is pure AssetsN → AssetsM renumber churn — the 15-arm union, Field(discriminator='asset_type'), and the Assets94 back-compat alias are intact, just repointed. registry_feed_response.py adds a required freshness but keeps extra='allow'. ad-tech-protocol-expert: sound-with-caveats, no fallback arm or open-union hatch dropped.
  • tmp/ fully removed, imports repointed. client.py now imports generated_poc.trusted_match.*. That direct generated import is sanctioned — client.py is on the layering exemption list (tests/test_import_layering.py:71), so this is the pre-existing pattern, not a new breach.
  • Injected validators are logically sound. _PROVIDER_ID_PATTERN ({1,64}) matches the provider_id max_length=64; the identity_match is True gating and AnyUrl.scheme check reference fields that exist; IdentityMatchResponse is the last class so the appended validator attaches. tests/test_trusted_match_validation.py exercises the real generated output on both happy and failure paths.
  • _generated.py renumbering is codegen churn, exactly the instability CLAUDE.md tells us to discard, not a schema delta.

Follow-ups (non-blocking — file as issues)

  • _require_https_endpoint fail-closes on wire-legal input. post_generate_fixes.py hard-rejects any non-https endpoint, but the 3.1.1 schema types it format: uri and the HTTPS requirement is prose-scoped to production. A schema-legal http://localhost:8080 dev/staging registration now raises ValueError on deserialization. Fail-closed beats fail-open, but a parsing SDK rejecting documents the schema accepts is the wrong cut here — scope it to non-loopback hosts or downgrade to a validation-mode check. (ad-tech-protocol-expert: the one genuine divergence.)
  • tmpx_providers key pattern is stricter than the wire. The response schema has no propertyNames constraint — the ^[A-Za-z0-9_]{1,64}$ check is reconstructed from provider_id. Defensible (provider_id is the only sanctioned key), but it will fail-closed if a future revision namespaces keys. Worth a comment noting the pattern is inferred, not normative.
  • Protocol enum drops the tmp wire value (tmp='tmp'trusted_match='trusted-match' in manifest_schema.py). A manifest declaring protocol: "tmp" will no longer deserialize. Sanctioned — this is the x-status: experimental TMP surface that upstream 3.1.1 itself renamed, so feat under the experimental carve-out is correct. Flagging it so the changelog prose calls out the experimental rename for anyone who prototyped against tmp.

Minor nits (non-blocking)

  1. Unchecked .replace() anchors in fix_trusted_match_runtime_validators. post_generate_fixes.py inserts the provider_registration validators by anchoring on "\n\nclass TmpProviderRegistration2(AdCPBaseModel):" etc. If codegen drifts either anchor, .replace() returns the string unchanged, the file is written, and it prints "added runtime validators" — a silent no-op. test_trusted_match_validation.py catches it in CI, but the failure surfaces in the test suite, not the generator. Assert the content changed (as the import-shape branch already does) so it fails loudly at generation time.
  2. dict\[StringConstraints\((.*?)\), non-greedy truncation. A StringConstraints argument whose pattern literally contains ), would truncate and emit broken Python. No 3.1.1 schema hits it — latent footgun, forward-looking normalization only.

Approving on the strength of the additive public surface plus a clean forward-compat audit. Follow-ups noted.

@aao-ipr-bot aao-ipr-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean 3.1.0→3.1.1 regen with the experimental tmp surface renamed to trusted_match. Additive on the wire — every public export name is stable, the snapshot only gains symbols — so feat(protocol): is the right semver signal.

Things I checked

  • Public surface is additive. tests/fixtures/public_api_snapshot.json gains only IdentityMatchTmpxMacro and ProviderRegistrationTmpxMacro; no name removed. ContextMatchRequest/Response, IdentityMatchRequest/Response keep their class names — only the module path moved tmptrusted_match. Non-breaking; feat: holds.
  • Collision aliases wired through every layer. TmpxMacro has two distinct wire shapes (emitted {name,value} pairs vs. registered-name RootModel[root]). consolidate_exports.py, _generated.py (disambiguated _TmpxMacroFrom*, no bare flat import), aliases.py, _eager.py, types/__init__.py, __init__.py, and the snapshot all agree. test_collision_aliases.py:206 pins the distinct field sets. Right shape.
  • Injected validators match the schema. provider_registration.py https enforcement and the identity_match is Truecountries/uid_types requirement land on both union arms (TmpProviderRegistration1/2); identity_match_response.py:_validate_tmpx_provider_ids iterates dict[str, TmpxProviders] keys against ^[A-Za-z0-9_]{1,64}$, faithfully reproducing provider-registration.json's provider_id min/max. ad-tech-protocol-expert: sound — every wire point matches upstream 3.1.1.
  • Regen, not hand-edit. Timestamps bumped to 2026-07-01, $schema refs to 3.1.1, ADCP_VERSION and README to 3.1.1. Validators are applied by post_generate_fixes.fix_trusted_match_runtime_validators() in the pipeline, consistent with the repo's post-gen architecture — not source edits to generated_poc/.
  • tmp is fully gone. generated_poc/tmp/* and schemas/cache/3.1/tmp/* removed, manifest_schema.Protocol enum tmptrusted-match, client.py import rename complete. No dangling reference.
  • identity_match_response shape change is sanctioned. tmpx_providers went Map<provider_id,string>Map<provider_id,{macros:[TmpxMacro]}> and root tmpx is deprecated (removed in 4.0) — the schema carries x-status: experimental and documents the change inline, citing upstream #5689. Not a stable-contract break.
  • Variant-number de-pin. test_decisioning_capabilities_submodule.py drops the brittle Idempotency3 assertion for an identity + prefix check. Correct per the codegen-numbering-is-unstable rule.

Follow-ups (non-blocking — file as issues)

  • https validator is stricter than the spec prose. provider-registration.json:endpoint is format: uri with "MUST be HTTPS in production"; _require_https_endpoint rejects any non-https scheme unconditionally, so a wire-valid http:// staging endpoint is refused. Defensible SSRF hardening — test_provider_registration_rejects_non_https_endpoint asserts it by design — but it's an SDK constraint beyond the wire shape. Document it in SCHEMA_DELTAS.md so it's an explicit decision, not a silent one.

Minor nits (non-blocking)

  1. Unguarded pattern-insert anchor. scripts/post_generate_fixes.py, identity_match_response branch: the _PROVIDER_ID_PATTERN insertion via .replace("...AdcpVersionEnvelope\n\n\n", ...) is unguarded, but the validator that references it is appended unconditionally. If a future codegen import reorder breaks that exact anchor, the pattern is silently not defined while the validator still lands — NameError at import instead of a clean skip. The sibling pydantic-import replace right above guards with else: return; mirror it. Anchor matches at head, so no active bug.
  2. _hold_unused_bundled_dirs runs before the try. scripts/generate_types.py: the hold shutil.move sits outside the try/finally, so a raise mid-hold strands bundled/ in ..._bundled_held with no restore, and the next run's rmtree(held) deletes it. Dev-time only (input reconstitutes from schemas/cache), but folding the hold into the try pairs it with the restore.

Approving on the strength of the additive public surface plus the faithful validator reproduction — two expert passes, no blocker. Follow-ups noted.

@bokelley bokelley merged commit 22ae047 into main Jul 1, 2026
40 of 41 checks passed
@bokelley bokelley deleted the upgrade-to-3-1-1 branch July 1, 2026 08:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant