diff --git a/.agent-loop/LOOP_STATE.md b/.agent-loop/LOOP_STATE.md index 4cf2f833..36b00488 100644 --- a/.agent-loop/LOOP_STATE.md +++ b/.agent-loop/LOOP_STATE.md @@ -4,20 +4,26 @@ - Active initiative: `WS-AUTH-001` - Workstream Authorization Service - Active planning chunk: none -- Active implementation chunk: none -- Branch: `codex/ws-auth-001-04b-post-merge-memory` -- Worktree: `/home/abiorh/flow/workstream-authorization-service` -- Status: AUTH-04B merged through PR #113 as `05a63c8` after Backend, Agent - Gates, CodeRabbit, required internal review, and explicit human approval - passed. +- Active implementation chunk: `WS-AUTH-001-05A` - Shared Audit Ownership And + Append-Only Authority Evidence +- Branch: `codex/ws-auth-001-05-authority-evidence` +- Worktree: `/home/abiorh/flow/workstream-auth-001-05` +- Status: AUTH-04B post-merge memory merged through PR #114 as `97cd0f5`. + The user explicitly started AUTH-05. Required L1 review rejected the combined + contract, then passed the first repaired AUTH-05A contract at `7a9023b`. + Exact-SHA implementation review reopened the contract for closed privacy + registries, non-echoing mapping admission, typed/SQL parity, and readable SQL. - Prior `WS-AUTH-001-01` reviewed implementation SHA: `be0b836` - Prior `WS-AUTH-001-01` final merged branch head: `b5217e1` -- Latest integrated `main` merge commit: `05a63c8` -- Current gate: publish and merge the AUTH-04B post-merge memory update, then - stop. -- Size checkpoint: implementation is 480 changed non-comment production lines; - the required 350-line inspection passed and the 500-line hard stop holds. -- Next chunk: none; do not start `WS-AUTH-001-05` automatically. +- Latest integrated `main` merge commit: `97cd0f5` +- Current gate: AUTH-05A runtime repair and focused evidence under the + human-approved semantic chunk boundary. +- Scope checkpoint: allowed files, non-goals, acceptance criteria, behavior + evidence, and required review bound AUTH-05A; there is no numeric line cap. + AUTH-05B remains a separate inactive chunk. +- Next chunk: AUTH-05B remains inactive until AUTH-05A merges, post-merge memory + is reconciled, and the user gives a separate explicit start. AUTH-06 remains + inactive. - Focused evidence: 77 isolated rate/config tests pass at 97 percent subsystem coverage, 59 final config/object-graph tests pass, and the exact migration proof and real API E2E pass. Final GitHub Backend passed 937 tests at 82.15 diff --git a/.agent-loop/REVIEW_LOG.md b/.agent-loop/REVIEW_LOG.md index 24a40adc..6265331c 100644 --- a/.agent-loop/REVIEW_LOG.md +++ b/.agent-loop/REVIEW_LOG.md @@ -1,5 +1,119 @@ # Review Log +## 2026-07-14 - WS-AUTH-001-05A CodeRabbit Response + +CodeRabbit run `30676bdc-7525-4deb-8f9e-a87d42c64f92` produced four actionable +comments on PR #115. The response makes the inactive AUTH-05B verification +commands executable, replaces fragile positional event mapping with explicit +identity-link event keys, types the task audit fixture session, and replaces +misleading reused-agent names with AUTH-05A-specific review references. Focused +behavior, Ruff, stale authorization docs, Markdown links, and diff integrity +pass. Exact-SHA internal re-review is required before publication of the repair. + +## 2026-07-14 - WS-AUTH-001-05A Semantic Scope Amendment + +The human replaced AUTH-05A's numeric production-line ceiling with the approved +chunk contract as the controlling boundary. Allowed files, explicit non-goals, +acceptance criteria, behavior evidence, and required internal review now govern +scope. Readable SQL remains mandatory; AUTH-05B, route activation, permission +evaluation, and grant lifecycle implementation remain inactive. + +## 2026-07-14 - WS-AUTH-001-05A Circuit Breaker Amendment Passed + +The human-approved AUTH-05A-only 500-line inspection and 1000-line hard stop +passed all required exact-SHA reviewers at `611abfc`. Review confirmed the typed +and PostgreSQL privacy matrices must remain atomic, the incomplete 689-line +draft was discarded, SQL packing remains prohibited, focused behavior and 90 +percent subsystem coverage gates remain, and AUTH-05B stays inactive. Bounded +runtime repair may resume. + +## 2026-07-14 - WS-AUTH-001-05A Circuit Breaker Reopened + +The first post-approval exact-registry dry run reached 689 changed non-comment +production lines before readable PostgreSQL parity was complete, crossing the +650 hard stop. The uncommitted runtime draft was discarded and the tree +returned to the 484-line approved implementation checkpoint. Splitting typed +and database enforcement into separately mergeable PRs would leave an +incomplete security boundary, so the user approved a 1000-line hard stop +with the existing 500-line inspection, 120-character migration line gate, and +no SQL packing. Runtime work remains paused for exact-SHA amendment review. + +## 2026-07-14 - WS-AUTH-001-05A Repaired Contract Review Passed + +The exact contract at `7cc6058` passed senior engineering, architecture, +product/ops, docs, reuse, QA, CI/test-delta, and security/privacy review. It +locks closed envelope/reason/fact registries, cross-field project integrity, +non-echoing Mapping/JSON admission, identical typed/direct-SQL behavior, +readable SQL, focused regressions, and the then-approved 500/650 +circuit breaker. Bounded runtime repair may proceed; AUTH-05B remains inactive. + +## 2026-07-14 - WS-AUTH-001-05A Repaired Contract Review Failed + +Required review of contract `c93cf14` rejected implicit security registries. +Senior engineering, QA, and security agreed that exact event-to-reason, +event-to-fact, permission, denial, and reference matrices must be normative; +lexical bounds and "spec-derived" values were insufficient. Review also +required sanitized malformed-JSON handling, the two known checker regression +nodes, deterministic size/line checks, and symmetric additive test-delta +checks. The contract was repaired without runtime edits and requires a new +exact-SHA review. + +Review of `2cd0fbe` then caught contradictory UUID rules for permission targets, +resource-ID nullability, admin role/scope compatibility, optional-versus-required +reason wording, and state-versus-decision fact meaning. It also required exact +added-plus-deleted size arithmetic and reuse of the repository's AST-aware test +weakening detector. Those contract defects were repaired before runtime work. +QA then found the AST command below the repository-root directory change; the +command was moved into its executable backend-relative position before final +contract review. +Senior review also required project-scoped grant facts to match the envelope +project, replacement facts to retain one scope, system-scope grant evidence to +omit project scope, and project resources to agree with the envelope project. + +## 2026-07-14 - WS-AUTH-001-05A Contract Reopened By Exact-SHA Review + +Exact-SHA review of implementation repair `6fbb1f8` closed append-only, +repository ownership, canonical actor-reference, and invalidation-cause +integrity concerns, but found valid remaining privacy and auditability gaps. +Arbitrary reason/fact strings could still impersonate opaque secrets, rejected +known values and non-dict mappings remained recoverable through structured +Pydantic errors, direct SQL did not enforce every typed token bound, and the +500-line ceiling had encouraged unreadable security SQL. + +The repaired contract raises only AUTH-05A's ceiling to 650, prohibits +long-line packing, requires closed reason/fact registries, requires a +non-echoing pre-admission boundary for every Mapping, clarifies invalidation +cause integrity versus inactive idempotency replay, and requires typed/direct +SQL parity. No further runtime repair is permitted until required L1 contract +review passes; AUTH-05B remains inactive. + +## 2026-07-14 - WS-AUTH-001-05A Preimplementation Review Passed + +The repaired AUTH-05A contract passed required senior engineering, +QA/CI/test-delta, security/privacy, product/ops, architecture, docs, and reuse +review at `7a9023b`. The contract fixes migration ownership at `0018`, retains +the existing `audit_events` ledger and `AuditRepository` as sole persistence +owner, defines exact legacy/authority compatibility and bounded event shapes, +and requires unconditional normal-DML append-only triggers plus downgrade and +production-role custody proof. Bounded implementation and deterministic +evidence are active; AUTH-05B remains inactive. + +## 2026-07-14 - WS-AUTH-001-05 Plan Review Split Required + +After AUTH-04B post-merge memory merged through PR #114 as `97cd0f5`, the user +explicitly started AUTH-05. Required L1 engineering, QA/CI, and +security/privacy/product plan review rejected the combined contract before any +runtime edit. Migration `0017` is already owned by AUTH-04B, dedicated audit and +authorization tests were absent, and shared audit custody plus idempotency and +invalidation were not credibly reviewable as one sub-500-line production diff. + +The repaired plan splits AUTH-05 into 05A shared audit ownership/append-only +authority evidence on migration `0018`, then 05B idempotency/invalidation on +`0019`. Both retain caller-owned transactions, focused 90 percent subsystem +coverage, the global 78 percent CI floor, full reviewer fanout, and separate +human merge/start gates. No runtime implementation is active pending repaired +contract review. + ## 2026-07-14 - WS-AUTH-001-04B Merged PR #113 merged `WS-AUTH-001-04B` to `main` as `05a63c8` after final branch head diff --git a/.agent-loop/WORK_QUEUE.md b/.agent-loop/WORK_QUEUE.md index 740ca009..3f45a464 100644 --- a/.agent-loop/WORK_QUEUE.md +++ b/.agent-loop/WORK_QUEUE.md @@ -4,7 +4,7 @@ | Chunk | Title | Risk | Status | |---|---|---:|---| -| - | No active implementation chunk | - | AUTH-04B post-merge memory in progress; no product implementation active | +| `WS-AUTH-001-05A` | Shared Audit Ownership And Append-Only Authority Evidence | L1 | Human-approved semantic chunk boundary; runtime repair/evidence | ## Planned Next @@ -57,9 +57,10 @@ ## Proposed Next -`WS-AUTH-001-04B` merged through PR #113 as `05a63c8` after all required checks -and reviews passed. Its post-merge memory update is active; no AUTH product -implementation is active. Do not start AUTH-05 or POL-002-04 automatically. +AUTH-04B post-merge memory merged through PR #114 as `97cd0f5`. The user +explicitly started AUTH-05. Required plan review rejected the combined contract +before runtime edits and required children 05A/05B. Only 05A contract repair is +active. Do not implement 05B, AUTH-06, or POL-002-04 automatically. Coverage R10 merged through PR #108. Do not start 01B2, chunk 02, or another coverage implementation chunk from this worktree. diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/CHUNK_MAP.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/CHUNK_MAP.md index 5f15136e..efd18cba 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/CHUNK_MAP.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/CHUNK_MAP.md @@ -18,7 +18,9 @@ stopped. | `WS-AUTH-001-04` | Request, Error, And API Control Foundation | L1 | Split before implementation into 04A and 04B | | `WS-AUTH-001-04A` | Request And Error Context | L1 | Merged through PR #111 as `90c9a28` | | `WS-AUTH-001-04B` | PostgreSQL Rate Controls | L1 | Merged through PR #113 as `05a63c8` | -| `WS-AUTH-001-05` | Authority Evidence And Idempotency Foundation | L1 | Proposed | +| `WS-AUTH-001-05` | Authority Evidence And Idempotency Foundation | L1 | Split before implementation into 05A and 05B | +| `WS-AUTH-001-05A` | Shared Audit Ownership And Append-Only Authority Evidence | L1 | Semantic chunk boundary approved; runtime repair/evidence | +| `WS-AUTH-001-05B` | Authority Idempotency And Invalidation Foundation | L1 | Inactive pending 05A merge/memory and separate explicit start | | `WS-AUTH-001-06` | Canonical Actor Profile And Identity Link | L1 | Proposed | | `WS-AUTH-001-07` | Authorization Kernel And Permission Registry | L1 | Proposed | | `WS-AUTH-001-08` | Bootstrap And Administrative Role Grants | L1 | Proposed | @@ -40,7 +42,8 @@ WS-AUTH-001-PLAN -> WS-AUTH-001-03 -> WS-AUTH-001-04A -> WS-AUTH-001-04B --> WS-AUTH-001-05 +-> WS-AUTH-001-05A +-> WS-AUTH-001-05B -> WS-AUTH-001-06 -> WS-AUTH-001-07 -> WS-AUTH-001-08 @@ -61,7 +64,9 @@ WS-AUTH-001-PLAN - Parent chunk 04 was split before implementation. Chunk 04A establishes request/correlation and additive error compatibility; chunk 04B later owns durable PostgreSQL rate controls and its migration. -- Chunk 05 evolves the shared audit path and idempotency before mutations. +- Parent chunk 05 was split before implementation. Chunk 05A owns shared audit + schema/writer custody and append-only authority evidence; chunk 05B owns + idempotency and typed invalidation orchestration. - Chunk 06 establishes canonical actor resolution while preserving only the enumerated non-authoritative legacy workflow-eligibility consumers required for intermediate-release operability. @@ -89,5 +94,11 @@ post-merge memory merged through PR #112 as `7749f54`. The user explicitly started AUTH-04B. Its repaired contract passed at `b5dceb1`; bounded implementation and all required internal review tracks passed, and PR #113 merged as `05a63c8` after Backend, Agent Gates, CodeRabbit, and explicit human -approval passed. AUTH-04B post-merge memory is the current gate. Stop; do not -start AUTH-05 or POL-002-04. +approval passed. AUTH-04B post-merge memory then merged through PR #114 as +`97cd0f5`, and the user explicitly started AUTH-05. Required plan review +rejected the combined contract before runtime edits and required 05A/05B. +The first 05A implementation review proved the original numeric ceiling +incompatible with readable typed/database privacy parity. Repaired 05A contract +review passed at `7cc6058`; the user subsequently replaced the line cap with +the semantic AUTH-05A boundary. Runtime repair is active. 05B remains inactive. +Do not start AUTH-06 or POL-002-04. diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/DISCOVERY.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/DISCOVERY.md index b4454949..99e56f08 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/DISCOVERY.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/DISCOVERY.md @@ -157,6 +157,24 @@ Production deployment inputs remain externally supplied: - Fixed internal system actors for server-owned automation. - One PR-sized chunk at a time with deterministic evidence and internal review. +## AUTH-05 Delta Discovery (2026-07-14) + +- Current migration head is `0017_api_controls`; the original AUTH-05 + `0017_*` allowance collides and cannot be implemented. +- `AuditEvent` remains in `tasks.models`. `AuditRepository` owns a shared insert + implementation, while `TaskRepository` still duplicates insert/read logic + used by task/checker services. +- Legacy audit rows require non-null external issuer/subject, token-role, and + claim fields. Authority events need a conditional privacy-safe envelope that + preserves legacy rows without copying those identity-provider fields. +- The combined shared-audit schema/writer, append-only database custody, + idempotency state machine, and concurrency proof cross two persistent + subsystem boundaries. Required L1 plan review split them into 05A (`0018`) + and 05B (`0019`) before runtime implementation. +- D13 provider-neutral verifier adoption does not block either child and remains + a separate reviewed chunk after the shared external-service adapter + foundation exists. + ## AUTH-04 Delta Discovery (2026-07-13) ### Current behavior diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/PLAN.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/PLAN.md index 6a8ab1a1..625a27bb 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/PLAN.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/PLAN.md @@ -130,7 +130,8 @@ proving the same token role alone no longer authorizes. evidence workflow. 4. Establish request/correlation context, structured errors, and API rate controls. -5. Evolve shared audit evidence and add canonical idempotency/invalidation. +5. Evolve shared audit evidence in 05A, then add canonical + idempotency/invalidation in 05B. 6. Migrate to canonical profile/link semantics and first-human resolution. 7. Implement the minimal registered permission and AuthorizationService kernel before protected authority-management APIs. diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/STATUS.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/STATUS.md index 9b959c54..3833bb5d 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/STATUS.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/STATUS.md @@ -43,7 +43,16 @@ internally approved at final SHA `922778b`; reviewed production SHA is `67484b5`. The Backend CI isolation repair passed exact review at `4fb846c`. Final branch head `94fb2fe` passed Backend, Agent Gates, and CodeRabbit, then explicit human approval merged PR #113 to `main` as `05a63c8` on 2026-07-14. -AUTH-04B post-merge memory is active; AUTH-05 remains inactive. +AUTH-04B post-merge memory merged through PR #114 as `97cd0f5`. The user then +explicitly started AUTH-05. Required L1 plan review rejected the combined +audit/idempotency contract before runtime edits and required children 05A and +05B. The first repaired AUTH-05A contract passed at `7a9023b`, but exact-SHA +implementation review demonstrated that closed privacy registries and readable +typed/SQL parity require a larger readable contract. The repaired contract +passed at `7cc6058`, but numeric hard stops repeatedly interrupted the atomic +typed/database parity boundary. The human replaced the line cap with AUTH-05A's +semantic scope, acceptance criteria, behavior evidence, and review gates; +runtime repair and focused evidence are active. ## Active planning chunk @@ -51,12 +60,12 @@ None. ## Active implementation chunk -None. +`WS-AUTH-001-05A` - Shared Audit Ownership And Append-Only Authority Evidence. ## Current implementation branch -None. Post-merge memory uses `codex/ws-auth-001-04b-post-merge-memory` and makes -no product implementation change. +`codex/ws-auth-001-05-authority-evidence` in +`/home/abiorh/flow/workstream-auth-001-05`. ## Chunk status @@ -69,7 +78,9 @@ no product implementation change. | `WS-AUTH-001-04` | Split | `codex/ws-auth-001-04-request-api-controls` | - | Parent split before runtime implementation. | | `WS-AUTH-001-04A` | Merged | `codex/ws-auth-001-04-request-api-controls` | #111 | Merged as `90c9a28`; production review `cdcaf77`; final branch head `36c4aa5`. | | `WS-AUTH-001-04B` | Merged | `codex/ws-auth-001-04b-postgres-rate-controls` | #113 | Merged as `05a63c8`; production review `67484b5`; final branch head `94fb2fe`. | -| `WS-AUTH-001-05` | Proposed | - | - | Authority evidence and idempotency foundation. | +| `WS-AUTH-001-05` | Split | `codex/ws-auth-001-05-authority-evidence` | - | Parent split before implementation into 05A and 05B. | +| `WS-AUTH-001-05A` | Runtime repair/evidence | `codex/ws-auth-001-05-authority-evidence` | - | Human-approved semantic chunk boundary; repair active. | +| `WS-AUTH-001-05B` | Inactive | - | - | Idempotency and invalidation; requires 05A merge/memory and separate explicit start. | | `WS-AUTH-001-06` | Proposed | - | - | Canonical actor profile and identity link. | | `WS-AUTH-001-07` | Proposed | - | - | Authorization kernel and permissions. | | `WS-AUTH-001-08` | Proposed | - | - | Bootstrap and administrative grants. | @@ -84,10 +95,15 @@ no product implementation change. ## Blockers -No active implementation blocker because no AUTH product chunk is active. -AUTH-05 may start only after this memory update merges and the user gives a -separate explicit start signal. AUTH-04B owns migration `0017`, following the -now-owned `0016` prefix on current main. Non-test +AUTH-05A has no preimplementation blocker under its human-approved semantic +chunk boundary. The combined AUTH-05 contract was +rejected before code changes because migration `0017` is already owned by +AUTH-04B and the shared-audit plus idempotency scope was not reviewable as one +L1 change. Repaired AUTH-05A owns migration `0018`; AUTH-05B later owns +migration `0019`. Exact-SHA review found valid arbitrary fact/reason privacy, +non-dict Mapping retention, DB token-parity, cause-contract ambiguity, and SQL +auditability findings; the repaired contract owns their closure without +reactivating 05B. Non-test operators must later supply explicit classification evidence rather than inferred kinds before the owning canonical actor migration. diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-05-authority-evidence-foundation.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-05-authority-evidence-foundation.md index e6547f67..12abd377 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-05-authority-evidence-foundation.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-05-authority-evidence-foundation.md @@ -1,125 +1,40 @@ # Chunk Contract: WS-AUTH-001-05 - Authority Evidence And Idempotency Foundation +## Status + +Split before runtime implementation after required L1 plan review. No parent +chunk implementation is permitted. + ## Parent initiative `WS-AUTH-001` - Workstream Authorization Service ## Goal -Evolve the existing `audit_events`/`AuditRepository` path with canonical -mutation idempotency and authority invalidation evidence before actor or grant -mutations use it. - -## Why this chunk exists - -Authority-changing APIs cannot ship with temporary provenance that a later -chunk would need to reconstruct. - -## Approved plan reference - -- INTENT: `.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/INTENT.md` -- PLAN: `.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/PLAN.md` -- CHUNK_MAP: `.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/CHUNK_MAP.md` - -## Risk class - -L1 - -## SLA - -P1 - -## Allowed files - -```text -backend/app/modules/audit/** -backend/app/modules/authorization/** -backend/app/db/models.py -backend/app/modules/tasks/models.py -backend/app/modules/tasks/repository.py -backend/app/modules/tasks/service.py -backend/alembic/versions/0017_*.py -backend/tests/test_auth.py -backend/tests/test_alembic.py -backend/tests/test_tasks.py -backend/scripts/api_contract_e2e.py -docs/operations_authorization_service.md -.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/** -.agent-loop/LOOP_STATE.md -.agent-loop/WORK_QUEUE.md -.agent-loop/REVIEW_LOG.md -``` - -## Not allowed - -```text -actor/profile migration or first-access behavior -permission/grant evaluation -admin/project grants -project/task/checker authorization cutover -backfilling fabricated authority events -``` - -## Acceptance criteria - -- Idempotency rows store canonical request hash, operation, actor reference, - status, and bounded committed response reference with unique replay guards. -- Authority events have final versioned fields for acting/target references, - matched grant, project, reason, idempotency key, bounded before/after state, - and database time. -- Existing `AuditEvent` is extended; no parallel AuthorityEvent table/model is - created. `TaskRepository.add_audit_event` delegates to or is consolidated - with `AuditRepository` so one shared writer owns persistence. -- Audit persistence exposes insert/read operations only through supported - application paths. Migration-level enforcement rejects authority-event - update/delete by the application database role; focused tests prove attempts - fail without changing prior rows. The runbook defines retention and the - separately controlled privileged database-maintenance path. -- Authorization-owned idempotency models, repository queries, and service - orchestration remain separate feature modules. Middleware, generic core - helpers, and ORM models do not own replay queries or transaction policy. -- The injected AsyncSession remains the UnitOfWork boundary. Caller services - own commit/rollback; no generic parallel UoW wrapper is introduced. -- Session-bound repository APIs let callers atomically persist business state, - idempotency result, authority event, and invalidation event. -- Migration owns idempotency/audit constraints, indexes, UTC/database-time - defaults, prior-head upgrade, downgrade, and re-upgrade. -- Each later authority-mutation chunk must test its specified successful and - denied audit events atomically when that behavior is introduced; chunk 16 - cannot backfill an omitted event. +Establish durable authority evidence before actor or grant mutations use it. -## Verification commands +## Split result -```bash -(cd backend && WORKSTREAM_DATABASE_URL= .venv/bin/alembic upgrade head) -(cd backend && WORKSTREAM_DATABASE_URL= .venv/bin/alembic downgrade -1) -(cd backend && WORKSTREAM_DATABASE_URL= .venv/bin/alembic upgrade head) -(cd backend && .venv/bin/python -m ruff check app tests) -(cd backend && WORKSTREAM_DATABASE_URL= .venv/bin/python -m pytest -q) -(cd backend && WORKSTREAM_DATABASE_URL= .venv/bin/python scripts/api_contract_e2e.py) -python3 scripts/check_stale_workstream_wording.py -python3 scripts/check_markdown_links.py -git diff --check -``` +The original contract combined two persistent subsystem boundaries and was not +credibly reviewable below the L1 size ceiling. Implementation is owned by: -## Required reviewers +- `WS-AUTH-001-05A` - Shared Audit Ownership And Append-Only Authority Evidence +- `WS-AUTH-001-05B` - Authority Idempotency And Invalidation Foundation -- senior engineering -- QA/test -- security/auth -- product/ops -- architecture -- CI integrity -- docs -- reuse/dedup -- test delta +AUTH-05A must merge and complete post-merge memory before AUTH-05B starts. +AUTH-06 must not start until both children merge and their memory is current. -## Human review focus +## Preserved boundaries -Review reuse of AuditEvent/AuditRepository, idempotency uniqueness/replay, -AsyncSession transaction ownership, and unchanged product authorization. +- Existing `AuditEvent` evolves in place; no parallel authority-event table. +- `AuditRepository` becomes the only supported insert/read implementation. +- Authorization owns idempotency and invalidation orchestration. +- The injected `AsyncSession` remains the unit-of-work boundary. +- No route, permission, grant, actor migration, or product cutover is activated. +- D13 provider-neutral verifier adoption remains a separate reviewed chunk + after the shared external-service adapter foundation exists. -## Stop conditions +## Stop condition -Stop if final evidence requires a canonical ActorProfile FK before migration; -use stable actor-reference fields rather than introducing a parallel actor. +Stop at this split record. Implement only one reviewed child contract at a +time. diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-05A-shared-audit-authority-evidence.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-05A-shared-audit-authority-evidence.md new file mode 100644 index 00000000..9f7ec94e --- /dev/null +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-05A-shared-audit-authority-evidence.md @@ -0,0 +1,394 @@ +# Chunk Contract: WS-AUTH-001-05A - Shared Audit Ownership And Append-Only Authority Evidence + +## Status + +Runtime repair and focused evidence. Repaired contract review passed on +`7cc6058`; the earlier numeric amendments passed at `611abfc`, and the human +subsequently replaced the line ceiling with this chunk's semantic scope, +acceptance criteria, behavior evidence, and required-review gates. + +## Parent initiative + +`WS-AUTH-001` - Workstream Authorization Service + +## Goal + +Evolve the existing `audit_events` path into one privacy-safe, append-only, +versioned authority-evidence envelope while preserving every legacy lifecycle +event and existing task/checker behavior. + +## Risk and circuit breaker + +- Risk: L1 / SLA P1. +- Scope is bounded by this chunk's allowed files, explicit non-goals, acceptance + criteria, and AUTH-05A ownership boundary rather than a production-line cap. +- Stop and replan if implementation crosses into AUTH-05B, authorization + evaluation, grant lifecycle, route activation, or another subsystem boundary. +- Security SQL must remain reviewable as named functions and multiline logical + clauses; line packing is prohibited. +- The scope amendment is limited to AUTH-05A's required privacy, custody, + registry, causation, and parity controls; it does not reactivate AUTH-05B. +- A readable typed-registry dry run reached 689 changed lines before the SQL + matrix was complete. The draft was discarded. Typed and database enforcement + are one atomic security boundary, so they are not split across mergeable PRs. + +## Allowed files + +```text +backend/app/modules/audit/** +backend/app/db/models.py +backend/app/modules/tasks/models.py +backend/app/modules/tasks/repository.py +backend/alembic/versions/0018_authority_audit_evidence.py +backend/tests/test_audit.py +backend/tests/test_alembic.py +backend/tests/test_tasks.py +docs/architecture_data_model.md +docs/operations_authorization_service.md +.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/** +.agent-loop/LOOP_STATE.md +.agent-loop/WORK_QUEUE.md +.agent-loop/REVIEW_LOG.md +``` + +`backend/tests/test_tasks.py` may receive only bounded delegation/regression +proof; the existing full task suite remains an unchanged regression target. + +## Not allowed + +```text +authorization idempotency records or replay orchestration +routes, dependencies, middleware, or generic core helpers +actor/profile migration or first-access behavior +permission definitions/evaluation or admin/project grants +project/task/submission/checker authorization cutover +invalidation consumers, caches, queues, or external delivery +token-role or IdentityIssuerVerifier/factory changes +workflow, dependency, coverage-threshold, skip, or exclusion changes +backfilled or fabricated authority history +repository or service commit/rollback/session creation +``` + +## Authority event envelope + +Migration `0018_authority_audit_evidence` revises `0017_api_controls` and +extends the existing `audit_events` table. It does not create a parallel event +table. Existing rows and legacy writers retain their current values and public +behavior; no legacy row is reclassified as authority evidence. + +New rows use `event_domain = authority`, `event_version = 1`, and a typed event +token from the adopted specification. Authority rows require: + +- database-owned `occurred_at`; +- `request_id` and `correlation_id` UUIDs; +- namespaced acting reference kind (`legacy_actor`, `actor_profile`, or + `system_principal`) plus the exact local reference defined below; +- optional target actor reference with all-or-none kind/reference presence; +- optional matched-grant, permission, project, resource type/id, and target + grant/link reference; +- required authority reason classification and optional denial code; +- optional idempotency-record reference; +- optional invalidation cause event and invalidation target kind/id; +- typed shallow before/after fact objects. + +Authority rows never persist external issuer, external subject, token roles, +claim snapshots, request bodies, raw tokens, JWKS documents, secrets, emails, +or URLs. Conditional database constraints reject mixed legacy/authority +shapes using this exact compatibility matrix: + +- existing `id` remains the event ID, `event_type` remains the typed event + token, and `created_at` remains the legacy timestamp; +- existing `actor_id` stores the stable local acting reference and the new + actor-reference-kind field supplies its namespace; +- existing `entity_type`/`entity_id` remain the primary audited aggregate + reference; optional project/resource fields provide additional exact scope; +- existing `reason` may carry only a registered authority reason code; +- `from_status` and `to_status` are `NULL` because typed before/after facts own + authority state transitions; +- `external_subject` and `external_issuer` are `NULL`; +- `actor_roles = '[]'`, `claim_snapshot = '{}'`, + `auth_source = 'local_authority'`, and `is_dev_auth = false`; +- `event_payload = '{}'`; authority evidence cannot use the legacy JSON payload + as an unbounded escape hatch. + +Only the external subject/issuer columns become nullable for the conditional +authority shape. Existing legacy rows and writers retain all current values +and non-authority requirements. + +Every authority string/reference field has one normative form: + +| Field | Allowed value | +|---|---| +| `entity_type` | `actor_profile`, `actor_identity_link`, `admin_role_grant`, `qualification_snapshot`, `project_role_grant`, `authorization_decision`, or `authority_invalidation` | +| `entity_id`, `matched_grant_id`, `project_id`, `resource_id` | canonical lowercase UUID; nullable optional fields remain nullable; decision/invalidation `entity_id` equals its event ID | +| acting `legacy_actor` or `actor_profile` reference | canonical lowercase UUID | +| acting `system_principal` reference | `workstream:system:bootstrap`; additional principals require an approved registry change | +| target actor | kind `actor_profile` plus canonical lowercase UUID | +| `resource_type` | `actor_profile`, `actor_identity_link`, `admin_role_grant`, `project`, `project_role_grant`, `task`, `submission`, `review`, `contribution`, `compensation_award`, `compensation_delivery`, `operations`, or `audit_event` | +| target/invalidation kind and reference | `actor_profile`, `actor_identity_link`, `admin_role_grant`, `qualification_snapshot`, or `project_role_grant` uses a canonical UUID; `permission_registry` uses one registered `permission_id` below | +| UUID envelope fields | native/canonical UUID for event, request, correlation, idempotency, and cause references | + +`permission_id` is closed to the exact specification Section 11 catalog: +`actor.profile.read_self`, `actor.profile.update_self`, +`actor.profile.read_any`, `actor.profile.suspend`, `actor.profile.reactivate`, +`actor.profile.deactivate`, `actor.identity_link.read`, +`actor.identity_link.revoke`, `actor.identity_link.reactivate`, +`actor.service.provision`, `admin_role.read`, `admin_role.grant`, +`admin_role.revoke`, `project.create`, `project.read`, `project.update`, +`project.archive`, `project.guide.manage`, `project.effective_policy.manage`, +`project.task.manage`, `project.review_policy.manage`, +`project.role_grant.read`, `project.role_grant.manage`, `task.queue.read`, +`task.claim`, `submission.create`, `submission.read_own`, +`submission.read_for_review`, `review.queue.read`, `review.queue.inspect`, +`review.claim`, `review.release`, `review.decline_preference`, +`review.decision`, `review.lease.force_release`, `review.chain.read`, +`contribution.read_self`, `contribution.read_project`, +`compensation.policy.manage`, `compensation.adapter_binding.manage`, +`compensation.award.read`, `compensation.delivery.reconcile`, +`operations.status.read`, `operations.timer.run`, +`operations.reconcile.run`, `operations.outbox.retry`, +`operations.projection.rebuild`, `audit.read`, and `audit.export`. + +`denial_code` is closed to the exact specification Section 22 codes. The +authority writer may use only: `required_scope_missing`, +`unsupported_subject_kind`, `service_actor_not_provisioned`, +`identity_link_revoked`, `actor_suspended`, `actor_deactivated`, +`permission_not_granted`, `scope_not_authorized`, `self_grant_forbidden`, +`self_role_revoke_forbidden`, `resource_guard_denied`, `actor_not_found`, +`grant_not_found`, `resource_not_found`, `actor_already_suspended`, +`actor_not_suspended`, `actor_deactivated_terminal`, +`last_access_administrator`, `admin_role_grant_exists`, +`project_role_grant_exists`, `identity_link_conflict`, +`resource_project_mismatch`, `idempotency_mismatch`, `invalid_role_scope`, +`invalid_project_role`, or `qualification_snapshot_invalid`. + +Audit `reason` is a privacy-safe classification, not operator-entered text and +not a replacement for the required rationale stored on the owning domain row. +The writer never copies domain reason text into audit evidence. Reason is +required and event-compatible as follows: + +| Event group | Allowed reason classification | +|---|---| +| human/service provisioning | `automatic_first_access` / `manual_service_provisioning`, respectively | +| identity link create/revoke/reactivate | `identity_lifecycle_change` | +| actor suspend/deactivate | `security_response` or `administrative_correction` | +| actor reactivate | `administrative_correction` | +| initial Access Administrator bootstrap | `initial_access_bootstrap` | +| admin/project grant issue | `authority_assignment` | +| project grant replacement | `authority_replacement` | +| admin/project grant revoke | `authority_revocation` | +| admin grant/last-admin denial | `authorization_policy_denial` | +| qualification snapshot | `qualification_evidence_captured` | +| sensitive allow/deny | `authorization_evaluation` | +| invalidation request | `authority_state_changed` | + +Before/after facts are JSON objects with at most eight allowlisted scalar keys, +no nested containers, and at most 4096 encoded bytes. The only fact keys and +values are: `status` in `active`, `suspended`, `deactivated`, `revoked`, or +`captured`; `subject_kind` in `human` or `service`; `provisioning_method` in +`automatic_first_access` or `manual_service_provisioning`; `role` in +`access_administrator`, `operator`, `project_manager`, `finance_authority`, +`audit_authority`, `submitter`, `reviewer`, or `both`; `scope_type` in `system` +or `project`; `scope_id` as a canonical UUID; and `effective` and `allowed` as +booleans. `effective` describes persisted authority state; `allowed` describes +an authorization decision and they are not interchangeable. + +The event-specific fact matrix is exact: + +| Event group | Before facts | After facts | +|---|---|---| +| human/service provisioning | `NULL` | `status=active`, matching `subject_kind` and `provisioning_method` | +| identity linked | `NULL` | `status=active`, `subject_kind` | +| identity revoked/reactivated | `status=active` / `status=revoked` | `status=revoked` / `status=active` | +| actor suspended/reactivated | `status=active` / `status=suspended` | `status=suspended` / `status=active` | +| actor deactivated | `status=active` or `suspended` | `status=deactivated` | +| initial admin bootstrap | `NULL` | `status=active`, `role=access_administrator`, `scope_type=system`, `effective=true` | +| admin/project grant issued | `NULL` | `status=active`, applicable `role`/scope, `effective=true` | +| project grant replaced | prior active role/scope with `effective=true` | replacement active role/scope with `effective=true` | +| admin/project grant revoked | active role/scope with `effective=true` | same role/scope, `status=revoked`, `effective=false` | +| qualification snapshot captured | `NULL` | `status=captured` | +| grant operation denied or last-admin denial | `NULL` | `NULL`; `denial_code` carries the attempted-operation result | +| sensitive allowed/denied | `NULL` | `allowed=true` / `allowed=false` | +| invalidation requested | `effective=true` | `effective=false` | + +For grant facts, `access_administrator` and `operator` use only `system`; +`project_manager`, `finance_authority`, and `audit_authority` use `system` or +`project`; `submitter`, `reviewer`, and `both` use only `project`. `scope_id` is +absent for system scope and required for project scope. System-scope grant +evidence has `project_id = NULL`. Project-scope grant evidence requires +`project_id`, and every before/after `scope_id` equals that envelope value; +replacement retains the same scope ID before and after. When +`resource_type = project` and `resource_id` is present, `resource_id` also +equals `project_id`. Typed and direct-SQL paths enforce these cross-field rules. +Typed validation and database constraints enforce the identical envelope and +fact matrices. Table-driven tests execute the same positive and negative cases +through Pydantic and direct SQL; lexical bounds alone are insufficient. + +The typed admission layer inspects every `collections.abc.Mapping`, including +non-dict mappings, before Pydantic owns rejected values. Unknown fields and +privacy-invalid known values raise one stable non-echoing error. A sanitized +`AuthorityAuditEventInput.model_validate_json` override replaces malformed, +scalar, list, string, bytes, and bytearray decode/type failures with that same +value-free error instead of exposing a decoder `.doc` or Pydantic input. +Constructor, `model_validate`, and `model_validate_json` tests traverse +structured errors, arguments, causes, contexts, and public object graphs to +prove forbidden input is not retained. + +AUTH-05A behavior tests exercise these exact foundation shapes: + +- `SensitiveAuthorizationAllowed` requires permission, forbids denial code and + invalidation cause/target, and permits an optional canonical UUID + idempotency reference for later AUTH-05B orchestration; +- `SensitiveAuthorizationDenied` requires permission plus a registered denial + code and forbids invalidation cause/target and idempotency reference; +- `AuthorityInvalidationRequested` requires cause-event ID plus all-or-none + target kind/reference, forbids denial code, and permits an optional canonical + UUID idempotency reference for later AUTH-05B orchestration. + +The 05A envelope and database constraints are final for the idempotency-reference +field: it has no foreign key to an idempotency table before AUTH-05B, and 05A +introduces no idempotency lookup or replay behavior. Tests persist both `NULL` +foundation use and a syntactically valid non-NULL future reference without +creating an idempotency record. AUTH-05B consumes this existing field without +changing audit schema, constraints, repository, or writer behavior. + +Invalidation causation is a separate integrity boundary. Self, nonexistent, +and legacy-domain cause references are rejected; an invalidation cause must be +an existing authority event visible in the caller transaction. The typed +service and database insertion boundary enforce that rule without activating +an invalidation consumer or AUTH-05B replay behavior. + +Target actor kind/reference and target/invalidation kind/reference are +all-or-none pairs. `resource_id` requires `resource_type`, but `resource_type` +may have a `NULL` ID for a system-wide decision as allowed by the adopted +`AuthorizationDecision` contract. + +## Shared writer and append-only custody + +- `AuditRepository` is the only supported insert/read implementation. +- `TaskRepository.add_audit_event` and `list_audit_events` remain temporary + compatibility methods but delegate the same session/event/query to + `AuditRepository`; task/checker call sites and responses do not change. +- Supported application APIs expose insert/read only. +- Named PostgreSQL triggers reject every `audit_events` UPDATE, DELETE, and + TRUNCATE, regardless of event domain or session flag. Failed attempts leave + both legacy and authority rows unchanged. +- No application-settable GUC or current-user bypass exists. +- The protection is a normal-DML custody boundary, not a defense against the + table owner or DDL credentials. Production rollout requires a distinct + non-owner runtime role without trigger/DDL privileges. The runbook documents + that deployment gate and a separately controlled DB-owner maintenance + procedure using an exclusive lock, explicit trigger disable/re-enable, + transaction, change record, and post-check. Migration teardown is the only + ordinary trigger-removal path. + +## Migration custody + +- Upgrade `0017 -> 0018` preserves representative populated prior-head domain + and legacy audit rows without fabricating authority fields. +- Tests assert exact columns, nullability, defaults, constraints, indexes, + trigger/function names, invalid direct inserts, and database time. +- Downgrade takes writer-blocking locks before inspection and refuses without + mutation when any authority-domain audit row exists. +- Privileged test cleanup may delete only explicit fixture authority rows after + deliberately disabling/re-enabling the trigger under lock; legacy rows are + never deleted by downgrade. +- Empty authority evidence permits `0018 -> 0017 -> 0018`; re-upgrade recreates + all triggers. Destructive migration tests restore Alembic `head` in `finally`. + +## Acceptance criteria + +- Legacy audit writers/readers and task/checker response projections remain + behaviorally unchanged. +- Typed allowed, denied, and invalidation authority event shapes persist only + bounded non-sensitive evidence and reject malformed/mixed shapes. +- Rejected provider/email/URL/token-like inputs are absent from every public + exception object graph for dict and non-dict mapping inputs. +- Typed and direct-SQL paths enforce closed reason/fact registries and matching + entity/type/reference privacy bounds. +- Application-role normal DML cannot update, delete, or truncate any audit row. +- Shared-writer tests instrument `AuditRepository` and prove TaskRepository + compatibility methods pass the same session/event and return the shared + result; row-count-only proof is insufficient. +- No new route, dependency, middleware, permission, grant, actor, product + authority behavior, or invalidation consumer exists. + +## Verification + +```bash +cd backend +tmp_dir=$(mktemp -d) +trap 'rm -rf "$tmp_dir"' EXIT +.venv/bin/python scripts/run_isolated_tests.py --metadata-json "$tmp_dir/05a.json" --timeout-seconds 1800 -- \ + .venv/bin/python -m pytest -q \ + tests/test_alembic.py::test_authority_audit_schema_preserves_legacy_and_guards_downgrade \ + tests/test_audit.py \ + tests/test_tasks.py::test_task_repository_delegates_audit_persistence \ + tests/test_tasks.py::test_manual_checker_run_cannot_bypass_failed_automatic_gate \ + tests/test_tasks.py::test_queued_gate_fails_closed_when_lock_audit_is_missing +.venv/bin/python scripts/run_isolated_tests.py --metadata-json "$tmp_dir/05a-coverage.json" --timeout-seconds 1800 -- \ + .venv/bin/python -m pytest -q tests/test_audit.py \ + tests/test_tasks.py::test_task_repository_delegates_audit_persistence \ + --cov=app.modules.audit --cov-report=term-missing --cov-fail-under=90 +.venv/bin/ruff check app tests +.venv/bin/docstr-coverage --config .docstr.yaml +.venv/bin/python - <<'PY' +from pathlib import Path +import subprocess +import sys + +sys.path.insert(0, "scripts") +from coverage_policy import weak_python # noqa: E402 + +paths = subprocess.check_output( + ["git", "diff", "--relative", "--name-only", "origin/main...HEAD", "--", "tests"], + text=True, +).splitlines() +blocked = [path for path in paths if path.endswith(".py") and weak_python(Path(path))] +raise SystemExit(f"test weakening: {blocked}" if blocked else 0) +PY +cd .. +python3 scripts/check_stale_workstream_wording.py +python3 scripts/check_stale_authorization_docs.py +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_markdown_links.py +python3 scripts/check_internal_review_evidence.py +changed_production_lines=$(git diff --unified=0 origin/main...HEAD -- backend/app backend/alembic | \ + awk '/^\+\+\+|^---/{next} /^[+-]/{line=substr($0,2); if (line !~ /^[[:space:]]*($|#)/) n++} END{print n+0}') +printf 'AUTH-05A changed production lines: %s\n' "$changed_production_lines" +awk 'length($0) > 120 { print FNR ":" $0; bad=1 } END { exit bad }' \ + backend/alembic/versions/0018_authority_audit_evidence.py +if git diff --unified=0 origin/main...HEAD -- backend/tests | \ + rg '^-(.*assert|.*pytest\.raises|.*pytest\.mark\.(skip|xfail)|.*pytest\.(skip|xfail)|.*skipTest)'; then exit 1; fi +git diff --check +``` + +Run the exact migration node before audit/delegation tests on the same isolated +database. Do not repeat the multi-hour local full suite after focused repairs; +GitHub Backend CI remains the final repository-wide `--cov-fail-under=78` gate. +Test delta must be additive: no assertion, raises, skip, xfail, threshold, +workflow, or exclusion weakening. + +## Required reviewers + +- senior engineering +- QA/test +- security/auth and privacy +- product/ops +- architecture +- CI integrity +- docs +- reuse/dedup +- test delta + +## Human review focus + +Review legacy audit compatibility, sole writer ownership, authority-envelope +privacy, unconditional normal-DML immutability, migration custody, and the +documented non-owner production role requirement. + +## Stop conditions + +Stop if legacy event behavior changes, authority evidence needs a canonical +ActorProfile FK, normal application DML can mutate history, production scope +exceeds the circuit breaker, or tests/CI must be weakened. diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-05B-idempotency-invalidation.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-05B-idempotency-invalidation.md new file mode 100644 index 00000000..7e399afc --- /dev/null +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-05B-idempotency-invalidation.md @@ -0,0 +1,209 @@ +# Chunk Contract: WS-AUTH-001-05B - Authority Idempotency And Invalidation Foundation + +## Status + +Inactive until AUTH-05A merges, its post-merge memory is current, and the user +gives a separate explicit start signal. + +## Parent initiative + +`WS-AUTH-001` - Workstream Authorization Service + +## Goal + +Add concurrency-safe authority-mutation reservation/replay and typed success, +denial, and invalidation orchestration on the shared AUTH-05A audit envelope. + +## Risk and circuit breaker + +- Risk: L1 / SLA P1. +- Inspect scope at 350 changed non-comment production lines. +- Hard stop at 500 changed non-comment production lines, counting + `backend/app/**` plus migration code. + +## Allowed files + +```text +backend/app/modules/authorization/** +backend/app/db/models.py +backend/alembic/versions/0019_authority_idempotency.py +backend/tests/test_authorization.py +backend/tests/test_audit.py +backend/tests/test_alembic.py +docs/architecture_data_model.md +docs/operations_authorization_service.md +.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/** +.agent-loop/LOOP_STATE.md +.agent-loop/WORK_QUEUE.md +.agent-loop/REVIEW_LOG.md +``` + +## Not allowed + +```text +routes, dependencies, middleware, or generic unit-of-work wrappers +actor/profile migration or first-access behavior +permissions, grant tables/APIs, or product authorization cutover +invalidation consumers, caches, queues, or external delivery +token-role or IdentityIssuerVerifier/factory changes +raw request/response bodies, URLs, PII, secrets, or arbitrary JSON persistence +workflow, dependency, threshold, skip, or exclusion changes +repository/service commit, rollback, or parallel session creation +``` + +## Idempotency schema and namespace + +Migration `0019_authority_idempotency` uses exact +`down_revision = "0018_authority_audit_evidence"` and creates +`authority_idempotency_records`. Each record contains: + +- UUID primary key and canonical UUID idempotency key; +- actor reference kind plus stable opaque actor reference; +- closed operation token; +- canonical request digest using existing + `app.core.hashing.canonical_json_hash` (`sha256:` plus 64 lowercase hex); +- status `pending` or `committed`; +- typed committed response reference: resource type, resource ID, optional + version, and successful HTTP status; never a body, URL, or free-form JSON; +- database-owned created/committed timestamps. + +The unique replay namespace is +`(actor_ref_kind, actor_ref, operation, idempotency_key)`. Different actors or +reference kinds may reuse a key without collision or visibility. Operation, +target project/resource, and mutation payload are included in the canonical +request object/digest. Request ID, correlation ID, and the idempotency key are +excluded from the digest so legitimate retries may carry new request context. +The raw request is never persisted. + +The canonical request object is a typed, bounded JSON-compatible dictionary. +It uses the shared canonical hash helper; no authorization-local encoder is +introduced. + +## State machine and transaction ownership + +- Reservation runs before any business-state flush. +- `INSERT ... ON CONFLICT` plus a locked read serializes concurrent users of + one replay namespace. +- A new reservation returns `claimed` with a `pending` row in the caller's + transaction. +- Same namespace and digest after commit returns `replay` with the existing + typed committed reference and does not append duplicate success/invalidation + events. +- Same namespace with a different digest returns a stable `mismatch` outcome; + caller-owned application orchestration records the required denial event in + a clean transaction before surfacing `idempotency_mismatch`. +- Completion atomically changes only the caller's pending row to `committed` + and writes its typed response reference. +- A deferred database constraint rejects commit while a record is still + pending. Rollback/cancellation removes the reservation and all co-transaction + business/audit/invalidation rows; retry may then claim normally. A pending + row is never stolen, expired, or treated as success. +- Repository/service methods may insert, lock, flush, and return typed outcomes + only. The injected caller session alone commits or rolls back; no parallel + unit-of-work abstraction or session is introduced. + +## Invalidation and denial evidence + +Invalidation is an `AuthorityInvalidationRequested` authority-domain +`AuditEvent`, not a second event table. Authorization owns typed orchestration; +AuditRepository remains the persistence owner. The event links its causing +authority event, exact target kind/reference, request/correlation IDs, actor, +reason, and idempotency record when applicable. AUTH-05B introduces persistence +and orchestration only: no cache, worker, adapter, or authority behavior consumes +the event. + +Allowed, denied, and invalidation events use stable tokens. Denial/mismatch +orchestration returns a typed result rather than committing or raising inside +the repository; the caller commits the denial evidence before translating the +result to an API error. Denials are not stored as successful replay results. + +## Migration custody + +- Upgrade preserves all AUTH-05A audit rows and creates no idempotency or + invalidation evidence. +- Tests assert exact table columns, constraints, indexes, deferred trigger, + defaults, invalid rows, and database time. +- Downgrade takes writer-blocking locks and refuses without mutation when any + idempotency row or audit row referencing idempotency/invalidation evidence + exists. Legacy and AUTH-05A audit rows remain intact. +- Privileged fixture cleanup permits `0019 -> 0018 -> 0019`; re-upgrade restores + the pending-commit guard. Destructive tests restore `head` in `finally`. + +## Acceptance criteria + +- Independent-session exact concurrent retries produce one committed result + and one success/invalidation event set. +- Concurrent mismatched requests produce one winner and one privacy-safe + mismatch outcome; different actors/kinds remain isolated. +- Injected failure after reservation, synthetic business flush, audit insert, + invalidation insert, completion, and commit attempt leaves either the complete + transaction or nothing. Repositories never commit/rollback. +- Retry after rollback claims normally; commit of unfinished pending state + fails closed and leaves no durable pending row after caller rollback. +- Exact replay never duplicates business, success, or invalidation evidence. +- No route, permission, grant, actor, product authority behavior, or + invalidation consumer exists. + +## Verification + +```bash +cd backend +tmp_dir=$(mktemp -d) +trap 'rm -rf "$tmp_dir"' EXIT +.venv/bin/python -m ruff check app tests +.venv/bin/python scripts/run_isolated_tests.py \ + --metadata-json "$tmp_dir/05b.json" --timeout-seconds 1800 -- \ + .venv/bin/python -m pytest -q \ + tests/test_alembic.py::test_authority_idempotency_schema_preserves_audit_and_guards_downgrade \ + tests/test_authorization.py tests/test_audit.py +.venv/bin/python scripts/run_isolated_tests.py \ + --metadata-json "$tmp_dir/05b-coverage.json" --timeout-seconds 1800 -- \ + .venv/bin/python -m pytest -q tests/test_authorization.py tests/test_audit.py \ + --cov=app.modules.authorization --cov-report=term-missing --cov-fail-under=90 +.venv/bin/python -m pytest -q tests/test_isolated_database_runner.py +.venv/bin/python scripts/run_isolated_tests.py \ + --metadata-json "$tmp_dir/05b-full.json" --timeout-seconds 10800 -- \ + .venv/bin/python -m pytest -q tests \ + --ignore=tests/test_isolated_database_runner.py \ + --cov=app --cov-report=term-missing \ + --cov-fail-under=78 +.venv/bin/docstr-coverage --config .docstr.yaml +cd .. +python3 scripts/check_stale_workstream_wording.py +python3 scripts/check_stale_authorization_docs.py +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_markdown_links.py +python3 scripts/check_internal_review_evidence.py +git diff --check +``` + +Run the exact migration node before audit/authorization tests on the same +isolated database. Test delta must be additive and all CI integrity controls +remain unchanged. + +## Required reviewers + +- senior engineering +- QA/test +- security/auth and privacy +- product/ops +- architecture +- CI integrity +- docs +- reuse/dedup +- test delta + +## Human review focus + +Review replay namespace, canonical hashing, pending-commit prevention, +concurrency/rollback behavior, typed result privacy, invalidation causation, +and caller-owned transaction boundaries. + +## Stop conditions + +Stop if exact concurrency cannot serialize without a committed pending row, +denial evidence requires committing unrelated work, a route/grant/permission +must be introduced, or tests/CI must be weakened. AUTH-05B consumes the merged +05A audit envelope and sole-writer contract without modifying its schema, +triggers, constraints, repository, or writer. A missing audit primitive requires +stop and replan rather than reopening 05A custody. diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/reviews/WS-AUTH-001-05A-external-review-response.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/reviews/WS-AUTH-001-05A-external-review-response.md new file mode 100644 index 00000000..77700776 --- /dev/null +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/reviews/WS-AUTH-001-05A-external-review-response.md @@ -0,0 +1,58 @@ +# WS-AUTH-001-05A External Review Response + +External review: CodeRabbit run `30676bdc-7525-4deb-8f9e-a87d42c64f92` +Pull request: `#115` + +## Comments Addressed + +1. Replaced `_REASONS` positional enum slicing with explicit identity-link + event keys while preserving `identity_lifecycle_change` behavior. +2. Added the `AsyncSession` annotation to the task audit owner-cleanup helper. +3. Replaced inactive AUTH-05B verification prose and placeholders with concrete + isolated migration, focused behavior, 90 percent subsystem coverage, and 78 + percent full-suite commands in the required order. +4. Replaced misleading reused AUTH-04B reviewer-session names with concrete + AUTH-05A final-review references tied to the reviewed code SHA. + +Required internal QA then found that the new full-suite example recursively +included the isolation-runner self-tests. The command now runs those self-tests +directly and excludes only that file from the isolated full suite, matching +Backend CI without changing either coverage threshold. + +## Comments Deferred + +None. + +## Non-Actionable Review Output + +CodeRabbit's generic docstring warning reports a diff-local percentage rather +than the repository's configured gate. No docstring-only churn was added: the +actual `docstr-coverage --config .docstr.yaml` result passes at 95.1 percent. + +## Human Decisions Needed + +None. + +## Commands Rerun + +```text +pytest test_authority_input_rejects_unbounded_or_inconsistent_evidence +pytest test_task_repository_delegates_audit_persistence +pytest test_authority_event_matrix_has_typed_direct_sql_parity +pytest tests/test_isolated_database_runner.py +ruff check backend/app/modules/audit/schemas.py backend/tests/test_tasks.py +bash -n +python3 scripts/check_markdown_links.py +python3 scripts/check_stale_authorization_docs.py +git diff --check +``` + +Focused result: 3 authority/delegation cases and the isolated-runner self-tests +passed. All listed static checks passed. Exact SHA +`ea16fd8bd2d9bc38b37c12003e51416c08a56678` then passed every required internal +review track with no remaining findings. + +## Remaining Risks + +GitHub Backend owns the full-suite and repository-wide coverage gate. AUTH-05B +remains an inactive contract; none of its runtime behavior is implemented. diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/reviews/WS-AUTH-001-05A-internal-review-evidence.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/reviews/WS-AUTH-001-05A-internal-review-evidence.md new file mode 100644 index 00000000..03e82b7f --- /dev/null +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/reviews/WS-AUTH-001-05A-internal-review-evidence.md @@ -0,0 +1,62 @@ +# WS-AUTH-001-05A Internal Review Evidence + +Reviewed code SHA: `ea16fd8bd2d9bc38b37c12003e51416c08a56678` +Reviewed at: `2026-07-14T13:18:49Z` +Reviewer run IDs: `AUTH-05A-ea16fd8-eng-external-fix`, `AUTH-05A-ea16fd8-qa-external-fix`, `AUTH-05A-ea16fd8-security-external-fix` + +## Deterministic Evidence + +- Exact migration upgrade/downgrade proof: 1 passed in 52.49 seconds. +- Audit behavior suite: 10 passed in 29.84 seconds. +- Audit plus shared-repository delegation coverage: 11 passed in 98.42 + seconds; `app.modules.audit` reached 94.55 percent against the 90 percent + gate. +- Task compatibility regressions for manual checker bypass and missing lock + audit: 2 passed in 154.57 seconds. Shared repository delegation also passed + in the coverage run. +- Ruff passed for `backend/app`, `backend/tests`, and `backend/alembic`. +- Docstring coverage passed at 95.1 percent overall. +- Test-delta analysis reported three changed test files and no weakening; + deleted assertion, raises, skip, and xfail scans passed. +- Migration lines remained within 120 characters. `git diff --check`, stale + Workstream/auth/artifact scans, and changed Markdown link checks passed. +- No dependency, workflow, coverage threshold, exclusion, or package-script + change was made. +- The multi-hour repository suite was not repeated locally. GitHub Backend CI + remains responsible for the full suite and repository-wide 78 percent floor; + the latest merged main evidence was 82.15 percent. + +## Reviewer Results + +| Reviewer | Result | Blocking findings | Notes | +|---|---|---|---| +| senior engineering | PASS | none | Final normalized Mapping custody and service snapshot reviewed. | +| qa/test | PASS | none | Full event matrix, warning assertion, and behavior regressions reviewed. | +| security/auth | PASS | none | Privacy non-retention, registries, causes, and append-only controls reviewed. | +| product/ops | PASS | none | Authority evidence semantics pass; no lifecycle activation entered scope. | +| architecture | PASS | none | Shared writer and caller-owned transaction boundaries pass. | +| ci integrity | PASS | none | CI, thresholds, dependencies, and exclusions were unchanged. | +| docs | PASS | none | Architecture, operations, and durable loop state match the implementation. | +| reuse/dedup | PASS | none | Task compatibility methods delegate to the shared audit repository. | +| test delta | PASS | none | No assertion, raise, skip, xfail, or behavior guard was weakened. | + +## Findings Resolved + +Review cycles repaired valid findings covering system-scope SQL `NULL` +semantics, canonical UUID event IDs, duplicate JSON keys, denial codes, +typed/direct-SQL event parity, mutable-model service custody, warning and +exception retention, state-changing top-level and nested mappings, and mutable +JSON buffers. External review repairs removed positional event coupling, typed +the task cleanup session, and made the inactive AUTH-05B verification block +executable and consistent with Backend CI. Every repair has focused behavior +proof. + +Valid findings addressed: yes + +Open sub-agent sessions: none + +## Remaining Gate + +GitHub Backend and CodeRabbit follow-up checks plus explicit human merge +approval remain pending. AUTH-05B and later authorization chunks remain +inactive. diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/reviews/WS-AUTH-001-05A-pr-trust-bundle.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/reviews/WS-AUTH-001-05A-pr-trust-bundle.md new file mode 100644 index 00000000..8124c4ba --- /dev/null +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/reviews/WS-AUTH-001-05A-pr-trust-bundle.md @@ -0,0 +1,141 @@ +# WS-AUTH-001-05A PR Trust Bundle + +## Chunk + +`WS-AUTH-001-05A` - Shared Audit Ownership And Append-Only Authority Evidence + +## Goal + +Add one typed, privacy-bounded, append-only authority evidence envelope to the +existing shared `audit_events` path without changing legacy task/checker +behavior or activating authorization decisions. + +## Human-Approved Intent + +The human approved AUTH-05A as one semantic chunk bounded by its allowed files, +non-goals, acceptance criteria, behavior evidence, and required reviews rather +than an arbitrary production-line ceiling. AUTH-05B remains separate. + +## What Changed + +- Migration `0018` extends the shared audit table with versioned authority + fields, closed registries, cross-field constraints, database-owned time, + authority-cause integrity, and append-only normal-DML triggers. +- A typed authority input and shared audit service enforce the same event, + reason, fact, reference, permission, denial, role, and project matrices. +- Task repository compatibility methods delegate to `AuditRepository` while + preserving caller session and transaction ownership. +- Behavior tests cover all 20 authority events through typed and direct-SQL + positive and negative paths, legacy compatibility, rollback, mutation + custody, non-retention, and migration downgrade guards. + +## Why It Changed + +Later authorization chunks require trustworthy evidence custody before actor, +grant, permission evaluation, route cutover, idempotency, or invalidation +consumers can be implemented. + +## Design Chosen + +The existing audit table remains the sole store. Application writes use one +typed service and shared repository. PostgreSQL independently enforces the +closed envelope so direct SQL cannot bypass privacy or integrity rules. +Rejected mappings and JSON are inspected from one stable snapshot, and service +re-admission never serializes untrusted mutated values before validation. + +## Alternatives Rejected + +- A parallel authority event table was rejected because it would split audit + custody and duplicate writer behavior. +- Application-only validation was rejected because direct SQL must enforce the + same boundary. +- Splitting typed and SQL matrices into mergeable partial states was rejected + because either half alone is an incomplete security boundary. +- AUTH-05B idempotency/replay work remains deferred to its own chunk. + +## Scope Control + +No route, dependency, middleware, permission evaluator, actor/grant lifecycle, +authorization cutover, idempotency record/replay path, invalidation consumer, +token verifier, workflow, dependency, or CI threshold was added or changed. + +## Product Behavior + +Legacy audit behavior and task/checker projections remain unchanged. New +authority evidence is internal foundation behavior only; it does not grant, +revoke, authorize, route, or invalidate product actions. + +## Acceptance Criteria Proof + +- All 20 registered events share typed and PostgreSQL positive/negative cases. +- System and project grant scopes, project/resource equality, UUID references, + registered tokens, denial codes, cause domain, self-cause, and append-only + custody have direct behavior proof. +- Malformed, scalar, duplicate-key, hostile mapping, mutable-buffer, secret + representation, and post-validation mutation cases retain no rejected value + and emit no serializer warning. +- Legacy rows survive upgrade/downgrade proof and existing repository behavior. + +## Tests And Checks Run + +- Exact migration proof: 1 passed. +- Audit suite: 10 passed. +- Audit plus delegation coverage: 11 passed at 94.55 percent. +- Two affected task lifecycle regressions passed. +- Ruff, 95.1 percent docstring coverage, test-delta, migration line length, + stale wording/auth/artifact, Markdown links, and diff integrity passed. +- Full local repository tests were intentionally not repeated; GitHub Backend + CI owns the full suite and repository-wide 78 percent coverage floor. + +## Test Delta + +Tests were added or strengthened only. No assertion, raises guard, skip, xfail, +coverage threshold, or exclusion was removed or weakened. + +## CI Integrity + +No workflow, package, dependency, coverage configuration, or test selection +file changed. GitHub checks must pass before merge. + +## Reviewer Results + +Exact SHA `ea16fd8bd2d9bc38b37c12003e51416c08a56678` passed senior engineering, +architecture, reuse/dedup, docs, product/ops, QA/test, CI integrity, test delta, +and security/auth review with no remaining findings. + +## External Review + +CodeRabbit run `30676bdc-7525-4deb-8f9e-a87d42c64f92` produced four +actionable comments. The response replaces positional reason mapping with +explicit event keys, types the task audit cleanup session, makes the inactive +AUTH-05B verification block executable, and records AUTH-05A-specific reviewer +references. Internal QA additionally aligned the full-suite example with +Backend CI by running isolation-runner self-tests outside the isolated suite. +The generic CodeRabbit docstring warning is not applicable because the +repository's configured docstring gate passes at 95.1 percent. GitHub Backend +CI remains the final full-suite gate. + +## Remaining Risks + +- PostgreSQL append-only triggers protect normal DML, not database-owner or DDL + credentials; production must use the documented non-owner runtime role. +- The full repository suite and global coverage floor remain GitHub CI gates. +- AUTH-05A records evidence but intentionally does not consume invalidations or + perform authorization. + +## Follow-Up Work + +After merge, post-merge memory, and a separate human start, AUTH-05B may add the +idempotency and invalidation foundation. Later chunks own actor profiles, +grants, permission evaluation, and route cutover. + +## Human Review Focus + +Review the migration constraints and downgrade ordering, typed/SQL matrix +parity, single-snapshot privacy boundary, shared writer transaction ownership, +and proof that no authorization behavior was activated. + +## Human Merge Ownership + +Only the human may approve and merge this PR. Completion of internal or +external review does not authorize merge. diff --git a/backend/alembic/versions/0018_authority_audit_evidence.py b/backend/alembic/versions/0018_authority_audit_evidence.py new file mode 100644 index 00000000..02700d4f --- /dev/null +++ b/backend/alembic/versions/0018_authority_audit_evidence.py @@ -0,0 +1,604 @@ +"""add append-only authority audit evidence + +Revision ID: 0018_authority_audit_evidence +Revises: 0017_api_controls +Create Date: 2026-07-14 +""" + +from alembic import op +import sqlalchemy as sa + +revision = "0018_authority_audit_evidence" +down_revision = "0017_api_controls" +branch_labels = depends_on = None + +AUTHORITY_EVENT_TYPES = ( + "ActorProfileProvisioned", + "ServiceActorProvisioned", + "ActorIdentityLinked", + "ActorIdentityLinkRevoked", + "ActorIdentityLinkReactivated", + "ActorProfileSuspended", + "ActorProfileReactivated", + "ActorProfileDeactivated", + "InitialAccessAdministratorBootstrapped", + "AdminRoleGrantIssued", + "AdminRoleGrantRevoked", + "AdminRoleGrantIssueDenied", + "LastAccessAdministratorOperationDenied", + "ProjectRoleQualificationSnapshotCaptured", + "ProjectRoleGrantIssued", + "ProjectRoleGrantReplaced", + "ProjectRoleGrantRevoked", + "SensitiveAuthorizationAllowed", + "SensitiveAuthorizationDenied", + "AuthorityInvalidationRequested", +) +PERMISSIONS = """actor.profile.read_self actor.profile.update_self actor.profile.read_any +actor.profile.suspend actor.profile.reactivate actor.profile.deactivate actor.identity_link.read +actor.identity_link.revoke actor.identity_link.reactivate actor.service.provision admin_role.read +admin_role.grant admin_role.revoke project.create project.read project.update project.archive +project.guide.manage project.effective_policy.manage project.task.manage project.review_policy.manage +project.role_grant.read project.role_grant.manage task.queue.read task.claim submission.create +submission.read_own submission.read_for_review review.queue.read review.queue.inspect review.claim +review.release review.decline_preference review.decision review.lease.force_release review.chain.read +contribution.read_self contribution.read_project compensation.policy.manage +compensation.adapter_binding.manage compensation.award.read compensation.delivery.reconcile +operations.status.read operations.timer.run operations.reconcile.run operations.outbox.retry +operations.projection.rebuild audit.read audit.export""".split() +DENIAL_CODES = """required_scope_missing unsupported_subject_kind service_actor_not_provisioned +identity_link_revoked actor_suspended actor_deactivated permission_not_granted scope_not_authorized +self_grant_forbidden self_role_revoke_forbidden resource_guard_denied actor_not_found grant_not_found +resource_not_found actor_already_suspended actor_not_suspended actor_deactivated_terminal +last_access_administrator admin_role_grant_exists project_role_grant_exists identity_link_conflict +resource_project_mismatch idempotency_mismatch invalid_role_scope invalid_project_role +qualification_snapshot_invalid""".split() +REASONS = { + "ActorProfileProvisioned": ("automatic_first_access",), + "ServiceActorProvisioned": ("manual_service_provisioning",), + "ActorIdentityLinked": ("identity_lifecycle_change",), + "ActorIdentityLinkRevoked": ("identity_lifecycle_change",), + "ActorIdentityLinkReactivated": ("identity_lifecycle_change",), + "ActorProfileSuspended": ("security_response", "administrative_correction"), + "ActorProfileReactivated": ("administrative_correction",), + "ActorProfileDeactivated": ("security_response", "administrative_correction"), + "InitialAccessAdministratorBootstrapped": ("initial_access_bootstrap",), + "AdminRoleGrantIssued": ("authority_assignment",), + "AdminRoleGrantRevoked": ("authority_revocation",), + "AdminRoleGrantIssueDenied": ("authorization_policy_denial",), + "LastAccessAdministratorOperationDenied": ("authorization_policy_denial",), + "ProjectRoleQualificationSnapshotCaptured": ("qualification_evidence_captured",), + "ProjectRoleGrantIssued": ("authority_assignment",), + "ProjectRoleGrantReplaced": ("authority_replacement",), + "ProjectRoleGrantRevoked": ("authority_revocation",), + "SensitiveAuthorizationAllowed": ("authorization_evaluation",), + "SensitiveAuthorizationDenied": ("authorization_evaluation",), + "AuthorityInvalidationRequested": ("authority_state_changed",), +} + + +def _sql_tokens(values) -> str: + return ", ".join(f"'{value}'" for value in values) + + +def upgrade() -> None: + """Extend the shared audit table and install normal-DML custody guards.""" + columns = ( + sa.Column( + "event_domain", + sa.String(24), + nullable=False, + server_default="legacy_lifecycle", + ), + sa.Column("event_version", sa.Integer(), nullable=True), + sa.Column("occurred_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("actor_ref_kind", sa.String(32), nullable=True), + sa.Column("request_id", sa.Uuid(), nullable=True), + sa.Column("correlation_id", sa.Uuid(), nullable=True), + sa.Column("target_actor_ref_kind", sa.String(32), nullable=True), + sa.Column("target_actor_ref", sa.String(100), nullable=True), + sa.Column("matched_grant_id", sa.String(100), nullable=True), + sa.Column("permission_id", sa.String(120), nullable=True), + sa.Column("project_id", sa.String(36), nullable=True), + sa.Column("resource_type", sa.String(80), nullable=True), + sa.Column("resource_id", sa.String(100), nullable=True), + sa.Column("target_ref_kind", sa.String(32), nullable=True), + sa.Column("target_ref_id", sa.String(100), nullable=True), + sa.Column("denial_code", sa.String(80), nullable=True), + sa.Column("idempotency_reference", sa.Uuid(), nullable=True), + sa.Column("invalidation_cause_event_id", sa.String(36), nullable=True), + sa.Column("invalidation_target_kind", sa.String(32), nullable=True), + sa.Column("invalidation_target_ref", sa.String(100), nullable=True), + sa.Column("before_facts", sa.JSON(), nullable=True), + sa.Column("after_facts", sa.JSON(), nullable=True), + ) + for column in columns: + op.add_column("audit_events", column) + op.alter_column("audit_events", "external_subject", existing_type=sa.String(200), nullable=True) + op.alter_column("audit_events", "external_issuer", existing_type=sa.String(200), nullable=True) + op.execute( + """ + create function authority_facts_are_safe(facts json) + returns boolean language sql immutable strict as $$ + select json_typeof(facts) = 'object' + and (select count(*) = count(distinct key) and count(*) <= 8 from json_each(facts)) + and not exists ( + select 1 from json_each(facts) item + where item.key not in ( + 'status', 'subject_kind', 'provisioning_method', 'role', + 'scope_type', 'scope_id', 'effective', 'allowed' + ) + or case item.key + when 'status' then item.value #>> '{}' not in ( + 'active', 'suspended', 'deactivated', 'revoked', 'captured' + ) + when 'subject_kind' then item.value #>> '{}' not in ('human', 'service') + when 'provisioning_method' then item.value #>> '{}' not in ( + 'automatic_first_access', 'manual_service_provisioning' + ) + when 'role' then item.value #>> '{}' not in ( + 'access_administrator', 'operator', 'project_manager', + 'finance_authority', 'audit_authority', 'submitter', 'reviewer', 'both' + ) + when 'scope_type' then item.value #>> '{}' not in ('system', 'project') + when 'scope_id' then (item.value #>> '{}') !~ + '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' + when 'effective' then json_typeof(item.value) <> 'boolean' + when 'allowed' then json_typeof(item.value) <> 'boolean' + else true + end + ) + $$ + """ + ) + op.execute( + """ + create function authority_grant_facts_are_safe( + facts json, roles text[], expected_status text, + expected_effective boolean, envelope_project_id text + ) returns boolean language sql immutable as $$ + select authority_facts_are_safe(facts) + and facts->>'role' = any(roles) + and facts->>'status' = expected_status + and (facts->>'effective')::boolean = expected_effective + and ( + ( + facts->>'scope_type' = 'system' + and envelope_project_id is null + and not facts::jsonb ? 'scope_id' + and facts->>'role' not in ('submitter', 'reviewer', 'both') + and (select count(*) from json_each(facts)) = 4 + ) or ( + facts->>'scope_type' = 'project' + and envelope_project_id is not null + and facts->>'scope_id' = envelope_project_id + and facts->>'role' not in ('access_administrator', 'operator') + and (select count(*) from json_each(facts)) = 5 + ) + ) + $$ + """ + ) + op.execute( + """ + create function authority_event_facts_are_safe( + event_name text, before_state json, after_state json, envelope_project_id text + ) returns boolean language plpgsql immutable as $$ + begin + if (before_state is not null and not authority_facts_are_safe(before_state)) + or (after_state is not null and not authority_facts_are_safe(after_state)) then + return false; + end if; + case event_name + when 'ActorProfileProvisioned' then + return before_state is null and after_state::jsonb = + '{"status":"active","subject_kind":"human",' + '"provisioning_method":"automatic_first_access"}'::jsonb; + when 'ServiceActorProvisioned' then + return before_state is null and after_state::jsonb = + '{"status":"active","subject_kind":"service",' + '"provisioning_method":"manual_service_provisioning"}'::jsonb; + when 'ActorIdentityLinked' then + return before_state is null and after_state::jsonb in ( + '{"status":"active","subject_kind":"human"}'::jsonb, + '{"status":"active","subject_kind":"service"}'::jsonb + ); + when 'ActorIdentityLinkRevoked' then + return before_state::jsonb = '{"status":"active"}'::jsonb + and after_state::jsonb = '{"status":"revoked"}'::jsonb; + when 'ActorIdentityLinkReactivated' then + return before_state::jsonb = '{"status":"revoked"}'::jsonb + and after_state::jsonb = '{"status":"active"}'::jsonb; + when 'ActorProfileSuspended' then + return before_state::jsonb = '{"status":"active"}'::jsonb + and after_state::jsonb = '{"status":"suspended"}'::jsonb; + when 'ActorProfileReactivated' then + return before_state::jsonb = '{"status":"suspended"}'::jsonb + and after_state::jsonb = '{"status":"active"}'::jsonb; + when 'ActorProfileDeactivated' then + return before_state::jsonb in ( + '{"status":"active"}'::jsonb, '{"status":"suspended"}'::jsonb + ) and after_state::jsonb = '{"status":"deactivated"}'::jsonb; + when 'InitialAccessAdministratorBootstrapped' then + return before_state is null and authority_grant_facts_are_safe( + after_state, array['access_administrator'], 'active', true, null + ); + when 'AdminRoleGrantIssued' then + return before_state is null and authority_grant_facts_are_safe( + after_state, + array[ + 'access_administrator', 'operator', 'project_manager', + 'finance_authority', 'audit_authority' + ], 'active', true, envelope_project_id + ); + when 'ProjectRoleGrantIssued' then + return before_state is null and authority_grant_facts_are_safe( + after_state, array['submitter', 'reviewer', 'both'], + 'active', true, envelope_project_id + ); + when 'AdminRoleGrantRevoked', 'ProjectRoleGrantRevoked' then + return authority_grant_facts_are_safe( + before_state, + case when event_name = 'AdminRoleGrantRevoked' then + array[ + 'access_administrator', 'operator', 'project_manager', + 'finance_authority', 'audit_authority' + ] + else array['submitter', 'reviewer', 'both'] end, + 'active', true, envelope_project_id + ) and authority_grant_facts_are_safe( + after_state, + case when event_name = 'AdminRoleGrantRevoked' then + array[ + 'access_administrator', 'operator', 'project_manager', + 'finance_authority', 'audit_authority' + ] + else array['submitter', 'reviewer', 'both'] end, + 'revoked', false, envelope_project_id + ) and before_state->>'role' = after_state->>'role' + and before_state->>'scope_type' = after_state->>'scope_type' + and coalesce(before_state->>'scope_id', '') = + coalesce(after_state->>'scope_id', ''); + when 'ProjectRoleGrantReplaced' then + return authority_grant_facts_are_safe( + before_state, array['submitter', 'reviewer', 'both'], + 'active', true, envelope_project_id + ) and authority_grant_facts_are_safe( + after_state, array['submitter', 'reviewer', 'both'], + 'active', true, envelope_project_id + ) and before_state->>'scope_id' = after_state->>'scope_id'; + when 'ProjectRoleQualificationSnapshotCaptured' then + return before_state is null + and after_state::jsonb = '{"status":"captured"}'::jsonb; + when 'AdminRoleGrantIssueDenied', 'LastAccessAdministratorOperationDenied' then + return before_state is null and after_state is null; + when 'SensitiveAuthorizationAllowed' then + return before_state is null and after_state::jsonb = '{"allowed": true}'::jsonb; + when 'SensitiveAuthorizationDenied' then + return before_state is null and after_state::jsonb = '{"allowed": false}'::jsonb; + when 'AuthorityInvalidationRequested' then + return before_state::jsonb = '{"effective": true}'::jsonb + and after_state::jsonb = '{"effective": false}'::jsonb; + else return false; + end case; + end + $$ + """ + ) + + event_tokens = _sql_tokens(AUTHORITY_EVENT_TYPES) + permission_tokens = _sql_tokens(PERMISSIONS) + denial_tokens = _sql_tokens(DENIAL_CODES) + reason_rules = " or ".join( + f"(event_type = '{event}' and reason in ({_sql_tokens(reasons)}))" + for event, reasons in REASONS.items() + ) + entity_tokens = _sql_tokens( + ( + "actor_profile", + "actor_identity_link", + "admin_role_grant", + "qualification_snapshot", + "project_role_grant", + "authorization_decision", + "authority_invalidation", + ) + ) + resource_tokens = _sql_tokens( + """actor_profile actor_identity_link admin_role_grant project project_role_grant task + submission review contribution compensation_award compensation_delivery operations + audit_event""".split() + ) + uuid_target_tokens = _sql_tokens( + ( + "actor_profile", + "actor_identity_link", + "admin_role_grant", + "qualification_snapshot", + "project_role_grant", + ) + ) + uuid_pattern = r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$" + op.create_check_constraint( + "domain_shape", + "audit_events", + """ + ( + event_domain = 'legacy_lifecycle' + and event_version is null and occurred_at is null and actor_ref_kind is null + and request_id is null and correlation_id is null and target_actor_ref_kind is null + and target_actor_ref is null and matched_grant_id is null and permission_id is null + and project_id is null and resource_type is null and resource_id is null + and target_ref_kind is null and target_ref_id is null and denial_code is null + and idempotency_reference is null and invalidation_cause_event_id is null + and invalidation_target_kind is null and invalidation_target_ref is null + and before_facts is null and after_facts is null + and external_subject is not null and external_issuer is not null + ) or ( + event_domain = 'authority' and event_version = 1 and occurred_at is not null + and actor_ref_kind in ('legacy_actor', 'actor_profile', 'system_principal') + and request_id is not null and correlation_id is not null + and from_status is null and to_status is null and reason is not null + and external_subject is null and external_issuer is null + and actor_roles::jsonb = '[]'::jsonb and claim_snapshot::jsonb = '{}'::jsonb + and auth_source = 'local_authority' and is_dev_auth = false + and event_payload::jsonb = '{}'::jsonb + ) + """, + ) + op.create_check_constraint( + "authority_tokens", + "audit_events", + f"event_domain <> 'authority' or event_type in ({event_tokens})", + ) + op.create_check_constraint( + "authority_registries", + "audit_events", + f""" + event_domain <> 'authority' or ( + reason is not null and ({reason_rules}) + and (permission_id is null or permission_id in ({permission_tokens})) + and (denial_code is null or denial_code in ({denial_tokens})) + ) + """, + ) + op.create_check_constraint( + "reference_pairs", + "audit_events", + """ + (target_actor_ref_kind is null) = (target_actor_ref is null) + and (resource_type is not null or resource_id is null) + and (target_ref_kind is null) = (target_ref_id is null) + and (invalidation_target_kind is null) = (invalidation_target_ref is null) + and (invalidation_cause_event_id is null or invalidation_cause_event_id <> id) + """, + ) + op.create_check_constraint( + "authority_privacy_bounds", + "audit_events", + f""" + event_domain <> 'authority' or ( + id ~ '{uuid_pattern}' and entity_type in ({entity_tokens}) and entity_id ~ '{uuid_pattern}' + and ( + (actor_ref_kind in ('legacy_actor', 'actor_profile') and actor_id ~ '{uuid_pattern}') + or (actor_ref_kind = 'system_principal' and actor_id = 'workstream:system:bootstrap') + ) + and ( + target_actor_ref is null + or (target_actor_ref_kind = 'actor_profile' and target_actor_ref ~ '{uuid_pattern}') + ) + and (matched_grant_id is null or matched_grant_id ~ '{uuid_pattern}') + and (project_id is null or project_id ~ '{uuid_pattern}') + and (resource_type is null or resource_type in ({resource_tokens})) + and (resource_id is null or resource_id ~ '{uuid_pattern}') + and ( + target_ref_kind is null + or (target_ref_kind in ({uuid_target_tokens}) and target_ref_id ~ '{uuid_pattern}') + or (target_ref_kind = 'permission_registry' and target_ref_id in ({permission_tokens})) + ) + and ( + invalidation_target_kind is null + or ( + invalidation_target_kind in ({uuid_target_tokens}) + and invalidation_target_ref ~ '{uuid_pattern}' + ) + or ( + invalidation_target_kind = 'permission_registry' + and invalidation_target_ref in ({permission_tokens}) + ) + ) + and ( + entity_type not in ('authorization_decision', 'authority_invalidation') + or entity_id = id + ) + and ( + resource_type <> 'project' or resource_id is null + or (project_id is not null and resource_id = project_id) + ) + ) + """, + ) + op.create_check_constraint( + "fact_bounds", + "audit_events", + """ + event_domain <> 'authority' or ( + (before_facts is null or octet_length(before_facts::text) <= 4096) + and (after_facts is null or octet_length(after_facts::text) <= 4096) + and coalesce( + authority_event_facts_are_safe(event_type, before_facts, after_facts, project_id), + false + ) + ) + """, + ) + op.create_foreign_key( + "fk_audit_events_invalidation_cause", + "audit_events", + "audit_events", + ["invalidation_cause_event_id"], + ["id"], + ) + op.create_check_constraint( + "foundation_shapes", + "audit_events", + """ + event_domain <> 'authority' + or event_type not in ( + 'SensitiveAuthorizationAllowed', 'SensitiveAuthorizationDenied', + 'AuthorityInvalidationRequested', 'AdminRoleGrantIssueDenied', + 'LastAccessAdministratorOperationDenied' + ) + or ( + event_type in ('AdminRoleGrantIssueDenied', 'LastAccessAdministratorOperationDenied') + and denial_code is not null + ) + or ( + event_type = 'SensitiveAuthorizationAllowed' and permission_id is not null + and denial_code is null and invalidation_cause_event_id is null + and invalidation_target_kind is null + ) + or ( + event_type = 'SensitiveAuthorizationDenied' and permission_id is not null + and denial_code is not null and invalidation_cause_event_id is null + and invalidation_target_kind is null and idempotency_reference is null + ) + or ( + event_type = 'AuthorityInvalidationRequested' + and invalidation_cause_event_id is not null + and invalidation_target_kind is not null and denial_code is null + ) + """, + ) + for name, fields in ( + ("ix_audit_events_request_id", ["request_id"]), + ("ix_audit_events_correlation_id", ["correlation_id"]), + ("ix_audit_events_occurred_at", ["occurred_at"]), + ("ix_audit_events_project_id", ["project_id"]), + ("ix_audit_events_actor_ref", ["actor_ref_kind", "actor_id"]), + ): + op.create_index(name, "audit_events", fields) + + op.execute( + """ + create function set_authority_audit_database_time() + returns trigger language plpgsql as $$ + begin + if new.event_domain = 'authority' then + if new.invalidation_cause_event_id is not null and not exists ( + select 1 from audit_events + where id = new.invalidation_cause_event_id and event_domain = 'authority' + ) then + raise exception 'invalid authority invalidation cause' using errcode = '23503'; + end if; + new.occurred_at = statement_timestamp(); + else + new.occurred_at = null; + end if; + return new; + end + $$ + """ + ) + op.execute( + """ + create trigger audit_events_set_authority_time + before insert on audit_events for each row + execute function set_authority_audit_database_time() + """ + ) + op.execute( + """ + create function reject_audit_event_mutation() + returns trigger language plpgsql as $$ + begin + raise exception 'audit events are append-only' using errcode = '55000'; + end + $$ + """ + ) + op.execute( + """ + create trigger audit_events_reject_update_delete + before update or delete on audit_events for each row + execute function reject_audit_event_mutation() + """ + ) + op.execute( + """ + create trigger audit_events_reject_truncate + before truncate on audit_events for each statement + execute function reject_audit_event_mutation() + """ + ) + + +def downgrade() -> None: + """Remove only unused authority-envelope schema while preserving legacy rows.""" + bind = op.get_bind() + bind.execute(sa.text("lock table audit_events in access exclusive mode")) + has_authority = bind.execute( + sa.text("select exists(select 1 from audit_events where event_domain = 'authority')") + ).scalar_one() + if has_authority: + raise RuntimeError("cannot downgrade non-empty authority audit evidence") + + for trigger in ( + "audit_events_reject_truncate", + "audit_events_reject_update_delete", + "audit_events_set_authority_time", + ): + op.execute(f"drop trigger {trigger} on audit_events") + op.execute("drop function reject_audit_event_mutation()") + op.execute("drop function set_authority_audit_database_time()") + for name in ( + "ix_audit_events_actor_ref", + "ix_audit_events_project_id", + "ix_audit_events_occurred_at", + "ix_audit_events_correlation_id", + "ix_audit_events_request_id", + ): + op.drop_index(name, table_name="audit_events") + op.drop_constraint("fk_audit_events_invalidation_cause", "audit_events", type_="foreignkey") + for name in ( + "foundation_shapes", + "fact_bounds", + "authority_privacy_bounds", + "reference_pairs", + "authority_registries", + "authority_tokens", + "domain_shape", + ): + op.drop_constraint(name, "audit_events", type_="check") + op.execute("drop function authority_event_facts_are_safe(text, json, json, text)") + op.execute("drop function authority_grant_facts_are_safe(json, text[], text, boolean, text)") + op.execute("drop function authority_facts_are_safe(json)") + for column in reversed( + ( + "event_version", + "occurred_at", + "actor_ref_kind", + "request_id", + "correlation_id", + "target_actor_ref_kind", + "target_actor_ref", + "matched_grant_id", + "permission_id", + "project_id", + "resource_type", + "resource_id", + "target_ref_kind", + "target_ref_id", + "denial_code", + "idempotency_reference", + "invalidation_cause_event_id", + "invalidation_target_kind", + "invalidation_target_ref", + "before_facts", + "after_facts", + "event_domain", + ) + ): + op.drop_column("audit_events", column) + op.alter_column("audit_events", "external_issuer", existing_type=sa.String(200), nullable=False) + op.alter_column("audit_events", "external_subject", existing_type=sa.String(200), nullable=False) diff --git a/backend/app/modules/audit/repository.py b/backend/app/modules/audit/repository.py index a1d2e75c..1d9a12f2 100644 --- a/backend/app/modules/audit/repository.py +++ b/backend/app/modules/audit/repository.py @@ -2,6 +2,9 @@ from __future__ import annotations +from collections.abc import Sequence + +from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession from app.modules.tasks.models import AuditEvent @@ -27,7 +30,36 @@ async def add_audit_event(self, event: AuditEvent) -> AuditEvent: Returns: Persisted audit event model. """ + if event.event_domain == "authority": + raise ValueError("authority events require the typed audit service") + return await self._persist(event) + + async def _add_validated_authority_event(self, event: AuditEvent) -> AuditEvent: + """Persist an authority event already validated by AuditService.""" + if event.event_domain != "authority": + raise ValueError("expected authority audit event") + return await self._persist(event) + + async def get_authority_event(self, event_id: str) -> AuditEvent | None: + """Return one authority cause visible in the caller transaction.""" + return await self._session.scalar( + select(AuditEvent).where( + AuditEvent.id == event_id, AuditEvent.event_domain == "authority" + ) + ) + + async def _persist(self, event: AuditEvent) -> AuditEvent: + """Flush one event without taking transaction ownership.""" self._session.add(event) await self._session.flush() await self._session.refresh(event) return event + + async def list_audit_events(self, entity_type: str, entity_id: str) -> Sequence[AuditEvent]: + """List events for one entity in database creation order.""" + result = await self._session.execute( + select(AuditEvent) + .where(AuditEvent.entity_type == entity_type, AuditEvent.entity_id == entity_id) + .order_by(AuditEvent.created_at.asc(), AuditEvent.id.asc()) + ) + return result.scalars().all() diff --git a/backend/app/modules/audit/schemas.py b/backend/app/modules/audit/schemas.py new file mode 100644 index 00000000..3433016f --- /dev/null +++ b/backend/app/modules/audit/schemas.py @@ -0,0 +1,383 @@ +"""Typed, privacy-bounded authority audit inputs.""" + +from __future__ import annotations + +from collections.abc import Mapping +from enum import StrEnum +import json +from typing import Annotated, Any, Self +from uuid import UUID + +from pydantic import BaseModel, ConfigDict, Field, model_validator + +_ENTITY_TYPES = frozenset( + { + "actor_profile", + "actor_identity_link", + "admin_role_grant", + "qualification_snapshot", + "project_role_grant", + "authorization_decision", + "authority_invalidation", + } +) +_RESOURCE_TYPES = frozenset( + """actor_profile actor_identity_link admin_role_grant project project_role_grant task + submission review contribution compensation_award compensation_delivery operations + audit_event""".split() +) +_UUID_TARGET_KINDS = frozenset( + {"actor_profile", "actor_identity_link", "admin_role_grant", "qualification_snapshot", "project_role_grant"} +) +_PERMISSIONS = frozenset( + """actor.profile.read_self actor.profile.update_self actor.profile.read_any + actor.profile.suspend actor.profile.reactivate actor.profile.deactivate + actor.identity_link.read actor.identity_link.revoke actor.identity_link.reactivate + actor.service.provision admin_role.read admin_role.grant admin_role.revoke project.create + project.read project.update project.archive project.guide.manage project.effective_policy.manage + project.task.manage project.review_policy.manage project.role_grant.read + project.role_grant.manage task.queue.read task.claim submission.create submission.read_own + submission.read_for_review review.queue.read review.queue.inspect review.claim review.release + review.decline_preference review.decision review.lease.force_release review.chain.read + contribution.read_self contribution.read_project compensation.policy.manage + compensation.adapter_binding.manage compensation.award.read compensation.delivery.reconcile + operations.status.read operations.timer.run operations.reconcile.run operations.outbox.retry + operations.projection.rebuild audit.read audit.export""".split() +) +_DENIAL_CODES = frozenset( + """required_scope_missing unsupported_subject_kind service_actor_not_provisioned + identity_link_revoked actor_suspended actor_deactivated permission_not_granted + scope_not_authorized self_grant_forbidden self_role_revoke_forbidden resource_guard_denied + actor_not_found grant_not_found resource_not_found actor_already_suspended actor_not_suspended + actor_deactivated_terminal last_access_administrator admin_role_grant_exists + project_role_grant_exists identity_link_conflict resource_project_mismatch idempotency_mismatch + invalid_role_scope invalid_project_role qualification_snapshot_invalid""".split() +) +_ADMIN_ROLES = frozenset( + {"access_administrator", "operator", "project_manager", "finance_authority", "audit_authority"} +) +_PROJECT_ROLES = frozenset({"submitter", "reviewer", "both"}) +_FACT_VALUES: dict[str, frozenset[str]] = { + "status": frozenset({"active", "suspended", "deactivated", "revoked", "captured"}), + "subject_kind": frozenset({"human", "service"}), + "provisioning_method": frozenset({"automatic_first_access", "manual_service_provisioning"}), + "role": _ADMIN_ROLES | _PROJECT_ROLES, + "scope_type": frozenset({"system", "project"}), +} + + +class AuthorityEventType(StrEnum): + """Closed authority event tokens from the adopted specification.""" + + ACTOR_PROFILE_PROVISIONED = "ActorProfileProvisioned" + SERVICE_ACTOR_PROVISIONED = "ServiceActorProvisioned" + ACTOR_IDENTITY_LINKED = "ActorIdentityLinked" + ACTOR_IDENTITY_LINK_REVOKED = "ActorIdentityLinkRevoked" + ACTOR_IDENTITY_LINK_REACTIVATED = "ActorIdentityLinkReactivated" + ACTOR_PROFILE_SUSPENDED = "ActorProfileSuspended" + ACTOR_PROFILE_REACTIVATED = "ActorProfileReactivated" + ACTOR_PROFILE_DEACTIVATED = "ActorProfileDeactivated" + INITIAL_ACCESS_ADMIN_BOOTSTRAPPED = "InitialAccessAdministratorBootstrapped" + ADMIN_ROLE_GRANT_ISSUED = "AdminRoleGrantIssued" + ADMIN_ROLE_GRANT_REVOKED = "AdminRoleGrantRevoked" + ADMIN_ROLE_GRANT_ISSUE_DENIED = "AdminRoleGrantIssueDenied" + LAST_ACCESS_ADMIN_OPERATION_DENIED = "LastAccessAdministratorOperationDenied" + PROJECT_ROLE_QUALIFICATION_CAPTURED = "ProjectRoleQualificationSnapshotCaptured" + PROJECT_ROLE_GRANT_ISSUED = "ProjectRoleGrantIssued" + PROJECT_ROLE_GRANT_REPLACED = "ProjectRoleGrantReplaced" + PROJECT_ROLE_GRANT_REVOKED = "ProjectRoleGrantRevoked" + SENSITIVE_AUTHORIZATION_ALLOWED = "SensitiveAuthorizationAllowed" + SENSITIVE_AUTHORIZATION_DENIED = "SensitiveAuthorizationDenied" + AUTHORITY_INVALIDATION_REQUESTED = "AuthorityInvalidationRequested" + + +class ActorReferenceKind(StrEnum): + """Stable namespaces usable before and after canonical actor migration.""" + + LEGACY_ACTOR = "legacy_actor" + ACTOR_PROFILE = "actor_profile" + SYSTEM_PRINCIPAL = "system_principal" + + +_REASONS = { + AuthorityEventType.ACTOR_PROFILE_PROVISIONED: {"automatic_first_access"}, + AuthorityEventType.SERVICE_ACTOR_PROVISIONED: {"manual_service_provisioning"}, + **dict.fromkeys( + ( + AuthorityEventType.ACTOR_IDENTITY_LINKED, + AuthorityEventType.ACTOR_IDENTITY_LINK_REVOKED, + AuthorityEventType.ACTOR_IDENTITY_LINK_REACTIVATED, + ), + {"identity_lifecycle_change"}, + ), + AuthorityEventType.ACTOR_PROFILE_SUSPENDED: {"security_response", "administrative_correction"}, + AuthorityEventType.ACTOR_PROFILE_REACTIVATED: {"administrative_correction"}, + AuthorityEventType.ACTOR_PROFILE_DEACTIVATED: {"security_response", "administrative_correction"}, + AuthorityEventType.INITIAL_ACCESS_ADMIN_BOOTSTRAPPED: {"initial_access_bootstrap"}, + AuthorityEventType.ADMIN_ROLE_GRANT_ISSUED: {"authority_assignment"}, + AuthorityEventType.ADMIN_ROLE_GRANT_REVOKED: {"authority_revocation"}, + AuthorityEventType.ADMIN_ROLE_GRANT_ISSUE_DENIED: {"authorization_policy_denial"}, + AuthorityEventType.LAST_ACCESS_ADMIN_OPERATION_DENIED: {"authorization_policy_denial"}, + AuthorityEventType.PROJECT_ROLE_QUALIFICATION_CAPTURED: {"qualification_evidence_captured"}, + AuthorityEventType.PROJECT_ROLE_GRANT_ISSUED: {"authority_assignment"}, + AuthorityEventType.PROJECT_ROLE_GRANT_REPLACED: {"authority_replacement"}, + AuthorityEventType.PROJECT_ROLE_GRANT_REVOKED: {"authority_revocation"}, + AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED: {"authorization_evaluation"}, + AuthorityEventType.SENSITIVE_AUTHORIZATION_DENIED: {"authorization_evaluation"}, + AuthorityEventType.AUTHORITY_INVALIDATION_REQUESTED: {"authority_state_changed"}, +} + + +def _enum_value(value: object, enum: type[StrEnum]) -> str | None: + raw = value.value if isinstance(value, enum) else value + return raw if isinstance(raw, str) and raw in {item.value for item in enum} else None + + +def _uuid(value: object) -> str | None: + raw = str(value) if isinstance(value, UUID) else value + try: + return raw if isinstance(raw, str) and str(UUID(raw)) == raw else None + except ValueError: + return None + + +def _registered(value: object, values: frozenset[str] | set[str]) -> bool: + return isinstance(value, str) and value in values + + +def _unique_object(pairs: list[tuple[str, object]]) -> dict[str, object]: + value = dict(pairs) + if len(value) != len(pairs): + raise ValueError + return value + + +def _facts(value: object) -> dict[str, object] | None: + if value is None: + return None + if not isinstance(value, Mapping): + return None + data = dict(value) + if len(data) > 8 or not set(data).issubset( + frozenset(_FACT_VALUES) | {"effective", "allowed", "scope_id"} + ): + return None + for key, item in data.items(): + if key in {"effective", "allowed"} and type(item) is not bool: + return None + if key == "scope_id" and _uuid(item) is None: + return None + if key in _FACT_VALUES and (not isinstance(item, str) or item not in _FACT_VALUES[key]): + return None + if len(json.dumps(data, separators=(",", ":"), ensure_ascii=False).encode()) > 4096: + return None + return data + + +def _grant_facts(facts: dict[str, object] | None, roles: frozenset[str], status: str, effective: bool) -> bool: + if facts is None or facts.get("role") not in roles: + return False + scope = facts.get("scope_type") + expected = {"status", "role", "scope_type", "effective"} | ({"scope_id"} if scope == "project" else set()) + return ( + set(facts) == expected + and facts["status"] == status + and facts["effective"] is effective + and scope in {"system", "project"} + and not (facts["role"] in {"access_administrator", "operator"} and scope != "system") + and not (facts["role"] in _PROJECT_ROLES and scope != "project") + ) + + +def _event_facts_valid(event: AuthorityEventType, before: dict[str, object] | None, after: dict[str, object] | None) -> bool: + exact = { + AuthorityEventType.ACTOR_PROFILE_PROVISIONED: (None, {"status": "active", "subject_kind": "human", "provisioning_method": "automatic_first_access"}), + AuthorityEventType.SERVICE_ACTOR_PROVISIONED: (None, {"status": "active", "subject_kind": "service", "provisioning_method": "manual_service_provisioning"}), + AuthorityEventType.ACTOR_IDENTITY_LINK_REVOKED: ({"status": "active"}, {"status": "revoked"}), + AuthorityEventType.ACTOR_IDENTITY_LINK_REACTIVATED: ({"status": "revoked"}, {"status": "active"}), + AuthorityEventType.ACTOR_PROFILE_SUSPENDED: ({"status": "active"}, {"status": "suspended"}), + AuthorityEventType.ACTOR_PROFILE_REACTIVATED: ({"status": "suspended"}, {"status": "active"}), + AuthorityEventType.PROJECT_ROLE_QUALIFICATION_CAPTURED: (None, {"status": "captured"}), + AuthorityEventType.ADMIN_ROLE_GRANT_ISSUE_DENIED: (None, None), + AuthorityEventType.LAST_ACCESS_ADMIN_OPERATION_DENIED: (None, None), + AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED: (None, {"allowed": True}), + AuthorityEventType.SENSITIVE_AUTHORIZATION_DENIED: (None, {"allowed": False}), + AuthorityEventType.AUTHORITY_INVALIDATION_REQUESTED: ({"effective": True}, {"effective": False}), + } + if event in exact: + return (before, after) == exact[event] + if event == AuthorityEventType.ACTOR_IDENTITY_LINKED: + return before is None and after in ({"status": "active", "subject_kind": "human"}, {"status": "active", "subject_kind": "service"}) + if event == AuthorityEventType.ACTOR_PROFILE_DEACTIVATED: + return before in ({"status": "active"}, {"status": "suspended"}) and after == {"status": "deactivated"} + if event == AuthorityEventType.INITIAL_ACCESS_ADMIN_BOOTSTRAPPED: + return before is None and _grant_facts(after, _ADMIN_ROLES, "active", True) and after["role"] == "access_administrator" + roles, action = { + AuthorityEventType.ADMIN_ROLE_GRANT_ISSUED: (_ADMIN_ROLES, "issued"), + AuthorityEventType.ADMIN_ROLE_GRANT_REVOKED: (_ADMIN_ROLES, "revoked"), + AuthorityEventType.PROJECT_ROLE_GRANT_ISSUED: (_PROJECT_ROLES, "issued"), + AuthorityEventType.PROJECT_ROLE_GRANT_REPLACED: (_PROJECT_ROLES, "replaced"), + AuthorityEventType.PROJECT_ROLE_GRANT_REVOKED: (_PROJECT_ROLES, "revoked"), + }[event] + if action == "issued": + return before is None and _grant_facts(after, roles, "active", True) + if action == "revoked": + return ( + _grant_facts(before, roles, "active", True) + and _grant_facts(after, roles, "revoked", False) + and (before["role"], before["scope_type"], before.get("scope_id")) + == (after["role"], after["scope_type"], after.get("scope_id")) + ) + return _grant_facts(before, roles, "active", True) and _grant_facts(after, roles, "active", True) + + +class AuthorityAuditEventInput(BaseModel): + """Validate one authority event without provider claims or request bodies.""" + + model_config = ConfigDict(extra="forbid", strict=True) + + event_id: UUID + event_type: AuthorityEventType + entity_type: str + entity_id: str + actor_ref_kind: ActorReferenceKind + actor_ref: Annotated[str, Field(max_length=120)] + request_id: UUID + correlation_id: UUID + target_actor_ref_kind: ActorReferenceKind | None = None + target_actor_ref: Annotated[str, Field(max_length=120)] | None = None + matched_grant_id: str | None = None + permission_id: str | None = None + project_id: str | None = None + resource_type: str | None = None + resource_id: str | None = None + target_ref_kind: str | None = None + target_ref_id: Annotated[str, Field(max_length=120)] | None = None + reason: str + denial_code: str | None = None + idempotency_reference: UUID | None = None + invalidation_cause_event_id: UUID | None = None + invalidation_target_kind: str | None = None + invalidation_target_ref: Annotated[str, Field(max_length=120)] | None = None + before_facts: dict[str, str | bool] | None = None + after_facts: dict[str, str | bool] | None = None + + @model_validator(mode="before") + @classmethod + def admit_privacy_safe_input(cls, value: object) -> object: + """Reject unsafe input before Pydantic can retain rejected values.""" + try: + admitted = cls._inspect_privacy_safe_input(value) + except Exception: # noqa: BLE001 - hostile Mapping methods are untrusted input + admitted = None + if admitted is None: + raise TypeError("invalid authority audit input") + return admitted + + @classmethod + def _inspect_privacy_safe_input(cls, value: object) -> dict | None: + if not isinstance(value, Mapping): + return None + data = dict(value) + if set(data) - cls.model_fields.keys(): + return None + event_raw = _enum_value(data.get("event_type"), AuthorityEventType) + kind = _enum_value(data.get("actor_ref_kind"), ActorReferenceKind) + event = AuthorityEventType(event_raw) if event_raw else None + actor_ref = data.get("actor_ref") + before_facts = _facts(data.get("before_facts")) + after_facts = _facts(data.get("after_facts")) + uuid_fields = ( + "event_id", "entity_id", "request_id", "correlation_id", "matched_grant_id", + "project_id", "resource_id", "idempotency_reference", "invalidation_cause_event_id", + ) + invalid = ( + event is None + or not _registered(data.get("entity_type"), _ENTITY_TYPES) + or any(data.get(key) is not None and _uuid(data[key]) is None for key in uuid_fields) + or kind is None + or (kind == "system_principal" and actor_ref != "workstream:system:bootstrap") + or (kind != "system_principal" and _uuid(actor_ref) is None) + or data.get("permission_id") is not None and not _registered(data["permission_id"], _PERMISSIONS) + or data.get("denial_code") is not None and not _registered(data["denial_code"], _DENIAL_CODES) + or data.get("resource_type") is not None and not _registered(data["resource_type"], _RESOURCE_TYPES) + or data.get("target_ref_kind") is not None + and not _registered(data["target_ref_kind"], _UUID_TARGET_KINDS | {"permission_registry"}) + or data.get("invalidation_target_kind") is not None + and not _registered(data["invalidation_target_kind"], _UUID_TARGET_KINDS | {"permission_registry"}) + or event is not None and not _registered(data.get("reason"), _REASONS[event]) + or before_facts is None and data.get("before_facts") is not None + or after_facts is None and data.get("after_facts") is not None + ) + for prefix in ("target_ref", "invalidation_target"): + ref_kind = data.get(f"{prefix}_kind") + ref = data.get(f"{prefix}_ref" if prefix == "invalidation_target" else f"{prefix}_id") + invalid |= (ref_kind is None) != (ref is None) + invalid |= _registered(ref_kind, _UUID_TARGET_KINDS) and ref is not None and _uuid(ref) is None + invalid |= ref_kind == "permission_registry" and not _registered(ref, _PERMISSIONS) + target_kind, target_ref = data.get("target_actor_ref_kind"), data.get("target_actor_ref") + invalid |= (target_kind is None) != (target_ref is None) + invalid |= target_kind is not None and ( + _enum_value(target_kind, ActorReferenceKind) != "actor_profile" or _uuid(target_ref) is None + ) + if invalid: + return None + data["before_facts"] = before_facts + data["after_facts"] = after_facts + return data + + @classmethod + def model_validate_json(cls, json_data: str | bytes | bytearray, **kwargs: Any) -> Self: + """Parse JSON without retaining malformed or non-object input.""" + try: + snapshot = ( + str.encode(json_data) + if isinstance(json_data, str) + else memoryview(json_data).tobytes() + ) + value = json.loads(snapshot, object_pairs_hook=_unique_object) + except Exception: # noqa: BLE001 - hostile buffer methods are untrusted input + value = None + if not isinstance(value, Mapping): + raise TypeError("invalid authority audit input") + return super().model_validate_json(json.dumps(value, separators=(",", ":")), **kwargs) + + @model_validator(mode="after") + def validate_shape(self) -> Self: + """Enforce event, reference, fact, and project-scope integrity.""" + if self.resource_id is not None and self.resource_type is None: + raise ValueError("resource ID requires resource type") + if self.entity_type in {"authorization_decision", "authority_invalidation"} and self.entity_id != str(self.event_id): + raise ValueError("decision entity ID must equal event ID") + if self.resource_type == "project" and self.resource_id is not None and self.resource_id != self.project_id: + raise ValueError("project resource must match project scope") + before, after = _facts(self.before_facts), _facts(self.after_facts) + if not _event_facts_valid(self.event_type, before, after): + raise ValueError("invalid authority event facts") + for facts in (before, after): + if facts and "scope_type" in facts: + if facts["scope_type"] == "system" and self.project_id is not None: + raise ValueError("system grant cannot carry project scope") + if facts["scope_type"] == "project" and facts.get("scope_id") != self.project_id: + raise ValueError("grant facts must match project scope") + if before and after and "scope_id" in before and before.get("scope_id") != after.get("scope_id"): + raise ValueError("replacement cannot change project scope") + invalidation = self.invalidation_cause_event_id is not None or self.invalidation_target_kind + if self.event_type == AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED: + if self.permission_id is None or self.denial_code is not None or invalidation: + raise ValueError("invalid allowed authorization evidence") + elif self.event_type == AuthorityEventType.SENSITIVE_AUTHORIZATION_DENIED: + if self.permission_id is None or self.denial_code is None or invalidation or self.idempotency_reference: + raise ValueError("invalid denied authorization evidence") + elif self.event_type in { + AuthorityEventType.ADMIN_ROLE_GRANT_ISSUE_DENIED, + AuthorityEventType.LAST_ACCESS_ADMIN_OPERATION_DENIED, + }: + if self.denial_code is None: + raise ValueError("denied authority operation requires denial code") + elif self.event_type == AuthorityEventType.AUTHORITY_INVALIDATION_REQUESTED: + if self.invalidation_cause_event_id is None or self.invalidation_target_kind is None or self.denial_code: + raise ValueError("invalid authority invalidation evidence") + if self.invalidation_cause_event_id == self.event_id: + raise ValueError("invalidation cannot reference its own event") + return self diff --git a/backend/app/modules/audit/service.py b/backend/app/modules/audit/service.py new file mode 100644 index 00000000..95b28db5 --- /dev/null +++ b/backend/app/modules/audit/service.py @@ -0,0 +1,46 @@ +"""Application service for typed authority audit evidence.""" + +from sqlalchemy.ext.asyncio import AsyncSession + +from app.modules.audit.repository import AuditRepository +from app.modules.audit.schemas import AuthorityAuditEventInput +from app.modules.tasks.models import AuditEvent + + +class AuditService: + """Build and persist privacy-safe authority events in the caller unit of work.""" + + def __init__(self, session: AsyncSession) -> None: + """Bind the shared audit repository to the caller's session.""" + self._repository = AuditRepository(session) + + async def add_authority_event(self, value: AuthorityAuditEventInput) -> AuditEvent: + """Persist one validated authority event without committing its transaction.""" + try: + fields = dict(object.__getattribute__(value, "__dict__")) + except Exception: # noqa: BLE001 - the service boundary accepts no caller diagnostics + fields = None + if fields is None: + raise TypeError("invalid authority audit input") + value = AuthorityAuditEventInput.model_validate(fields) + cause_id = value.invalidation_cause_event_id + if cause_id is not None and await self._repository.get_authority_event(str(cause_id)) is None: + raise ValueError("invalidation cause must be an existing authority event") + fields = value.model_dump(mode="json") + fields["id"] = fields.pop("event_id") + fields["actor_id"] = fields.pop("actor_ref") + event = AuditEvent( + **fields, + from_status=None, + to_status=None, + external_subject=None, + external_issuer=None, + actor_roles=[], + claim_snapshot={}, + auth_source="local_authority", + is_dev_auth=False, + event_payload={}, + event_domain="authority", + event_version=1, + ) + return await self._repository._add_validated_authority_event(event) diff --git a/backend/app/modules/tasks/models.py b/backend/app/modules/tasks/models.py index 19d88d61..64f13704 100644 --- a/backend/app/modules/tasks/models.py +++ b/backend/app/modules/tasks/models.py @@ -18,6 +18,7 @@ String, Text, UniqueConstraint, + Uuid, text, ) from sqlalchemy.orm import Mapped, mapped_column, relationship @@ -453,7 +454,7 @@ class EvidenceItem(Base): class AuditEvent(Base): - """Audit event for actor-attributed task lifecycle changes.""" + """Shared append-only lifecycle and authority audit evidence.""" __tablename__ = "audit_events" @@ -464,8 +465,8 @@ class AuditEvent(Base): from_status: Mapped[str | None] = mapped_column(String(30)) to_status: Mapped[str | None] = mapped_column(String(30)) actor_id: Mapped[str] = mapped_column(String(100), nullable=False, index=True) - external_subject: Mapped[str] = mapped_column(String(200), nullable=False) - external_issuer: Mapped[str] = mapped_column(String(200), nullable=False) + external_subject: Mapped[str | None] = mapped_column(String(200)) + external_issuer: Mapped[str | None] = mapped_column(String(200)) actor_roles: Mapped[list[str]] = mapped_column(JSON, nullable=False, default=list) claim_snapshot: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict) auth_source: Mapped[str] = mapped_column(String(30), nullable=False) @@ -473,3 +474,27 @@ class AuditEvent(Base): reason: Mapped[str | None] = mapped_column(Text) event_payload: Mapped[dict] = mapped_column(JSON, nullable=False, default=dict) created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) + event_domain: Mapped[str] = mapped_column( + String(24), nullable=False, default="legacy_lifecycle", server_default="legacy_lifecycle" + ) + event_version: Mapped[int | None] = mapped_column(Integer) + occurred_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) + actor_ref_kind: Mapped[str | None] = mapped_column(String(32)) + request_id: Mapped[str | None] = mapped_column(Uuid(as_uuid=False)) + correlation_id: Mapped[str | None] = mapped_column(Uuid(as_uuid=False)) + target_actor_ref_kind: Mapped[str | None] = mapped_column(String(32)) + target_actor_ref: Mapped[str | None] = mapped_column(String(100)) + matched_grant_id: Mapped[str | None] = mapped_column(String(100)) + permission_id: Mapped[str | None] = mapped_column(String(120)) + project_id: Mapped[str | None] = mapped_column(String(36)) + resource_type: Mapped[str | None] = mapped_column(String(80)) + resource_id: Mapped[str | None] = mapped_column(String(100)) + target_ref_kind: Mapped[str | None] = mapped_column(String(32)) + target_ref_id: Mapped[str | None] = mapped_column(String(100)) + denial_code: Mapped[str | None] = mapped_column(String(80)) + idempotency_reference: Mapped[str | None] = mapped_column(Uuid(as_uuid=False)) + invalidation_cause_event_id: Mapped[str | None] = mapped_column(String(36)) + invalidation_target_kind: Mapped[str | None] = mapped_column(String(32)) + invalidation_target_ref: Mapped[str | None] = mapped_column(String(100)) + before_facts: Mapped[dict | None] = mapped_column(JSON(none_as_null=True)) + after_facts: Mapped[dict | None] = mapped_column(JSON(none_as_null=True)) diff --git a/backend/app/modules/tasks/repository.py b/backend/app/modules/tasks/repository.py index 983e2e88..57bcbd54 100644 --- a/backend/app/modules/tasks/repository.py +++ b/backend/app/modules/tasks/repository.py @@ -9,6 +9,7 @@ from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.orm import selectinload +from app.modules.audit.repository import AuditRepository from app.modules.tasks.models import ( AuditEvent, EvidenceItem, @@ -28,6 +29,7 @@ def __init__(self, session: AsyncSession) -> None: session: Async SQLAlchemy session for the current unit of work. """ self._session = session + self._audit_repository = AuditRepository(session) async def add_task(self, task: WorkstreamTask) -> WorkstreamTask: """Persist a new task and refresh generated database fields. @@ -207,10 +209,7 @@ async def add_audit_event(self, event: AuditEvent) -> AuditEvent: Returns: Persisted audit event model. """ - self._session.add(event) - await self._session.flush() - await self._session.refresh(event) - return event + return await self._audit_repository.add_audit_event(event) async def list_audit_events(self, entity_type: str, entity_id: str) -> Sequence[AuditEvent]: """List audit events for one entity in creation order. @@ -222,9 +221,4 @@ async def list_audit_events(self, entity_type: str, entity_id: str) -> Sequence[ Returns: Matching audit events ordered by creation time. """ - result = await self._session.execute( - select(AuditEvent) - .where(AuditEvent.entity_type == entity_type, AuditEvent.entity_id == entity_id) - .order_by(AuditEvent.created_at.asc(), AuditEvent.id.asc()) - ) - return result.scalars().all() + return await self._audit_repository.list_audit_events(entity_type, entity_id) diff --git a/backend/tests/test_alembic.py b/backend/tests/test_alembic.py index 646b35f7..ffbac20b 100644 --- a/backend/tests/test_alembic.py +++ b/backend/tests/test_alembic.py @@ -17,6 +17,7 @@ from app.adapters.auth.dev import actor_id_from_external_identity + def test_alembic_upgrade_and_downgrade(isolated_database_env: str, migration_lock) -> None: project_root = Path(__file__).resolve().parents[1] config = Config(str(project_root / "alembic.ini")) @@ -143,7 +144,9 @@ def test_post_submit_policy_upgrade_blocks_pre_provenance_runtime_rows( config = Config(str(project_root / "alembic.ini")) config.set_main_option("script_location", str(project_root / "alembic")) - ids = {name: str(uuid4()) for name in ("project", "guide", "policy", "task", "submission", "run")} + ids = { + name: str(uuid4()) for name in ("project", "guide", "policy", "task", "submission", "run") + } with migration_lock(): try: command.downgrade(config, "base") @@ -287,9 +290,7 @@ def test_artifact_foundation_enforces_immutable_facts_and_guarded_downgrade( command.upgrade(config, "0016_artifact_domain") assert all( count == 0 - for count in asyncio.run( - _artifact_table_counts(isolated_database_env) - ).values() + for count in asyncio.run(_artifact_table_counts(isolated_database_env)).values() ) finally: command.downgrade(config, "base") @@ -313,20 +314,14 @@ def test_api_rate_control_schema_preserves_domain_and_guards_downgrade( command.upgrade(config, "0015_post_submit_correction") asyncio.run(_seed_artifact_prior_head(isolated_database_env, project_id)) command.upgrade(config, "0016_artifact_domain") - before = asyncio.run( - _artifact_prior_head_project(isolated_database_env, project_id) - ) + before = asyncio.run(_artifact_prior_head_project(isolated_database_env, project_id)) artifact_before = asyncio.run( _seed_and_fetch_0016_artifact(isolated_database_env, artifact_id) ) command.upgrade(config, "0017_api_controls") - after = asyncio.run( - _artifact_prior_head_project(isolated_database_env, project_id) - ) - artifact_after = asyncio.run( - _fetch_0016_artifact(isolated_database_env, artifact_id) - ) + after = asyncio.run(_artifact_prior_head_project(isolated_database_env, project_id)) + artifact_after = asyncio.run(_fetch_0016_artifact(isolated_database_env, artifact_id)) schema = asyncio.run(_api_rate_control_schema(isolated_database_env)) asyncio.run(_assert_api_rate_control_guards(isolated_database_env, digest)) @@ -365,9 +360,7 @@ def guarded_downgrade() -> None: refused_state = asyncio.run(_api_rate_control_state(isolated_database_env)) asyncio.run(_clear_api_rate_controls(isolated_database_env)) command.downgrade(config, "0016_artifact_domain") - downgraded_state = asyncio.run( - _api_rate_control_state(isolated_database_env) - ) + downgraded_state = asyncio.run(_api_rate_control_state(isolated_database_env)) command.upgrade(config, "0017_api_controls") finally: asyncio.run(_clear_api_rate_controls(isolated_database_env)) @@ -410,6 +403,111 @@ def guarded_downgrade() -> None: } +def test_authority_audit_schema_preserves_legacy_and_guards_downgrade( + isolated_database_env: str, + migration_lock, +) -> None: + """Prove 0018 preserves legacy evidence and refuses destructive downgrade.""" + project_root = Path(__file__).resolve().parents[1] + config = Config(str(project_root / "alembic.ini")) + config.set_main_option("script_location", str(project_root / "alembic")) + legacy_id = str(uuid4()) + authority_id = str(uuid4()) + + with migration_lock(): + try: + command.downgrade(config, "base") + command.upgrade(config, "0017_api_controls") + before = asyncio.run(_seed_and_fetch_legacy_audit(isolated_database_env, legacy_id)) + + command.upgrade(config, "0018_authority_audit_evidence") + after = asyncio.run(_fetch_audit_row(isolated_database_env, legacy_id)) + schema = asyncio.run(_authority_audit_schema(isolated_database_env)) + occurred_at = asyncio.run( + _insert_authority_audit_fixture(isolated_database_env, authority_id) + ) + + with pytest.raises(RuntimeError, match="non-empty authority audit"): + command.downgrade(config, "0017_api_controls") + refused = asyncio.run(_authority_audit_state(isolated_database_env)) + + asyncio.run(_remove_authority_audit_fixture(isolated_database_env, authority_id)) + command.downgrade(config, "0017_api_controls") + downgraded = asyncio.run(_fetch_audit_row(isolated_database_env, legacy_id)) + command.upgrade(config, "0018_authority_audit_evidence") + restored_schema = asyncio.run(_authority_audit_schema(isolated_database_env)) + finally: + asyncio.run(_remove_authority_audit_fixture(isolated_database_env, authority_id)) + command.downgrade(config, "base") + command.upgrade(config, "head") + + assert after == {**before, "event_domain": "legacy_lifecycle"} + assert downgraded == before + assert occurred_at.year >= 2026 + assert refused == { + "revision": "0018_authority_audit_evidence", + "authority_rows": 1, + } + assert schema == restored_schema + assert schema == { + "columns": { + "actor_ref_kind:varchar:YES", + "after_facts:json:YES", + "before_facts:json:YES", + "correlation_id:uuid:YES", + "denial_code:varchar:YES", + "event_domain:varchar:NO", + "event_version:int4:YES", + "idempotency_reference:uuid:YES", + "invalidation_cause_event_id:varchar:YES", + "invalidation_target_kind:varchar:YES", + "invalidation_target_ref:varchar:YES", + "matched_grant_id:varchar:YES", + "occurred_at:timestamptz:YES", + "permission_id:varchar:YES", + "project_id:varchar:YES", + "request_id:uuid:YES", + "resource_id:varchar:YES", + "resource_type:varchar:YES", + "target_actor_ref:varchar:YES", + "target_actor_ref_kind:varchar:YES", + "target_ref_id:varchar:YES", + "target_ref_kind:varchar:YES", + }, + "constraints": { + "ck_audit_events_authority_privacy_bounds", + "ck_audit_events_authority_registries", + "ck_audit_events_authority_tokens", + "ck_audit_events_domain_shape", + "ck_audit_events_fact_bounds", + "ck_audit_events_foundation_shapes", + "ck_audit_events_reference_pairs", + "fk_audit_events_invalidation_cause", + }, + "indexes": { + "ix_audit_events_actor_ref", + "ix_audit_events_correlation_id", + "ix_audit_events_occurred_at", + "ix_audit_events_project_id", + "ix_audit_events_request_id", + }, + "triggers": { + "audit_events_reject_truncate", + "audit_events_reject_update_delete", + "audit_events_set_authority_time", + }, + "functions": { + "authority_facts_are_safe", + "authority_grant_facts_are_safe", + "authority_event_facts_are_safe", + "reject_audit_event_mutation", + "set_authority_audit_database_time", + }, + "legacy_default": True, + "external_identity_nullable": True, + } + + async def _fetch_columns(database_url: str) -> set[str]: """Return current public table columns as table.column names.""" engine = create_async_engine(database_url) @@ -899,7 +997,9 @@ async def _seed_pre_provenance_runtime_rows(database_url: str, ids: dict[str, st await engine.dispose() -async def _seed_pre_provenance_policies(connection, project_id: str, checker_policy_id: str) -> None: +async def _seed_pre_provenance_policies( + connection, project_id: str, checker_policy_id: str +) -> None: """Seed v0.1 guide policies required by locked task foreign keys.""" await connection.execute( text( @@ -1410,11 +1510,15 @@ async def _artifact_prior_head_project(database_url: str, project_id: str) -> di try: async with engine.connect() as connection: row = ( - await connection.execute( - text("select id, name, slug, status from projects where id = :id"), - {"id": project_id}, + ( + await connection.execute( + text("select id, name, slug, status from projects where id = :id"), + {"id": project_id}, + ) ) - ).mappings().one() + .mappings() + .one() + ) return dict(row) finally: await engine.dispose() @@ -1858,8 +1962,7 @@ async def _api_rate_control_schema(database_url: str) -> dict[str, set[str]]: ).scalars() return { "columns": { - f"{row.column_name}:{row.data_type}:{row.is_nullable}" - for row in columns + f"{row.column_name}:{row.data_type}:{row.is_nullable}" for row in columns }, "constraints": set(constraints), "indexes": set(indexes), @@ -1868,9 +1971,7 @@ async def _api_rate_control_schema(database_url: str) -> dict[str, set[str]]: await engine.dispose() -async def _seed_and_fetch_0016_artifact( - database_url: str, artifact_id: str -) -> dict[str, object]: +async def _seed_and_fetch_0016_artifact(database_url: str, artifact_id: str) -> dict[str, object]: """Seed one representative 0016 row and return its exact persisted value.""" engine = create_async_engine(database_url) try: @@ -1888,22 +1989,24 @@ async def _seed_and_fetch_0016_artifact( return await _fetch_0016_artifact(database_url, artifact_id) -async def _fetch_0016_artifact( - database_url: str, artifact_id: str -) -> dict[str, object]: +async def _fetch_0016_artifact(database_url: str, artifact_id: str) -> dict[str, object]: """Fetch the representative 0016 artifact-domain row.""" engine = create_async_engine(database_url) try: async with engine.begin() as connection: row = ( - await connection.execute( - text( - "select id, sha256, byte_count, media_type, " - "normalized_display_name from artifact_contents where id = :id" - ), - {"id": artifact_id}, + ( + await connection.execute( + text( + "select id, sha256, byte_count, media_type, " + "normalized_display_name from artifact_contents where id = :id" + ), + {"id": artifact_id}, + ) ) - ).mappings().one() + .mappings() + .one() + ) return dict(row) finally: await engine.dispose() @@ -2006,3 +2109,242 @@ async def _clear_api_rate_controls(database_url: str) -> None: await connection.execute(text("delete from api_rate_control_counters")) finally: await engine.dispose() + + +async def _seed_and_fetch_legacy_audit(database_url: str, event_id: str) -> dict[str, object]: + """Insert one prior-head lifecycle event and return its legacy fields.""" + engine = create_async_engine(database_url) + try: + async with engine.begin() as connection: + await connection.execute( + text( + "insert into audit_events " + "(id, entity_type, entity_id, event_type, from_status, to_status, " + "actor_id, external_subject, external_issuer, actor_roles, " + "claim_snapshot, auth_source, is_dev_auth, reason, event_payload) " + "values (:id, 'task', :entity_id, 'task_created', null, 'draft', " + "'legacy-actor', 'opaque-subject', 'https://issuer.example.test', " + "'[\"project_manager\"]'::json, '{\"bounded\": true}'::json, " + "'verified_token', false, 'created', '{\"source\": \"manual\"}'::json)" + ), + {"id": event_id, "entity_id": str(uuid4())}, + ) + finally: + await engine.dispose() + return await _fetch_audit_row(database_url, event_id) + + +async def _fetch_audit_row(database_url: str, event_id: str) -> dict[str, object]: + """Fetch stable legacy fields and the authority domain when it exists.""" + engine = create_async_engine(database_url) + try: + async with engine.begin() as connection: + has_domain = await connection.scalar( + text( + "select exists(select 1 from information_schema.columns " + "where table_name = 'audit_events' and column_name = 'event_domain')" + ) + ) + domain = ", event_domain" if has_domain else "" + row = ( + ( + await connection.execute( + text( + "select id, entity_type, entity_id, event_type, from_status, " + "to_status, actor_id, external_subject, external_issuer, " + "actor_roles, claim_snapshot, auth_source, is_dev_auth, reason, " + f"event_payload{domain} from audit_events where id = :id" + ), + {"id": event_id}, + ) + ) + .mappings() + .one() + ) + return dict(row) + finally: + await engine.dispose() + + +async def _authority_audit_schema(database_url: str) -> dict[str, object]: + """Return the exact 0018 authority-audit schema surface.""" + new_columns = { + "event_domain", + "event_version", + "occurred_at", + "actor_ref_kind", + "request_id", + "correlation_id", + "target_actor_ref_kind", + "target_actor_ref", + "matched_grant_id", + "permission_id", + "project_id", + "resource_type", + "resource_id", + "target_ref_kind", + "target_ref_id", + "denial_code", + "idempotency_reference", + "invalidation_cause_event_id", + "invalidation_target_kind", + "invalidation_target_ref", + "before_facts", + "after_facts", + } + engine = create_async_engine(database_url) + try: + async with engine.begin() as connection: + columns = ( + ( + await connection.execute( + text( + "select column_name, udt_name, is_nullable, column_default " + "from information_schema.columns where table_schema = 'public' " + "and table_name = 'audit_events'" + ) + ) + ) + .mappings() + .all() + ) + by_name = {row["column_name"]: row for row in columns} + constraints = set( + ( + await connection.execute( + text( + "select conname from pg_constraint where " + "conrelid = 'audit_events'::regclass " + "and (conname like 'ck_audit_events_%' or " + "conname = 'fk_audit_events_invalidation_cause')" + ) + ) + ).scalars() + ) + indexes = set( + ( + await connection.execute( + text( + "select indexname from pg_indexes where schemaname = 'public' " + "and tablename = 'audit_events' and indexname in " + "('ix_audit_events_request_id', 'ix_audit_events_correlation_id', " + "'ix_audit_events_occurred_at', 'ix_audit_events_project_id', " + "'ix_audit_events_actor_ref')" + ) + ) + ).scalars() + ) + triggers = set( + ( + await connection.execute( + text( + "select tgname from pg_trigger where " + "tgrelid = 'audit_events'::regclass and not tgisinternal" + ) + ) + ).scalars() + ) + functions = set( + ( + await connection.execute( + text( + "select proname from pg_proc where proname in " + "('authority_facts_are_safe', 'authority_grant_facts_are_safe', " + "'authority_event_facts_are_safe', 'reject_audit_event_mutation', " + "'set_authority_audit_database_time')" + ) + ) + ).scalars() + ) + return { + "columns": { + f"{name}:{by_name[name]['udt_name']}:{by_name[name]['is_nullable']}" + for name in new_columns + }, + "constraints": constraints, + "indexes": indexes, + "triggers": triggers, + "functions": functions, + "legacy_default": "legacy_lifecycle" + in (by_name["event_domain"]["column_default"] or ""), + "external_identity_nullable": all( + by_name[name]["is_nullable"] == "YES" + for name in ("external_subject", "external_issuer") + ), + } + finally: + await engine.dispose() + + +async def _insert_authority_audit_fixture(database_url: str, event_id: str): + """Insert valid authority evidence while proving database-owned time.""" + engine = create_async_engine(database_url) + try: + async with engine.begin() as connection: + return await connection.scalar( + text( + "insert into audit_events " + "(id, entity_type, entity_id, event_type, actor_id, actor_roles, " + "claim_snapshot, auth_source, is_dev_auth, event_payload, event_domain, " + "event_version, occurred_at, actor_ref_kind, request_id, correlation_id, " + "permission_id, reason, after_facts) values (:id, " + "'authorization_decision', :id, 'SensitiveAuthorizationAllowed', " + "'workstream:system:bootstrap', '[]'::json, " + "'{}'::json, 'local_authority', false, '{}'::json, 'authority', 1, " + "'2000-01-01T00:00:00Z', 'system_principal', :request_id, " + ":correlation_id, 'actor.profile.read_any', " + "'authorization_evaluation', '{\"allowed\": true}') returning occurred_at" + ), + { + "id": event_id, + "request_id": str(uuid4()), + "correlation_id": str(uuid4()), + }, + ) + finally: + await engine.dispose() + + +async def _authority_audit_state(database_url: str) -> dict[str, object]: + """Return migration revision and retained authority evidence count.""" + engine = create_async_engine(database_url) + try: + async with engine.begin() as connection: + return { + "revision": await connection.scalar( + text("select version_num from alembic_version") + ), + "authority_rows": await connection.scalar( + text("select count(*) from audit_events where event_domain = 'authority'") + ), + } + finally: + await engine.dispose() + + +async def _remove_authority_audit_fixture(database_url: str, event_id: str) -> None: + """Perform explicit owner-only fixture cleanup under the documented lock.""" + engine = create_async_engine(database_url) + try: + async with engine.begin() as connection: + has_domain = await connection.scalar( + text( + "select exists(select 1 from information_schema.columns " + "where table_name = 'audit_events' and column_name = 'event_domain')" + ) + ) + if not has_domain: + return + await connection.execute(text("lock table audit_events in access exclusive mode")) + await connection.execute( + text("alter table audit_events disable trigger audit_events_reject_update_delete") + ) + await connection.execute( + text("delete from audit_events where id = :id and event_domain = 'authority'"), + {"id": event_id}, + ) + await connection.execute( + text("alter table audit_events enable trigger audit_events_reject_update_delete") + ) + finally: + await engine.dispose() diff --git a/backend/tests/test_audit.py b/backend/tests/test_audit.py new file mode 100644 index 00000000..d8e57d94 --- /dev/null +++ b/backend/tests/test_audit.py @@ -0,0 +1,1105 @@ +from __future__ import annotations + +from collections import UserDict +from collections.abc import Mapping +import json +from pathlib import Path +from types import MappingProxyType +from uuid import UUID, uuid4 +import warnings + +from alembic import command +from alembic.config import Config +from pydantic import ValidationError +import pytest +from sqlalchemy import delete, text +from sqlalchemy.exc import DBAPIError, IntegrityError +from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine + +from app.modules.audit.repository import AuditRepository +from app.modules.audit.schemas import ( + ActorReferenceKind, + AuthorityAuditEventInput, + AuthorityEventType, +) +from app.modules.audit.service import AuditService +from app.modules.tasks.models import AuditEvent + + +def _assert_value_not_retained(value: object, forbidden: str, seen: set[int] | None = None) -> None: + """Traverse public exception state and prove rejected input is absent.""" + seen = seen or set() + if id(value) in seen: + return + seen.add(id(value)) + if isinstance(value, str): + assert forbidden not in value + elif isinstance(value, BaseException): + for item in (value.args, vars(value), value.__cause__, value.__context__): + _assert_value_not_retained(item, forbidden, seen) + elif isinstance(value, Mapping): + for key, item in value.items(): + _assert_value_not_retained(key, forbidden, seen) + _assert_value_not_retained(item, forbidden, seen) + elif isinstance(value, (list, tuple, set)): + for item in value: + _assert_value_not_retained(item, forbidden, seen) + + +@pytest.fixture +def audit_database_env(postgres_database_url: str, migration_lock) -> str: + """Ensure audit tests do not inherit another migration test's schema state.""" + project_root = Path(__file__).resolve().parents[1] + config = Config(str(project_root / "alembic.ini")) + config.set_main_option("script_location", str(project_root / "alembic")) + with migration_lock(): + command.upgrade(config, "head") + return postgres_database_url + + +@pytest.fixture +async def audit_factory(audit_database_env: str): + """Provide independent sessions over the isolated migrated database.""" + engine = create_async_engine(audit_database_env) + async with engine.connect() as connection: + existing = set( + ( + await connection.execute( + text("select id from audit_events where event_domain = 'authority'") + ) + ).scalars() + ) + try: + yield async_sessionmaker(engine, expire_on_commit=False) + finally: + async with engine.begin() as connection: + created = ( + set( + ( + await connection.execute( + text("select id from audit_events where event_domain = 'authority'") + ) + ).scalars() + ) + - existing + ) + if created: + await connection.execute(text("lock table audit_events in access exclusive mode")) + await connection.execute( + text( + "alter table audit_events disable trigger audit_events_reject_update_delete" + ) + ) + await connection.execute(delete(AuditEvent).where(AuditEvent.id.in_(created))) + await connection.execute( + text( + "alter table audit_events enable trigger audit_events_reject_update_delete" + ) + ) + await engine.dispose() + + +def _authority_input(event_type: AuthorityEventType, **overrides) -> AuthorityAuditEventInput: + event_id = overrides.pop("event_id", uuid4()) + defaults = { + AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED: { + "entity_type": "authorization_decision", + "reason": "authorization_evaluation", + "after_facts": {"allowed": True}, + }, + AuthorityEventType.SENSITIVE_AUTHORIZATION_DENIED: { + "entity_type": "authorization_decision", + "reason": "authorization_evaluation", + "after_facts": {"allowed": False}, + }, + AuthorityEventType.AUTHORITY_INVALIDATION_REQUESTED: { + "entity_type": "authority_invalidation", + "reason": "authority_state_changed", + "before_facts": {"effective": True}, + "after_facts": {"effective": False}, + }, + }[event_type] + values = { + "event_id": event_id, + "event_type": event_type, + "entity_id": str(event_id), + "actor_ref_kind": ActorReferenceKind.SYSTEM_PRINCIPAL, + "actor_ref": "workstream:system:bootstrap", + "request_id": uuid4(), + "correlation_id": uuid4(), + "permission_id": "actor.profile.read_any", + **defaults, + } + values.update(overrides) + return AuthorityAuditEventInput(**values) + + +def _authority_event_matrix() -> list[dict]: + """Return every closed event with one valid and one fact-invalid shape.""" + project_id = str(uuid4()) + system_active = { + "status": "active", + "role": "access_administrator", + "scope_type": "system", + "effective": True, + } + project_active = { + "status": "active", + "role": "submitter", + "scope_type": "project", + "scope_id": project_id, + "effective": True, + } + + def row(event_type, entity_type, reason, before, after, invalid_after, **extra): + event_id = uuid4() + return { + "event_id": event_id, + "event_type": event_type, + "entity_type": entity_type, + "entity_id": str(event_id) + if entity_type in {"authorization_decision", "authority_invalidation"} + else str(uuid4()), + "actor_ref_kind": ActorReferenceKind.SYSTEM_PRINCIPAL, + "actor_ref": "workstream:system:bootstrap", + "request_id": uuid4(), + "correlation_id": uuid4(), + "reason": reason, + "before_facts": before, + "after_facts": after, + "invalid_patch": {"after_facts": invalid_after}, + **extra, + } + + rows = [ + row( + AuthorityEventType.ACTOR_PROFILE_PROVISIONED, + "actor_profile", + "automatic_first_access", + None, + { + "status": "active", + "subject_kind": "human", + "provisioning_method": "automatic_first_access", + }, + { + "status": "active", + "subject_kind": "service", + "provisioning_method": "automatic_first_access", + }, + ), + row( + AuthorityEventType.SERVICE_ACTOR_PROVISIONED, + "actor_profile", + "manual_service_provisioning", + None, + { + "status": "active", + "subject_kind": "service", + "provisioning_method": "manual_service_provisioning", + }, + { + "status": "active", + "subject_kind": "human", + "provisioning_method": "manual_service_provisioning", + }, + ), + row( + AuthorityEventType.ACTOR_IDENTITY_LINKED, + "actor_identity_link", + "identity_lifecycle_change", + None, + {"status": "active", "subject_kind": "human"}, + {"status": "revoked", "subject_kind": "human"}, + ), + row( + AuthorityEventType.ACTOR_IDENTITY_LINK_REVOKED, + "actor_identity_link", + "identity_lifecycle_change", + {"status": "active"}, + {"status": "revoked"}, + {"status": "active"}, + ), + row( + AuthorityEventType.ACTOR_IDENTITY_LINK_REACTIVATED, + "actor_identity_link", + "identity_lifecycle_change", + {"status": "revoked"}, + {"status": "active"}, + {"status": "suspended"}, + ), + row( + AuthorityEventType.ACTOR_PROFILE_SUSPENDED, + "actor_profile", + "security_response", + {"status": "active"}, + {"status": "suspended"}, + {"status": "deactivated"}, + ), + row( + AuthorityEventType.ACTOR_PROFILE_REACTIVATED, + "actor_profile", + "administrative_correction", + {"status": "suspended"}, + {"status": "active"}, + {"status": "deactivated"}, + ), + row( + AuthorityEventType.ACTOR_PROFILE_DEACTIVATED, + "actor_profile", + "security_response", + {"status": "active"}, + {"status": "deactivated"}, + {"status": "suspended"}, + ), + row( + AuthorityEventType.INITIAL_ACCESS_ADMIN_BOOTSTRAPPED, + "admin_role_grant", + "initial_access_bootstrap", + None, + system_active, + system_active | {"effective": False}, + ), + row( + AuthorityEventType.ADMIN_ROLE_GRANT_ISSUED, + "admin_role_grant", + "authority_assignment", + None, + system_active, + system_active | {"status": "revoked"}, + ), + row( + AuthorityEventType.ADMIN_ROLE_GRANT_REVOKED, + "admin_role_grant", + "authority_revocation", + system_active, + system_active | {"status": "revoked", "effective": False}, + system_active | {"role": "operator", "status": "revoked", "effective": False}, + ), + row( + AuthorityEventType.ADMIN_ROLE_GRANT_ISSUE_DENIED, + "admin_role_grant", + "authorization_policy_denial", + None, + None, + {"allowed": False}, + denial_code="admin_role_grant_exists", + ), + row( + AuthorityEventType.LAST_ACCESS_ADMIN_OPERATION_DENIED, + "admin_role_grant", + "authorization_policy_denial", + None, + None, + {"allowed": False}, + denial_code="last_access_administrator", + ), + row( + AuthorityEventType.PROJECT_ROLE_QUALIFICATION_CAPTURED, + "qualification_snapshot", + "qualification_evidence_captured", + None, + {"status": "captured"}, + {"status": "active"}, + project_id=project_id, + ), + row( + AuthorityEventType.PROJECT_ROLE_GRANT_ISSUED, + "project_role_grant", + "authority_assignment", + None, + project_active, + { + "status": "active", + "role": "submitter", + "scope_type": "system", + "effective": True, + }, + project_id=project_id, + ), + row( + AuthorityEventType.PROJECT_ROLE_GRANT_REPLACED, + "project_role_grant", + "authority_replacement", + project_active, + project_active | {"role": "reviewer"}, + project_active | {"scope_id": str(uuid4()), "role": "reviewer"}, + project_id=project_id, + ), + row( + AuthorityEventType.PROJECT_ROLE_GRANT_REVOKED, + "project_role_grant", + "authority_revocation", + project_active, + project_active | {"status": "revoked", "effective": False}, + project_active | {"role": "reviewer", "status": "revoked", "effective": False}, + project_id=project_id, + ), + row( + AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED, + "authorization_decision", + "authorization_evaluation", + None, + {"allowed": True}, + {"allowed": False}, + permission_id="actor.profile.read_any", + ), + row( + AuthorityEventType.SENSITIVE_AUTHORIZATION_DENIED, + "authorization_decision", + "authorization_evaluation", + None, + {"allowed": False}, + {"allowed": True}, + permission_id="actor.profile.read_any", + denial_code="actor_suspended", + ), + ] + rows[11]["invalid_patch"] = {"denial_code": None} + rows[12]["invalid_patch"] = {"denial_code": None} + rows.append( + row( + AuthorityEventType.AUTHORITY_INVALIDATION_REQUESTED, + "authority_invalidation", + "authority_state_changed", + {"effective": True}, + {"effective": False}, + {"effective": True}, + invalidation_cause_event_id=rows[0]["event_id"], + invalidation_target_kind="actor_profile", + invalidation_target_ref=str(uuid4()), + ) + ) + return rows + + +def _authority_sql_values(candidate: dict) -> dict: + """Convert one shared matrix row into direct-SQL bind values.""" + value = candidate.copy() + value.pop("invalid_patch", None) + optional = ( + "target_actor_ref_kind", + "target_actor_ref", + "matched_grant_id", + "permission_id", + "project_id", + "resource_type", + "resource_id", + "target_ref_kind", + "target_ref_id", + "denial_code", + "idempotency_reference", + "invalidation_cause_event_id", + "invalidation_target_kind", + "invalidation_target_ref", + ) + return { + **{key: value.get(key) for key in optional}, + "id": str(value["event_id"]), + "entity_type": value["entity_type"], + "entity_id": value["entity_id"], + "event_type": value["event_type"].value, + "actor_ref_kind": value["actor_ref_kind"].value, + "actor_id": value["actor_ref"], + "request_id": str(value["request_id"]), + "correlation_id": str(value["correlation_id"]), + "reason": value["reason"], + "invalidation_cause_event_id": str(value["invalidation_cause_event_id"]) + if value.get("invalidation_cause_event_id") is not None + else None, + "before_facts": json.dumps(value["before_facts"]) + if value.get("before_facts") is not None + else None, + "after_facts": json.dumps(value["after_facts"]) + if value.get("after_facts") is not None + else None, + } + + +def test_authority_input_rejects_unbounded_or_inconsistent_evidence() -> None: + secret = "secret-bearer-value" + + class HostileMapping(Mapping): + def __getitem__(self, key): + raise KeyError(key) + + def __iter__(self): + raise RuntimeError(secret) + + def __len__(self): + return 1 + + payload = _authority_input(AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED).model_dump( + mode="json" + ) | {"raw_token": secret} + constructors = ( + lambda value: AuthorityAuditEventInput(**value), + AuthorityAuditEventInput.model_validate, + ) + mappings = (payload, UserDict(payload), MappingProxyType(payload)) + for constructor in constructors: + for mapping in mappings: + with pytest.raises(TypeError, match="invalid authority audit input") as caught: + constructor(mapping) + _assert_value_not_retained(caught.value, secret) + with pytest.raises(TypeError, match="invalid authority audit input") as caught: + AuthorityAuditEventInput.model_validate_json(json.dumps(payload)) + _assert_value_not_retained(caught.value, secret) + with pytest.raises(TypeError, match="invalid authority audit input") as caught: + AuthorityAuditEventInput.model_validate(HostileMapping()) + _assert_value_not_retained(caught.value, secret) + + safe = _authority_input(AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED).model_dump( + mode="json" + ) + safe_python = _authority_input( + AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED + ).model_dump() + + class ChangingMapping(Mapping): + def __init__(self): + self.iterations = 0 + + def __getitem__(self, key): + return safe_python[key] if key in safe_python else secret + + def __iter__(self): + self.iterations += 1 + keys = ( + safe_python + if self.iterations == 1 + else safe_python | {"raw_token": secret} + ) + return iter(keys) + + def __len__(self): + return len(safe_python) + + class HostileBytearray(bytearray): + def decode(self, *args, **kwargs): + raise RuntimeError(secret) + + class LyingBytearray(bytearray): + def decode(self, *args, **kwargs): + return json.dumps(safe) + + class ChangingFacts(Mapping): + def __init__(self): + self.iterations = 0 + + def __getitem__(self, key): + return True if key == "allowed" else secret + + def __iter__(self): + self.iterations += 1 + return iter(("allowed",) if self.iterations == 1 else ("raw_token",)) + + def __len__(self): + return 1 + + def __repr__(self): + return f"<{secret}>" + + changing = ChangingMapping() + assert AuthorityAuditEventInput.model_validate(changing).event_type == safe_python["event_type"] + assert changing.iterations == 1 + changing_facts = ChangingFacts() + with warnings.catch_warnings(record=True) as caught_warnings: + normalized = AuthorityAuditEventInput.model_validate( + safe_python | {"after_facts": changing_facts} + ) + assert normalized.after_facts == {"allowed": True} + assert changing_facts.iterations == 1 + assert caught_warnings == [] + assert ( + AuthorityAuditEventInput.model_validate_json(json.dumps(safe)).model_dump(mode="json") + == safe + ) + duplicate = json.dumps(safe).replace( + '"allowed": true', + '"allowed": false, "allowed": true, "raw_token": "secret-bearer-value"', + ) + for raw in ( + "{secret-bearer-value", + '"secret-bearer-value"', + '["secret-bearer-value"]', + "0", + "true", + "false", + "null", + duplicate, + ): + for document in (raw, raw.encode(), bytearray(raw.encode())): + with pytest.raises(TypeError, match="invalid authority audit input") as caught: + AuthorityAuditEventInput.model_validate_json(document) + _assert_value_not_retained(caught.value, secret) + for document in (HostileBytearray(b"{}"), LyingBytearray(duplicate.encode())): + with pytest.raises(TypeError, match="invalid authority audit input") as caught: + AuthorityAuditEventInput.model_validate_json(document) + _assert_value_not_retained(caught.value, secret) + + for patch, forbidden in ( + ( + { + "actor_ref_kind": ActorReferenceKind.LEGACY_ACTOR, + "actor_ref": "provider@example.com", + }, + "provider@example.com", + ), + ({"after_facts": {"allowed": "secret-bearer-value"}}, secret), + ({"after_facts": {"allowed": {"secret": secret}}}, secret), + ( + {"after_facts": {"decision_code": "https://issuer.example/private"}}, + "https://issuer.example/private", + ), + ({"reason": "secret-bearer-value"}, secret), + ({"entity_type": "secret-bearer-value"}, secret), + ({"permission_id": [secret]}, secret), + ): + candidate = safe | patch + for constructor in constructors: + with pytest.raises(TypeError, match="invalid authority audit input") as caught: + constructor(candidate) + _assert_value_not_retained(caught.value, forbidden) + with pytest.raises(TypeError, match="invalid authority audit input") as caught: + AuthorityAuditEventInput.model_validate_json(json.dumps(candidate)) + _assert_value_not_retained(caught.value, forbidden) + with pytest.raises(ValidationError, match="resource ID requires"): + _authority_input( + AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED, + resource_id=str(uuid4()), + ) + with pytest.raises(ValidationError, match="invalid denied authorization"): + _authority_input( + AuthorityEventType.SENSITIVE_AUTHORIZATION_DENIED, + denial_code="actor_suspended", + idempotency_reference=uuid4(), + ) + with pytest.raises(ValidationError, match="invalidation evidence"): + _authority_input(AuthorityEventType.AUTHORITY_INVALIDATION_REQUESTED) + event_id = uuid4() + with pytest.raises(ValidationError, match="own event"): + _authority_input( + AuthorityEventType.AUTHORITY_INVALIDATION_REQUESTED, + event_id=event_id, + permission_id=None, + invalidation_cause_event_id=event_id, + invalidation_target_kind="actor_profile", + invalidation_target_ref=str(uuid4()), + ) + + +def test_authority_input_enforces_grant_scope_matrix() -> None: + """Grant evidence cannot contradict role, project, or replacement scope.""" + source = _authority_input(AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED).model_dump() + event_id = uuid4() + project_id = str(uuid4()) + source.update( + event_id=event_id, + event_type=AuthorityEventType.ADMIN_ROLE_GRANT_ISSUED, + entity_type="admin_role_grant", + entity_id=str(uuid4()), + permission_id=None, + reason="authority_assignment", + project_id=project_id, + after_facts={ + "status": "active", + "role": "project_manager", + "scope_type": "project", + "scope_id": project_id, + "effective": True, + }, + ) + assert AuthorityAuditEventInput.model_validate(source).project_id == project_id + + for patch in ( + {"after_facts": source["after_facts"] | {"role": "access_administrator"}}, + {"after_facts": source["after_facts"] | {"scope_id": str(uuid4())}}, + {"project_id": None}, + ): + with pytest.raises(ValidationError, match="facts|scope"): + AuthorityAuditEventInput.model_validate(source | patch) + + replacement = source | { + "event_type": AuthorityEventType.PROJECT_ROLE_GRANT_REPLACED, + "entity_type": "project_role_grant", + "reason": "authority_replacement", + "before_facts": { + "status": "active", + "role": "submitter", + "scope_type": "project", + "scope_id": project_id, + "effective": True, + }, + "after_facts": { + "status": "active", + "role": "reviewer", + "scope_type": "project", + "scope_id": str(uuid4()), + "effective": True, + }, + } + with pytest.raises(ValidationError, match="scope"): + AuthorityAuditEventInput.model_validate(replacement) + with pytest.raises(ValidationError, match="project resource"): + AuthorityAuditEventInput.model_validate( + source + | { + "resource_type": "project", + "resource_id": str(uuid4()), + } + ) + + +async def test_authority_event_matrix_has_typed_direct_sql_parity(audit_factory) -> None: + """Every registered event accepts and rejects the same fact shape at both boundaries.""" + insert = text( + "insert into audit_events " + "(id, entity_type, entity_id, event_type, actor_id, actor_roles, claim_snapshot, " + "auth_source, is_dev_auth, event_payload, event_domain, event_version, actor_ref_kind, " + "request_id, correlation_id, target_actor_ref_kind, target_actor_ref, matched_grant_id, " + "permission_id, project_id, resource_type, resource_id, target_ref_kind, target_ref_id, " + "reason, denial_code, idempotency_reference, invalidation_cause_event_id, " + "invalidation_target_kind, invalidation_target_ref, before_facts, after_facts) values " + "(:id, :entity_type, :entity_id, :event_type, :actor_id, '[]', '{}', " + "'local_authority', false, '{}', 'authority', 1, :actor_ref_kind, :request_id, " + ":correlation_id, :target_actor_ref_kind, :target_actor_ref, :matched_grant_id, " + ":permission_id, :project_id, :resource_type, :resource_id, :target_ref_kind, " + ":target_ref_id, :reason, :denial_code, :idempotency_reference, " + ":invalidation_cause_event_id, :invalidation_target_kind, :invalidation_target_ref, " + "cast(:before_facts as json), cast(:after_facts as json))" + ) + cases = _authority_event_matrix() + assert {case["event_type"] for case in cases} == set(AuthorityEventType) + + async with audit_factory() as session: + for case in cases: + candidate = {key: value for key, value in case.items() if key != "invalid_patch"} + assert ( + AuthorityAuditEventInput.model_validate(candidate).event_type == case["event_type"] + ) + await session.execute(insert, _authority_sql_values(candidate)) + await session.commit() + + for case in cases: + event_id = uuid4() + candidate = { + **{key: value for key, value in case.items() if key != "invalid_patch"}, + **case["invalid_patch"], + "event_id": event_id, + } + if candidate["entity_type"] in {"authorization_decision", "authority_invalidation"}: + candidate["entity_id"] = str(event_id) + with pytest.raises(ValidationError): + AuthorityAuditEventInput.model_validate(candidate) + with pytest.raises(IntegrityError): + await session.execute(insert, _authority_sql_values(candidate)) + await session.rollback() + + +async def test_authority_service_readmits_mutated_inputs_without_retention(audit_factory) -> None: + """The service never forwards post-validation mutations to SQLAlchemy.""" + secret = "secret-bearer-value" + + class HostileFacts(Mapping): + def __getitem__(self, key): + raise KeyError(key) + + def __iter__(self): + raise RuntimeError(secret) + + def __len__(self): + return 1 + + def __repr__(self): + return f"<{secret}>" + + class ChangingFacts(Mapping): + def __init__(self): + self.iterations = 0 + + def __getitem__(self, key): + return True if key == "allowed" else secret + + def __iter__(self): + self.iterations += 1 + return iter(("allowed",) if self.iterations == 1 else ("raw_token",)) + + def __len__(self): + return 1 + + def __repr__(self): + return f"<{secret}>" + + values = [ + _authority_input(AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED), + _authority_input(AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED), + _authority_input(AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED), + ] + values[0].actor_ref = secret + values[1].after_facts["allowed"] = secret + values[2].after_facts = HostileFacts() + valid_hostile_repr = _authority_input(AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED) + changing_facts = ChangingFacts() + valid_hostile_repr.after_facts = changing_facts + + async with audit_factory() as session: + for value in values: + with warnings.catch_warnings(record=True) as caught_warnings: + with pytest.raises(TypeError, match="invalid authority audit input") as caught: + await AuditService(session).add_authority_event(value) + _assert_value_not_retained(caught.value, secret) + assert caught_warnings == [] + assert await session.get(AuditEvent, str(value.event_id)) is None + with warnings.catch_warnings(record=True) as caught_warnings: + stored = await AuditService(session).add_authority_event(valid_hostile_repr) + assert stored.after_facts == {"allowed": True} + assert changing_facts.iterations == 1 + assert caught_warnings == [] + + +async def test_authority_writer_persists_typed_privacy_neutral_events(audit_factory) -> None: + future_idempotency = uuid4() + project_id = str(uuid4()) + allowed = _authority_input( + AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED, + project_id=project_id, + resource_type="project", + resource_id=project_id, + ) + denied = _authority_input( + AuthorityEventType.SENSITIVE_AUTHORIZATION_DENIED, + denial_code="permission_not_granted", + ) + invalidation = _authority_input( + AuthorityEventType.AUTHORITY_INVALIDATION_REQUESTED, + permission_id=None, + invalidation_cause_event_id=allowed.event_id, + invalidation_target_kind="actor_profile", + invalidation_target_ref=str(uuid4()), + idempotency_reference=future_idempotency, + ) + + async with audit_factory() as session: + service = AuditService(session) + stored = [ + await service.add_authority_event(value) for value in (allowed, denied, invalidation) + ] + await session.commit() + + assert [event.event_type for event in stored] == [ + "SensitiveAuthorizationAllowed", + "SensitiveAuthorizationDenied", + "AuthorityInvalidationRequested", + ] + assert all(event.occurred_at is not None for event in stored) + for event in stored: + assert event.event_domain == "authority" + assert event.event_version == 1 + assert event.external_subject is None + assert event.external_issuer is None + assert event.actor_roles == [] + assert event.claim_snapshot == {} + assert event.event_payload == {} + assert event.auth_source == "local_authority" + assert event.is_dev_auth is False + assert stored[0].idempotency_reference is None + assert UUID(stored[2].idempotency_reference) == future_idempotency + + +async def test_legacy_writer_and_reader_remain_compatible(audit_factory) -> None: + event = AuditEvent( + id=str(uuid4()), + entity_type="task", + entity_id=str(uuid4()), + event_type="task_created", + from_status=None, + to_status="draft", + actor_id="legacy-actor", + external_subject="opaque-subject", + external_issuer="https://issuer.example.test", + actor_roles=["project_manager"], + claim_snapshot={"bounded": True}, + auth_source="verified_token", + is_dev_auth=False, + event_payload={"source_type": "manual"}, + ) + async with audit_factory() as session: + repository = AuditRepository(session) + stored = await repository.add_audit_event(event) + await session.commit() + listed = await repository.list_audit_events("task", event.entity_id) + + assert listed == [stored] + assert stored.event_domain == "legacy_lifecycle" + assert stored.event_version is None + assert stored.occurred_at is None + assert stored.event_payload == {"source_type": "manual"} + + +async def test_authority_event_rollback_leaves_no_row(audit_factory) -> None: + value = _authority_input(AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED) + async with audit_factory() as session: + await AuditService(session).add_authority_event(value) + await session.rollback() + async with audit_factory() as session: + assert await session.get(AuditEvent, str(value.event_id)) is None + + +async def test_authority_invalidation_requires_an_existing_authority_cause(audit_factory) -> None: + legacy_id = str(uuid4()) + invalidation = _authority_input( + AuthorityEventType.AUTHORITY_INVALIDATION_REQUESTED, + permission_id=None, + invalidation_cause_event_id=uuid4(), + invalidation_target_kind="actor_profile", + invalidation_target_ref=str(uuid4()), + ) + async with audit_factory() as session: + with pytest.raises(ValueError, match="existing authority event"): + await AuditService(session).add_authority_event(invalidation) + await AuditRepository(session).add_audit_event( + AuditEvent( + id=legacy_id, + entity_type="task", + entity_id=str(uuid4()), + event_type="task_created", + actor_id="legacy", + external_subject="opaque", + external_issuer="https://issuer.test", + actor_roles=[], + claim_snapshot={}, + auth_source="verified_token", + is_dev_auth=False, + event_payload={}, + ) + ) + invalidation.invalidation_cause_event_id = UUID(legacy_id) + with pytest.raises(ValueError, match="existing authority event"): + await AuditService(session).add_authority_event(invalidation) + assert await session.get(AuditEvent, str(invalidation.event_id)) is None + + +async def test_legacy_repository_rejects_unvalidated_authority_rows(audit_factory) -> None: + raw = AuditEvent( + id=str(uuid4()), + entity_type="actor", + entity_id=str(uuid4()), + event_type="SensitiveAuthorizationAllowed", + actor_id="provider@example.com", + actor_roles=[], + claim_snapshot={}, + auth_source="local_authority", + is_dev_auth=False, + event_payload={}, + event_domain="authority", + ) + async with audit_factory() as session: + with pytest.raises(ValueError, match="typed audit service"): + await AuditRepository(session).add_audit_event(raw) + assert raw not in session + + +async def test_database_rejects_malformed_and_mutated_audit_rows(audit_factory) -> None: + value = _authority_input(AuthorityEventType.SENSITIVE_AUTHORIZATION_ALLOWED) + legacy_id = str(uuid4()) + async with audit_factory() as session: + await AuditService(session).add_authority_event(value) + await AuditRepository(session).add_audit_event( + AuditEvent( + id=legacy_id, + entity_type="task", + entity_id=str(uuid4()), + event_type="task_created", + actor_id="legacy-actor", + external_subject="opaque-subject", + external_issuer="https://issuer.example.test", + actor_roles=[], + claim_snapshot={}, + auth_source="verified_token", + is_dev_auth=False, + event_payload={}, + ) + ) + await session.commit() + + async with audit_factory() as session: + for event_id in (str(value.event_id), legacy_id): + with pytest.raises(DBAPIError, match="append-only"): + await session.execute( + text("update audit_events set reason = 'changed' where id = :id"), + {"id": event_id}, + ) + await session.rollback() + with pytest.raises(DBAPIError, match="append-only"): + await session.execute( + text("delete from audit_events where id = :id"), + {"id": event_id}, + ) + await session.rollback() + with pytest.raises(DBAPIError, match="append-only"): + await session.execute(text("truncate table audit_events cascade")) + await session.rollback() + rows = ( + await session.execute( + text( + "select id, reason, event_domain from audit_events " + "where id in (:authority_id, :legacy_id) order by id" + ), + {"authority_id": str(value.event_id), "legacy_id": legacy_id}, + ) + ).all() + assert {(row.id, row.reason, row.event_domain) for row in rows} == { + (str(value.event_id), "authorization_evaluation", "authority"), + (legacy_id, None, "legacy_lifecycle"), + } + + with pytest.raises(IntegrityError): + await session.execute( + text( + "insert into audit_events " + "(id, entity_type, entity_id, event_type, actor_id, actor_roles, " + "claim_snapshot, auth_source, is_dev_auth, event_payload, event_domain) " + "values (:id, 'actor', :entity_id, 'SensitiveAuthorizationAllowed', " + "'legacy', '[]', '{}', 'local_authority', false, '{}', 'authority')" + ), + {"id": str(uuid4()), "entity_id": str(uuid4())}, + ) + await session.rollback() + insert = text( + "insert into audit_events " + "(id, entity_type, entity_id, event_type, actor_id, actor_roles, claim_snapshot, " + "auth_source, is_dev_auth, event_payload, event_domain, event_version, actor_ref_kind, " + "request_id, correlation_id, permission_id, reason, after_facts) values " + "(:id, :entity_type, :id, 'SensitiveAuthorizationAllowed', :actor_id, '[]', '{}', " + "'local_authority', false, '{}', 'authority', 1, :actor_kind, :request_id, " + ":correlation_id, :permission_id, :reason, cast(:facts as json))" + ) + for patch in ( + {"actor_kind": "legacy_actor", "actor_id": "provider@example.com"}, + {"facts": '{"decision_code":"https://issuer.test"}'}, + {"facts": '{"allowed":"secret-bearer-value"}'}, + {"facts": '{"allowed":false,"allowed":true}'}, + {"entity_type": "provider@example.com"}, + {"id": "secret-bearer-value"}, + {"permission_id": "secret-bearer-value"}, + {"reason": "secret-bearer-value"}, + ): + with pytest.raises(IntegrityError): + values = { + "id": str(uuid4()), + "actor_id": "workstream:system:bootstrap", + "actor_kind": "system_principal", + "request_id": str(uuid4()), + "correlation_id": str(uuid4()), + "entity_type": "authorization_decision", + "permission_id": "actor.profile.read_any", + "reason": "authorization_evaluation", + "facts": '{"allowed":true}', + } + values.update(patch) + await session.execute( + insert, + values, + ) + await session.rollback() + grant_insert = text( + "insert into audit_events " + "(id, entity_type, entity_id, event_type, actor_id, actor_roles, claim_snapshot, " + "auth_source, is_dev_auth, event_payload, event_domain, event_version, actor_ref_kind, " + "request_id, correlation_id, project_id, resource_type, resource_id, reason, " + "after_facts) values " + "(:id, 'admin_role_grant', :entity_id, 'AdminRoleGrantIssued', " + "'workstream:system:bootstrap', '[]', '{}', 'local_authority', false, '{}', " + "'authority', 1, 'system_principal', :request_id, :correlation_id, :project_id, " + ":resource_type, :resource_id, 'authority_assignment', cast(:facts as json))" + ) + project_id = str(uuid4()) + + def grant_values(facts: str, *, scope_project_id: str | None = project_id) -> dict: + return { + "id": str(uuid4()), + "entity_id": str(uuid4()), + "request_id": str(uuid4()), + "correlation_id": str(uuid4()), + "project_id": scope_project_id, + "resource_type": "project" if scope_project_id is not None else None, + "resource_id": scope_project_id, + "facts": facts, + } + + valid_facts = json.dumps( + { + "status": "active", + "role": "project_manager", + "scope_type": "project", + "scope_id": project_id, + "effective": True, + } + ) + await session.execute(grant_insert, grant_values(valid_facts)) + system_facts = json.dumps( + { + "status": "active", + "role": "access_administrator", + "scope_type": "system", + "effective": True, + } + ) + await session.execute(grant_insert, grant_values(system_facts, scope_project_id=None)) + await session.commit() + for values in ( + grant_values(valid_facts.replace(project_id, str(uuid4()))), + grant_values(valid_facts) | {"resource_id": str(uuid4())}, + grant_values(valid_facts.replace("project_manager", "access_administrator")), + grant_values( + json.dumps( + { + "status": "active", + "role": "project_manager", + "scope_type": "system", + "effective": True, + } + ) + ), + ): + with pytest.raises(IntegrityError): + await session.execute(grant_insert, values) + await session.rollback() + invalidation_insert = text( + "insert into audit_events (id, entity_type, entity_id, event_type, actor_id, " + "actor_roles, claim_snapshot, auth_source, is_dev_auth, event_payload, event_domain, " + "event_version, actor_ref_kind, request_id, correlation_id, invalidation_cause_event_id, " + "invalidation_target_kind, invalidation_target_ref, reason, before_facts, after_facts) " + "values (:id, 'authority_invalidation', :id, " + "'AuthorityInvalidationRequested', 'workstream:system:bootstrap', '[]', '{}', " + "'local_authority', false, '{}', 'authority', 1, 'system_principal', :request_id, " + ":correlation_id, :cause_id, 'actor_profile', :target_ref, " + "'authority_state_changed', '{\"effective\": true}', '{\"effective\": false}')" + ) + for cause_id in (str(uuid4()), legacy_id): + with pytest.raises(IntegrityError): + await session.execute( + invalidation_insert, + { + "id": str(uuid4()), + "request_id": str(uuid4()), + "correlation_id": str(uuid4()), + "cause_id": cause_id, + "target_ref": str(uuid4()), + }, + ) + await session.rollback() + self_id = str(uuid4()) + with pytest.raises(IntegrityError): + await session.execute( + invalidation_insert, + { + "id": self_id, + "request_id": str(uuid4()), + "correlation_id": str(uuid4()), + "cause_id": self_id, + "target_ref": str(uuid4()), + }, + ) + await session.rollback() diff --git a/backend/tests/test_tasks.py b/backend/tests/test_tasks.py index 4701faef..89f70cd5 100644 --- a/backend/tests/test_tasks.py +++ b/backend/tests/test_tasks.py @@ -7,15 +7,17 @@ from datetime import UTC, datetime, timedelta from decimal import Decimal from pathlib import Path +from unittest.mock import AsyncMock, MagicMock from uuid import uuid4 import pytest from alembic import command from alembic.config import Config from httpx import ASGITransport, AsyncClient -from sqlalchemy import inspect, select +from sqlalchemy import inspect, select, text from sqlalchemy.dialects import postgresql from sqlalchemy.exc import IntegrityError +from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.schema import CreateIndex from app.adapters.auth.dev import actor_id_from_external_identity @@ -57,6 +59,47 @@ from app.schemas.auth import ActorContext +async def test_task_repository_delegates_audit_persistence() -> None: + """Keep legacy task audit methods as same-session shared-writer adapters.""" + session = MagicMock() + repository = TaskRepository(session) + event = MagicMock(spec=AuditEvent) + persisted = MagicMock(spec=AuditEvent) + listed = [persisted] + repository._audit_repository.add_audit_event = AsyncMock(return_value=persisted) + repository._audit_repository.list_audit_events = AsyncMock(return_value=listed) + + assert repository._audit_repository._session is session + assert await repository.add_audit_event(event) is persisted + assert await repository.list_audit_events("task", "task-1") is listed + repository._audit_repository.add_audit_event.assert_awaited_once_with(event) + repository._audit_repository.list_audit_events.assert_awaited_once_with( + "task", "task-1" + ) + + +async def delete_audit_fixture_as_owner(session: AsyncSession, event_id: str) -> None: + """Construct missing-evidence corruption under explicit test-owner custody.""" + await session.execute(text("lock table audit_events in access exclusive mode")) + await session.execute( + text( + "alter table audit_events disable trigger " + "audit_events_reject_update_delete" + ) + ) + await session.execute( + text("delete from audit_events where id = :event_id"), + {"event_id": event_id}, + ) + await session.execute( + text( + "alter table audit_events enable trigger " + "audit_events_reject_update_delete" + ) + ) + await session.commit() + + @pytest.fixture def task_database_env( monkeypatch: pytest.MonkeyPatch, @@ -4821,8 +4864,7 @@ async def test_manual_checker_run_cannot_bypass_failed_automatic_gate( ) assert queued_run is not None assert lock_audit is not None - await session.delete(lock_audit) - await session.commit() + await delete_audit_fixture_as_owner(session, lock_audit.id) with pytest.raises(CheckerExecutionBlocked): run_pre_review_gate.run( @@ -5075,8 +5117,7 @@ async def test_queued_gate_fails_closed_when_lock_audit_is_missing( ) assert queued_run is not None assert lock_audit is not None - await session.delete(lock_audit) - await session.commit() + await delete_audit_fixture_as_owner(session, lock_audit.id) with pytest.raises(CheckerExecutionBlocked): run_pre_review_gate.run( diff --git a/docs/architecture_data_model.md b/docs/architecture_data_model.md index 07c11827..1223479f 100644 --- a/docs/architecture_data_model.md +++ b/docs/architecture_data_model.md @@ -1497,16 +1497,26 @@ Fields: Audit events are append-only. -`actor_roles` and `claim_snapshot` are legacy-only columns. New authority events -must leave them empty and use only the bounded actor, matched grant/permission, -scope, resource, reason, and before/after fields. Raw claims and unnecessary -profile data are prohibited. +Authority evidence extends the same row with `event_domain`, `event_version`, +database-owned `occurred_at`, request/correlation IDs, an actor-reference +namespace, and optional bounded target actor, matched grant, permission, +project, resource, denial, invalidation, idempotency-reference, and shallow +before/after fact fields. It does not create a parallel event table. + +`actor_roles`, `claim_snapshot`, and `event_payload` are legacy-only data +surfaces. Authority rows keep them at `[]`, `{}`, and `{}` respectively, use +`auth_source = local_authority`, and leave external issuer, external subject, +and lifecycle status fields null. They never store tokens, claims, emails, +request bodies, issuer URLs, key material, or policy bodies. Typed validation +and database constraints reject mixed legacy/authority shapes. v0.1 audit storage is the existing Workstream `audit_events` ledger. Task Lifecycle events and authority events share the canonical audit repository so operators can reconstruct why an actor was allowed or denied. Authority events record bounded actor, matched grant/permission, scope, resource, reason, and -before/after facts without raw claims or unnecessary profile data. +before/after facts without raw claims or unnecessary profile data. The shared +repository participates in its caller's transaction and does not commit or +open an independent session. ## Required Invariants diff --git a/docs/operations_authorization_service.md b/docs/operations_authorization_service.md index 3b9915c4..60dfd9e0 100644 --- a/docs/operations_authorization_service.md +++ b/docs/operations_authorization_service.md @@ -440,6 +440,37 @@ metrics/configuration evidence for the normal security incident period. Never use token, subject, email, `jti`, raw URL, key material, or unbounded resource IDs as metric labels. +## Authority Audit Custody + +Authority decisions and invalidation requests use the shared `audit_events` +ledger. Database triggers reject normal `UPDATE`, `DELETE`, and `TRUNCATE` for +both lifecycle and authority rows. There is no runtime flag, session setting, +or application-user bypass. Authority timestamps come from the database, and +the application writer joins the caller's transaction so evidence cannot +commit separately from the authority mutation it explains. + +Production must run Workstream with a dedicated non-owner database role that +has no DDL, trigger-management, table-owner, superuser, or bypass privileges. +Release verification must fail if the application credential owns +`audit_events` or can disable its triggers. Schema migration credentials are +separate and unavailable to the running service. + +Owner-level maintenance is exceptional and requires an approved change record, +an outage or writer drain, and an explicit list of rows and purpose. The +database owner must: + +1. begin one transaction and take `ACCESS EXCLUSIVE` lock on `audit_events`; +2. record the pre-change row count and trigger state without exporting private data; +3. disable only the named mutation trigger needed for the approved operation; +4. perform only the listed maintenance statements and verify affected IDs/counts; +5. re-enable the trigger before commit and prove all three audit triggers enabled; +6. retain redacted change evidence and return credentials to controlled storage. + +Do not use owner maintenance to revise authority history, erase a denial, or +fabricate evidence. Normal migration downgrade refuses while authority rows +exist; destructive cleanup requires a separately reviewed retention or legal +procedure and is not an application operation. + ## Incident Response For suspected authority misuse: