Skip to content

perf(perm-compile): read only the profile's catalogs' grants and rules - #845

Merged
viethqb merged 1 commit into
mainfrom
perf/opa-load-ep-scoped
Nov 8, 2026
Merged

viethqb merged 1 commit into
mainfrom
perf/opa-load-ep-scoped

Conversation

@viethqb

@viethqb viethqb commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Summary

This is the first Task 26 follow-up: a profile's compile and its OPA rebuild now read only the grants and rules that profile can lower.

The cost (measured first).

  • ir_load::load_ep read every grant and every rule in the installation, on every compile and every OPA rebuild. The lowerings then skipped every grant and path rule on a catalog the profile does not reach.
  • Timed alone in the shared-tag harness, load_ep was 55.9 ms of the 61.2 ms snapshot_parts_via at N = 10, and it grew with the grants read (820 × N).
  • So a refresh moving N profiles cost O(N²).

Changes

  • egret-perm gains two reads:

    • list_grants_for_catalogs_via: the grants on the given catalogs;
    • list_rules_for_catalogs_via: every tag-selected rule, plus the path rules on the given catalogs.

    They share row parsing with list_grants_via and list_rules_via, which are unchanged and still serve the API listings.

  • load_ep reads through them, using the catalogs the profile reaches (catalogs_for_service_via, already read just before).

  • Compiled output is unchanged.

    • lower_grants and lower_rules already skip every row on another catalog.
    • A tag rule names no catalog, so every tag rule is still read.
    • No other consumer of the loaded rows looks at an out-of-scope row.
  • Side effect: fewer usernames minted. Principals are now expanded for these rows only, so a compile mints a username only for someone its own policies name. premint_model_usernames still sweeps the whole model, and its doc comment now says it covers a superset of what each compile reads.

  • Harness:

    • opa_shared_tag_service_refresh_latency now times load_ep alone and prints how many grants it read;
    • new helper harness_support::load_ep_model_ids_for_test.
  • Report: docs/en/reports/2026-11-06-gov-sp2b-opa-latency.md gains the section "Follow-up 2026-11-09: the loader reads only the profile's catalogs".

Results

Shared-tag refresh, release build, 10 runs per N, p50 in ms:

N one rebuild (before → after) last profile current (before → after) grants read (before → after)
1 60.9 → 49.8 60.9 → 49.8 820 → 820
5 79.7 → 50.9 403.3 → 257.5 4 100 → 820
10 94.8 → 47.4 954.2 → 493.0 (max 541.8) 8 200 → 820
20 150.0 → 46.9 3112.8 → 971.2 16 400 → 820
  • One rebuild is now flat, so the refresh is linear in N.
  • N = 10 is inside the 1 s threshold, including its max.

Test plan

  • New test the_loader_reads_only_the_grants_and_rules_its_catalogs_reach (tests/metadata_tables.rs). It sets up two Trino profiles, each on its own catalog, with a grant and a path rule on each catalog and one tag rule. It checks that each profile reads its own grant and path rule and the tag rule, and neither of the other profile's rows.
  • Mutants, each failing that test:
    • M1, load_ep back on the installation-wide reads: fails at "not the other catalog's grant";
    • M2, the tag clause dropped from the rule read: fails at "every tag rule".
  • Suites:
    • cargo test -p egret-perm: 69 passed;
    • cargo test -p egret-perm-compile --features harness: 1019 passed, 9 ignored, including golden_compile on Ranger;
    • golden_compile with EGRET_GOLDEN_MECHANISM=opa: passed.
  • cargo fmt --check and clippy -D warnings on both crates (all targets, with harness) are clean.

Documentation

egret-data/docs#72 updates the "Shared tag service" note on the Trino on OPA page.

Checklist

  • Linked to an issue (no tracking issue)
  • Tests added or updated
  • Docs updated in egret-data/docs
  • Commit messages follow Conventional Commits
  • No secrets, tokens or real data added

🤖 Generated with Claude Code

`ir_load::load_ep` read every grant and rule in the installation on each
profile's compile and OPA rebuild, though the lowerings skip every grant
and path rule on a catalog the profile does not reach. With N profiles
moving together that cost O(N^2): measured, `load_ep` was 56 of 61 ms of
`snapshot_parts_via` at N = 10 and grew with the grants read.

It now reads the grants and path rules on the profile's reachable
catalogs plus every tag rule (`list_grants_for_catalogs_via`,
`list_rules_for_catalogs_via`). One rebuild is flat at about 47 ms; the
last of 10 profiles sharing a tag service goes current in 493 ms p50
(954 ms before), 20 in 971 ms (3113 ms).

Principals are expanded for those rows only, so a compile mints a
username only for someone its own policies name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@viethqb
viethqb merged commit 26016f9 into main Nov 8, 2026
9 checks passed
@viethqb
viethqb deleted the perf/opa-load-ep-scoped branch November 8, 2026 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant