feat(privacy): add quick privacy check to run, bench all generators

.github/workflows/bench.yml

@@ -81,6 +81,8 @@ jobs:
         with: { persist-credentials: false }
       - uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
         with: { python-version: "3.12" }
+      # bench imports relsynth (relsynth._http since #50) and defusedxml (the bench extra)
+      - run: pip install -e ".[bench]"
       - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
         # one folder per job: parallel jobs can write records of the same name
         with: { pattern: bench-history-*, path: bench-out }
@@ -103,6 +105,8 @@ jobs:
         with: { persist-credentials: false }
       - uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
         with: { python-version: "3.12" }
+      # bench imports relsynth (relsynth._http since #50) and defusedxml (the bench extra)
+      - run: pip install -e ".[bench]"
       - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
         with:
           pattern: bench-history-*

CHANGELOG.md

@@ -5,13 +5,16 @@ All notable user-visible changes are recorded here, following [Keep a Changelog
 ## Unreleased
 ### Added
 
+- **`relsynth run` ends with a quick privacy check by default** (G5, T06 decision 4; not in DP mode, and skipped without a privacy unit of at least 50 people or when the unit's tables were fitted by pushdown). It fits the spec's generator on a third of up to 3000 people of the privacy unit and runs two membership attacks against people the proxy never saw: DOMIAS and a population-calibrated closest-distance attack, which catches copied people that DOMIAS's kernel density misses. One line is printed (`privacy check: PASS - membership DOMIAS AUC ... vs limit 0.65 (internal) ...`) and the result goes into the diagnostics bundle and the `--json` output as `privacy_check`; the exit code and the scorecard are unchanged. The check runs after every file is written on its own seeded stream, so the synthetic files are byte-identical with and without it (Props. 1740-1741, checks C1740-C1741). Turn it off with `--no-privacy-check`, `rs.run(..., privacy_check=False)` or `synthesis.evaluation.privacy_check: false`; `synthesis.evaluation.privacy_check_people` sets the sample. See [docs/privacy.md](docs/privacy.md#the-quick-check-in-every-relsynth-run).
+- **The benchmark attacks every built-in generator.** The privacy suite of ACS and Berka now runs once per generator plug-in (copula, cart, independent) with the same settings; the rows of the other generators carry the generator in their ids (`privacy[cart]:...`, `goal:privacy_internal[cart]`), so the copula rows keep their baseline. Every bench run also tracks the quick check's AUCs (`metric:privacy_check_domias_auc`, `metric:privacy_check_closest_distance_auc`).
 - **Several privacy units in one DP run** (gap plan G1). `synthesis.privacy.dp.unit` takes a list (`[DimCustomer, DimEmployee]`), and `--dp-unit` a comma-separated list. Each unit owns the tables whose path to it passes through no other unit, has its own accountant with the run's budget, and gets its own (ε, δ) guarantee; the budgets do not add up because the units' tables are disjoint (Prop. 1703). The plan refuses a table with data of two units and links between the units' data before any read. The statement of such a run has schema `relsynth.dp-statement/2` (`unit` is the list, `units` each unit's ε, ρ and tables), each charge names its unit, and `DPResult.per_unit` holds the same figures. A unit's budget can be set by hand (`epsilon: {DimCustomer: 2, DimEmployee: 1}`).
 - **Governed runs with several units** (RFC-G1-01, Prop. 1704). A policy domain can name several units (`unit: {entities: [...]}`); the ledger keeps one account per unit, the domain's budget applies to each unit, and `unit_budgets` sets a unit's budget by hand. A run reserves every unit's budget before it reads data (all or nothing), touches and settles each account, and its manifest lists the units (`privacy.unit` as a list, `charges[].unit`, `privacy.units`). The release rules run once per unit with that unit's figures and must all allow; no Rego change. `gov budget` and `gov policy check` show each unit's account. Ledger entries of such a domain carry `unit`; one-unit domains are unchanged.
 - **Real composite keys in the DP seed database.** When an id groups an owned table's rows (an order number with a sequence column `within` it, or a fact's header key), stage 1 releases a histogram of rows per group as one more block of the table's marginals (Prop. 1702). The seed has orders with several lines numbered within each order, and children point to them by the full composite key; other free key columns number the rows instead of the old `(i, 1)` placeholder, attribute parts of a composite key (a date) are drawn like attributes, and seed rows that repeat a declared composite key are dropped.
 - Checks C1700–C1706 (`tests/science/test_privacy_units_checks.py`), Props. 1700–1704 (`docs/proofs/sec_g1_units.tex`), docs/dp.md section 8; C1705 replays the two-unit sensitivity check on AdventureWorks DW when the bench cache has it; C1706 replays 120 governed two-unit releases against the per-unit ledger accounts.
 
 ### Changed
 
+- **`cart` smooths continuous values by default** (`smoothing: density`, was `none`; discrete codes are drawn as before). Without smoothing, cart releases the numbers of its donor rows exactly, so a value seen once in the source (the largest income, say) appears in the release; the new per-generator benchmark caught it on ACS (univariate singling-out risk 0.53 against a limit of 0.10, internal verdict `fail`), and with smoothing the risk is 0.003 and the verdict `review` (`docs/assurance/evidence/G5/cart_acs.json`). Cart output changes for specs that do not set the option; `generator: {name: cart, options: {smoothing: none}}` restores the old draws. Saved cart models keep the option they were fitted with.
 - **Privacy-unit ownership comes from the IR's trace** (`relsynth.ir.privacy`, RFC-T06-01). The attack suite (`relsynth privacy`) now follows composite foreign keys, so AdventureWorks `FactInternetSalesReason` is owned by `DimCustomer` and appears in its profiles and report. A row that reaches the unit along several paths belongs to every person they reach (union ownership, Prop. 1700); on the bench specs whose paths all agree (Contoso, bi/COREP) the subsets and profiles are unchanged. New `relsynth.privacy.unit.owners`; `owner_ids` keeps one owner per row along the shortest path.
 - The attack suite keeps people who share a row (union ownership) on the same side of its training/control/reference split, so no control profile holds a row the release was fitted on, and control and reference never grow past their sizes. If the shared rows join almost everyone, the split falls back to the plain cut: the report then warns (`split_shared_rows`) and every context is at least `review`. Without shared rows the split is unchanged.
 - DP mode: a parent table measures and truncates only the children on their ownership links, so a link dropped with `drop_columns` (to another unit's table, or off the path) is never read; a declared unique key with a null part no longer counts as a repeat in the seed; `--dp-unit` with no table name (including `--dp-unit ""`) is an argument error. When every parent draws degree 0 from a child's noisy degree histogram (rare, small tables), the seed keeps one child row, as a root table keeps at least one row; stage 2 failed on such an empty fact with children (post-processing, no budget).
@@ -22,6 +25,7 @@ All notable user-visible changes are recorded here, following [Keep a Changelog
 - **`relsynth[spark]` exists.** The spark executor of the partitioned runtime told users to `pip install 'relsynth[spark]'`, but the package had no such extra. It now installs PySpark (3.5.8 or later for CVE-2026-32773, and not 4.0.0, following the Spark security advisories) and cloudpickle; cloudpickle is needed because the fitted model holds closures, which the standard pickle cannot send to Spark tasks. CI runs the spark executor's identity test through the extra.
 - **Dependabot now moves the service image's base digest.** `deploy/docker/Dockerfile.service` kept its digest in a build argument (`ARG PYTHON=...`), which Dependabot's docker updates do not read. The digest is now on a `FROM` line of its own (stage `python-upstream`) and `PYTHON` names that stage by default; `--build-arg PYTHON=<mirror image>` still works for air-gapped builds, and BuildKit then does not pull the upstream image.
 - **The SBOM's scope is stated correctly.** docs/development/ci.md and supply-chain.md said the release SBOM covers a wheel installed with every extra; it covers the `[all]` install (database and file-format connectors), which leaves out `spark`, `service`, `dp`, the text extras and the other optional extras.
+- The `compare` and `baseline` jobs of `bench.yml` install the package: `bench/datasets.py` imports `relsynth` since 2.1.0, and both jobs had failed with `ModuleNotFoundError` on every bench run since then.
 
 ### Changed
 

README.md

@@ -62,7 +62,7 @@ relsynth run spec.yaml --target "sqlite:synthetic.db" --fail-under 0.98
 - **Statistics.** Each table's marginals and dependence come from a Bayesian Gaussian copula over mixed types. Links between tables come from latent segments, so customers who buy more of one product kind keep doing so. Counts, participation and multiplicities follow fitted laws.
 - **Bayesian.** Each synthetic dataset is one draw from the posterior predictive distribution. Several datasets show the model's own uncertainty, and `posterior.mode: mean` gives stable test data.
 - **Scale.** You set the size per table: `--scale 0.1` for a small test database, or `--scale 10` for load tests. Facts follow the tables they hang from by the fitted rates.
-- **Privacy.** Rows are drawn from a fitted model; only tables you declare `mode: copy` are copied. Distance-to-closest-record checks compare synthetic rows with real ones, against the real data's own spacing. Text cannot reproduce rare source content, and memorisation guards are on by default. `relsynth privacy` attacks a release per person (membership, attribute inference, singling out, linkability) and writes a report per release context.
+- **Privacy.** Rows are drawn from a fitted model; only tables you declare `mode: copy` are copied. Distance-to-closest-record checks compare synthetic rows with real ones, against the real data's own spacing. Text cannot reproduce rare source content, and memorisation guards are on by default. `relsynth privacy` attacks a release per person (membership, attribute inference, singling out, linkability) and writes a report per release context. By default `relsynth run` ends with a quick membership check on a proxy release and prints one privacy line (except in DP mode, or when it is skipped: see [docs/privacy.md](docs/privacy.md#the-quick-check-in-every-relsynth-run)); it never changes the synthetic files.
 - **Differential privacy (optional).** `relsynth run --epsilon 1` gives a user-level (ε, δ) guarantee for each person of the privacy unit and all their rows. The organisation sets ε; there is no default. Releases outside the organisation are refused without it.
 - **Governance.** Signed, hash-chained manifests of every run, a privacy budget ledger per domain, an organisation policy and human approvals before release (`relsynth gov`).
 - **Documents.** MongoDB, Cosmos DB, DynamoDB (single-table designs included), Couchbase and JSON collections keep their shape: optional fields, unions, nested arrays with their length laws, polymorphic sub-documents, maps and references between collections (`relsynth docs`).
@@ -429,7 +429,7 @@ relsynth ships no model weights; `relsynth text models` lists models with their
 - **SQL.** Every SQL statement is built with SQLAlchemy Core, with quoted identifiers and bound values. Table and column names cannot inject SQL; tests use hostile names on every database.
 - **Specs.** A spec is YAML read with a safe loader that also refuses aliases, so an alias bomb fails. Rule expressions run in a sandboxed evaluator: no attribute access, calls only to a fixed list of functions, and size and depth limits. Regular expressions with catastrophic backtracking are refused.
 - **Files.** Table names cannot leave the target folder (`..`, separators, absolute paths and symlinks are refused).
-- **Data.** Rows never leave the source database for pushed-down facts. Bundles and profiles hold aggregates only, and `profile --top` must be asked for explicitly. Copy guards run by default; they are not a privacy check ([docs/privacy.md](docs/privacy.md) measures privacy).
+- **Data.** Rows never leave the source database for pushed-down facts. Bundles and profiles hold aggregates only, and `profile --top` must be asked for explicitly. Copy guards run by default; they are not a privacy check ([docs/privacy.md](docs/privacy.md) measures privacy). The quick privacy check of `relsynth run` is a smoke test, not an assessment.
 - **Supply chain.** The code is scanned with bandit and ruff's security rules, and dependencies with pip-audit, in CI.
 - **Code scanning.** Every pull request also runs CodeQL (security-extended and security-and-quality suites), Semgrep with project rules, gitleaks, zizmor for the workflows and GitHub dependency review; Dependabot watches dependencies and the digest-pinned base images ([docs/development/ci.md](docs/development/ci.md#security-scanning)).
 
@@ -491,6 +491,7 @@ The method is described in two papers shipped with the package:
 | Time and scale (v2.0) | **1200–1206** time change, gamma renewal bridge, sequence posterior, calibrated limits, incremental runs; degree laws of link tables; **950–954** worker independence, partitions keep the law, keys across partitions, drift tests |
 | Synthetic text (v2.1) | **1500–1507** canary exposure under DP; DP n-gram release; user-level DP-GD; Private Evolution votes; expected exposure; near-duplicate search; PII placeholders; consistency with the row |
 | Agents (v2.1) | **1600–1604** data minimisation, containment of any model output, evidence first, grounded wording, no authority without a person |
+| Quick privacy check (v2.2) | **1740–1741** false alarms of the quick check are bounded under no membership signal; the check leaves the release unchanged |
 
 [docs/theory.md](docs/theory.md) gives a readable summary of the model and of the six v1.0 propositions, with the proofs in full.
 

bench/assess.py

@@ -18,7 +18,9 @@
   (`privacy:<criterion>|<attack>`, passing when the upper confidence bound is
   under the limit), the verdict of each context (`goal:privacy_<context>`,
   where an external context must stay blocked without DP) and the attack values
-  as tracked values.
+  as tracked values.  Datasets that list privacy.generators are attacked once
+  more per listed generator (G5), with the generator in each id
+  (`privacy[cart]:<criterion>|<attack>`).
 
 Only aggregates leave this module: values, limits, verdicts.
 """
@@ -133,36 +135,42 @@ def run_quality(name, location, syn_dir, overrides) -> list[dict]:
     return quality_checks(rep)
 
 
-def privacy_checks(rep: dict) -> list[dict]:
+def privacy_checks(rep: dict, generator: str | None = None) -> list[dict]:
+    """Harness checks of one privacy report.  generator: a generator other than
+    the spec's own, whose checks carry it in brackets (`privacy[cart]:...`,
+    `goal:privacy_internal[cart]`, `metric:privacy_..._cart`) so that the default
+    generator's ids stay those of earlier baselines."""
+    tag = f"[{generator}]" if generator else ""
+    sfx = f"_{generator}" if generator else ""
     out = []
     for ctx_name, ctx in rep["contexts"].items():
         if ctx_name == "internal":
             for c in ctx["checks"]:
-                cid = f"privacy:{c['criterion']}|{c['attack']}"
+                cid = f"privacy{tag}:{c['criterion']}|{c['attack']}"
                 ci = c.get("ci") or [None, None]
                 out.append({"id": cid, "pass": None if c["status"] == "info" else c["status"] == "pass",
                             "value": _r(c["value"]), "limit": _r(c["limit"]), "upper": _r(ci[1]),
                             "status": c["status"]})
-                out.append(tracked(f"privacy_{c['criterion']}_{c['attack']}", c["value"], "lower"))
-            out.append({"id": "goal:privacy_internal", "pass": ctx["verdict"] != "fail", "value": None,
+                out.append(tracked(f"privacy_{c['criterion']}_{c['attack']}{sfx}", c["value"], "lower"))
+            out.append({"id": f"goal:privacy_internal{tag}", "pass": ctx["verdict"] != "fail", "value": None,
                         "status": ctx["verdict"]})
         elif ctx.get("external"):
             # without DP an external release must be refused (user decision of 2026-10-26)
             dp = (rep.get("release") or {}).get("dp")
-            out.append({"id": f"goal:privacy_{ctx_name}_needs_dp", "pass": bool(dp) or ctx["verdict"] == "blocked",
+            out.append({"id": f"goal:privacy_{ctx_name}_needs_dp{tag}", "pass": bool(dp) or ctx["verdict"] == "blocked",
                         "value": None, "status": ctx["verdict"]})
     g = (rep.get("guards") or {}).get("dcr")
     if isinstance(g, dict):
         for k, v in g.items():
             if isinstance(v, (int, float)) and not isinstance(v, bool):
-                out.append(tracked(f"privacy_guard_dcr_{k}", v, "lower" if "match" in k else "higher"))
+                out.append(tracked(f"privacy_guard_dcr_{k}{sfx}", v, "lower" if "match" in k else "higher"))
     return out
 
 
-def run_privacy(name, location, seed, overrides) -> list[dict]:
-    from relsynth.privacy import assess
+def privacy_config(name: str) -> dict:
+    """benchmarks/privacy.yaml: the defaults merged with the database's block,
+    and the database's shadow-model setting from bench/datasets.py."""
     from .datasets import DATASETS
-    spec = load_spec(name, location, overrides)
     allc = yaml.safe_load(PRIVACY.read_text(encoding="utf-8")) or {}
     cfg = json.loads(json.dumps({**(allc.get("defaults") or {})}))
     for k, v in (allc.get(name) or {}).items():
@@ -174,8 +182,34 @@ def run_privacy(name, location, seed, overrides) -> list[dict]:
     shadow = DATASETS[name].privacy.get("shadow")
     if shadow is not None:
         cfg["attacks"]["membership"]["shadow"] = {**cfg["attacks"]["membership"].get("shadow", {}), **shadow}
-    rep = assess(spec, cfg, seed=int(seed))
-    return privacy_checks(rep)
+    return cfg
+
+
+def with_generator(spec: dict, generator: str) -> dict:
+    """The spec with every table on one generator plug-in (G5: each built-in
+    generator is attacked with the same settings)."""
+    import copy
+    s = copy.deepcopy(spec)
+    s.setdefault("synthesis", {})["generator"] = generator
+    for ts in s["tables"].values():
+        ts.pop("generator", None)
+    return s
+
+
+def run_privacy(name, location, seed, overrides) -> list[dict]:
+    """The attacks on the spec's own generator, then on every other generator
+    the dataset lists under privacy.generators (cart, independent)."""
+    from relsynth.privacy import assess
+    from .datasets import DATASETS
+    spec = load_spec(name, location, overrides)
+    own = (spec.get("synthesis") or {}).get("generator", "copula")
+    # a table override (--set tables.x.generator=...) means the first run is not the default generator's
+    mixed = any((ts or {}).get("generator") not in (None, own) for ts in spec["tables"].values())
+    out = []
+    for g in [None, *[g for g in DATASETS[name].privacy.get("generators", ()) if mixed or g != own]]:
+        rep = assess(spec if g is None else with_generator(spec, g), privacy_config(name), seed=int(seed))
+        out += privacy_checks(rep, g)
+    return out
 
 
 def main(argv=None) -> int: