feat(privacy): privacy margin for small dimension tables (G3)

CHANGELOG.md

@@ -36,6 +36,12 @@ All notable user-visible changes are recorded here, following [Keep a Changelog
 - **Three claims now have tests of their own.** A diagnostics bundle, a verify report and default profile output (without `--top`, which lists frequent values and so can show identifier values) hold no customer identifier of the test scenario and at most a few summary values of each continuous column (SEC-10 and README RM-27 move from partial to supported; literals written in the spec itself, such as a filter value, stay in a bundle's masked spec); every statement sent to a SQLite source by introspect, profile, run (pushdown off and on) and verify, through SQLAlchemy or on the DBAPI cursor, is a `SELECT`, a `PRAGMA` that only reads or the connection's `PRAGMA foreign_keys=ON`, and the source file is unchanged (new claim SEC-17, partial as only SQLite is measured; docs/security.md now says relsynth never writes to a source, as long as the run's target is not the source itself, and names the reflection and session statements, and it and SEC-10 now list every kind of pushdown read: aggregates, the seeded sample, which is the whole table for small tables, and the first row per order or running-balance group); `relsynth profile` lists frequent values only with `--top` (RM-27); `doctor --apply` refuses a DP spec (DP-12). The register holds 148 claims.
 - **Module-level import cycles fail a test** (`tests/unit/test_import_graph.py`). The 76 CodeQL `py/cyclic-import` notes all ran through imports deferred into functions, which cannot fail at import time; the note is now filtered out in the CodeQL config, the test guards the cycles that can fail, and `py/unsafe-cyclic-import` stays on.
 
+### Added
+
+- **Segment marginal shrinkage for small dimensions** (`synthesis.marginals.segment_prior_rows`, gap plan G3, opt-in; default 0 keeps the 2.1 output). Each segment marginal of an entity table gives the table's marginal the weight of that many rows: a segment of `n` rows keeps the share `n / (n + rows)` of its own distribution, so segments of a few dozen rows reproduce their own rows less often (shrinkage does not rule out exact copies), while segments of thousands of rows move by at most `rows / n` (Props. 1720-1721). With 20 rows, the AdventureWorks `DimProduct` distance-to-closest-record ratio rises from 0.50-0.65 to 0.96-1.10 over five seeds (limit 0.5), at a cost of 7.8 of 1,267 scorecard items on average. It also weakens the link between a dimension's segments and its facts (science checks C09, C19 and C43 fail with it on), so it is not on by default. Numbers in `docs/research/small-dimensions.md`.
+- **Near-copy guard** (`synthesis.privacy.near_copy: auto`, off by default; `near_copy_ratio`, `near_copy_rounds`, `near_copy_max_rows`). It redraws entity rows that sit closer to a real row than that row's nearest distinct real neighbour, and exact copies of unique real rows, from their own segment, and counts them in the sample log (`near_copies`). It is a distance rule, not a privacy guarantee: Prop. 1723 shows that a guard redrawing until it succeeds is not differentially private for any ε, the guard adds no DP guarantee of its own (in DP mode it is post-processing of the seed database), and the `dcr_ratio` of a guarded table no longer tests the generator on its own ([docs/privacy.md](docs/privacy.md#small-dimensions)).
+- Propositions 1720-1723 (`docs/proofs/sec_g3_smalldims.tex`) with checks C1720-C1723 (`tests/science/test_small_dims_checks.py`).
+
 ## 2.1.0 (2026-11-07)
 ### Security
 

README.md

@@ -369,7 +369,7 @@ The relational part of the model is unchanged: Proposition 100 proves it, and a
 
 - **Time series** ([docs/timeseries.md](docs/timeseries.md)). `tables.<t>.sequence` redraws the times of an entity's events from a point-process model (calendar in operational time, life-time copula, gamma renewal gaps), keeping keys, counts and rules. `tables.<t>.series` synthesises regular readings with daily, weekly and yearly profiles. `relsynth extend` adds the events after a dataset day by day; a stepwise run equals one long run. TimescaleDB hypertables are read and created.
 - **Generator plug-ins** ([docs/generators.md](docs/generators.md), [docs/compute.md](docs/compute.md)). `tables.<t>.generator` picks the model of a table's columns; the engine keeps keys, links, segments, counts and rules whatever the generator. Third-party generators install through the `relsynth.generators` entry point; `compute.yaml` presets control GPUs.
-- **Engine settings and autotune** ([docs/settings.md](docs/settings.md), [docs/autotune.md](docs/autotune.md)). 101 settings with type, range and the checks they affect, from defaults, a settings file, presets (`fast`, `thorough`), the spec or `--set`. With default settings the output is unchanged byte for byte. `relsynth autotune` separates systematic failures from noise over several seeds and changes settings until your objectives hold, or says why they cannot.
+- **Engine settings and autotune** ([docs/settings.md](docs/settings.md), [docs/autotune.md](docs/autotune.md)). 101 settings with type, range and the checks they affect, from defaults, a settings file, presets (`fast`, `thorough`), the spec or `--set`. Adding the settings changed no output: their defaults reproduce the earlier engine byte for byte. The small-dimension settings of release 2.2 (`marginals.segment_prior_rows`, `privacy.near_copy`) are off by default and keep that output. `relsynth autotune` separates systematic failures from noise over several seeds and changes settings until your objectives hold, or says why they cannot.
 - **Partitioned generation** ([docs/runtime.md](docs/runtime.md)). `synthesis.runtime` runs fact rows in partitions on the `local`, `process` or `spark` executor. Each partition draws from a Philox stream keyed by seed, dataset, fact and partition, so the output is the same for 1, 8 or 64 workers and for Spark (Prop. 950).
 - **IR 2.0** ([docs/ir-guide.md](docs/ir-guide.md)). One versioned document describes a data estate: entities with shapes (record, document, node, edge, event), nested arrays, keys, references, hierarchies, time and versioning, partitions, rules and the privacy unit. 1.0 specs convert without loss (same synthetic data); ODCS v3 data contracts and OpenLineage are exported and checked.
 

docs/assurance/claims.yaml

@@ -1311,14 +1311,17 @@ claims:
     doc: README.md
     area: statistics
     quote:
-      - "With default settings the output is unchanged byte for byte."
+      - "Adding the settings changed no output: their defaults reproduce the earlier engine byte for byte."
+      - "The small-dimension settings of release 2.2 (`marginals.segment_prior_rows`, `privacy.near_copy`) are off by default and keep that output."
       - "relsynth autotune separates systematic failures from noise over several seeds"
     status: supported
     evidence:
       - test: tests/unit/test_settings.py::test_promoted_constants_keep_the_1_0_values
       - test: tests/unit/test_runtime.py::test_default_runtime_keeps_the_1_0_sampler
       - test: tests/unit/test_generators.py::test_copula_is_the_default_and_naming_it_changes_nothing
       - test: tests/unit/test_autotune.py::test_systematic_versus_sporadic
+      - test: tests/unit/test_small_dims.py::test_segment_prior_is_wired_and_guard_off_by_default
+      - check: C1720
       - code: tools/identity_check.py
 
   - id: RM-25
@@ -2009,6 +2012,156 @@ claims:
       - check: C1740
       - check: C1741
 
+  # ------------------------------------------------------------------ docs/privacy.md, small dimensions (G3)
+
+  - id: PRV-20
+    doc: docs/privacy.md
+    area: statistics
+    quote: "Two settings deal with this (Props. 1720-1723 in `docs/proofs/sec_g3_smalldims.tex`)."
+    status: supported
+    evidence:
+      - prop: prop:segshrink
+      - prop: prop:ncguard
+      - report: docs/research/small-dimensions.md
+
+  - id: PRV-21
+    doc: docs/privacy.md
+    area: statistics
+    quote: "a segment of `n` rows keeps the share `n / (n + rows)` of its own distribution and takes the rest from the table's (Prop. 1720)"
+    status: supported
+    evidence:
+      - prop: prop:segshrink
+      - check: C1720
+
+  - id: PRV-22
+    doc: docs/privacy.md
+    area: statistics
+    quote: "the change is at most `rows / (n + rows)` in total variation (Prop. 1721)"
+    status: supported
+    evidence:
+      - prop: prop:seglarge
+      - check: C1721
+
+  - id: PRV-23
+    doc: docs/privacy.md
+    area: privacy-measured
+    quote: "An exact copy of a real row that no other real row shares in the guard's encoding also counts (two real rows that differ only in rare levels share an encoding, so a copy of either is not flagged as exact)."
+    status: supported
+    evidence:
+      - test: tests/unit/test_small_dims.py::test_guard_rules
+      - check: C1722
+
+  - id: PRV-24
+    doc: docs/privacy.md
+    area: privacy-formal
+    quote: "It is not a privacy guarantee."
+    status: limitation
+    evidence:
+      - prop: prop:ncnotdp
+      - check: C1723
+
+  - id: PRV-25
+    doc: docs/privacy.md
+    area: privacy-formal
+    quote: "If the guard redraws until it succeeds, someone who knows the rule learns that no synthetic row sits inside a real record's ball, and Prop. 1723 shows that this rules out differential privacy for any ε, given a generator that can produce rows near the record."
+    status: limitation
+    evidence:
+      - prop: prop:ncnotdp
+      - check: C1723
+
+  - id: PRV-32
+    doc: docs/privacy.md
+    area: scope
+    quote: "The guard runs on the tables the distance check covers, up to `privacy.near_copy_max_rows` rows (default 20,000; the search compares every synthetic row with every distinct real row, so its time grows with the square of the rows), and only when at least three of the distance columns are sampled by the table's model: a table with only two sampled distance columns and the rest computed (derived or aggregated columns) is not guarded."
+    status: supported
+    evidence:
+      - test: tests/unit/test_small_dims.py::test_settings_are_registered_with_defaults
+      - test: tests/unit/test_small_dims.py::test_nearest_matches_brute_force_in_small_blocks
+      - test: tests/unit/test_small_dims.py::test_guard_skips_small_or_narrow_tables
+      - code: src/relsynth/engine.py::Synthesizer::_near_copy_guard
+
+  - id: PRV-33
+    doc: docs/privacy.md
+    area: privacy-measured
+    quote:
+      - "Rows that are redrawn later to satisfy an attribute rule or a unique key, and rows whose values change through them (a group effect), are checked again in the same rejection, for at most `privacy.near_copy_rounds` rounds and half of `rules.max_rejection_rounds`, so the remaining rounds enforce the rules alone."
+      - "`kept` counts the near copies among the rows emitted, recounted after the rule redraws and group effects of the last step."
+      - "For an entity with a version history (SCD type 2), it is counted before the versions are added."
+      - "A row redrawn only for being a near copy that ends the rule budget breaking a rule gets its earlier rule-valid row back, hidden draws included, and the log counts those rows (`restored_after_rules`)."
+    status: supported
+    evidence:
+      - test: tests/unit/test_small_dims.py::test_rule_redraws_are_guarded_and_kept_counts_emitted_rows
+      - test: tests/unit/test_small_dims.py::test_kept_counts_emitted_rows_after_a_group_effect
+      - test: tests/unit/test_small_dims.py::test_near_copy_rejection_respects_both_round_limits
+      - test: tests/unit/test_small_dims.py::test_group_effect_rows_join_the_near_copy_recheck
+      - test: tests/unit/test_small_dims.py::test_near_copy_redraw_never_trades_a_valid_row_for_a_violation
+      - test: tests/unit/test_small_dims.py::test_restored_rows_keep_their_hidden_draws
+      - code: src/relsynth/engine.py::Synthesizer::_finish
+
+  - id: PRV-34
+    doc: docs/privacy.md
+    area: privacy-measured
+    quote:
+      - "It uses the distance check's encoding (rank scores of numbers and one-hot codes of the 20 most frequent levels of each categorical, so two rows that differ only in rarer levels look the same to it), but fits the ranks and levels on the real rows only, while the scorecard fits them on real and synthetic rows together; the two distances, and so the guard's decisions and the scorecard's ratio, can differ."
+    status: supported
+    evidence:
+      - test: tests/unit/test_small_dims.py::test_guard_encoding_is_fitted_on_real_rows_only
+      - code: src/relsynth/nearcopy.py::NearCopyGuard::encode
+      - code: src/relsynth/evaluate.py
+
+  - id: PRV-30
+    doc: docs/privacy.md
+    area: privacy-formal
+    quote: "Prop. 1723 does not cover that case, and the guard gives no differential-privacy guarantee of its own."
+    status: limitation
+    evidence:
+      - prop: prop:ncnotdp
+
+  - id: PRV-31
+    doc: docs/privacy.md
+    area: privacy-formal
+    quote: "In DP mode (`synthesis.privacy.dp`) the engine is fitted on the DP seed database, so the guard measures distances to seed rows and is post-processing of the DP release: the DP guarantee stands, and the guard adds nothing to it."
+    status: supported
+    evidence:
+      - prop: prop:post
+      - test: tests/unit/dp/test_dp.py::test_stage2_keeps_the_near_copy_guard_on_the_seed
+      - code: src/relsynth/dp/__init__.py::synthesize
+
+  - id: PRV-26
+    doc: docs/privacy.md
+    area: scope
+    quote: "Ganev and De Cristofaro (2023) reconstruct records from exactly such holes."
+    status: scope
+    evidence:
+      - source: "G. Ganev, E. De Cristofaro, On the inadequacy of similarity-based privacy metrics, arXiv:2312.05114 (2023)"
+
+  - id: PRV-27
+    doc: docs/privacy.md
+    area: privacy-formal
+    quote: "The attacks above remain the privacy test, and DP mode remains the guarantee."
+    status: supported
+    evidence:
+      - claim: PRV-01
+      - prop: prop:ncnotdp
+
+  - id: PRV-28
+    doc: docs/privacy.md
+    area: privacy-measured
+    quote: "the scorecard's `dcr_ratio` for `DimProduct` was 0.40 to 0.65 at the 2.1 defaults, against a limit of 0.5"
+    status: supported
+    note: 0.40 is seed 12 and 0.64 seed 11 of the autotune report (CI runners); 0.42-0.65 are the full fits and paired draws of the small-dimensions report.
+    evidence:
+      - report: docs/research/small-dimensions.md
+      - report: docs/research/autotune-adventureworks.md
+
+  - id: PRV-29
+    doc: docs/privacy.md
+    area: privacy-measured
+    quote: "After the guard, the `dcr_ratio` of a guarded table no longer tests the generator on its own."
+    status: limitation
+    evidence:
+      - test: tests/science/test_small_dims_checks.py::test_c1722_scorecard_marks_a_guarded_table
+
 not_claims:
   - {doc: docs/privacy.md, quote: "Every key is optional.", reason: "describes the configuration file format"}
   - {doc: docs/privacy.md, quote: "Viswanath, The composition theorem for differential privacy, ICML 2015.", reason: "bibliography entry"}