diff --git a/CHANGELOG.md b/CHANGELOG.md index 017b5cc..1260321 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,6 +36,12 @@ All notable user-visible changes are recorded here, following [Keep a Changelog - **Three claims now have tests of their own.** A diagnostics bundle, a verify report and default profile output (without `--top`, which lists frequent values and so can show identifier values) hold no customer identifier of the test scenario and at most a few summary values of each continuous column (SEC-10 and README RM-27 move from partial to supported; literals written in the spec itself, such as a filter value, stay in a bundle's masked spec); every statement sent to a SQLite source by introspect, profile, run (pushdown off and on) and verify, through SQLAlchemy or on the DBAPI cursor, is a `SELECT`, a `PRAGMA` that only reads or the connection's `PRAGMA foreign_keys=ON`, and the source file is unchanged (new claim SEC-17, partial as only SQLite is measured; docs/security.md now says relsynth never writes to a source, as long as the run's target is not the source itself, and names the reflection and session statements, and it and SEC-10 now list every kind of pushdown read: aggregates, the seeded sample, which is the whole table for small tables, and the first row per order or running-balance group); `relsynth profile` lists frequent values only with `--top` (RM-27); `doctor --apply` refuses a DP spec (DP-12). The register holds 148 claims. - **Module-level import cycles fail a test** (`tests/unit/test_import_graph.py`). The 76 CodeQL `py/cyclic-import` notes all ran through imports deferred into functions, which cannot fail at import time; the note is now filtered out in the CodeQL config, the test guards the cycles that can fail, and `py/unsafe-cyclic-import` stays on. +### Added + +- **Segment marginal shrinkage for small dimensions** (`synthesis.marginals.segment_prior_rows`, gap plan G3, opt-in; default 0 keeps the 2.1 output). Each segment marginal of an entity table gives the table's marginal the weight of that many rows: a segment of `n` rows keeps the share `n / (n + rows)` of its own distribution, so segments of a few dozen rows reproduce their own rows less often (shrinkage does not rule out exact copies), while segments of thousands of rows move by at most `rows / n` (Props. 1720-1721). With 20 rows, the AdventureWorks `DimProduct` distance-to-closest-record ratio rises from 0.50-0.65 to 0.96-1.10 over five seeds (limit 0.5), at a cost of 7.8 of 1,267 scorecard items on average. It also weakens the link between a dimension's segments and its facts (science checks C09, C19 and C43 fail with it on), so it is not on by default. Numbers in `docs/research/small-dimensions.md`. +- **Near-copy guard** (`synthesis.privacy.near_copy: auto`, off by default; `near_copy_ratio`, `near_copy_rounds`, `near_copy_max_rows`). It redraws entity rows that sit closer to a real row than that row's nearest distinct real neighbour, and exact copies of unique real rows, from their own segment, and counts them in the sample log (`near_copies`). It is a distance rule, not a privacy guarantee: Prop. 1723 shows that a guard redrawing until it succeeds is not differentially private for any ε, the guard adds no DP guarantee of its own (in DP mode it is post-processing of the seed database), and the `dcr_ratio` of a guarded table no longer tests the generator on its own ([docs/privacy.md](docs/privacy.md#small-dimensions)). +- Propositions 1720-1723 (`docs/proofs/sec_g3_smalldims.tex`) with checks C1720-C1723 (`tests/science/test_small_dims_checks.py`). + ## 2.1.0 (2026-11-07) ### Security diff --git a/README.md b/README.md index 131a405..2ba32b8 100644 --- a/README.md +++ b/README.md @@ -369,7 +369,7 @@ The relational part of the model is unchanged: Proposition 100 proves it, and a - **Time series** ([docs/timeseries.md](docs/timeseries.md)). `tables..sequence` redraws the times of an entity's events from a point-process model (calendar in operational time, life-time copula, gamma renewal gaps), keeping keys, counts and rules. `tables..series` synthesises regular readings with daily, weekly and yearly profiles. `relsynth extend` adds the events after a dataset day by day; a stepwise run equals one long run. TimescaleDB hypertables are read and created. - **Generator plug-ins** ([docs/generators.md](docs/generators.md), [docs/compute.md](docs/compute.md)). `tables..generator` picks the model of a table's columns; the engine keeps keys, links, segments, counts and rules whatever the generator. Third-party generators install through the `relsynth.generators` entry point; `compute.yaml` presets control GPUs. -- **Engine settings and autotune** ([docs/settings.md](docs/settings.md), [docs/autotune.md](docs/autotune.md)). 101 settings with type, range and the checks they affect, from defaults, a settings file, presets (`fast`, `thorough`), the spec or `--set`. With default settings the output is unchanged byte for byte. `relsynth autotune` separates systematic failures from noise over several seeds and changes settings until your objectives hold, or says why they cannot. +- **Engine settings and autotune** ([docs/settings.md](docs/settings.md), [docs/autotune.md](docs/autotune.md)). 101 settings with type, range and the checks they affect, from defaults, a settings file, presets (`fast`, `thorough`), the spec or `--set`. Adding the settings changed no output: their defaults reproduce the earlier engine byte for byte. The small-dimension settings of release 2.2 (`marginals.segment_prior_rows`, `privacy.near_copy`) are off by default and keep that output. `relsynth autotune` separates systematic failures from noise over several seeds and changes settings until your objectives hold, or says why they cannot. - **Partitioned generation** ([docs/runtime.md](docs/runtime.md)). `synthesis.runtime` runs fact rows in partitions on the `local`, `process` or `spark` executor. Each partition draws from a Philox stream keyed by seed, dataset, fact and partition, so the output is the same for 1, 8 or 64 workers and for Spark (Prop. 950). - **IR 2.0** ([docs/ir-guide.md](docs/ir-guide.md)). One versioned document describes a data estate: entities with shapes (record, document, node, edge, event), nested arrays, keys, references, hierarchies, time and versioning, partitions, rules and the privacy unit. 1.0 specs convert without loss (same synthetic data); ODCS v3 data contracts and OpenLineage are exported and checked. diff --git a/docs/assurance/claims.yaml b/docs/assurance/claims.yaml index 9be17a2..c0196d1 100644 --- a/docs/assurance/claims.yaml +++ b/docs/assurance/claims.yaml @@ -1311,7 +1311,8 @@ claims: doc: README.md area: statistics quote: - - "With default settings the output is unchanged byte for byte." + - "Adding the settings changed no output: their defaults reproduce the earlier engine byte for byte." + - "The small-dimension settings of release 2.2 (`marginals.segment_prior_rows`, `privacy.near_copy`) are off by default and keep that output." - "relsynth autotune separates systematic failures from noise over several seeds" status: supported evidence: @@ -1319,6 +1320,8 @@ claims: - test: tests/unit/test_runtime.py::test_default_runtime_keeps_the_1_0_sampler - test: tests/unit/test_generators.py::test_copula_is_the_default_and_naming_it_changes_nothing - test: tests/unit/test_autotune.py::test_systematic_versus_sporadic + - test: tests/unit/test_small_dims.py::test_segment_prior_is_wired_and_guard_off_by_default + - check: C1720 - code: tools/identity_check.py - id: RM-25 @@ -2009,6 +2012,156 @@ claims: - check: C1740 - check: C1741 + # ------------------------------------------------------------------ docs/privacy.md, small dimensions (G3) + + - id: PRV-20 + doc: docs/privacy.md + area: statistics + quote: "Two settings deal with this (Props. 1720-1723 in `docs/proofs/sec_g3_smalldims.tex`)." + status: supported + evidence: + - prop: prop:segshrink + - prop: prop:ncguard + - report: docs/research/small-dimensions.md + + - id: PRV-21 + doc: docs/privacy.md + area: statistics + quote: "a segment of `n` rows keeps the share `n / (n + rows)` of its own distribution and takes the rest from the table's (Prop. 1720)" + status: supported + evidence: + - prop: prop:segshrink + - check: C1720 + + - id: PRV-22 + doc: docs/privacy.md + area: statistics + quote: "the change is at most `rows / (n + rows)` in total variation (Prop. 1721)" + status: supported + evidence: + - prop: prop:seglarge + - check: C1721 + + - id: PRV-23 + doc: docs/privacy.md + area: privacy-measured + quote: "An exact copy of a real row that no other real row shares in the guard's encoding also counts (two real rows that differ only in rare levels share an encoding, so a copy of either is not flagged as exact)." + status: supported + evidence: + - test: tests/unit/test_small_dims.py::test_guard_rules + - check: C1722 + + - id: PRV-24 + doc: docs/privacy.md + area: privacy-formal + quote: "It is not a privacy guarantee." + status: limitation + evidence: + - prop: prop:ncnotdp + - check: C1723 + + - id: PRV-25 + doc: docs/privacy.md + area: privacy-formal + quote: "If the guard redraws until it succeeds, someone who knows the rule learns that no synthetic row sits inside a real record's ball, and Prop. 1723 shows that this rules out differential privacy for any ε, given a generator that can produce rows near the record." + status: limitation + evidence: + - prop: prop:ncnotdp + - check: C1723 + + - id: PRV-32 + doc: docs/privacy.md + area: scope + quote: "The guard runs on the tables the distance check covers, up to `privacy.near_copy_max_rows` rows (default 20,000; the search compares every synthetic row with every distinct real row, so its time grows with the square of the rows), and only when at least three of the distance columns are sampled by the table's model: a table with only two sampled distance columns and the rest computed (derived or aggregated columns) is not guarded." + status: supported + evidence: + - test: tests/unit/test_small_dims.py::test_settings_are_registered_with_defaults + - test: tests/unit/test_small_dims.py::test_nearest_matches_brute_force_in_small_blocks + - test: tests/unit/test_small_dims.py::test_guard_skips_small_or_narrow_tables + - code: src/relsynth/engine.py::Synthesizer::_near_copy_guard + + - id: PRV-33 + doc: docs/privacy.md + area: privacy-measured + quote: + - "Rows that are redrawn later to satisfy an attribute rule or a unique key, and rows whose values change through them (a group effect), are checked again in the same rejection, for at most `privacy.near_copy_rounds` rounds and half of `rules.max_rejection_rounds`, so the remaining rounds enforce the rules alone." + - "`kept` counts the near copies among the rows emitted, recounted after the rule redraws and group effects of the last step." + - "For an entity with a version history (SCD type 2), it is counted before the versions are added." + - "A row redrawn only for being a near copy that ends the rule budget breaking a rule gets its earlier rule-valid row back, hidden draws included, and the log counts those rows (`restored_after_rules`)." + status: supported + evidence: + - test: tests/unit/test_small_dims.py::test_rule_redraws_are_guarded_and_kept_counts_emitted_rows + - test: tests/unit/test_small_dims.py::test_kept_counts_emitted_rows_after_a_group_effect + - test: tests/unit/test_small_dims.py::test_near_copy_rejection_respects_both_round_limits + - test: tests/unit/test_small_dims.py::test_group_effect_rows_join_the_near_copy_recheck + - test: tests/unit/test_small_dims.py::test_near_copy_redraw_never_trades_a_valid_row_for_a_violation + - test: tests/unit/test_small_dims.py::test_restored_rows_keep_their_hidden_draws + - code: src/relsynth/engine.py::Synthesizer::_finish + + - id: PRV-34 + doc: docs/privacy.md + area: privacy-measured + quote: + - "It uses the distance check's encoding (rank scores of numbers and one-hot codes of the 20 most frequent levels of each categorical, so two rows that differ only in rarer levels look the same to it), but fits the ranks and levels on the real rows only, while the scorecard fits them on real and synthetic rows together; the two distances, and so the guard's decisions and the scorecard's ratio, can differ." + status: supported + evidence: + - test: tests/unit/test_small_dims.py::test_guard_encoding_is_fitted_on_real_rows_only + - code: src/relsynth/nearcopy.py::NearCopyGuard::encode + - code: src/relsynth/evaluate.py + + - id: PRV-30 + doc: docs/privacy.md + area: privacy-formal + quote: "Prop. 1723 does not cover that case, and the guard gives no differential-privacy guarantee of its own." + status: limitation + evidence: + - prop: prop:ncnotdp + + - id: PRV-31 + doc: docs/privacy.md + area: privacy-formal + quote: "In DP mode (`synthesis.privacy.dp`) the engine is fitted on the DP seed database, so the guard measures distances to seed rows and is post-processing of the DP release: the DP guarantee stands, and the guard adds nothing to it." + status: supported + evidence: + - prop: prop:post + - test: tests/unit/dp/test_dp.py::test_stage2_keeps_the_near_copy_guard_on_the_seed + - code: src/relsynth/dp/__init__.py::synthesize + + - id: PRV-26 + doc: docs/privacy.md + area: scope + quote: "Ganev and De Cristofaro (2023) reconstruct records from exactly such holes." + status: scope + evidence: + - source: "G. Ganev, E. De Cristofaro, On the inadequacy of similarity-based privacy metrics, arXiv:2312.05114 (2023)" + + - id: PRV-27 + doc: docs/privacy.md + area: privacy-formal + quote: "The attacks above remain the privacy test, and DP mode remains the guarantee." + status: supported + evidence: + - claim: PRV-01 + - prop: prop:ncnotdp + + - id: PRV-28 + doc: docs/privacy.md + area: privacy-measured + quote: "the scorecard's `dcr_ratio` for `DimProduct` was 0.40 to 0.65 at the 2.1 defaults, against a limit of 0.5" + status: supported + note: 0.40 is seed 12 and 0.64 seed 11 of the autotune report (CI runners); 0.42-0.65 are the full fits and paired draws of the small-dimensions report. + evidence: + - report: docs/research/small-dimensions.md + - report: docs/research/autotune-adventureworks.md + + - id: PRV-29 + doc: docs/privacy.md + area: privacy-measured + quote: "After the guard, the `dcr_ratio` of a guarded table no longer tests the generator on its own." + status: limitation + evidence: + - test: tests/science/test_small_dims_checks.py::test_c1722_scorecard_marks_a_guarded_table + not_claims: - {doc: docs/privacy.md, quote: "Every key is optional.", reason: "describes the configuration file format"} - {doc: docs/privacy.md, quote: "Viswanath, The composition theorem for differential privacy, ICML 2015.", reason: "bibliography entry"} diff --git a/docs/privacy.md b/docs/privacy.md index 275826f..a961389 100644 --- a/docs/privacy.md +++ b/docs/privacy.md @@ -139,6 +139,16 @@ Shadow modelling fits the generator `n_train + n_test` times (64 by default) on The scorecard's distance-to-closest-record ratio and exact-match rate stay in every run (`relsynth run`) for continuity, but they are **not a privacy check**: calibration showed the distance guard misses even 20% copied records, which DOMIAS catches at 5%. The privacy report lists them in the annex and no verdict depends on them. `benchmarks/privacy/calibrate_guards.py` measures which leaks each one detects: see the T06 report for the numbers. The text model's memorisation guards (vocabulary k-anonymity, no rare n-gram copies) are unchanged. +## Small dimensions + +A small dimension, such as a product list of a few hundred rows, is fitted by latent segments of a few dozen rows each. AdventureWorks `DimProduct` (606 rows) gets 24 segments of 6 to 49 rows. Each segment's marginals then follow its own few rows closely, and synthetic rows land closer to real rows than real rows sit to each other: the scorecard's `dcr_ratio` for `DimProduct` was 0.40 to 0.65 at the 2.1 defaults, against a limit of 0.5 (gap plan G3, [docs/research/small-dimensions.md](research/small-dimensions.md)). Two settings deal with this (Props. 1720-1723 in `docs/proofs/sec_g3_smalldims.tex`). + +**`marginals.segment_prior_rows`** (default 0, off; 20 is the studied value) gives the table's marginal the weight of that many rows in each segment's marginal: a segment of `n` rows keeps the share `n / (n + rows)` of its own distribution and takes the rest from the table's (Prop. 1720). Segments of a few dozen rows are pulled toward the table, and segments of thousands of rows hardly move: the change is at most `rows / (n + rows)` in total variation (Prop. 1721). It applies to entity tables only. Each segment still keeps its own copula only when it has at least `copula.min_rows` rows, so small segments already share the pooled copula. It is off by default because it also pulls each segment toward the table in the columns that drive the fact tables: with 20 rows, the fake-data science checks C09 (product price against sale amount through a snowflake), C19 and C43 fail, and the AdventureWorks scorecard loses 7.8 of 1,267 items on average. Turn it on for small dimensions whose privacy margin matters more than their joins, per table under `per_table` if needed. + +**`privacy.near_copy`** (default `off`): with `auto`, the engine redraws entity rows that are *near copies* of a real row. A near copy is closer to its nearest real row than that real row is to its own nearest distinct real neighbour, times `privacy.near_copy_ratio` (default 1). An exact copy of a real row that no other real row shares in the guard's encoding also counts (two real rows that differ only in rare levels share an encoding, so a copy of either is not flagged as exact). The guard runs on the tables the distance check covers, up to `privacy.near_copy_max_rows` rows (default 20,000; the search compares every synthetic row with every distinct real row, so its time grows with the square of the rows), and only when at least three of the distance columns are sampled by the table's model: a table with only two sampled distance columns and the rest computed (derived or aggregated columns) is not guarded. It uses the distance check's encoding (rank scores of numbers and one-hot codes of the 20 most frequent levels of each categorical, so two rows that differ only in rarer levels look the same to it), but fits the ranks and levels on the real rows only, while the scorecard fits them on real and synthetic rows together; the two distances, and so the guard's decisions and the scorecard's ratio, can differ. It redraws a near copy from its segment up to `privacy.near_copy_rounds` times. Rows that are redrawn later to satisfy an attribute rule or a unique key, and rows whose values change through them (a group effect), are checked again in the same rejection, for at most `privacy.near_copy_rounds` rounds and half of `rules.max_rejection_rounds`, so the remaining rounds enforce the rules alone. The sample log counts near copies, redraws and kept rows under `near_copies` (with `redrawn_by_rules`, `changed_by_rules` and `near_copy_rule_rounds` when rule redraws ran); `kept` counts the near copies among the rows emitted, recounted after the rule redraws and group effects of the last step. For an entity with a version history (SCD type 2), it is counted before the versions are added. A row redrawn only for being a near copy that ends the rule budget breaking a rule gets its earlier rule-valid row back, hidden draws included, and the log counts those rows (`restored_after_rules`). + +The guard is a rule about distances. It is **not** a privacy guarantee. Whether a row is redrawn depends on the real rows near it. If the guard redraws until it succeeds, someone who knows the rule learns that no synthetic row sits inside a real record's ball, and Prop. 1723 shows that this rules out differential privacy for any ε, given a generator that can produce rows near the record. With a finite number of rounds some near copies are kept, and the sample log counts them. Prop. 1723 does not cover that case, and the guard gives no differential-privacy guarantee of its own. In DP mode (`synthesis.privacy.dp`) the engine is fitted on the DP seed database, so the guard measures distances to seed rows and is post-processing of the DP release: the DP guarantee stands, and the guard adds nothing to it. Ganev and De Cristofaro (2023) reconstruct records from exactly such holes. After the guard, the `dcr_ratio` of a guarded table no longer tests the generator on its own. The attacks above remain the privacy test, and DP mode remains the guarantee. + ## Limits - Results hold for the attacks run, not for every possible attack. Only differential privacy gives a guarantee against all attacks; this module measures, it does not protect. diff --git a/docs/proofs/relsynth-proofs.tex b/docs/proofs/relsynth-proofs.tex index bfa96cf..e3da535 100644 --- a/docs/proofs/relsynth-proofs.tex +++ b/docs/proofs/relsynth-proofs.tex @@ -406,6 +406,7 @@ \subsection{Defaults} \input{sec_v40_documents.tex} \input{sec_v50_text.tex} \input{sec_v60_agents.tex} +\input{sec_g3_smalldims.tex} \input{sec_v22_quickcheck.tex} %===================================================================== @@ -556,5 +557,6 @@ \section{Results}\label{sec:results} \bibitem{mironov2012} I.~Mironov. On significance of the least significant bits for differential privacy. \emph{ACM CCS}, 2012. \bibitem{wilson2020} R.~J.~Wilson, C.~Y.~Zhang, W.~Lam, D.~Desfontaines, D.~Simmons-Marengo and B.~Gipson. Differentially private SQL with bounded user contribution. \emph{PoPETs}, 2020(2), 2020. \bibitem{desfontaines2022} D.~Desfontaines, J.~Voss, B.~Gipson and C.~Mandayam. Differentially private partition selection. \emph{PoPETs}, 2022(1), 2022. +\bibitem{ganev2023dcr} G.~Ganev, E.~De~Cristofaro (2023). On the inadequacy of similarity-based privacy metrics: reconstruction attacks against ``truly anonymous synthetic data''. arXiv:2312.05114. \end{thebibliography} \end{document} diff --git a/docs/proofs/sec_g3_smalldims.tex b/docs/proofs/sec_g3_smalldims.tex new file mode 100644 index 0000000..bcabce5 --- /dev/null +++ b/docs/proofs/sec_g3_smalldims.tex @@ -0,0 +1,43 @@ +%===================================================================== +\section{Small dimensions: segment shrinkage and the near-copy guard (v2.2)}\label{sec:g3small} +% numbered from 1720 (gap plan G3 range); the shared counter is restored at the end +\newcounter{savedpropgthree}\setcounter{savedpropgthree}{\value{proposition}} +A small dimension (AdventureWorks \texttt{DimProduct}, 606 rows) is fitted by many latent segments of few rows: its strata give 24 segments of 6 to 49 rows, none large enough for its own copula (\texttt{copula.min\_rows}), so every segment already uses the pooled copula and only its marginals are its own. With the Dirichlet weight $a_0=0.25$ (\texttt{marginals.dirichlet\_prior}) a segment marginal is close to the empirical law of its rows, and synthetic rows land closer to real rows than real rows are to each other (\texttt{dcr\_ratio} 0.40--0.65 at the defaults, limit 0.5). This section gives the two remedies of \texttt{relsynth} 2.2 (guide \texttt{docs/privacy.md}, section ``Small dimensions''; code \texttt{relsynth/generators/copula.py}, \texttt{relsynth/nearcopy.py}). Each proposition is tested by a check in \texttt{tests/science/test\_small\_dims\_checks.py} (C1720--C1723). + +\setcounter{proposition}{1719} +\begin{proposition}[Segment shrinkage]\label{prop:segshrink} +Let a segment of an entity table have $n$ rows, empirical law $e$ of a column, and let $F_0$ be the table's marginal of that column. With $a=\max(a_0,\texttt{segment\_prior\_rows})$, the segment marginal is the posterior $\mathrm{DP}(a+n,\,(aF_0+ne)/(a+n))$ for a continuous column and $\mathrm{Dir}(c+ap_0)$ for a discrete one ($c$ the level counts, $p_0$ the table's level probabilities). (a) Its posterior predictive is $w\,e+(1-w)F_0$ with $w=n/(n+a)$: a synthetic value comes from the segment's own rows with probability $w$. (b) The number of random numbers drawn does not depend on $a$, so changing $a$ changes values, not the random stream. +\end{proposition} +\begin{proof} +(a) For the discrete case $E[p]=(c+ap_0)/(n+a)=w\,e+(1-w)p_0$. For the continuous case the code draws $V\sim\mathrm{Beta}(n,a)$, Bayesian-bootstrap weights on the $n$ rows and equal weights on the quantile knots of $F_0$ (the representation of the DP posterior in Prop.~\ref{prop:dp}); $E[V]=n/(n+a)=w$. (b) One Dirichlet (discrete) or one Beta plus one Dirichlet over the $n$ rows (continuous) is drawn per segment and column whatever $a$ is. +\end{proof} +\noindent Check C1720 draws 20{,}000 posterior marginals of a segment for $a\in\{0.25,10,20\}$ and compares the share of predictive mass on the segment's own atoms with $w$, and replays one fit with two values of $a$ to show that the generator's random state ends at the same place. + +\setcounter{proposition}{1720} +\begin{proposition}[Large segments are unchanged]\label{prop:seglarge} +In the notation of Prop.~\ref{prop:segshrink}, the total variation distance between the predictive laws under $a$ and under $a_00$), $M_B(X)$ puts positive probability on the ball $\{s: d(s,x)<\rho\nu(x)\}$ minus the guard regions of $X$, and $M_B(X\cup\{x\})$ puts none there, unless that set is empty. So $M_B$ is not $\varepsilon$-differentially private for any finite $\varepsilon$, and the absence of synthetic rows around a real record is evidence of the record. +\end{proposition} +\begin{proof} +Under $X\cup\{x\}$ every synthetic row whose nearest real row is $x$ and lies within $\rho\nu(x)$ of it is in $B$ and is never accepted; a row nearer to another real row than to $x$ lies outside that ball. Under $X$ the same set is outside $B$ (the rows of $X$ are farther than their own guard radii, by the choice of the set) and has positive probability by the density assumption. A ratio of probabilities $p/0$ is unbounded. +\end{proof} +\noindent With a finite number of rounds $R$ some near copies are kept, and the proposition does not apply; the guard still adds no guarantee of its own. In DP mode stage~2 is fitted on the DP seed database, so the guard reads seed rows only and is post-processing of the DP release, which keeps its $(\varepsilon,\delta)$. This is the hole that Ganev and De Cristofaro \cite{ganev2023dcr} use to reconstruct records from outputs filtered by distance to closest record. relsynth therefore reports the guard as a distance rule, never as privacy: \texttt{relsynth privacy} (Section~\ref{sec:v25}) remains the test and DP mode (Section~\ref{sec:v25dp}) the guarantee, and the scorecard marks the \texttt{dcr\_ratio} of a guarded table as not independent of the generation. Check C1723 builds the example in one dimension and finds the empty ball. + +\setcounter{proposition}{\value{savedpropgthree}} diff --git a/docs/research/small-dimensions.md b/docs/research/small-dimensions.md new file mode 100644 index 0000000..4844014 --- /dev/null +++ b/docs/research/small-dimensions.md @@ -0,0 +1,74 @@ +# Small dimensions: privacy margin of DimProduct (gap plan G3) + +Gap plan G3 (release 2.2.0). The AdventureWorks autotune runs ([autotune-adventureworks.md](autotune-adventureworks.md)) found the distance-to-closest-record ratio (`dcr_ratio`) of `DimProduct` close to its 0.5 limit at the 2.1 defaults (0.40 on one seed in five). This page records the study of the remedies, the choice, and the validation. Propositions 1720-1723 are in `docs/proofs/sec_g3_smalldims.tex`, with checks C1720-C1723 in `tests/science/test_small_dims_checks.py`. The user guide is [docs/privacy.md](../privacy.md#small-dimensions). + +## Cause + +`DimProduct` has 606 rows. Its strata give 24 latent segments of 6 to 49 rows. Every segment is below `copula.min_rows` (80), so every segment already uses the table's pooled copula, and only the marginals are the segment's own. With the Dirichlet weight 0.25 (`marginals.dirichlet_prior`), a segment marginal is almost the empirical law of its few rows. A third of the real rows are exact duplicates (components with every optional attribute null), so the check compares the positive parts of the distance distributions. Synthetic rows that match a real product's categories and land between two of its family's prices sit closer to that product than any other real product does. + +## Options studied + +All numbers are `DimProduct` `dcr_ratio` (limit 0.5) and the AdventureWorks scorecard (1,257 items in this evaluation, without the privacy items). They come from two fitted models (seeds 1 and 12) and two sampling streams each. Every variant re-samples the same fitted model with the same random streams, so the comparisons are paired. + +| Variant | DCR (4 paired draws) | Scorecard change vs defaults (paired) | +|---|---|---| +| 2.1 defaults | 0.473, 0.567, 0.421, 0.575 | — | +| segment cap 4 for DimProduct (`segments.tables`) | 0.498 (one full fit) | −3 | +| copula pooled toward the table | no effect: no segment of DimProduct has its own copula | — | +| marginal shrinkage, 10 rows | 0.902, 1.079, 0.932, 0.808 | −6, −3, +1, +2 | +| **marginal shrinkage, 20 rows** | **1.026, 0.985, 1.037, 0.943** | −2, −10, −1, +1 | +| near-copy guard alone (ratio 1) | 1.105, 0.518, 0.774, 0.529 | −6, −1, −3, +3 | +| shrinkage 10 rows + guard | 1.135, 1.137 | −3, −4 | + +- **Segment cap.** The cap applies per stratum, and the strata make the segments, so a lower cap barely changes them. It did not help, and it cost joins. +- **Copula pooling.** Small segments already share the pooled copula, so pooling has nothing to act on here. It was dropped from the change. +- **Marginal shrinkage** (`marginals.segment_prior_rows`) raises the weight of the table marginal in each segment's Dirichlet / Dirichlet-process posterior to that many rows. It is the hierarchical prior relsynth already has, made stronger for small segments. Segments of thousands of rows move by at most `rows / n` in total variation (Prop. 1721), and the random stream is unchanged (Prop. 1720). It works on every draw. +- **Near-copy guard** (`privacy.near_copy`). It redraws rows that sit closer to a real row than that row's nearest distinct neighbour. Alone it is unreliable (0.52 on two draws), because the evaluation's metric ranks real and synthetic rows together and looks only at positive distances. It also rewrote rows in six AdventureWorks tables (24% of `ProspectiveBuyer`, 5% of `DimCustomer`), and on Chinook it broke `Track` associations on 4 of 5 seeds. Redrawing until it succeeds, it is not differentially private for any ε (Prop. 1723); with finite rounds it gives no DP guarantee of its own, and in DP mode it is post-processing of the seed database. It makes the DCR item of a guarded table circular. It ships as an opt-in safety net, off by default. + +**Choice:** marginal shrinkage with 20 rows was the first choice for the default, on entity tables only. The full validation below and the science suite changed that: both settings ship opt-in, and the 2.1 output stays the default (see "Science checks"). + +**Cost.** Over the four paired draws the AdventureWorks scorecard drops by 3 items on average out of 1,257. The draw-to-draw spread at the defaults is about ±5. The items that fail more often are systematic: `FactInternetSales` joined product prices (`ListPrice`, `DealerPrice`, `StandardCost` `join_ks`, 3 of 4 draws), `StartDate` `join_ks`, and the `FactInternetSales` discriminator. Internet sales are concentrated on a few products. When those products' prices are shrunk toward the catalogue, the price distribution weighted by sales moves. This is the price of not reproducing small product families. A deployment that needs these joins exactly can set `marginals.segment_prior_rows: 0` (the 2.1 behaviour), or set it per table under `per_table`. + +## Other golden databases + +| Database | Defaults 2.1 | Shrinkage 20 | Guard on (with shrinkage) | +|---|---|---|---| +| Chinook, 5 seeds, scorecard | 142/142 on every seed | 142/142 on every seed | 142/142 on seed 1, 141/142 on seeds 2-5 (`Track` association) | +| Chinook `Track` DCR | 1.63-1.79 | 1.83-2.10 | 2.83-3.35 | + +Contoso and Berka run in the CI bench job of the pull request. + +## Validation of shrinkage 20 (AdventureWorks, 5 seeds) + +Five full fits per setting (seeds 1, 2, 3, 11, 12), each fitted and sampled once with the same seed, scored on the 1,267-item AdventureWorks scorecard. "2.1" is `marginals.segment_prior_rows: 0`. + +| Seed | DimProduct DCR, 2.1 | DimProduct DCR, shrinkage 20 | Scorecard, 2.1 | Scorecard, shrinkage 20 | Change | +|---|---|---|---|---|---| +| 1 | 0.501 | 0.969 | 1215 | 1202 | −13 | +| 2 | 0.648 | 0.963 | 1202 | 1202 | 0 | +| 3 | 0.576 | 1.010 | 1214 | 1201 | −13 | +| 11 | 0.549 | 0.963 | 1207 | 1198 | −9 | +| 12 | 0.581 | 1.101 | 1202 | 1198 | −4 | +| Mean | 0.571 | 1.001 | 1208.0 | 1200.2 | −7.8 | + +The DCR ratio of `DimProduct` is at least 0.96 on every seed (limit 0.5), against 0.50-0.65 at the 2.1 defaults. The scorecard cost on full fits is larger than on the paired re-samples above: 7.8 items on average (0.6% of 1,267), at most 13. The other dimension tables keep DCR ratios of at least 0.97 (`DimReseller`, seed 12). + +With shrinkage on, `tools/identity_check.py` reports every scenario as different, as expected. With `marginals.segment_prior_rows: 0`, now the default, it reproduces `main` byte for byte on all 24 runs. + +## Science checks + +With shrinkage 20 on by default, three fake-data checks of the science suite fail, and pass again with 0 (same code, local run): + +| Check | With 0 (2.1) | With 20 | +|---|---|---| +| C09 snowflake: Spearman(sale amount, product price) across 3 tables, real 0.909 | 0.910 ± 0.002 | 0.799 ± 0.026 (z = 4.2) | +| C19 doctor: planted problems left after tuning | none | `D02 customer` | +| C43 sales warehouse: bike share in segment A / C orders, target 0.60 / 0.03 ± 0.08 | pass | 0.51 / 0.04 | + +The Berka bench also failed one calibrated item (`order` association, 0.155 against a limit of 0.15). The cause is the same as the AdventureWorks price joins: the segments of a dimension carry what the facts depend on (which products sell, at what price), and shrinking each segment toward the table weakens exactly that link. Shrinkage that keeps each segment's location and shrinks only its shape is a possible next step; it has not been tried. + +## Open items and decisions for Cod3r + +- **Default (decision).** Both settings are opt-in in this change, so the default output is byte-identical with 2.1 and the `DimProduct` margin at the defaults is unchanged (0.50-0.65). Turning shrinkage on by default needs Cod3r's approval and the science checks above revised or a variant that keeps the segment links. +- The autotune rerun the gap plan asks for (`segments.default_cap`, the doctor's parts, `facts.*`) needs the CI autotune workflow (4.5 hours). It is best started after this pull request merges. `marginals.segment_prior_rows` is registered as tunable. +- `dcr_ratio` remains a guard, not a privacy test (T06). The privacy attacks (`relsynth privacy`) on AdventureWorks have not been rerun for this change. diff --git a/docs/settings-reference.md b/docs/settings-reference.md index a3d08c1..b4791f8 100644 --- a/docs/settings-reference.md +++ b/docs/settings-reference.md @@ -48,6 +48,7 @@ Generated from `relsynth.settings` (`python -m relsynth.settings > docs/settings | `marginals.discrete_int_max` | int | `30` | 0 … 1e+07 | | Integer columns with at most this many distinct values are modelled as discrete. | | `marginals.max_obs` | int | `20000` | 10 … 1e+09 | yes (log) | Observations kept per continuous marginal (a seeded subsample above it). | | `marginals.fd_sample_rows` | int | `50000` | 100 … 1e+10 | | Rows of the seeded subsample on which functional dependencies are tested. | +| `marginals.segment_prior_rows` | float | `0.0` | 0 … 200 | yes | Prior weight, in rows, of an entity table's marginal in each segment's marginal (hierarchical shrinkage; a segment of n rows keeps about n / (n + this) of its own distribution). 0 = off (2.1); 20 makes near copies in small dimensions rarer but weakens the link between their segments and the facts. | | `facts.min_group_rows` | int | `30` | 0 … 1e+09 | yes (log) | Rows below which a block borrows the coarser dimension's marginals. | | `facts.frailty` | choice | `auto` | auto, on, off | yes | auto | on | off: heavy-tailed rows per parent (gamma frailty). | | `facts.frailty_threshold` | float | `1.5` | 1 … 1e+06 | yes (log) | Dispersion index above which frailty is used. | @@ -100,6 +101,10 @@ Generated from `relsynth.settings` (`python -m relsynth.settings > docs/settings | `settings_file` | str | `None` | | | A settings file merged under this block (a path relative to the spec, or preset:fast / preset:thorough). The spec's own keys win. See docs/settings.md. | | `privacy.suppress_small_counts` | int | `5` | 0 … 1e+06 | | Counts below this are suppressed in the diagnostics bundle. | | `privacy.dp` | map | `None` | | | Differential privacy mode: unit, epsilon or rho, delta, public_tables, public, max_degree, spec_is_public and expert keys; validated by relsynth.dp (docs/dp.md). Off when absent. | +| `privacy.near_copy` | choice | `off` | auto, off | | auto: redraw entity rows that sit closer to a real row than that row's nearest distinct real neighbour (tables the distance check covers, up to near_copy_max_rows rows); off. A distance rule, not a privacy guarantee (docs/privacy.md). | +| `privacy.near_copy_ratio` | float | `1.0` | 0 … 100 | | A synthetic row is a near copy when its distance to its nearest real row is below this times that row's nearest-distinct-neighbour distance. | +| `privacy.near_copy_rounds` | int | `20` | 0 … 1e+06 | | Redraws of a near copy before it is kept and counted in the sample log. | +| `privacy.near_copy_max_rows` | int | `20000` | 0 … 1e+12 | | Largest entity table the near-copy guard runs on (it compares every synthetic row with every distinct real row, in memory-bounded blocks; time grows with the square of the rows). | | `pushdown.mode` | choice | `auto` | auto, on, off | | auto | on | off: fit eligible fact tables from exact statistics computed where the data lives (row counts, rows per key and key pair, distinct keys, rank-score sums) and a uniform sample for their attribute models, instead of loading them. auto: facts with at least min_rows rows; on: every eligible fact; off: load every table. | | `pushdown.min_rows` | int | `1000000` | 0 … 1e+15 | | Rows from which auto mode pushes a fact table down. | | `pushdown.sample_rows` | int | `500000` | 1 … 1e+12 | yes (log) | Rows of a pushed-down fact sampled (seeded, uniform) for its attribute models; the relational model always uses every row through the statistics. | diff --git a/docs/spec-reference.md b/docs/spec-reference.md index 86c96f7..b33d6c9 100644 --- a/docs/spec-reference.md +++ b/docs/spec-reference.md @@ -682,6 +682,7 @@ Every key below is checked by `validate`. Any other key is an error, with the cl | `marginals.discrete_int_max` | `30` | Integer columns with at most this many distinct values are modelled as discrete. | | `marginals.max_obs` | `20000` | Observations kept per continuous marginal (a seeded subsample above it). | | `marginals.fd_sample_rows` | `50000` | Rows of the seeded subsample on which functional dependencies are tested. | +| `marginals.segment_prior_rows` | `0.0` | Prior weight, in rows, of an entity table's marginal in each segment's marginal (hierarchical shrinkage; a segment of n rows keeps about n / (n + this) of its own distribution). 0 = off (2.1); 20 makes near copies in small dimensions rarer but weakens the link between their segments and the facts. | | `facts.min_group_rows` | `30` | Rows below which a block borrows the coarser dimension's marginals. | | `facts.frailty` | `auto` | auto | on | off: heavy-tailed rows per parent (gamma frailty). | | `facts.frailty_threshold` | `1.5` | Dispersion index above which frailty is used. | @@ -734,6 +735,10 @@ Every key below is checked by `validate`. Any other key is an error, with the cl | `settings_file` | `None` | A settings file merged under this block (a path relative to the spec, or preset:fast / preset:thorough). The spec's own keys win. See docs/settings.md. | | `privacy.suppress_small_counts` | `5` | Counts below this are suppressed in the diagnostics bundle. | | `privacy.dp` | `None` | Differential privacy mode: unit, epsilon or rho, delta, public_tables, public, max_degree, spec_is_public and expert keys; validated by relsynth.dp (docs/dp.md). Off when absent. | +| `privacy.near_copy` | `off` | auto: redraw entity rows that sit closer to a real row than that row's nearest distinct real neighbour (tables the distance check covers, up to near_copy_max_rows rows); off. A distance rule, not a privacy guarantee (docs/privacy.md). | +| `privacy.near_copy_ratio` | `1.0` | A synthetic row is a near copy when its distance to its nearest real row is below this times that row's nearest-distinct-neighbour distance. | +| `privacy.near_copy_rounds` | `20` | Redraws of a near copy before it is kept and counted in the sample log. | +| `privacy.near_copy_max_rows` | `20000` | Largest entity table the near-copy guard runs on (it compares every synthetic row with every distinct real row, in memory-bounded blocks; time grows with the square of the rows). | | `pushdown.mode` | `auto` | auto | on | off: fit eligible fact tables from exact statistics computed where the data lives (row counts, rows per key and key pair, distinct keys, rank-score sums) and a uniform sample for their attribute models, instead of loading them. auto: facts with at least min_rows rows; on: every eligible fact; off: load every table. | | `pushdown.min_rows` | `1000000` | Rows from which auto mode pushes a fact table down. | | `pushdown.sample_rows` | `500000` | Rows of a pushed-down fact sampled (seeded, uniform) for its attribute models; the relational model always uses every row through the statistics. | diff --git a/src/relsynth/engine.py b/src/relsynth/engine.py index 62ae621..4c69344 100644 --- a/src/relsynth/engine.py +++ b/src/relsynth/engine.py @@ -380,6 +380,7 @@ def fit(self, tables: dict[str, pd.DataFrame], stats=None): samples to their exact statistics (relsynth.pushdown.stats.TableStats): the relational model reads those facts only through the statistics.""" t0 = time.time() + self.__dict__.pop("_nc_guards", None) # near-copy guards are built from this fit's rows spec, cfg, rng = self.spec, self.cfg, self.rng T = spec["tables"] self.fstats = {t: s for t, s in (stats or {}).items() if t in T and T[t]["role"] == "fact"} @@ -2542,6 +2543,7 @@ def _sample_entities(self, scale=None, index=None): z = allocate(ent.pi / ent.pi.sum(), n, rng) par = None df = self.tm[e].sample(z, rng) + df = self._guard_near_copies(e, df, z, rng) # must stay before keys and _computed pk = T[e]["primary_key"] keys[e] = self._make_keys(e, len(z)) ko = T[e].get("key_order") @@ -2624,6 +2626,64 @@ def _sample_end(self, ctx): ext = sample_series(self, ext, ctx["prng"], ctx["index"], ctx["mode"]) return ext + def _near_copy_guard(self, e): + """The near-copy guard of entity e (relsynth.nearcopy), built on first use + from the fitted source rows; None when the guard does not apply: off, a + table the evaluation's distance check does not cover (fewer than 50 rows, + fewer than 3 distance columns, privacy: false), or more than + privacy.near_copy_max_rows rows.""" + cache = self.__dict__.setdefault("_nc_guards", {}) + if e in cache: + return cache[e] + pcfg = self.cfg.get("privacy") or {} + T = self.spec["tables"] + real = getattr(self, "_tables_for_fit", {}).get(e) + g = None + if pcfg.get("near_copy", "off") != "off" and real is not None and T[e].get("privacy", True) \ + and 50 <= len(real) <= int(pcfg.get("near_copy_max_rows", 20_000)): + from .nearcopy import NearCopyGuard, guard_columns + keycols = {(r["child"], c) for r in self.spec["relationships"] + for c in (r["child_key"] if r.get("composite") else [r["child_key"]])} + out_cols = set(self.tm[e].out_cols) + cols = [c for c in guard_columns(T[e], real, keycols, e) if c in out_cols] + if len(cols) >= 3: + g = NearCopyGuard(real, T[e], cols, rho=float(pcfg.get("near_copy_ratio", 1.0))) + cache[e] = g + return g + + def _guard_near_copies(self, e, df, z, rng): + """Redraw, from their own segments, the rows that are near copies of a real + row (Prop. 1722); rows still near copies after privacy.near_copy_rounds + rounds are kept and counted in the sample log. No random number is drawn + when no row is a near copy, so such tables are unchanged.""" + g = self._near_copy_guard(e) + if g is None or not len(df): + return df + bad = np.flatnonzero(g.near_copies(df)) + first = len(bad) + rounds = int((self.cfg.get("privacy") or {}).get("near_copy_rounds", 20)) + r = 0 + while len(bad) and r < rounds: + new = self.tm[e].sample(np.asarray(z)[bad], rng) + new.index = df.index[bad] + # column by column, keeping each column's dtype; the guard runs on the + # attribute frame, before keys, links and computed (hidden) columns exist + for c in new.columns: + col = df[c].to_numpy(dtype=object, copy=True) + col[bad] = new[c].to_numpy(dtype=object) + try: + df[c] = pd.Series(col, index=df.index).astype(df[c].dtype) + except (TypeError, ValueError): + df[c] = col + still = g.near_copies(df.iloc[bad]) + bad = bad[still] + r += 1 + if first: + self.sample_log.setdefault("near_copies", {})[e] = { + "rows": int(len(df)), "near_copies": first, "redrawn": first - int(len(bad)), + "kept": int(len(bad)), "rounds": r, "ratio": g.rho} + return df + def _stratum_keys(self, e, z, out): """Strata are rows of small copied parents and segments are nested in them, so a row's stratum columns are determined by its segment (exact by @@ -4285,7 +4345,29 @@ def _finish(self, t, df, z, out): xrules = [r_ for r_ in cons.get("key_rules", []) if T["role"] != "fact" and names_in(r_["expr"]) & set(tm.out_cols)] if arules or akeys else [] - def attr_bad(frame): + # near-copy guard (G3): rows whose attributes are redrawn here must not become + # near copies, so the guard joins the rejection for those rows only + ncg = self._near_copy_guard(t) if T["role"] != "fact" else None + redrawn = np.zeros(len(df), bool) + # the near-copy budget (privacy.near_copy_rounds), and at most half the rule + # rounds, so the rounds after it enforce the rules and keys alone + nc_rounds = min(int((self.cfg.get("privacy") or {}).get("near_copy_rounds", 20)), + int(self.cfg["rules"]["max_rejection_rounds"]) // 2) + rounds = 0 + nc_active = set() # rule rounds in which the near-copy rejection ran + fb_vals = {} # whole rows, hidden draws included (restored as they were) + has_fb = np.zeros(len(df), bool) + nc_base = df[ncg.cols].astype(object).where(df[ncg.cols].notna(), None) if ncg is not None else None + + def touched(frame): + # rows redrawn here, and rows whose guard columns changed through them + # (a group effect recomputed by _computed) + if nc_base is None: + return redrawn + cur = frame[ncg.cols].astype(object).where(frame[ncg.cols].notna(), None) + return redrawn | (cur.to_numpy() != nc_base.to_numpy()).any(axis=1) + + def attr_bad(frame, nc=True): ok = np.ones(len(frame), bool) res_ = self._resolver(t, frame, out) for r_ in arules + xrules: @@ -4294,15 +4376,28 @@ def attr_bad(frame): sub = frame[u].astype(object) sub = sub.where(sub.notna(), None) ok &= ~(sub.duplicated(keep="first").to_numpy() & sub.notna().all(axis=1).to_numpy()) + if nc and ncg is not None and redrawn.any() and rounds < nc_rounds: + nc_active.add(rounds) + ri = np.flatnonzero(touched(frame)) + ok[ri] &= ~ncg.near_copies(frame.iloc[ri]) return ~ok if arules or akeys: # rejection on the attributes jointly for all attribute rules (each redraw # is from the row's own conditional, so the accepted law is Q(x | rules)) bad = attr_bad(df) rounds = 0 - redrawn = np.zeros(len(df), bool) while bad.any() and rounds < self.cfg["rules"]["max_rejection_rounds"] and tm.out_cols: idx = np.flatnonzero(bad) + if ncg is not None: + # rows redrawn only for being near copies keep their rule- and + # key-valid values as a fallback (restored below if the budget ends + # on a redraw that breaks a rule) + ok_rule = idx[~attr_bad(df, nc=False)[idx]] + for c in df.columns: + if c not in fb_vals: + fb_vals[c] = np.empty(len(df), dtype=object) + fb_vals[c][ok_rule] = df[c].astype(object).to_numpy()[ok_rule] + has_fb[ok_rule] = True redrawn[idx] = True gv = getattr(self, "_given", {}).get(t) new = tm.sample(groups[idx], rng, gv.iloc[idx] if gv is not None else None) @@ -4317,6 +4412,27 @@ def attr_bad(frame): df = self._computed(t, df, out) bad = attr_bad(df) rounds += 1 + if ncg is not None and has_fb.any(): + back = np.flatnonzero(attr_bad(df, nc=False) & has_fb) + if len(back): + # the whole stashed row, with its hidden draws and their flag, so the + # recompute reproduces the rule-valid values instead of drawing again + for c in fb_vals: + if c not in df.columns: + continue + col = df[c].astype(object).to_numpy(copy=True) + col[back] = fb_vals[c][back] + try: + df[c] = pd.Series(col, index=df.index).astype(df[c].dtype) + except (TypeError, ValueError): + df[c] = col + df = self._computed(t, df, out) + log_r = self.sample_log.setdefault("near_copies", {}).setdefault(t, { + "rows": int(len(df)), "near_copies": 0, "redrawn": 0, "kept": 0, "rounds": 0, + "ratio": ncg.rho}) + log_r["restored_after_rules"] = int(len(back)) + # rechecked after the recompute (rules and keys): expected 0 + log_r["restored_still_bad"] = int(attr_bad(df, nc=False)[back].sum()) anch = [r for r in self.fk[t] if r.get("anchor") and r["anchor"] in df.columns] if redrawn.any() and anch and z is not None: # rows whose attributes were redrawn: their anchored links (the area of @@ -4338,6 +4454,21 @@ def attr_bad(frame): res_ = self._resolver(t, df, out) for r in arules: viol[r["expr"]] = int((~check_mask(df, r["expr"], res_)).sum()) + if ncg is not None: + # the log describes the rows emitted: near copies among the final attributes, + # after rule redraws and group effects (computed columns) + kept_final = int(ncg.near_copies(df).sum()) + log_all = self.sample_log.setdefault("near_copies", {}) + if t in log_all or kept_final or redrawn.any(): + log_nc = log_all.setdefault(t, { + "rows": int(len(df)), "near_copies": 0, "redrawn": 0, "kept": 0, "rounds": 0, "ratio": ncg.rho}) + if redrawn.any(): + log_nc["redrawn_by_rules"] = int(redrawn.sum()) + log_nc["changed_by_rules"] = int(touched(df).sum()) + log_nc["near_copy_rule_rounds"] = len(nc_active) + log_nc["kept"] = kept_final + elif not log_all: + self.sample_log.pop("near_copies") if T["role"] == "fact" and cons.get("key_rules") and t not in self.fact_entity: # rows still invalid after the per-row redraws of _refill: removed and reported keep = np.ones(len(df), bool) diff --git a/src/relsynth/evaluate.py b/src/relsynth/evaluate.py index 5e8862e..5a32216 100644 --- a/src/relsynth/evaluate.py +++ b/src/relsynth/evaluate.py @@ -988,6 +988,11 @@ def _size_factor(t): if privacy and ts["role"] == "entity" and ts.get("privacy", True) and ts.get("mode") != "copy" \ and len(pcols) >= 3 and len(real[t]) >= 50: rep["privacy"][t] = dcr(real[t], syn[t], ts, pcols) + nc = ((sample_log or {}).get("near_copies") or {}).get(t) + if nc and rep["privacy"][t]: + # the near-copy guard (relsynth.nearcopy) read these distances while + # generating: the ratio no longer tests the generator on its own + rep["privacy"][t]["near_copy_guard"] = nc for c, cs in ts["columns"].items(): if cs["type"] == "text": o = cs.get("text") or {} @@ -1438,7 +1443,11 @@ def tvd_alpha(lim, mu, sd): add(n, "text_tstr_ratio", d["tstr_ratio"], th["tstr_ratio"], d["tstr_ratio"] >= th["tstr_ratio"]) for n, d in rep["privacy"].items(): if d: + before = len(items) add(n, "dcr_ratio", d["dcr_ratio"], th["dcr_ratio"], d["dcr_ratio"] >= th["dcr_ratio"]) + if d.get("near_copy_guard") and len(items) > before: + items[-1]["note"] = ("near-copy guard on (privacy.near_copy): rows were redrawn by this distance, " + "so the ratio is not independent evidence (docs/privacy.md)") if "exact_match_rate_real_any" in d: # null value: the share of real rows equal to another real row, plus 2.5 sd e = d["exact_match_rate_real_any"] diff --git a/src/relsynth/generators/copula.py b/src/relsynth/generators/copula.py index d1c5a78..ed85ec3 100644 --- a/src/relsynth/generators/copula.py +++ b/src/relsynth/generators/copula.py @@ -36,21 +36,33 @@ def _fit_marginals(self, rng, bootstrap): self.group_m, self.group_n = {}, {} self.coarse_m = {} min_rows = ccfg["min_rows"] + # prior weight, in rows, of the table's (or coarse group's) marginal in each + # segment's marginal (G3, Prop. 1720; opt-in, default 0): a = max(dirichlet_prior, segment_prior_rows) + a_seg = max(mcfg["dirichlet_prior"], self._segment_prior()) + kw_seg = {**kw, "dirichlet_a": a_seg} if parent: cg = np.array([parent.get(g, -1) for g in groups]) if n else np.array([]) for c in np.unique(cg): if c < 0: continue idx = cg == c - self.coarse_m[c] = [Marginal.fit(cd, x[idx], rng, prior=gm, min_rows=min_rows, **kw) + self.coarse_m[c] = [Marginal.fit(cd, x[idx], rng, prior=gm, min_rows=min_rows, **kw_seg) for cd, x, gm in zip(d.codecs, d.X, self.global_m)] for g in np.unique(groups): idx = groups == g prior = self.coarse_m.get(parent.get(g, -1), self.global_m) self.group_n[g] = int(idx.sum()) - self.group_m[g] = [Marginal.fit(cd, x[idx], rng, prior=pm, min_rows=min_rows, **kw) + self.group_m[g] = [Marginal.fit(cd, x[idx], rng, prior=pm, min_rows=min_rows, **kw_seg) for cd, x, pm in zip(d.codecs, d.X, prior)] + def _segment_prior(self): + """Prior weight, in rows, of the table's marginal in each segment's + marginal (marginals.segment_prior_rows); entity tables only, since the + segments of a fact table are blocks of the relational model, not records.""" + if getattr(self.data, "role", None) != "entity": + return 0.0 + return float(self.data.cfg["marginals"].get("segment_prior_rows", 0) or 0) + def _redraw_marginals(self, rng, mode): if self.data.cfg["marginals"]["bayesian_bootstrap"]: # mean mode runs the posterior draws with their means (and median diff --git a/src/relsynth/nearcopy.py b/src/relsynth/nearcopy.py new file mode 100644 index 0000000..423c06b --- /dev/null +++ b/src/relsynth/nearcopy.py @@ -0,0 +1,178 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Near-copy guard for entity tables (G3; docs/proofs/sec_g3_smalldims.tex, Props. 1720-1723). + +A small dimension (a product list of a few hundred rows) is fitted by a few +segments of a few dozen rows each. Its segment marginals are then close to +the empirical distribution of those rows, and a synthetic row can land closer +to a real row than any other real row is: the case the distance-to-closest-record +check (``dcr_ratio``) flags. + +The guard redraws such rows. A synthetic row ``s`` with nearest real row ``r`` +at distance ``d`` is a *near copy* when + +* ``0 < d < rho * nn(r)``, where ``nn(r)`` is the distance from ``r`` to its + nearest real row at a positive distance (``rho`` = ``privacy.near_copy_ratio``), or +* ``d == 0`` and ``r`` has no exact twin among the real rows (a copy of a unique + record). A copy of a real row that other real rows share (a coarse attribute + space, such as components with every optional attribute null) is not a near + copy: those rows are judged by the exact-match rate. + +A near copy is drawn again from its own segment, up to ``privacy.near_copy_rounds`` +times; a row that is still a near copy after the last round is kept and counted. + +What this does and does not mean (Prop. 1723): after the guard no synthetic row +of the table is closer, in this metric, to a real record than that record's +nearest distinct real neighbour (times rho), unless the rounds ran out, which +the sample log reports. It is a rule about distances, not a privacy guarantee: +it adds no differential privacy of its own, and when it redraws until it +succeeds an attacker who knows the rule learns that no synthetic row sits inside +a real record's ball (Prop. 1723; Ganev & De Cristofaro 2023 reconstruct +records from such holes). In DP mode the engine is fitted on the DP seed +database, so the guard reads seed rows and is post-processing of the DP release. +The membership and attribute-inference attacks of ``relsynth privacy`` (T06) +remain the privacy test, and DP mode (``synthesis.privacy.dp``) the guarantee. +The DCR check that reads the same distances is no longer independent evidence +for a guarded table; the scorecard says so. + +The metric is the evaluation's (relsynth.evaluate.dcr): rank normal scores +``z, z^2`` and a null flag for numeric columns, one-hot top-20 levels for +categorical ones, over the table's determining columns, except that the scores +are fitted on the real rows only (the synthetic rows do not move the scale). +""" +from __future__ import annotations + +from typing import Any + +import numpy as np +import pandas as pd +from scipy.special import ndtri + +MODELLED = ("integer", "float", "categorical", "boolean", "date", "datetime") +TOP_LEVELS = 20 + + +def _is_cat(cs, s): + """True when the evaluation treats the column as categorical (one-hot).""" + return cs["type"] in ("categorical", "boolean") or (cs["type"] == "integer" and s.nunique() <= 30) + + +def _num(s: pd.Series) -> np.ndarray: + """The column as floats (datetimes as nanoseconds), NaN where missing.""" + if pd.api.types.is_datetime64_any_dtype(s): + v = s.to_numpy(dtype="datetime64[ns]").astype("int64").astype(float) + v[s.isna().to_numpy()] = np.nan + return v + return pd.to_numeric(s, errors="coerce").to_numpy(dtype=float) + + +def nearest(A: np.ndarray, B: np.ndarray, exclude_self: bool = False, positive: bool = False, + max_cells: int = 4_000_000): + """(index, distance) of each row of A's nearest row of B. exclude_self: A is + B and a row is not its own neighbour (an exact twin of it still is); + positive: only rows at a distance > 0 count (inf when there is none). + Distances to the chosen row are recomputed by subtraction, so exact copies + have distance exactly 0. + + The search runs over the distinct rows of B, in blocks of at most max_cells + distances (about 32 MB), so memory is bounded whatever the table size; time + is O(|A| |distinct B| p), which privacy.near_copy_max_rows bounds.""" + n = len(A) + j_out = np.zeros(n, dtype=np.int64) + d_out = np.full(n, np.inf) + if not len(B): + return j_out, d_out + U, first, inv, cnt = np.unique(B, axis=0, return_index=True, return_inverse=True, return_counts=True) + inv = np.asarray(inv).reshape(-1) + # a second occurrence of each distinct row (its twin), for exclude_self + order = np.argsort(inv, kind="stable") + starts = np.concatenate([[0], np.cumsum(cnt)[:-1]]) + second = np.where(cnt > 1, order[np.minimum(starts + 1, len(order) - 1)], -1) + uu = (U * U).sum(1) + step = max(1, int(max_cells) // max(1, len(U))) + for i in range(0, n, step): + a = A[i:i + step] + rows = np.arange(len(a)) + D = np.maximum((a * a).sum(1)[:, None] + uu[None] - 2 * a @ U.T, 0.0) + # exact ties computed by subtraction (the expansion above is inexact) + same = np.zeros_like(D, dtype=bool) + for q in np.flatnonzero((D < 1e-9).any(1)): + close = np.flatnonzero(D[q] < 1e-9) + same[q, close] = ((U[close] - a[q]) ** 2).sum(1) == 0 + if positive: + D[same] = np.inf + elif exclude_self: + own = inv[i:i + len(a)] + D[rows[cnt[own] == 1], own[cnt[own] == 1]] = np.inf # a unique row is not its own neighbour + k = D.argmin(1) + d = np.sqrt(((a - U[k]) ** 2).sum(1)) + d[~np.isfinite(D[rows, k])] = np.inf + j = first[k] + if exclude_self: + # the chosen row is the row itself: report its twin + selfpick = j == i + rows + j[selfpick] = np.where(second[k[selfpick]] >= 0, second[k[selfpick]], j[selfpick]) + j_out[i:i + len(a)], d_out[i:i + len(a)] = j, d + return j_out, d_out + + +class NearCopyGuard: + """The near-copy rule of one table, fitted on its real rows.""" + + def __init__(self, real: pd.DataFrame, tspec: dict, cols: list[str], rho: float = 1.0): + """Fit the encoding on the real rows and store each real row's distance to + its nearest distinct real row and whether it has an exact twin.""" + self.cols, self.rho = list(cols), float(rho) + self.enc: list[tuple[str, str, Any]] = [] + for c in self.cols: + cs, s = tspec["columns"][c], real[c] + if _is_cat(cs, s): + self.enc.append(("cat", c, list(s.astype(str).value_counts().index[:TOP_LEVELS]))) + else: + x = _num(s) + ok = np.sort(x[~np.isnan(x)]) + self.enc.append(("num", c, ok)) + self.X = self.encode(real) + _, self.nn = nearest(self.X, self.X, exclude_self=True, positive=True) + _, d0 = nearest(self.X, self.X, exclude_self=True) + self.has_twin = d0 == 0 + self.n_real = len(real) + + def encode(self, df: pd.DataFrame) -> np.ndarray: + """The rows of df in the distance metric: one-hot top levels for categorical + columns; normal score, its square and a null flag for numeric ones.""" + feats = [] + for kind, c, ref in self.enc: + s = df[c] + if kind == "cat": + v = s.astype(str).to_numpy() + feats.append(np.column_stack([(v == t).astype(float) for t in ref]) if ref + else np.zeros((len(df), 0))) + else: + x = _num(s) + ok = ~np.isnan(x) + sc = np.zeros(len(x)) + if ok.any() and len(ref): + # mid-rank of x among the real values, as the evaluation's rank / (n + 1) + lo = np.searchsorted(ref, x[ok], side="left") + hi = np.searchsorted(ref, x[ok], side="right") + sc[ok] = ndtri(np.clip(((lo + hi + 1) / 2) / (len(ref) + 1), 1e-6, 1 - 1e-6)) + feats.append(np.column_stack([sc, sc ** 2, (~ok).astype(float)])) + return np.column_stack(feats) if feats else np.zeros((len(df), 0)) + + def near_copies(self, df: pd.DataFrame) -> np.ndarray: + """Boolean mask of the rows of df that are near copies of a real row.""" + if not len(df) or not self.X.shape[1]: + return np.zeros(len(df), dtype=bool) + j, d = nearest(self.encode(df), self.X) + exact = (d == 0) & ~self.has_twin[j] + close = (d > 0) & (d < self.rho * self.nn[j]) + return exact | close + + +def guard_columns(tspec: dict, real: pd.DataFrame, keycols: set, table: str) -> list[str]: + """The columns the evaluation measures distances on (relsynth.evaluate): + modelled, not a key, not determined by another categorical.""" + from .evaluate import determining_columns + cols = [c for c, cs in tspec["columns"].items() if cs["type"] in MODELLED and (table, c) not in keycols + and c != tspec.get("primary_key") and c in real.columns] + return determining_columns(real, tspec, cols) diff --git a/src/relsynth/schema.py b/src/relsynth/schema.py index 96d821d..80b65dc 100644 --- a/src/relsynth/schema.py +++ b/src/relsynth/schema.py @@ -434,6 +434,10 @@ "marginals.min_dependence_rows": "Rows away from the mode needed to model a column's dependence, in the whole table and in each segment's copula sample.", "marginals.fd_bf": "Bayes factor to treat a categorical dependency as functional.", "marginals.joint_max_levels": "Largest joint of dependent categoricals.", + "marginals.segment_prior_rows": "Prior weight, in rows, of an entity table's marginal in each segment's marginal " + "(hierarchical shrinkage; a segment of n rows keeps about n / (n + this) of its own " + "distribution). 0 = off (2.1); 20 makes near copies in small dimensions rarer but weakens " + "the link between their segments and the facts.", "marginals.joint_min_cramers_v": "Association that joins two categoricals.", "facts.min_group_rows": "Rows below which a block borrows the coarser dimension's marginals.", "facts.frailty": "auto | on | off: heavy-tailed rows per parent (gamma frailty).", @@ -486,6 +490,15 @@ "runtime.start": "process executor: fork (default on Linux), spawn or forkserver.", "runtime.master": "spark executor: the master URL when no Spark session is active (default local[workers]).", "privacy.suppress_small_counts": "Counts below this are suppressed in the diagnostics bundle.", + "privacy.near_copy": "auto: redraw entity rows that sit closer to a real row than that row's nearest distinct " + "real neighbour (tables the distance check covers, up to near_copy_max_rows rows); off. " + "A distance rule, not a privacy guarantee (docs/privacy.md).", + "privacy.near_copy_ratio": "A synthetic row is a near copy when its distance to its nearest real row is below " + "this times that row's nearest-distinct-neighbour distance.", + "privacy.near_copy_rounds": "Redraws of a near copy before it is kept and counted in the sample log.", + "privacy.near_copy_max_rows": "Largest entity table the near-copy guard runs on (it compares every synthetic " + "row with every distinct real row, in memory-bounded blocks; time grows with the " + "square of the rows).", "privacy.dp": "Differential privacy mode: unit, epsilon or rho, delta, public_tables, public, max_degree, " "spec_is_public and expert keys; validated by relsynth.dp (docs/dp.md). Off when absent.", "pushdown.mode": "auto | on | off: fit eligible fact tables from exact statistics computed where the data lives " diff --git a/src/relsynth/settings.py b/src/relsynth/settings.py index 98d33ef..4ff6d47 100644 --- a/src/relsynth/settings.py +++ b/src/relsynth/settings.py @@ -188,6 +188,9 @@ def _p(key, **kw): _p("marginals.max_obs", type="int", low=10, high=1e9, tunable=True, scale="log", affects=_FID_MARG, direction={f: +1 for f in _FID_MARG}), _p("marginals.fd_sample_rows", type="int", low=100, high=1e10), + _p("marginals.segment_prior_rows", type="float", low=0, high=200, tunable=True, scale="linear", + affects=_FID_ASSOC + _FID_MARG + _JOIN + _PRIV, + direction={f: -1 for f in _FID_ASSOC + _FID_MARG + _JOIN} | {f: +1 for f in _PRIV}), _p("facts.min_group_rows", type="int", low=0, high=1e9, tunable=True, scale="log", affects=_JOIN + _FID_ASSOC, direction={f: -1 for f in _JOIN + _FID_ASSOC}), _p("facts.frailty", type="choice", choices=("auto", "on", "off"), tunable=True, affects=_CARD), @@ -252,6 +255,10 @@ def _p(key, **kw): _p("evaluation.privacy_check_people", type="int", low=50, high=1e7), _p("privacy.suppress_small_counts", type="int", low=0, high=1e6), _p("privacy.dp", type="map", nullable=True), + _p("privacy.near_copy", type="choice", choices=("auto", "off")), + _p("privacy.near_copy_ratio", type="float", low=0, high=100), + _p("privacy.near_copy_rounds", type="int", low=0, high=1e6), + _p("privacy.near_copy_max_rows", type="int", low=0, high=1e12), _p("pushdown.mode", type="choice", choices=("auto", "on", "off")), _p("pushdown.min_rows", type="int", low=0, high=1e15), _p("pushdown.sample_rows", type="int", low=1, high=1e12, tunable=True, scale="log", affects=_FID_ASSOC + _FID_MARG, diff --git a/src/relsynth/spec.py b/src/relsynth/spec.py index 2d10af9..f2533eb 100644 --- a/src/relsynth/spec.py +++ b/src/relsynth/spec.py @@ -22,6 +22,12 @@ AGGREGATES = {"count", "sum", "min", "max", "mean", "exists", "count_distinct"} PII_GENERATORS = {"email", "person_name", "phone", "uuid", "pattern"} +# G3 defaults (docs/proofs/sec_g3_smalldims.tex): prior rows of the table marginal in each +# segment marginal of an entity table, and the near-copy guard +SEGMENT_PRIOR_ROWS = 0.0 +NEAR_COPY = "off" + + DEFAULT_SYNTHESIS = { "seed": 7, # one posterior draw of every parameter per dataset (mode draw), or posterior @@ -39,7 +45,7 @@ "marginals": {"bayesian_bootstrap": True, "dirichlet_prior": 0.25, "max_levels": 200, "tail_trim": 0.0, "min_dependence_rows": 5, "fd_bf": 100.0, "joint_max_levels": 64, "joint_min_cramers_v": 0.3, "discrete_int_max": 30, "max_obs": 20000, - "fd_sample_rows": 50000}, + "fd_sample_rows": 50000, "segment_prior_rows": SEGMENT_PRIOR_ROWS}, "facts": {"min_group_rows": 30, "frailty": "auto", "frailty_threshold": 1.5, "pairs": "auto", "multiplicity": "auto", "count_model": "auto", "count_model_bf": 100.0, "group_fallback": "auto", "link_degree": "auto", @@ -67,7 +73,10 @@ # a settings file merged under this block (relsynth.settings; relative to the spec) "settings_file": None, # dp: differential privacy mode (relsynth.dp, docs/dp.md); None = off - "privacy": {"suppress_small_counts": 5, "dp": None}, + # near_copy (G3, relsynth.nearcopy): redraw entity rows closer to a real row than + # its nearest distinct real neighbour (a distance rule, not a privacy guarantee) + "privacy": {"suppress_small_counts": 5, "dp": None, "near_copy": NEAR_COPY, "near_copy_ratio": 1.0, + "near_copy_rounds": 20, "near_copy_max_rows": 20_000}, # fit large fact tables from statistics computed in the database and a bounded # uniform sample (relsynth.pushdown); auto: facts with at least min_rows rows "pushdown": {"mode": "auto", "min_rows": 1_000_000, "sample_rows": 500_000}, diff --git a/src/relsynth/table_model.py b/src/relsynth/table_model.py index b43069c..0a7019e 100644 --- a/src/relsynth/table_model.py +++ b/src/relsynth/table_model.py @@ -107,6 +107,7 @@ def __init__(self, tname, tspec, df: pd.DataFrame, cfg, exclude=(), aux=None): joined onto each row. They enter the copula, and at synthesis the row's own columns are drawn conditionally on its synthetic parent's values.""" self.tname = tname + self.role = tspec.get("role") self.cfg = cfg self.cols = modelled_columns(tspec, exclude) mcfg = cfg["marginals"] diff --git a/tests/science/test_small_dims_checks.py b/tests/science/test_small_dims_checks.py new file mode 100644 index 0000000..6f17ae7 --- /dev/null +++ b/tests/science/test_small_dims_checks.py @@ -0,0 +1,220 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Checks C1720-C1723: small-dimension privacy (gap plan G3; +docs/proofs/sec_g3_smalldims.tex, Props. 1720-1723), tested on the shipped +code (`marginals.segment_prior_rows`, `relsynth.nearcopy`, the engine's guard). + + pytest -m science tests/science/test_small_dims_checks.py +""" +from __future__ import annotations + +import copy +import sys +from pathlib import Path + +import numpy as np +import pandas as pd +import pytest + +pytestmark = [pytest.mark.science] + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +from conftest import write_scenario # noqa: E402 + +import relsynth as rs # noqa: E402 +from relsynth.marginals import Codec, Marginal # noqa: E402 +from relsynth.nearcopy import NearCopyGuard, nearest # noqa: E402 + + +def _own_mass(m, own): + """Predictive mass of a continuous marginal draw on the segment's own atoms: + the share of quantile knots that fall on the segment's values.""" + u = np.linspace(0, 1, 4001)[1:-1] + v = m.from_uniform(u) + return np.isin(np.round(v, 9), np.round(own, 9)).mean() + + +# --------------------------------------------------------------------- C1720 +def test_c1720_segment_shrinkage_weight_and_stream(): + """Prop. 1720: the predictive mass on the segment's own rows is n / (n + a) + (discrete: exactly, by the Dirichlet mean; continuous: E[V]), and the number + of random numbers drawn does not depend on a.""" + rng = np.random.default_rng(5) + table = pd.Series(rng.choice(list("abcdefgh"), 3000, p=[.3, .2, .15, .1, .1, .05, .05, .05])) + cd = Codec("x", {"type": "categorical"}, table) + X = cd.encode(table) + top = Marginal.fit(cd, X, rng, dirichlet_a=0.25, bootstrap=False) + seg = cd.encode(pd.Series(list("aaaaabbbcc"))) # n = 10, levels a, b, c only + n = len(seg) + for a in (0.25, 10.0, 20.0): + draws = np.array([Marginal.fit(cd, seg, rng, prior=top, dirichlet_a=a).p for _ in range(20000)]) + own = draws[:, :3].sum(1) if list(cd.levels[:3]) == list("abc") else \ + draws[:, [cd.levels.index(v) for v in "abc"]].sum(1) + w = n / (n + a) + expect = w + (1 - w) * top.p[[cd.levels.index(v) for v in "abc"]].sum() + assert abs(own.mean() - expect) < 4 * own.std() / np.sqrt(len(own)) + 1e-3, (a, own.mean(), expect) + # continuous: E[V] = n / (n + a), V ~ Beta(n, a) + vals = rng.gamma(2.0, size=3000) + cc = Codec("y", {"type": "float"}, pd.Series(vals)) + topc = Marginal.fit(cc, vals, rng, bootstrap=False, grid=64) + own = np.round(rng.gamma(2.0, size=12) * 100) / 100 + 50.0 # atoms far right of the table law + for a in (0.25, 20.0): + st0 = np.random.default_rng(11) + Marginal.fit(cc, own, st0, prior=topc, dirichlet_a=a, grid=64) + st1 = np.random.default_rng(11) + Marginal.fit(cc, own, st1, prior=topc, dirichlet_a=0.25 if a != 0.25 else 20.0, grid=64) + assert st0.random() == st1.random() # same stream whatever a is + mass = [] + for _ in range(3000): + m = Marginal.fit(cc, own, rng, prior=topc, dirichlet_a=a, grid=256) + mass.append((m.from_uniform(np.linspace(0, 1, 2001)[1:-1]) >= 50.0).mean()) + w = len(own) / (len(own) + a) + assert abs(np.mean(mass) - w) < 0.03, (a, np.mean(mass), w) + + +# --------------------------------------------------------------------- C1721 +def test_c1721_large_segments_move_by_at_most_a_over_n(): + """Prop. 1721: TV(p_a, p_a0) = n (a - a0) / ((n + a)(n + a0)) TV(e, p0) <= (a - a0) / (n + a).""" + rng = np.random.default_rng(3) + for _ in range(500): + V = int(rng.integers(2, 30)) + p0 = rng.dirichlet(np.ones(V)) + n = int(rng.integers(1, 5000)) + c = rng.multinomial(n, rng.dirichlet(np.ones(V))) + a0, a = 0.25, float(rng.uniform(0.25, 100)) + pa, pb = (c + a * p0) / (n + a), (c + a0 * p0) / (n + a0) + tv = 0.5 * np.abs(pa - pb).sum() + exact = n * (a - a0) / ((n + a) * (n + a0)) * 0.5 * np.abs(c / n - p0).sum() + assert abs(tv - exact) < 1e-12 and tv <= (a - a0) / (n + a) + 1e-12 + + +# --------------------------------------------------------------------- C1722 +@pytest.fixture(scope="module") +def shop(tmp_path_factory): + d = tmp_path_factory.mktemp("shop") + spec = write_scenario("S9_small_shop", {"kind": "csv", "path": str(d / "src")}) + spec["target"] = {"kind": "csv", "path": str(d / "out")} + return spec + + +@pytest.fixture(scope="module") +def catalogue(tmp_path_factory): + """A small product list in families (30 models x 10 variants that share a price, + a cost and a line), the shape of DimProduct: as CSV with its spec.""" + d = tmp_path_factory.mktemp("catalogue") + rng = np.random.default_rng(12) + rows = [] + for m in range(30): + price = float(np.round(rng.lognormal(5, 1), 2)) + line = rng.choice(list("MRTS")) + for v in range(10): + rows.append({"id": len(rows) + 1, "price": price, "cost": round(price * 0.6, 2), + "weight": float(np.round(rng.normal(10 + m, 0.3), 2)), + "color": rng.choice(["red", "black", "silver", "blue"]), + "size": str(38 + 2 * (v % 5)), "line": line}) + (d / "src").mkdir() + pd.DataFrame(rows).to_csv(d / "src" / "product.csv", index=False) + spec = {"version": 1, "name": "catalogue", "source": {"kind": "csv", "path": str(d / "src")}, + "target": {"kind": "csv", "path": str(d / "out")}, + "tables": {"product": {"role": "entity", "primary_key": "id", "columns": { + "id": {"type": "id"}, "price": {"type": "float"}, "cost": {"type": "float"}, + "weight": {"type": "float"}, "color": {"type": "categorical"}, + "size": {"type": "categorical"}, "line": {"type": "categorical"}}}}, + "relationships": [], "rules": []} + return spec + + +def _fit(spec, **priv): + s = copy.deepcopy(spec) + s.setdefault("synthesis", {}).setdefault("privacy", {}).update(priv) + return rs.fit(s, seed=4) + + +def test_c1722_guard_accepts_only_rows_outside_B(catalogue): + """Prop. 1722 (a, b): every accepted row of a guarded table is outside B, by + brute force; on a one-dimensional example the accepted law is P(. | B^c).""" + m = _fit(catalogue, near_copy="auto") + syn = m.synthesizer + out = syn.sample(index=0) + log = syn.sample_log.get("near_copies", {}) + assert log, "no guarded table in the scenario" + for t, r in log.items(): + g = syn._nc_guards[t] + assert g is not None + bad = g.near_copies(out[t].reset_index(drop=True)) if set(g.cols) <= set(out[t].columns) else None + if bad is not None: + # rows kept after the last round are the only rows allowed in B + assert int(bad.sum()) <= r["kept"], (t, int(bad.sum()), r) + # 1-D rejection sampling: law of accepted draws is the normal restricted to B^c + real = pd.DataFrame({"a": [-1.0, 0.0, 0.4, 2.0], "b": list("xxxx"), "c": [0.0] * 4, "d": list("pppp")}) + ts = {"columns": {"a": {"type": "float"}, "b": {"type": "categorical"}, "c": {"type": "float"}, + "d": {"type": "categorical"}}} + g = NearCopyGuard(real, ts, ["a", "b", "c", "d"]) + rng = np.random.default_rng(9) + prop = pd.DataFrame({"a": rng.normal(size=200000), "b": "x", "c": 0.0, "d": "p"}) + inB = g.near_copies(prop) + acc = prop.a[~inB].to_numpy() + # reference: the same proposal filtered by B computed by brute force from its + # definition (every real row, its own positive-neighbour distance), without + # relsynth.nearcopy.nearest + Xr = g.X + nu = np.array([min(np.linalg.norm(Xr[i] - Xr[k]) for k in range(len(Xr)) + if np.linalg.norm(Xr[i] - Xr[k]) > 0) for i in range(len(Xr))]) + keep = [] + while len(keep) < 20000: + x = pd.DataFrame({"a": rng.normal(size=5000), "b": "x", "c": 0.0, "d": "p"}) + D = np.linalg.norm(g.encode(x)[:, None, :] - Xr[None], axis=2) + j = D.argmin(1) + d = D[np.arange(len(x)), j] + inb = ((d > 0) & (d < nu[j])) | ((d == 0) & ~g.has_twin[j]) + keep += x.a[~inb].tolist() + from scipy.stats import ks_2samp + assert ks_2samp(acc[:20000], np.array(keep[:20000])).pvalue > 1e-3 + assert 0 < inB.mean() < 1 + + +def test_c1722_no_near_copy_draws_nothing(shop): + """Prop. 1722 (c): when no row is a near copy the guard draws no random + number, so the tables equal the unguarded ones byte for byte.""" + base = dict(_fit(shop, near_copy="off").sample()) + m = _fit(shop, near_copy="auto", near_copy_ratio=0.0) + zero = dict(m.sample()) + flagged = sum(r["near_copies"] for r in (m.synthesizer.sample_log.get("near_copies") or {}).values()) + assert flagged == 0, "the scenario must have no near copies at ratio 0 for this check" + for t in base: + pd.testing.assert_frame_equal(base[t], zero[t]) + + +# --------------------------------------------------------------------- C1723 +def test_c1723_guard_leaves_a_hole_around_a_record(): + """Prop. 1723: adding a record x empties the ball around it in the guarded + output; without x the same set has positive probability. No finite epsilon.""" + ts = {"columns": {"a": {"type": "float"}, "b": {"type": "categorical"}, "c": {"type": "float"}, + "d": {"type": "categorical"}}} + X = pd.DataFrame({"a": [-2.0, -1.0, 1.0, 2.0], "b": list("xxxx"), "c": [0.0] * 4, "d": list("pppp")}) + x = pd.DataFrame({"a": [10.0], "b": ["x"], "c": [0.0], "d": ["p"]}) # an outlying record + rng = np.random.default_rng(1) + # a generator with atoms (as discrete-valued columns have): a tenth of its rows at 10 + a = rng.uniform(-3, 20, 400000) + a[rng.random(len(a)) < 0.1] = 10.0 + prop = pd.DataFrame({"a": a, "b": "x", "c": 0.0, "d": "p"}) + with_x = NearCopyGuard(pd.concat([X, x], ignore_index=True), ts, ["a", "b", "c", "d"]) + without = NearCopyGuard(X, ts, ["a", "b", "c", "d"]) + in_with, in_without = with_x.near_copies(prop), without.near_copies(prop) + # the ball around x in the guard's metric (rows whose nearest real row is x), + # minus the guard regions of X + j, d = nearest(with_x.encode(prop), with_x.X) + hole = (j == len(X)) & (d < with_x.nn[len(X)]) & ~in_without + assert hole.mean() > 0.01 # positive probability without x + assert in_with[hole].all() # probability 0 with x + + +def test_c1722_scorecard_marks_a_guarded_table(catalogue): + """The dcr_ratio item of a guarded table says that it is not independent evidence.""" + m = _fit(catalogue, near_copy="auto") + data = m.sample() + rep = m.evaluate(data) + items = [i for i in rep.scorecard["items"] if i["metric"] == "dcr_ratio"] + guarded = set(data.sample_log.get("near_copies") or {}) + assert items and guarded + for i in items: + assert ("near-copy guard" in i.get("note", "")) == (i["area"] in guarded) diff --git a/tests/unit/dp/test_dp.py b/tests/unit/dp/test_dp.py index fb6d2c2..7f60f99 100644 --- a/tests/unit/dp/test_dp.py +++ b/tests/unit/dp/test_dp.py @@ -185,3 +185,46 @@ def test_cli_refuses_doctor_apply_on_a_dp_spec(star_sqlite, tmp_path, capsys): assert "doctor --apply reads real data outside the DP accounting" in capsys.readouterr().err assert not out.exists() assert main(["doctor", str(tmp_path / "bundle.json"), "--spec", str(p)]) == 0 + + +def test_stage2_keeps_the_near_copy_guard_on_the_seed(star): + """docs/privacy.md (small dimensions): in DP mode the near-copy guard runs in + stage 2, which is fitted on the seed database only, so it is post-processing.""" + _, s = star + spec = _spec(s) + spec["synthesis"]["privacy"]["near_copy"] = "auto" + s2 = dp.stage2_spec(spec) + assert s2["synthesis"]["privacy"]["near_copy"] == "auto" and "dp" not in s2["synthesis"]["privacy"] + + +def test_dp_guard_reads_the_seed_rows_not_the_real_ones(star, monkeypatch): + """In DP mode every near-copy guard is built from the decoded seed database: + the rows it receives are the seed's, never the real tables.""" + pytest.importorskip("opendp") + import relsynth.nearcopy as nc + t, s = star + seeds, seen = {}, [] + real_decode = dp.decode + + def spy_decode(spec, model): + db = real_decode(spec, model) + seeds.update(db) + return db + real_init = nc.NearCopyGuard.__init__ + + def spy_init(self, real, tspec, cols, rho=1.0): + seen.append((real, list(cols))) + real_init(self, real, tspec, cols, rho) + monkeypatch.setattr(dp, "decode", spy_decode) + monkeypatch.setattr(nc.NearCopyGuard, "__init__", spy_init) + spec = _spec(s, epsilon=1.0, spec_is_public=True) + spec["synthesis"]["privacy"]["near_copy"] = "auto" + dp.synthesize(spec, tables=t, write=False, evaluate=False) + assert seen, "the scenario must build at least one guard" + for real, cols in seen: + match = [n for n, df in seeds.items() if len(df) == len(real) and set(cols) <= set(df.columns) + and all(sorted(map(str, df[c])) == sorted(map(str, real[c])) for c in cols)] + assert match, "a guard was built from rows that are not a seed table" + orig = [n for n in match if n in t and len(t[n]) == len(real) + and all(sorted(map(str, t[n][c])) == sorted(map(str, real[c])) for c in cols)] + assert not orig, "a guard was built from the real rows" diff --git a/tests/unit/test_small_dims.py b/tests/unit/test_small_dims.py new file mode 100644 index 0000000..fe1b0e7 --- /dev/null +++ b/tests/unit/test_small_dims.py @@ -0,0 +1,307 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Small-dimension privacy (G3): segment marginal shrinkage +(marginals.segment_prior_rows) and the near-copy guard (privacy.near_copy, +relsynth.nearcopy). The statistical checks are C1720-C1723 in +tests/science/test_small_dims_checks.py.""" +from __future__ import annotations + +import copy + +import numpy as np +import pandas as pd +import pytest + +import relsynth as rs +from relsynth.marginals import Codec, Marginal +from relsynth.nearcopy import NearCopyGuard, nearest +from relsynth.settings import registry +from relsynth.spec import DEFAULT_SYNTHESIS + +from conftest import write_scenario + +TS = {"columns": {"a": {"type": "float"}, "b": {"type": "categorical"}, "c": {"type": "float"}, + "d": {"type": "categorical"}}} + + +def _real(n=120, seed=0): + rng = np.random.default_rng(seed) + df = pd.DataFrame({"a": rng.normal(size=n), "b": rng.choice(list("xyz"), n), + "c": rng.gamma(2.0, size=n), "d": rng.choice(list("pq"), n)}) + # a block of identical rows (a coarse attribute space) + df.loc[:9, ["a", "b", "c", "d"]] = [0.0, "x", 1.0, "p"] + return df + + +def test_settings_are_registered_with_defaults(): + for k in ("marginals.segment_prior_rows", "privacy.near_copy", "privacy.near_copy_ratio", + "privacy.near_copy_rounds", "privacy.near_copy_max_rows"): + assert k in registry() + assert DEFAULT_SYNTHESIS["privacy"]["near_copy_ratio"] == 1.0 + assert DEFAULT_SYNTHESIS["privacy"]["near_copy_max_rows"] == 20_000 + + +def test_nearest_exact_and_positive(): + B = np.array([[0.0, 0.0], [0.0, 0.0], [3.0, 4.0]]) + j, d = nearest(B, B, exclude_self=True) + assert d.tolist() == [0.0, 0.0, 5.0] + j, d = nearest(B, B, exclude_self=True, positive=True) + assert d.tolist() == [5.0, 5.0, 5.0] and j.tolist() == [2, 2, 0] + + +def test_nearest_matches_brute_force_in_small_blocks(): + """The block search over distinct rows gives the brute-force distances, never + picks the row itself under exclude_self, and stays exact with blocks of a few + cells (memory bound).""" + rng = np.random.default_rng(0) + for _ in range(100): + B = rng.integers(0, 3, (rng.integers(1, 40), 3)).astype(float) + A = rng.integers(0, 3, (rng.integers(1, 30), 3)).astype(float) + for ex, pos, AA in [(False, False, A), (False, True, A), (True, False, B), (True, True, B)]: + j, d = nearest(AA, B, exclude_self=ex, positive=pos, max_cells=7) + D = np.sqrt(((AA[:, None] - B[None]) ** 2).sum(2)) + if ex: + np.fill_diagonal(D, np.inf) + if pos: + D[D == 0] = np.inf + np.testing.assert_allclose(d, D.min(1)) + fin = np.isfinite(d) + np.testing.assert_allclose(np.sqrt(((AA[fin] - B[j[fin]]) ** 2).sum(1)), d[fin]) + if ex: + assert (j[fin] != np.arange(len(AA))[fin]).all() + + +def test_guard_rules(): + real = _real() + g = NearCopyGuard(real, TS, ["a", "b", "c", "d"]) + assert g.has_twin[:10].all() and not g.has_twin[10:].any() + syn = pd.concat([real.iloc[[0]], # copy of a shared row: allowed + real.iloc[[20]], # copy of a unique row: near copy + real.iloc[[30]].assign(a=real.a.iloc[30] + 1e-6), # a hair away from a unique row + pd.DataFrame({"a": [40.0], "b": ["z"], "c": [99.0], "d": ["q"]})], ignore_index=True) + assert g.near_copies(syn).tolist() == [False, True, True, False] + # rho = 0 keeps every positive distance (only copies of unique rows remain near copies) + g0 = NearCopyGuard(real, TS, ["a", "b", "c", "d"], rho=0.0) + assert g0.near_copies(syn).tolist() == [False, True, False, False] + + +def test_segment_prior_is_the_dirichlet_mean(): + """mean mode: p = (counts + a p0) / (n + a) with a = max(dirichlet_prior, segment_prior_rows).""" + s = pd.Series(list("aaab") + list("bbbbbbbbbbbb")) + cd = Codec("x", {"type": "categorical"}, s) + x = cd.encode(s) + rng = np.random.default_rng(0) + top = Marginal.fit(cd, x, rng, dirichlet_a=0.25, bootstrap=False) + seg = x[:4] + for a in (0.25, 20.0): + m = Marginal.fit(cd, seg, rng, prior=top, dirichlet_a=a, bootstrap=False) + cnt = np.bincount(seg, minlength=2) + assert np.allclose(m.p, (cnt + a * top.p + 1e-9) / (cnt.sum() + a + 2e-9)) + + +@pytest.fixture(scope="module") +def shop(tmp_path_factory): + d = tmp_path_factory.mktemp("shop") + spec = write_scenario("S9_small_shop", {"kind": "csv", "path": str(d / "src")}) + spec["target"] = {"kind": "csv", "path": str(d / "out")} + return spec + + +def _run(spec, **priv): + s = copy.deepcopy(spec) + syn = s.setdefault("synthesis", {}) + for k, v in priv.items(): + g, key = k.split("__") + syn.setdefault(g, {})[key] = v + m = rs.fit(s, seed=3) + return m, m.sample() + + +def test_segment_prior_is_wired_and_guard_off_by_default(shop): + """Both settings are opt-in: shrinkage 0 and the guard off reproduce 2.1.""" + m, base = _run(shop, marginals__segment_prior_rows=20.0) + assert m.spec["synthesis"]["marginals"]["segment_prior_rows"] == 20.0 + m0, _ = _run(shop) + assert m0.spec["synthesis"]["marginals"]["segment_prior_rows"] == 0.0 # opt-in: 2.1 output by default + assert m.spec["synthesis"]["privacy"]["near_copy"] == "off" + assert "near_copies" not in m.synthesizer.sample_log + _, zero = _run(shop, marginals__segment_prior_rows=0.0) + b, z = dict(base), dict(zero) + assert any(not b[t].equals(z[t]) for t in b) + # the same random stream (Prop. 1720 b): same row counts in every table + assert {t: len(v) for t, v in b.items()} == {t: len(v) for t, v in z.items()} + + +def _catalogue(d): + """A product list in families (30 models x 10 variants sharing price, cost and line).""" + rng = np.random.default_rng(12) + rows = [] + for m in range(30): + price = float(np.round(rng.lognormal(5, 1), 2)) + line = rng.choice(list("MRTS")) + for v in range(10): + rows.append({"id": len(rows) + 1, "price": price, "cost": round(price * 0.6, 2), + "weight": float(np.round(rng.normal(10 + m, 0.3), 2)), + "color": rng.choice(["red", "black", "silver", "blue"]), "size": str(38 + 2 * (v % 5)), + "line": line}) + (d / "src").mkdir() + pd.DataFrame(rows).to_csv(d / "src" / "product.csv", index=False) + cols = {"id": "id", "price": "float", "cost": "float", "weight": "float", "color": "categorical", + "size": "categorical", "line": "categorical"} + return {"version": 1, "name": "catalogue", "source": {"kind": "csv", "path": str(d / "src")}, + "target": {"kind": "csv", "path": str(d / "out")}, + "tables": {"product": {"role": "entity", "primary_key": "id", + "columns": {c: {"type": t} for c, t in cols.items()}}}, + "relationships": [], "rules": []} + + +def test_guard_redraws_and_logs(tmp_path): + spec = _catalogue(tmp_path) + m, d = _run(spec, privacy__near_copy="auto", marginals__segment_prior_rows=0.0) + log = m.synthesizer.sample_log.get("near_copies", {}) + assert set(log) == {"product"} + r = log["product"] + assert r["near_copies"] > 0 and r["near_copies"] == r["redrawn"] + r["kept"] and r["rows"] == 300 + # dtypes survive the redraw + _, off = _run(spec, privacy__near_copy="off", marginals__segment_prior_rows=0.0) + assert dict(d)["product"].dtypes.to_dict() == dict(off)["product"].dtypes.to_dict() + + +def test_constant_real_table_flags_every_other_row(): + """Every real row identical: no positive neighbour distance (nn = inf), so a + synthetic row at any positive distance is a near copy and an exact copy is not + (the real row has twins). The engine never guards such a table (fewer than 3 + distinct columns), but the rule is defined.""" + real = pd.DataFrame({"a": [1.0] * 60, "b": ["x"] * 60, "c": [2.0] * 60, "d": ["p"] * 60}) + g = NearCopyGuard(real, TS, ["a", "b", "c", "d"]) + assert np.isinf(g.nn).all() and g.has_twin.all() + syn = pd.DataFrame({"a": [1.0, 5.0], "b": ["x", "y"], "c": [2.0, 2.0], "d": ["p", "p"]}) + assert g.near_copies(syn).tolist() == [False, True] + + +def test_rule_redraws_are_guarded_and_kept_counts_emitted_rows(tmp_path): + """Rows redrawn later for an attribute rule join the near-copy rejection, and + the log's `kept` is the number of near copies among the rows emitted.""" + spec = _catalogue(tmp_path) + spec["rules"] = [{"table": "product", "type": "check", "expr": "weight < 30"}] + m, d = _run(spec, privacy__near_copy="auto", marginals__segment_prior_rows=0.0) + syn = m.synthesizer + r = syn.sample_log["near_copies"]["product"] + assert r.get("redrawn_by_rules", 0) > 0 + g = syn._near_copy_guard("product") + assert r["kept"] == int(g.near_copies(dict(d)["product"]).sum()) + + +def test_near_copy_redraw_never_trades_a_valid_row_for_a_violation(tmp_path, monkeypatch): + """With a short rule budget (2 rounds, near-copy budget 1), a row redrawn only + for being a near copy and then drawn into a rule violation gets its rule-valid + values back: the guard adds no rule violations.""" + import relsynth.nearcopy as nc + spec = _catalogue(tmp_path) + spec["rules"] = [{"table": "product", "type": "check", "expr": "weight < 30"}] + # every redrawn row counts as a near copy, so rule-valid rows of round 1 are redrawn + monkeypatch.setattr(nc.NearCopyGuard, "near_copies", lambda self, df: np.ones(len(df), bool)) + m, d = _run(spec, privacy__near_copy="auto", privacy__near_copy_rounds=1, marginals__segment_prior_rows=0.0, + rules__max_rejection_rounds=2) + log = m.synthesizer.sample_log + assert log["near_copies"]["product"].get("restored_after_rules", 0) > 0 + # restored rows keep their hidden draws, so they pass the rules and keys again + assert log["near_copies"]["product"]["restored_still_bad"] == 0 + out = dict(d)["product"] + assert log["rule_violations"]["product"]["weight < 30"] == int((out["weight"] >= 30).sum()) + + +def test_restored_rows_keep_their_hidden_draws(tmp_path, monkeypatch): + """A rule on a column with its own distribution (a hidden draw): restored rows + get back the draw they passed with, not a fresh one, so none fails again.""" + import relsynth.nearcopy as nc + spec = _catalogue(tmp_path) + src = tmp_path / "src" / "product.csv" + df = pd.read_csv(src) + df["promo"] = np.random.default_rng(0).uniform(0, 1, len(df)).round(3) + df.to_csv(src, index=False) + spec["tables"]["product"]["columns"]["promo"] = {"type": "float", + "distribution": {"kind": "uniform", "min": 0, "max": 1}} + spec["rules"] = [{"table": "product", "type": "check", "expr": "promo < 0.5"}] + monkeypatch.setattr(nc.NearCopyGuard, "near_copies", lambda self, df: np.ones(len(df), bool)) + m, d = _run(spec, privacy__near_copy="auto", privacy__near_copy_rounds=1, marginals__segment_prior_rows=0.0, + rules__max_rejection_rounds=2) + log = m.synthesizer.sample_log["near_copies"]["product"] + assert log.get("restored_after_rules", 0) > 0 + assert log["restored_still_bad"] == 0 + + +def test_kept_counts_emitted_rows_after_a_group_effect(tmp_path): + """No rule redraw, but a group effect (one price per line) rewrites guarded + columns after the guard: `kept` is the near-copy count of the emitted rows.""" + spec = _catalogue(tmp_path) + spec["tables"]["product"]["group_effects"] = [{"column": "price", "by": "line"}] + m, d = _run(spec, privacy__near_copy="auto", marginals__segment_prior_rows=0.0) + out = dict(d)["product"] + assert out.groupby("line")["price"].nunique().max() == 1 # the group effect applied + g = m.synthesizer._near_copy_guard("product") + log = m.synthesizer.sample_log.get("near_copies", {}).get("product", {"kept": 0}) + assert log["kept"] == int(g.near_copies(out).sum()) + + +@pytest.mark.parametrize("nc_rounds,max_rounds,expect", [(1, 10, 1), (3, 4, 2), (5, 20, 5)]) +def test_near_copy_rejection_respects_both_round_limits(tmp_path, monkeypatch, nc_rounds, max_rounds, expect): + """Every row counts as a near copy, so the near-copy rejection would run in every + rule round: it runs in min(near_copy_rounds, max_rejection_rounds // 2) rounds.""" + import relsynth.nearcopy as nc + spec = _catalogue(tmp_path) + spec["rules"] = [{"table": "product", "type": "check", "expr": "weight < 10.5"}] + monkeypatch.setattr(nc.NearCopyGuard, "near_copies", lambda self, df: np.ones(len(df), bool)) + m, _ = _run(spec, privacy__near_copy="auto", privacy__near_copy_rounds=nc_rounds, + marginals__segment_prior_rows=0.0, rules__max_rejection_rounds=max_rounds) + assert m.synthesizer.sample_log["near_copies"]["product"]["near_copy_rule_rounds"] == expect + + +def test_group_effect_rows_join_the_near_copy_recheck(tmp_path): + """Rule redraws change a group's shared price, so rows that were not redrawn + change too; those rows are part of the near-copy recheck (changed_by_rules).""" + spec = _catalogue(tmp_path) + spec["tables"]["product"]["group_effects"] = [{"column": "price", "by": "line"}] + spec["rules"] = [{"table": "product", "type": "check", "expr": "weight < 10.5"}] + m, _ = _run(spec, privacy__near_copy="auto", marginals__segment_prior_rows=0.0) + log = m.synthesizer.sample_log["near_copies"]["product"] + assert log["changed_by_rules"] > log["redrawn_by_rules"] > 0 + + +def test_guard_encoding_is_fitted_on_real_rows_only(tmp_path): + """The guard ranks numbers and picks levels among the real rows; the scorecard's + design ranks real and synthetic rows together, so the two encodings differ.""" + from relsynth.evaluate import _design + from relsynth.nearcopy import NearCopyGuard + tspec = {"columns": {"x": {"type": "float"}, "c": {"type": "categorical"}}} + real = pd.DataFrame({"x": np.arange(10.0), "c": list("ab") * 5}) + syn = pd.DataFrame({"x": np.arange(100.0, 110.0), "c": ["z"] * 10}) + g = NearCopyGuard(real, tspec, ["x", "c"]) + eg = g.encode(syn) + assert np.unique(eg, axis=0).shape[0] == 1 # all above every real value, level z unknown + assert not eg[:, 3:].any() # z is not one of the real levels + es = _design([real, syn], tspec, ["x", "c"])[10:] + assert np.unique(es, axis=0).shape[0] == 10 # ranked among real and synthetic rows + np.testing.assert_array_equal(g.encode(real), NearCopyGuard(real, tspec, ["x", "c"]).X) + + +def test_guard_skips_small_or_narrow_tables(tmp_path): + """The guard is not built for a table of fewer than 50 rows, nor for one with + fewer than three sampled distance columns (claim PRV-32).""" + spec = _catalogue(tmp_path) + m, _ = _run(spec, privacy__near_copy="auto", marginals__segment_prior_rows=0.0) + assert m.synthesizer._near_copy_guard("product") is not None + # two sampled distance columns only + narrow = copy.deepcopy(spec) + cols = narrow["tables"]["product"]["columns"] + for c in ("cost", "weight", "size", "line"): + cols.pop(c) + m2, _ = _run(narrow, privacy__near_copy="auto", marginals__segment_prior_rows=0.0) + assert m2.synthesizer._near_copy_guard("product") is None + # fewer than 50 rows + src = tmp_path / "small" + (src / "src").mkdir(parents=True) + pd.read_csv(tmp_path / "src" / "product.csv").head(40).to_csv(src / "src" / "product.csv", index=False) + small = copy.deepcopy(spec) + small["source"]["path"] = str(src / "src") + m3, _ = _run(small, privacy__near_copy="auto", marginals__segment_prior_rows=0.0) + assert m3.synthesizer._near_copy_guard("product") is None