Skip to content

[6.42][RF] RooFit HS3 backports for ATLAS Higgs discovery roundtripping - #24289

Open
guitargeek wants to merge 9 commits into
root-project:v6-42-00-patchesfrom
guitargeek:bp_2
Open

guitargeek wants to merge 9 commits into
root-project:v6-42-00-patchesfrom
guitargeek:bp_2

Conversation

@guitargeek

Copy link
Copy Markdown
Contributor

Bring RooFit HS3 from ROOT master to 6.42 as well.

guitargeek and others added 9 commits November 21, 2026 14:55
Fix the argument comments that don't match the parameter names.

Assisted-by: claude-code:claude-fable-5-2
(cherry picked from commit 097df95)
Assisted-by: claude-code:claude-fable-5-2
(cherry picked from commit 3c759b4)
Make the parameter names in the declarations match the ones in the
definitions. In most cases this is done with the fix-its of the check,
which rename the parameters in the declaration.

Where the name in the header was the better one, the parameter in the
definition was renamed by hand instead:

  * HS3: readValue() for ParameterValue, importHistFactoryNode()

Assisted-by: claude-code:claude-fable-5-2
Cherry-picked from 82e73e3
…t-project#24231)

RooFitHS3 grew a lot when it was synced with the upcoming HS3 0.3
standard and moved to the typed document representation. Now that the
right abstractions and the required public interface are clearer, this
reduces roofit/hs3 by about 1000 lines without intended changes in
functionality. The exported documents are the same, and the tests pass
without changes.

Mostly, this removes unused functions and code that repeated what the
parser, the writer and the validation of the document already do. The
check for shared variates moves to the new HS3Variates.cxx in hs3cpp,
which is compiled without RooFit.

Removed from the public headers; what was in a release is listed in
the release notes:
* RooJSONFactoryWSTool: createNewJSONTree(), makeVariablesNode(),
  getRooFitInternal(), exportAxis(), exportCategoricalAxis(), and
  rebuildModelConfigInWorkspace(), which was only declared.
  hs3VersionInput() is replaced by hs3Version(), which gives the
  parsed version instead of the string.
* JSONNode: has_key(), operator>>(std::string &), the fill_seq()
  overloads with a maximum size or a transformation, the decrement
  and arrow operators of the child iterators, and the operators that
  stream a map into a node, the children of a non-const node into a
  std::vector, or a node into a std::ostream.

What does change:
* removeExporters() and removeImporters() find the built-in ones by
  the names of their classes, which are different now. The exporters
  of JSONFactories_RooFitCore.cxx match the exported class (e.g.
  "RooGaussian" instead of "exportGaussian"), its function importers
  share one class, and the HistFactory exporters are
  HistFactoryStreamer instead of HistFactoryStreamer_ProdPdf and
  HistFactoryStreamer_SumPdf.
* Diagnostics: malformed "axes" of a generic formula get the messages
  of the parser, which warns about unknown components and reads null
  as left out. Interpolations in error messages have their components
  sorted by name, and some diagnostics come in a different order.
* HistFactory export: if a sample has several ParamHistFuncs named
  "mc_stat_*", the constraints of all but the last one are not
  written as constraint terms of the channel anymore. HistFactory
  doesn't create such models.
* Variates: in a HistFactory distribution that the document
  interprets, the "constraints" of a custom modifier don't count as
  terms of the likelihood anymore. A combined distribution that no
  likelihood lists is also checked if a likelihood lists a data set
  of the same name.

Assisted-by: claude-code:claude-fable-5-2
(cherry picked from commit 0d79bf7)
Bring the HS3 import and export in line with the fixes that went into
the standard before v0.3 (HS3 PR 154, with the decisions of the HS3
issues 156 and 157):
hep-statistics-serialization-standard/hep-statistics-serialization-standard#154

Where 0.3 forbids or changes something that older documents rely on,
the new behaviour applies to documents with hs3_version 0.3 or later.
Older documents are read as before, usually with a warning.

HS3 0.3:

* Data axes: an infinite bound of a continuous axis is left out instead
  of written as null or as the largest double, and an axis with one
  bound sets only that bound on import. The validator reports binned
  axes with infinite bounds, data axes with a fixed "value", and data
  axes whose range differs from the domain of the same variate.
* Point data is read and written as a RooDataSet with one entry of one
  variable. Without axes, and for a number given in-line in "data", the
  variable is the single variate of the paired distribution.
* A ModelConfig without data is written as a likelihood with
  "data": null. The old "data": [0] is read as "no data" only in
  documents before 0.3, and is an observation of 0 otherwise.
* landau_dist: the location is written as "mu"; "mean" is still read.
* mixture_dist: "extended" is required with one coefficient per
  summand. An explicit false is now honoured, with a RooAddPdf of the
  normalized coefficients that can't be extended.
* Expressions: TRUE, FALSE, === and !== are translated for TFormula.
  The validator reports objects with a reserved name (PI, EULER, TRUE,
  FALSE, ifelse, functions). A generic expression that refers to a
  distribution is rejected on import, and the export fails for what it
  would have to write as one, e.g. a RooFormulaVar that depends on a
  pdf.
* HistFactory: the "constraints" of a modifier add no term. Every named
  constraint must contribute exactly once to each likelihood with the
  channel, counted by the entries of the data it is paired with.
* HistFactory: staterror gammas with named constraints keep the range
  and constness of the document, and are not configured from the errors
  of the samples.

Auxiliary measurements (constraint terms), independent of HS3 0.3:

* A RooLognormal in RooFit's default parametrization (median, shape)
  can be a component of an array_dist, so config().exportArrayDist and
  config().exportConstraintData also apply to such constraints.
* The helper functions of literal constants get valid identifiers,
  "const_1p2_lognormal_log" instead of "1.2_lognormal_log", with all
  digits of the value. The old names are still read.
* Objects that the import creates for something RooFit has no class
  for, like the RooProdPdf that holds an array_dist together, carry the
  attribute "hs3_helper" and are only written if something refers to
  them. Written as a product_dist, that product kept the constraints
  out of arrays in every later export.
* With exportConstraintData, an array of constraints of a
  RooSimultaneous is a component of the combined_dist, with its data
  set in the combined data set, instead of a separate likelihood term.
  A channel that consists of constraint terms only can now be written.
  These components are marked in misc.ROOT_internal.auxiliary_channels
  and read back as constraint terms, or kept as channels with the new,
  EXPERIMENTAL config().importConstraintsAsChannels.
* On import, a constraint term with a global observable becomes a
  factor of the channels that use its parameters again, instead of an
  external constraint of the ModelConfig. A channel that is no product
  is wrapped in a RooProdPdf "<channel>_with_constraints". The pdf of
  the model stays the sampling distribution of the auxiliary
  measurements in a round trip, which Asimov data and toys rely on.
* Which of x and mean of a Gaussian constraint is the observable is
  determined before any pdf is written, so that it can't disagree with
  the axis of the constraint's data set.

Documents before 0.3 that are published on HEPData:

* Two histosys modifiers of the same name in a sample, as ROOT 6.37
  wrote them, are a warning instead of an error, and are read as one
  parameter that interpolates the sample once per modifier.
* The HistFactory export writes the formulas of a channel as custom
  modifiers if splitting them into one modifier per factor makes the
  channel invalid, e.g. two normfactors of the same name for "b*b".

Each change has a test in testRooFitHS3. The new
testHS3AuxiliaryMeasurements writes a three-channel HistFactory
workspace in all four layouts of exportArrayDist x exportConstraintData
and compares the documents, workspaces, likelihood scans and toys. The
likelihoods of rf515_hfJSON.json and of the hand-written 0.3 documents
of the tests now list their constraints in "aux_distributions".

Assisted-by: claude-code:claude-opus-5-5
(cherry picked from commit 2bacb4b)
The HS3 tests have grown a bit too much, so it is time for a refactoring that
keeps the coverage the same or higher.

Assisted-by: claude-code:claude-fable-5-2
(cherry picked from commit cb01aa8)
…t#24261)

Follow-up to root-project#24201 in the spirit of root-project#24231: this reduces roofit/hs3/inc
and roofit/hs3/src by about 820 lines (1916 added, 2735 deleted). The
JSON layer stays independent of RooFit, and the tests pass unchanged.

* A "combined_dist" is part of the typed document now and shares its
  representation with combined data sets, so the parser, the writer and
  the validation handle both with the same code.
* A document is validated as it was written, then the variates and
  constraints of its likelihoods are checked, and only then is it
  adapted and imported. The adaptations (auxiliary channels read as
  constraint terms, in-line numbers as point data) move from
  RooJSONFactoryWSTool to the JSON layer.
* All importers and exporters are registered in the same way
  (static_execute.h is removed), and 13 classes that are read and
  written field by field are described once for both directions with
  registerFields().
* Simplifications in the tool, the HistFactory code, the parser, the
  writer, the validation, the upgrade of older documents and JSONNode.

Fixes:
* JSONNode::val() kept six significant digits of a number and cast
  integers to 32 bit, so literal numbers in a document lost precision
  on import (1.23456789 became 1.23457).
* With config().importConstraintsAsChannels, a data set shared by
  several components was silently not copied once the room reserved
  for the copies was used up.

What changes for users:
* A document that fails the checks of its variates or constraints, or
  whose auxiliary components are inconsistent, is rejected before
  anything is imported into the workspace.
* RooFit no longer warns about components without data when it reads
  back auxiliary measurements that it wrote with
  config().exportConstraintData.
* A user's own importer for "combined_dist" is not called for the
  distributions of a document anymore, and an importer registered for
  the replacement of a deprecated type key is now also used for the
  deprecated key.
* The names of the built-in importers and exporters, as
  printImporters() and removeImporters() see them, changed.
* Public headers: RooJSONFactoryWSTool::CombinedData and the operator
  that streams node.children() into a std::vector are removed (see the
  release notes), as are the unreleased
  isAuxiliaryChannelReadAsConstraint() and otherLikelihoodTerms().
  RooJSONFactoryWSTool::channelConstraints() is new.

Assisted-by: claude-code:claude-fable-5-2
(cherry picked from commit a3cdefd)
The test took about 40 s and wrote 315 ROOT files: every instance built
the HistFactory model through files, and most of the time went into
sampling the original model with generateSimGlobal(), which sets up a
generator for every toy and channel, once per layout.

Now the model is built in memory, and the exports, scan, moments and
toys that don't depend on the layout are computed once per model kind.
The moments sample the global observables per channel, and the data
toys reuse one generator. ConstraintsAsChannels runs only for the layout
it applies to. The test takes 5 s and catches the same libRooFitHS3
mutants as before.

Assisted-by: claude-code:claude-opus-5-5
(cherry picked from commit bd12160)
…ect#24250)

Implement the ROOT side of HS3 PR 161 ("Define identity and composition
of factors"): combined and array distributions compose their components
by union, so identical factors count once, however often and under
whatever names they occur, and non-identical factors must not share a
variate. The likelihood stays the plain product of its terms, so
distributions that share factors have to be joined in a (possibly
nested) combined distribution. The main use case is the combination of
several documents, e.g. two HistFactory models written by RooFit that
share a systematic or the luminosity.

Identical components of array distributions:

  * Import: a component over an observable that another array already
    created is reused if it is identical (same type and fields after
    applying defaults, numbers compared as numbers), so
    getAllConstraints() finds it once. Otherwise it is an error.
  * Distinct variates: each term of a likelihood is the union of its
    parts (recursively through combined and array distributions), with
    identical parts counted once. A variate shared by two terms is an
    error. Components with missing or empty sub-data sets are ignored.
  * Export: a component of several arrays is written in each of them.
    An external constraint that is an array component is written as the
    whole array, which must then consist of external constraints only.

One RooSimultaneous and one data set per likelihood:

Without the pdf name in the RooFit metadata, the importer used to take
the pdf named "simPdf" and silently dropped the other likelihood terms.
Now the model becomes one RooSimultaneous "<analysis>_simPdf":

  * Combined distributions are flattened recursively into its channels.
    The index has the columns of all of them (a RooSuperCategory if
    there are several), with the label "" where a column doesn't apply.
    A column "<analysis>_categoryIndex" with the term names is added if
    a term is not combined or the channels can't be told apart.
  * Array channels of one component type with one data entry each (as
    written with exportConstraintData) are merged into one array.
    Differing observed values of a shared variate are an error.
  * The data sets of the terms are merged into "<analysis>_data".
    Combined data sets may now contain combined data sets.
  * The export writes one combined distribution and one combined data
    set. Replaced objects are helpers, written only if something else
    uses them. This is transitive, and a skipped array dissolves into
    its components.
  * A pdf named in the RooFit metadata (pre-HS3 0.3 files) is still
    used.

Follow-ups to root-project#24201:

  * Only terms that can be constraint terms (not a combined
    distribution, paired with no data, in-line values, point data or an
    unbinned data set of one entry) are candidates for re-attachment to
    channels.
  * The checks of the "constraints" of HistFactory modifiers descend
    into nested combined distributions, and their violations are
    reported together with those of distinct variates.

Without RooFit metadata, constraint terms written as data are channels
rather than external constraints; see root-project#24251.

New tests: ArrayDistIdenticalComponents,
MergedHistFactoryDocumentsWithArrayConstraints and
LikelihoodOfCombinedAndPlainTerms. The round trip of the 283 HS3
documents on HEPData still works.

Assisted-by: claude-code:claude-opus-5-5
(cherry picked from commit 63fa6a7)
@guitargeek guitargeek self-assigned this Nov 21, 2026
@guitargeek
guitargeek requested a review from bellenot as a code owner November 21, 2026 14:14
@guitargeek guitargeek added this to the 6.42.00 milestone Nov 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants