Repository navigation
[6.42][RF] RooFit HS3 backports for ATLAS Higgs discovery roundtripping - #24289
Open
guitargeek wants to merge 9 commits into
Open
guitargeek wants to merge 9 commits into
guitargeek wants to merge 9 commits into
Conversation
Fix the argument comments that don't match the parameter names. Assisted-by: claude-code:claude-fable-5-2 (cherry picked from commit 097df95)
Assisted-by: claude-code:claude-fable-5-2 (cherry picked from commit 3c759b4)
Make the parameter names in the declarations match the ones in the definitions. In most cases this is done with the fix-its of the check, which rename the parameters in the declaration. Where the name in the header was the better one, the parameter in the definition was renamed by hand instead: * HS3: readValue() for ParameterValue, importHistFactoryNode() Assisted-by: claude-code:claude-fable-5-2 Cherry-picked from 82e73e3
…t-project#24231) RooFitHS3 grew a lot when it was synced with the upcoming HS3 0.3 standard and moved to the typed document representation. Now that the right abstractions and the required public interface are clearer, this reduces roofit/hs3 by about 1000 lines without intended changes in functionality. The exported documents are the same, and the tests pass without changes. Mostly, this removes unused functions and code that repeated what the parser, the writer and the validation of the document already do. The check for shared variates moves to the new HS3Variates.cxx in hs3cpp, which is compiled without RooFit. Removed from the public headers; what was in a release is listed in the release notes: * RooJSONFactoryWSTool: createNewJSONTree(), makeVariablesNode(), getRooFitInternal(), exportAxis(), exportCategoricalAxis(), and rebuildModelConfigInWorkspace(), which was only declared. hs3VersionInput() is replaced by hs3Version(), which gives the parsed version instead of the string. * JSONNode: has_key(), operator>>(std::string &), the fill_seq() overloads with a maximum size or a transformation, the decrement and arrow operators of the child iterators, and the operators that stream a map into a node, the children of a non-const node into a std::vector, or a node into a std::ostream. What does change: * removeExporters() and removeImporters() find the built-in ones by the names of their classes, which are different now. The exporters of JSONFactories_RooFitCore.cxx match the exported class (e.g. "RooGaussian" instead of "exportGaussian"), its function importers share one class, and the HistFactory exporters are HistFactoryStreamer instead of HistFactoryStreamer_ProdPdf and HistFactoryStreamer_SumPdf. * Diagnostics: malformed "axes" of a generic formula get the messages of the parser, which warns about unknown components and reads null as left out. Interpolations in error messages have their components sorted by name, and some diagnostics come in a different order. * HistFactory export: if a sample has several ParamHistFuncs named "mc_stat_*", the constraints of all but the last one are not written as constraint terms of the channel anymore. HistFactory doesn't create such models. * Variates: in a HistFactory distribution that the document interprets, the "constraints" of a custom modifier don't count as terms of the likelihood anymore. A combined distribution that no likelihood lists is also checked if a likelihood lists a data set of the same name. Assisted-by: claude-code:claude-fable-5-2 (cherry picked from commit 0d79bf7)
Bring the HS3 import and export in line with the fixes that went into the standard before v0.3 (HS3 PR 154, with the decisions of the HS3 issues 156 and 157): hep-statistics-serialization-standard/hep-statistics-serialization-standard#154 Where 0.3 forbids or changes something that older documents rely on, the new behaviour applies to documents with hs3_version 0.3 or later. Older documents are read as before, usually with a warning. HS3 0.3: * Data axes: an infinite bound of a continuous axis is left out instead of written as null or as the largest double, and an axis with one bound sets only that bound on import. The validator reports binned axes with infinite bounds, data axes with a fixed "value", and data axes whose range differs from the domain of the same variate. * Point data is read and written as a RooDataSet with one entry of one variable. Without axes, and for a number given in-line in "data", the variable is the single variate of the paired distribution. * A ModelConfig without data is written as a likelihood with "data": null. The old "data": [0] is read as "no data" only in documents before 0.3, and is an observation of 0 otherwise. * landau_dist: the location is written as "mu"; "mean" is still read. * mixture_dist: "extended" is required with one coefficient per summand. An explicit false is now honoured, with a RooAddPdf of the normalized coefficients that can't be extended. * Expressions: TRUE, FALSE, === and !== are translated for TFormula. The validator reports objects with a reserved name (PI, EULER, TRUE, FALSE, ifelse, functions). A generic expression that refers to a distribution is rejected on import, and the export fails for what it would have to write as one, e.g. a RooFormulaVar that depends on a pdf. * HistFactory: the "constraints" of a modifier add no term. Every named constraint must contribute exactly once to each likelihood with the channel, counted by the entries of the data it is paired with. * HistFactory: staterror gammas with named constraints keep the range and constness of the document, and are not configured from the errors of the samples. Auxiliary measurements (constraint terms), independent of HS3 0.3: * A RooLognormal in RooFit's default parametrization (median, shape) can be a component of an array_dist, so config().exportArrayDist and config().exportConstraintData also apply to such constraints. * The helper functions of literal constants get valid identifiers, "const_1p2_lognormal_log" instead of "1.2_lognormal_log", with all digits of the value. The old names are still read. * Objects that the import creates for something RooFit has no class for, like the RooProdPdf that holds an array_dist together, carry the attribute "hs3_helper" and are only written if something refers to them. Written as a product_dist, that product kept the constraints out of arrays in every later export. * With exportConstraintData, an array of constraints of a RooSimultaneous is a component of the combined_dist, with its data set in the combined data set, instead of a separate likelihood term. A channel that consists of constraint terms only can now be written. These components are marked in misc.ROOT_internal.auxiliary_channels and read back as constraint terms, or kept as channels with the new, EXPERIMENTAL config().importConstraintsAsChannels. * On import, a constraint term with a global observable becomes a factor of the channels that use its parameters again, instead of an external constraint of the ModelConfig. A channel that is no product is wrapped in a RooProdPdf "<channel>_with_constraints". The pdf of the model stays the sampling distribution of the auxiliary measurements in a round trip, which Asimov data and toys rely on. * Which of x and mean of a Gaussian constraint is the observable is determined before any pdf is written, so that it can't disagree with the axis of the constraint's data set. Documents before 0.3 that are published on HEPData: * Two histosys modifiers of the same name in a sample, as ROOT 6.37 wrote them, are a warning instead of an error, and are read as one parameter that interpolates the sample once per modifier. * The HistFactory export writes the formulas of a channel as custom modifiers if splitting them into one modifier per factor makes the channel invalid, e.g. two normfactors of the same name for "b*b". Each change has a test in testRooFitHS3. The new testHS3AuxiliaryMeasurements writes a three-channel HistFactory workspace in all four layouts of exportArrayDist x exportConstraintData and compares the documents, workspaces, likelihood scans and toys. The likelihoods of rf515_hfJSON.json and of the hand-written 0.3 documents of the tests now list their constraints in "aux_distributions". Assisted-by: claude-code:claude-opus-5-5 (cherry picked from commit 2bacb4b)
The HS3 tests have grown a bit too much, so it is time for a refactoring that keeps the coverage the same or higher. Assisted-by: claude-code:claude-fable-5-2 (cherry picked from commit cb01aa8)
…t#24261) Follow-up to root-project#24201 in the spirit of root-project#24231: this reduces roofit/hs3/inc and roofit/hs3/src by about 820 lines (1916 added, 2735 deleted). The JSON layer stays independent of RooFit, and the tests pass unchanged. * A "combined_dist" is part of the typed document now and shares its representation with combined data sets, so the parser, the writer and the validation handle both with the same code. * A document is validated as it was written, then the variates and constraints of its likelihoods are checked, and only then is it adapted and imported. The adaptations (auxiliary channels read as constraint terms, in-line numbers as point data) move from RooJSONFactoryWSTool to the JSON layer. * All importers and exporters are registered in the same way (static_execute.h is removed), and 13 classes that are read and written field by field are described once for both directions with registerFields(). * Simplifications in the tool, the HistFactory code, the parser, the writer, the validation, the upgrade of older documents and JSONNode. Fixes: * JSONNode::val() kept six significant digits of a number and cast integers to 32 bit, so literal numbers in a document lost precision on import (1.23456789 became 1.23457). * With config().importConstraintsAsChannels, a data set shared by several components was silently not copied once the room reserved for the copies was used up. What changes for users: * A document that fails the checks of its variates or constraints, or whose auxiliary components are inconsistent, is rejected before anything is imported into the workspace. * RooFit no longer warns about components without data when it reads back auxiliary measurements that it wrote with config().exportConstraintData. * A user's own importer for "combined_dist" is not called for the distributions of a document anymore, and an importer registered for the replacement of a deprecated type key is now also used for the deprecated key. * The names of the built-in importers and exporters, as printImporters() and removeImporters() see them, changed. * Public headers: RooJSONFactoryWSTool::CombinedData and the operator that streams node.children() into a std::vector are removed (see the release notes), as are the unreleased isAuxiliaryChannelReadAsConstraint() and otherLikelihoodTerms(). RooJSONFactoryWSTool::channelConstraints() is new. Assisted-by: claude-code:claude-fable-5-2 (cherry picked from commit a3cdefd)
The test took about 40 s and wrote 315 ROOT files: every instance built the HistFactory model through files, and most of the time went into sampling the original model with generateSimGlobal(), which sets up a generator for every toy and channel, once per layout. Now the model is built in memory, and the exports, scan, moments and toys that don't depend on the layout are computed once per model kind. The moments sample the global observables per channel, and the data toys reuse one generator. ConstraintsAsChannels runs only for the layout it applies to. The test takes 5 s and catches the same libRooFitHS3 mutants as before. Assisted-by: claude-code:claude-opus-5-5 (cherry picked from commit bd12160)
…ect#24250) Implement the ROOT side of HS3 PR 161 ("Define identity and composition of factors"): combined and array distributions compose their components by union, so identical factors count once, however often and under whatever names they occur, and non-identical factors must not share a variate. The likelihood stays the plain product of its terms, so distributions that share factors have to be joined in a (possibly nested) combined distribution. The main use case is the combination of several documents, e.g. two HistFactory models written by RooFit that share a systematic or the luminosity. Identical components of array distributions: * Import: a component over an observable that another array already created is reused if it is identical (same type and fields after applying defaults, numbers compared as numbers), so getAllConstraints() finds it once. Otherwise it is an error. * Distinct variates: each term of a likelihood is the union of its parts (recursively through combined and array distributions), with identical parts counted once. A variate shared by two terms is an error. Components with missing or empty sub-data sets are ignored. * Export: a component of several arrays is written in each of them. An external constraint that is an array component is written as the whole array, which must then consist of external constraints only. One RooSimultaneous and one data set per likelihood: Without the pdf name in the RooFit metadata, the importer used to take the pdf named "simPdf" and silently dropped the other likelihood terms. Now the model becomes one RooSimultaneous "<analysis>_simPdf": * Combined distributions are flattened recursively into its channels. The index has the columns of all of them (a RooSuperCategory if there are several), with the label "" where a column doesn't apply. A column "<analysis>_categoryIndex" with the term names is added if a term is not combined or the channels can't be told apart. * Array channels of one component type with one data entry each (as written with exportConstraintData) are merged into one array. Differing observed values of a shared variate are an error. * The data sets of the terms are merged into "<analysis>_data". Combined data sets may now contain combined data sets. * The export writes one combined distribution and one combined data set. Replaced objects are helpers, written only if something else uses them. This is transitive, and a skipped array dissolves into its components. * A pdf named in the RooFit metadata (pre-HS3 0.3 files) is still used. Follow-ups to root-project#24201: * Only terms that can be constraint terms (not a combined distribution, paired with no data, in-line values, point data or an unbinned data set of one entry) are candidates for re-attachment to channels. * The checks of the "constraints" of HistFactory modifiers descend into nested combined distributions, and their violations are reported together with those of distinct variates. Without RooFit metadata, constraint terms written as data are channels rather than external constraints; see root-project#24251. New tests: ArrayDistIdenticalComponents, MergedHistFactoryDocumentsWithArrayConstraints and LikelihoodOfCombinedAndPlainTerms. The round trip of the 283 HS3 documents on HEPData still works. Assisted-by: claude-code:claude-opus-5-5 (cherry picked from commit 63fa6a7)
guitargeek
requested review from
dpiparo,
hageboeck and
silverweed
as code owners
November 21, 2026 14:14
dpiparo
approved these changes
Nov 21, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bring RooFit HS3 from ROOT
masterto 6.42 as well.