Skip to content

[RF][HF] Experimental storage efficient HistFactory rewrite - #24218

Draft
guitargeek wants to merge 5 commits into
root-project:masterfrom
guitargeek:histfactory-pdf
Draft

guitargeek wants to merge 5 commits into
root-project:masterfrom
guitargeek:histfactory-pdf

Conversation

@guitargeek

Copy link
Copy Markdown
Contributor

This is an experiment on making HistFactory models up to 20x smaller in memory and never be bothered by the 1GB limit on disk, by using an RNTuple to store array data from the model.

@guitargeek guitargeek self-assigned this Nov 17, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Test Results

    21 files      21 suites   3d 8h 59m 48s ⏱️
 3 890 tests  3 890 ✅ 0 💤 0 ❌
72 311 runs  72 311 ✅ 0 💤 0 ❌

Results for commit 5e58d6b.

♻️ This comment has been updated with latest results.

A RooHistFactorySample is one sample of a HistFactory channel: the nominal
histogram with all its modifiers, in a single object. Its histogram data
is shared between clones. A RooHistFactoryPdf is the channel, a
RooRealSumPdf of its samples, and corresponds to the HS3 histfactory_dist.

The arithmetic is in RooFit::Detail::MathFuncs, so that the evaluation
and code generation use the same functions, in the order of operations
of the HistFactory models so far.

Assisted-by: claude-code:claude-opus-5-5
HistFactory and the HS3 importer now create one RooHistFactorySample per
sample instead of several RooFit objects per histogram. The previous
structure is still available with
HistoToWorkspaceFactoryFast::Configuration::useHistFactoryPdf,
`hist2workspace -legacy_model_structure`, and
RooJSONFactoryWSTool::config().useHistFactoryPdf.

The HS3 exporter writes RooHistFactoryPdf directly, keeping the names of
the modifiers. The tests that inspect the RooFit objects of the models
are adapted, and the ones about exporting the previous structure use it
explicitly.

Assisted-by: claude-code:claude-opus-5-5
RooWorkspace::Write() and the reading in RooWorkspace::Streamer() open a
WorkspaceStreamingContext, in which the streamers of the workspace objects
can share state. With it, they can write large payloads to the file after
the workspace, which is limited to 1 GB like all objects in ROOT files.

Assisted-by: claude-code:claude-opus-5-5
When a workspace is written with RooWorkspace::Write(), the data of all
its RooHistFactorySamples goes to the RNTuple
`<workspace>_histfactory_arrays`, committed after the workspace itself.
HistFactory workspaces are then no longer limited to 1 GB.
hist2workspace writes its workspaces like this.

Assisted-by: claude-code:claude-opus-5-5
With HistoToWorkspaceFactoryFast::Configuration::writeMeasurement set to
false, MakeModelAndMeasurementFast() writes only the workspaces to the
files of the channels and of the combined model, without the Measurement
and the directories with copies of its histograms. The Measurement holds
all histograms, so it is subject to the 1 GB limit of single objects in
ROOT files like the workspace was before its histograms could go to an
RNTuple, and the workspaces don't need it.

Assisted-by: claude-code:claude-opus-5-5

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant