Skip to content

study_saturation: record DDSketch mean relative value error - #176

Merged
milindsrivastava1997 merged 1 commit into
mainfrom
175-saturation-study-records-ddsketch-max-relative-value-error-optimizer-reads-mean
Oct 7, 2026
Merged

milindsrivastava1997 merged 1 commit into
mainfrom
175-saturation-study-records-ddsketch-max-relative-value-error-optimizer-reads-mean

Conversation

@milindsrivastava1997

Copy link
Copy Markdown
Contributor

Closes #175 (step 1).

scripts/study_saturation.py now records DD curves as mean_relative_value_error (was max_relative_value_error), matching what rqe-optimizer reads for dd (#173).

Design decisions (please approve)

  • DD is judged on mean relative value error over probes, not max. KLL and DD answer the same quantile query, so they're both judged on average error.

Not in this PR

  • Rerunning the DD saturation study and regenerating its curves (step 2). Existing DD curves still carry max_* until then.
  • Checking that small_problem runs with a Quantile RAQE and no panic (step 3). This depends on step 2.

Test

python3 -m pytest scripts/test_study_saturation.py: 26 passed.

🤖 Generated with Claude Code

The optimizer reads DD by mean_relative_value_error; the study wrote
max_relative_value_error, so SaturationCurves::accuracy would panic on
the metric mismatch. Judge DD on mean error, like KLL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@milindsrivastava1997
milindsrivastava1997 merged commit 67b7484 into main Oct 7, 2026
2 checks passed
@milindsrivastava1997
milindsrivastava1997 deleted the 175-saturation-study-records-ddsketch-max-relative-value-error-optimizer-reads-mean branch October 7, 2026 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Saturation study records DDSketch max relative value error; optimizer reads mean

1 participant