Repository navigation
fix(cli): fit downstream bias without reapplying checkpoint presets #6074
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
6e3554b
d085730
8cffa35
f682c8d
79beec0
b72a481
4f5af64
0be5abf
8a13bce
a75e7fb
fce767f
e0edaa4
528803f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -926,6 +926,7 @@ def change_out_bias( | |
| self, | ||
| merged: Any, | ||
| bias_adjust_mode: str = "change-by-statistic", | ||
| preset_out_bias: dict | None = None, | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Non-blocking: the new |
||
| ) -> None: | ||
| """Change the output bias according to the input data and the pretrained model. | ||
|
|
||
|
|
@@ -936,8 +937,17 @@ def change_out_bias( | |
| bias_adjust_mode : str | ||
| The mode for changing output bias: | ||
| 'change-by-statistic' or 'set-by-statistic'. | ||
| preset_out_bias : dict, optional | ||
| Per-call preset override. Omit to use the configured preset; | ||
| pass an empty dict to fit without fixing preset outputs. | ||
| Non-finite stored cells covered by the checkpoint preset are | ||
| recovered before residual fitting; finite values are retained. | ||
| """ | ||
| self.atomic_model.change_out_bias(merged, bias_adjust_mode=bias_adjust_mode) | ||
| self.atomic_model.change_out_bias( | ||
| merged, | ||
| bias_adjust_mode=bias_adjust_mode, | ||
| preset_out_bias=preset_out_bias, | ||
| ) | ||
|
|
||
| def _input_type_cast( | ||
| self, | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Blocking: with an explicit
preset_out_bias={}— which is exactly whatdp change-biasnow passes —presetnormalizes to{}and thisif preset:gate is False, so Step 1 is skipped entirely. Step 1 is not only where preset-assigned types receive their preset values;preset_out_bias_rows(...)also recovers non-finite stored bias cells of preset-assigned types back to the finite preset (behavior pinned bytest_nonfinite_assigned_bias_from_checkpointinsource/tests/pt/model/test_atomic_model_global_stat.py, which sets stored out_bias cells to nan/±inf and expects [-10.0, -13.6, 0.0] back). A checkpoint whose stored bias has NaN cells for a preset-assigned type is the normal case for a zero-sample type covered by the preset: on master,dp change-biasrecovers the finite preset; after this PR the CLI path skips Step 1 and change-by-statistic folds its shift onto the NaN stored cell (add=True), so NaN propagates into the adjusted model. I see the Copilot threads on this were resolved, but the mechanism is unchanged at this head. If dropping that recovery on the CLI path is intended, please make that explicit (the pinned test's CLI analogue would need to change too); otherwise recover non-finite stored cells before skipping Step 1 when an explicit empty preset is passed.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Confirmed at exact head
fce767f2554895f65782858492725ed244ab5d02with a bounded NumPy reproduction using the exact-source bias/statistics methods and an atomic-forward test double, not a native CLI/backend run. With stored O/H biases[NaN, NaN]and finite presets, base668b60cand the current omitted-override path restore the presets;preset_out_bias={}instead evaluates the non-finite baseline and leaves non-finite output. Positive/negative infinity reproduce too, and a preset-covered absent type retains its non-finite stored cell after adding a zero shift. All 26 comparison/control cases passed. The existing non-finite checkpoint test omits the override, so it does not cover this CLI path.One scope qualification: this is a recovery regression for an already non-finite checkpoint. Ordinary preset initialization writes assigned values even for absent types, so zero samples alone do not establish that such a checkpoint is normal. A recovery step before residual fitting should replace only non-finite assigned cells, preserve finite adjusted baselines, and still allow
{}to fit the output from data.Agent: dot