From 801ad2a602ca9d8c8a827e966fb689809859147b Mon Sep 17 00:00:00 2001 From: Spencer Bryngelson Date: Wed, 9 Sep 2026 09:08:27 -0500 Subject: [PATCH] A relative tolerance at a zero-valued optimum fails on the runs that search best The nightly CI has failed since 8 September on 3.14, and since 9 September on 3.13, in test_the_result_records_everything_it_evaluated. Master has not changed since 19 August; numpy 2.4.6 -> 2.5.3 did. The test compared best_value, the GP posterior mean at the best point, against the raw observation there, with rel=1e-3. On a noiseless objective those differ by the interpolation floor -- an ABSOLUTE error of 1e-10 to 7e-08, set by _JITTER and the conditioning of the Cholesky. But _bowl peaks at exactly zero, so the closer the search lands to the optimum, the smaller values[chosen] gets while the numerator stays put. The relative bound therefore tightens without limit as the optimiser improves, and fails precisely on the best runs. That is what happened. The numpy upgrade perturbed the L-BFGS-B rounding enough to move seed 0 from |x*| = 0.053 to |x*| = 0.0029 -- a more accurate answer -- and values[chosen] fell from -2.8e-03 to -8.6e-06. The absolute error was unchanged at 3e-08; the relative error rose from 1.3e-05 to 3.5e-03 and tripped the bound. Pinning numpy 2.5.3 and scipy 1.18.1 locally reproduces the CI failure exactly. bayesian_maximize is not at fault and is not touched. The companion assertion, that the chosen point is still the best observation, passed throughout CI. Adds abs=1e-6, the invariant that actually holds: ~14x the worst interpolation error measured over ten seeds, and six orders of magnitude below the data range, so it still tests that the GP interpolates. The docstring's "~3e-05 relative" was never a property of the code, only of where seed 0 happened to land, and is replaced by the absolute floor and the reason the relative form degenerates. The other twenty rel= assertions in the suite were checked; none compares against a quantity that approaches zero, so the pattern is isolated to this test. --- tests/test_optimize.py | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/tests/test_optimize.py b/tests/test_optimize.py index 5e8e7f6..8e5822a 100644 --- a/tests/test_optimize.py +++ b/tests/test_optimize.py @@ -81,8 +81,15 @@ def noisy(point): def test_the_result_records_everything_it_evaluated(): """`best_value` is the posterior mean at `best_point`, not the raw observation there. - On a noiseless objective the GP interpolates, so the two agree to ~3e-05 relative and - the chosen point is still the best observation -- the distinction only bites under noise. + On a noiseless objective the GP interpolates, so the two agree to ~1e-07 absolute -- the + interpolation floor set by `_JITTER` and the conditioning of the Cholesky -- and the + chosen point is still the best observation. The distinction only bites under noise. + + The tolerance here must be absolute, not relative. `_bowl` peaks at exactly zero, so the + closer the search lands to the optimum the smaller `values[chosen]` gets, while the + interpolation error stays put; a purely relative bound would tighten without limit and + fail precisely on the *best* runs. Seeds 6 and 9 land inside 0.004 of the optimum and + reach 4e-03 and 4e-04 relative on an absolute error of 4e-08 and 5e-09. """ result = bayesian_maximize(_bowl(np.array([0.0, 0.0])), _BOX, evaluations=12, initial=4, seed=0) assert result.points.shape == (12, 2) @@ -92,7 +99,8 @@ def test_the_result_records_everything_it_evaluated(): chosen = int(np.argmin(np.linalg.norm(result.points - result.best_point, axis=1))) np.testing.assert_allclose(result.points[chosen], result.best_point) assert chosen == int(result.values.argmax()), "noiseless, the best posterior mean is the best observation" - assert result.best_value == pytest.approx(result.values[chosen], rel=1e-3) + # rel guards the large-|value| end, abs the optimum, where the relative form degenerates. + assert result.best_value == pytest.approx(result.values[chosen], rel=1e-3, abs=1e-6) @pytest.mark.parametrize(