Repository navigation
fix: keep degree-1 and degree-2 fits finite and positive on tied, gappy data - #42
Merged
Merged
Conversation
…py data Three defects, found on one column of a density-estimation benchmark (1023 observations, 75 distinct values, a clump of ties at a cap, and a gap of about ten bandwidths below it). A point in the gap got a density of exactly zero, so any held-out observation there had log-likelihood -inf. Bandwidth selection. `ll_ibias2` averages the bias functional over all bins, weighted by their counts. At degree 1 and 2 the functional divides by the pilot density, which is zero up to FFT round-off in empty bins far from the data, so `0 * inf` or `0 * NaN` made the sum NaN on some builds and finite on others. `select_bandwidth` then took the normal-reference fallback (0.710) on the first and the plug-in value (0.2245) on the second. Only occupied bins contribute now, so every build selects the plug-in value. The degree-2 fit. The local quadratic of Hjort and Jones exists only where `1 + h^2 D > 0`, with `D` the second derivative of the pilot log-density. Next to a clump of ties the pilot log-density is as curved as the kernel's own and the condition fails. `sqrt` of a negative number then gave NaN, which `fit_lp` wrote as a density of zero. Where the local quadratic does not exist or is not finite, the local linear fit now stands in, with its own influence value for the degrees of freedom. The degree-1 fit. The local linear estimate is `f * exp(-h^2 b^2 / 2)` with `b = f' / f`, but `fit_lp` used `h` for `h^2`. The estimate was therefore not scale equivariant: on the same standard-normal sample rescaled by 0.1, 1 and 10, the density at 1.5 read 0.066, 0.148 and 0.155. It reads 0.1538 at every scale now. Every degree-1 fit changes, bounded ones included: those are fitted on the transformed scale, where the same `h` for `h^2` applies. Positivity. The binned kernel is truncated at four bandwidths, so a point in a wide enough gap still had density zero at any degree. The grid values between the smallest and largest observation are floored at machine epsilon times the largest value. The interpolating spline preserves positivity, so the density is positive everywhere in that range. Tests: scale equivariance of unbounded fits at every degree (the existing tests covered only bounded fits, where the transform hid the degree-1 error), positivity across a synthetic gap of twenty bandwidths, and the benchmark column as test data, pinning the plug-in bandwidth and positivity at every degree. All three fail before this change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
tvatter
added a commit
to vinecopulib/pyvinecopulib
that referenced
this pull request
Sep 24, 2026
Pins lib/kde1d at the head of vinecopulib/kde1d-cpp#42, which fixes three defects found on a benchmark column with 75 distinct values in 1023 rows, a clump of ties at a cap, and a gap of about ten bandwidths: - the degree-2 plug-in bandwidth summed 0 * inf over empty bins, so it came out NaN on some builds and took the normal-reference fallback there only (0.710 against the plug-in 0.2245 on that column); - the degree-2 local quadratic does not exist where 1 + h^2 D <= 0, as next to a clump of ties, and the resulting NaN was written out as a density of zero; the local linear fit stands in there now; - the degree-1 correction used h for h^2, so a degree-1 fit was not scale equivariant. Grid values between the smallest and largest observation are also floored at machine epsilon times the largest, because the binned kernel is truncated at four bandwidths and a point in a wide enough gap got density zero at any degree. A held-out observation there had log-likelihood -inf. Across 1,500 benchmark columns, 11 bandwidths move, each from the fallback to the plug-in value, and no held-out point has density zero. The pin is a branch head; re-pin to the merged commit and its v1.2.3 tag before this merges. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On a column with many ties and a wide gap,
Kde1dcould return a density of exactly zero inside the data range, so a held-out observation there had log-likelihood-inf. The bandwidth chosen for that column also depended on the build. The column comes from a density benchmark: 1023 observations, 75 distinct values, a clump of ties at a cap, and no observations for about ten bandwidths below it.Changes
dpik.hpp).ll_ibias2now sums only over occupied bins. The pilot density is zero up to FFT round-off in empty bins far from the data, and0 * infthere made the bias term NaN on some builds and finite on others. A NaN falls back to the normal-reference rule, so on that column one build selected 0.710 and another 0.2245, the plug-in value. Every build now selects 0.2245.fit_lp). The Hjort–Jones local quadratic exists only where1 + h²D > 0. That condition fails next to a clump of ties, and the resulting NaN fromsqrtused to be written out as a density of zero. At those points the local linear fit is used instead, together with its own influence value.fit_lp). The local linear estimate isf·exp(-h²b²/2), but the code usedhwhere it should useh², so the estimate was not scale equivariant. On one standard-normal sample rescaled by 0.1, 1 and 10, the density at 1.5 was 0.066, 0.148 and 0.155. It is now 0.1538 at every scale. This changes every degree-1 fit.fit). The binned kernel is truncated at 4h, so a point in a wide enough gap still got exactly zero at any degree. Grid values between the smallest and largest observation are now floored ateps × max. The interpolating spline preserves positivity (Schmidt–Hess), so the density is positive across the whole data range.Tests
test/data/tied_gap_sample.hpp, selects the plug-in bandwidth and has positive density and finite loglik and edf at every degree.All three fail on
main: 7 failed assertions. All pass with this change.Downstream, pyvinecopulib's kde1d suites (
test_kde1d,test_torch_kde1d,test_margins,test_sklearn_margins,test_plots; 234 tests) pass against this branch. Across 1,500 benchmark columns, 11 bandwidths change, all from the NaN fallback to the plug-in value, and no held-out point has density zero any more (previously one).🤖 Generated with Claude Code