Repository navigation
fix: avoid zero-size realloc in remove_nans - #41
Merged
Merged
Conversation
`remove_nans` shrinks `x` and `weights` with `conservativeResize`, which for a dynamic vector reaches Eigen's `aligned_realloc` and calls `std::realloc(ptr, new_size)` directly. When every row is dropped -- every observation NaN, every weight a drop marker, or an already empty `x` -- `new_size` is zero. `realloc(ptr, 0)` is deprecated in C17 and undefined in C23, and valgrind reports it as an error. CRAN's valgrind check of the R package hit this through a zero-inflated fit whose observations were all zero: `fit` zeroes the weight of every zero observation, marks those rows NaN, and calls `remove_nans` a second time, which empties both vectors. https://www.stats.ox.ac.uk/pub/bdr/memtests/valgrind/kde1d/tests/testthat.Rout Nothing in this suite noticed, because a zero-size `realloc` returns normally -- the tests only fail under a memory checker. So this adds both the cases and a CI job that runs the suite under `valgrind --error-exitcode=1`. Both currently fail there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When every row is a drop marker, `last` wraps to `SIZE_MAX` and the new size is zero. Eigen's `conservativeResize` reaches `aligned_realloc` for a dynamic vector, so that became `std::realloc(ptr, 0)` -- deprecated in C17, undefined in C23, and an error under valgrind. Swap in a fresh empty vector instead. That only exchanges storage pointers, and the old buffer is released by `free` when the temporary dies. The resulting sizes are unchanged, so every caller sees what it saw before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Valgrind catches the zero-size `realloc` this branch fixes, but that is one narrow class of defect. Signed overflow, bad shifts, invalid casts, stack errors and leaks are all things it either cannot see or does not report, and the suite has never been run under a sanitizer in CI -- every existing job passes `-DOPT_ASAN=OFF`. `OPT_ASAN` could not have been turned on instead: it only appends to `CMAKE_CXX_FLAGS_DEBUG`, so it is silently inert in the release build the tests actually run, and it cannot request anything but the address sanitizer. So this adds a `SANITIZERS` cache variable taking an explicit `-fsanitize=` list, applied to every configuration, with `-fno-sanitize-recover=all` so a finding exits nonzero -- UBSan otherwise prints a diagnostic and carries on, and CI would pass. The valgrind job becomes a `memory-checks` matrix: valgrind, and address+undefined under both GCC and Clang (the leak sanitizer comes with the address sanitizer on Linux). Valgrind and the address sanitizer both instrument allocation and cannot share a process, hence separate entries; GCC and Clang disagree often enough about what UBSan instruments to be worth both. All three entries pass locally on this branch: 587 + 721 assertions, no findings. Verified the sanitizer entries fail as intended by temporarily introducing a signed overflow and a heap overread into the suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
CRAN's valgrind check of the R package kde1d reports two errors:
https://www.stats.ox.ac.uk/pub/bdr/memtests/valgrind/kde1d/tests/testthat.Rout
remove_nansmoves every dropped row to the end and then shrinks both vectors:When every row is a drop marker -- a NaN observation, a NaN weight or a zero weight --
lastwraps toSIZE_MAXand the new size is zero. (An already emptyxgets there the same way.) For a dynamic vector, Eigen'sconservativeResizereachesaligned_realloc, which callsstd::realloc(ptr, 0). That is deprecated in C17 and undefined in C23, so valgrind flags it. Two contexts: one forx, one forweights.The reported trigger is a zero-inflated fit whose observations are all zero.
Kde1d::fitzeroes the weight of every zero observation, marks those rows NaN and callsremove_nansa second time, which empties both vectors. In R that iswhich the R package's test suite exercises directly. The
fitpaths that drop every row and then throw reach the same call, sokde1d(c(NA, NA))and all-zero weights hit it too.Nothing here is a memory error -- no bad access, no leak -- but CRAN treats the valgrind report as a check failure, and the call is undefined under C23 regardless.
The fix
Handle the empty result without asking Eigen to reallocate: swap in a fresh empty vector, which only exchanges storage pointers and releases the old buffer through
free. Resulting sizes are unchanged, so every caller sees exactly what it saw before.Tests
test/test.cppgains a case covering all the drop-everything paths, including the zero-inflated fit CRAN reported.CI:
memory-checksA zero-size
reallocreturns normally, so the suite could never have caught this on its own -- it only fails under a checker, and the suite has never been run under one here. Every existing job passes-DOPT_ASAN=OFF, andOPT_ASANcould not simply have been turned on instead: it only appends toCMAKE_CXX_FLAGS_DEBUG, so it is inert in the release build the tests actually run, and it cannot request anything but the address sanitizer.So this adds a
SANITIZERScache variable taking an explicit-fsanitize=list, applied to every configuration, with-fno-sanitize-recover=allso a finding exits nonzero -- UBSan otherwise prints a diagnostic and carries on, and CI would pass.OPT_ASANis left alone.The new
memory-checksjob is a matrix of three entries, each building the release suite and running both test binaries:valgrindreallocabove, which no sanitizer flagsasan+lsan+ubsan GNUasan+lsan+ubsan ClangValgrind and the address sanitizer both instrument allocation and cannot share a process, hence separate entries. The leak sanitizer comes with the address sanitizer on Linux.
All three pass locally on this branch (587 + 721 assertions, no findings). Checked that they fail as intended:
Downstream verification
Rebuilt the R package against these headers and ran its test suite under valgrind, in CRAN's configuration:
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 501 ],ERROR SUMMARY: 0 errors from 0 contexts(was 2).Other sites of the same kind
Checked every other
resize/conservativeResizeininclude/; none can reach a zero-sizerealloc:stats.hpp:194,tab.conservativeResize(lev, 2)--levis 0 for a single observation, buttabis a column-major matrix whose row count changes, and Eigen cannot reallocate in place for that:conservative_resize_like_implbuilds a temporary and swaps. Verified clean under valgrind.interpolation.hpp:357,interpolation.hpp:378,kde1d.hpp:1258-- plainresize, whoseDenseStoragepath frees and allocates, guarded on a nonzero size. Noreallocinvolved.