Repository navigation
Respect rounding mode when quantizing sub-ULP values - #152
Conversation
Quantize truncated any value with fewer digits than the target exponent straight to zero, ignoring the context's Rounding mode. RoundCeiling of a positive sub-ULP value (e.g. 0.002 to exponent -1) returned 0.0 instead of 0.1, and RoundFloor of a negative one lost the rounding entirely. Round the discarded coefficient through the configured Rounder instead.
When the coefficient has fewer digits than the exponent difference, the discarded value is below a tenth of a unit in the last place, so it always compares less than one half. Pass that result to ShouldAddOne directly instead of building a Decimal to compare. Add Round05Up and RoundHalfDown rows to TestQuantizeRounding, assert the Inexact and Rounded conditions, check a separate destination alongside the aliased call, and cover RoundToIntegralValue and RoundToIntegralExact, which reach the same branch through toIntegral. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
rafiss
left a comment
There was a problem hiding this comment.
The fix is right. I checked the branch against Python's decimal for all eight rounding modes on sub-ULP inputs and signed zeros, including through RoundToIntegralValue and RoundToIntegralExact, and results and flags match. PostgreSQL numeric agrees on the cases it can express (half away from zero, trunc, ceil and floor), differing only in the sign of zero, which numeric does not have.
I pushed c3295ad on top. In this branch the coefficient has fewer digits than the exponent difference, so the discarded value is always below one half and discard.Cmp(decimalHalf) is always -1; the commit passes -1 directly with a comment saying why, which also makes it clear that only ceiling, floor, up and 05up can round to one unit here. It adds Round05Up and RoundHalfDown rows, asserts the Inexact and Rounded conditions, checks a separate destination, and adds a RoundToIntegral test since toIntegral reaches the same branch.
One small thing for the record: the branch fires for values below a tenth of a ULP, not below one ULP. 0.05 quantized to one decimal place already rounded correctly before this change.
|
Thanks for the thorough cross-check and the follow-up commit — and good catch on the threshold: it's below a tenth of a ULP, not one ULP. Appreciate the merge. |
Fixes #140
What
Context.Quantizeignored the rounding mode when the input had fewer significant digits than the target exponent (i.e. its magnitude was smaller than one unit in the last place). It truncated straight to zero regardless of the configuredRounder:RoundCeiling(1.002, -1)already returned1.1; only the sub-ULP case was wrong.Fix
In the
p < 0branch ofquantize, run the discarded coefficient through the context'sRounding.ShouldAddOne(the same mechanism the normal rounding path uses) instead of unconditionally zeroing it.Tests
Added
TestQuantizeRoundingcovering the reported case plus every rounding mode (ceiling/floor/up/down/half-up/half-even, positive and negative). Verified genuine RED→GREEN:Full suite (including the GDA conformance tests) is green:
Happy to adjust.