Skip to content

Respect rounding mode when quantizing sub-ULP values - #152

Merged
rafiss merged 2 commits into
cockroachdb:masterfrom
chiliec:fix-quantize-subulp-rounding
Oct 22, 2026
Merged

rafiss merged 2 commits into
cockroachdb:masterfrom
chiliec:fix-quantize-subulp-rounding

Conversation

@chiliec

@chiliec chiliec commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Fixes #140

What

Context.Quantize ignored the rounding mode when the input had fewer significant digits than the target exponent (i.e. its magnitude was smaller than one unit in the last place). It truncated straight to zero regardless of the configured Rounder:

ctx := apd.Context{Precision: 100, Rounding: apd.RoundCeiling, MaxExponent: 1000, MinExponent: -1000, Traps: apd.DefaultTraps}
var d apd.Decimal
d.SetString("0.002")
ctx.Quantize(&d, &d, -1)
// got "0.0", want "0.1"

RoundCeiling(1.002, -1) already returned 1.1; only the sub-ULP case was wrong.

Fix

In the p < 0 branch of quantize, run the discarded coefficient through the context's Rounding.ShouldAddOne (the same mechanism the normal rounding path uses) instead of unconditionally zeroing it.

Tests

Added TestQuantizeRounding covering the reported case plus every rounding mode (ceiling/floor/up/down/half-up/half-even, positive and negative). Verified genuine RED→GREEN:

# with fix
$ go test -run TestQuantizeRounding .
ok  github.com/cockroachdb/apd/v3

# revert context.go, keep the test
$ git stash push context.go && go test -run TestQuantizeRounding .
--- FAIL: TestQuantizeRounding/0.002:_-1_ceiling
    expected: 0.1, got: 0.0
--- FAIL: TestQuantizeRounding/-0.002:_-1_floor
    expected: -0.1, got: -0.0
FAIL

Full suite (including the GDA conformance tests) is green:

$ go test ./...
ok  github.com/cockroachdb/apd/v3  1.524s

Happy to adjust.

Quantize truncated any value with fewer digits than the target exponent
straight to zero, ignoring the context's Rounding mode. RoundCeiling of a
positive sub-ULP value (e.g. 0.002 to exponent -1) returned 0.0 instead of
0.1, and RoundFloor of a negative one lost the rounding entirely.

Round the discarded coefficient through the configured Rounder instead.
When the coefficient has fewer digits than the exponent difference,
the discarded value is below a tenth of a unit in the last place, so
it always compares less than one half. Pass that result to
ShouldAddOne directly instead of building a Decimal to compare.

Add Round05Up and RoundHalfDown rows to TestQuantizeRounding, assert
the Inexact and Rounded conditions, check a separate destination
alongside the aliased call, and cover RoundToIntegralValue and
RoundToIntegralExact, which reach the same branch through toIntegral.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@rafiss rafiss left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix is right. I checked the branch against Python's decimal for all eight rounding modes on sub-ULP inputs and signed zeros, including through RoundToIntegralValue and RoundToIntegralExact, and results and flags match. PostgreSQL numeric agrees on the cases it can express (half away from zero, trunc, ceil and floor), differing only in the sign of zero, which numeric does not have.

I pushed c3295ad on top. In this branch the coefficient has fewer digits than the exponent difference, so the discarded value is always below one half and discard.Cmp(decimalHalf) is always -1; the commit passes -1 directly with a comment saying why, which also makes it clear that only ceiling, floor, up and 05up can round to one unit here. It adds Round05Up and RoundHalfDown rows, asserts the Inexact and Rounded conditions, checks a separate destination, and adds a RoundToIntegral test since toIntegral reaches the same branch.

One small thing for the record: the branch fires for values below a tenth of a ULP, not below one ULP. 0.05 quantized to one decimal place already rounded correctly before this change.

@rafiss
rafiss merged commit e9c7a71 into cockroachdb:master Oct 22, 2026
18 checks passed
@chiliec

chiliec commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough cross-check and the follow-up commit — and good catch on the threshold: it's below a tenth of a ULP, not one ULP. Appreciate the merge.

@chiliec
chiliec deleted the fix-quantize-subulp-rounding branch October 23, 2026 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RoundCeil(0.002, -1) != 0.1

2 participants