Tracking Issue: Fix Canonical Decimal

This is a tracking issue for fixing Vortex's canonical Decimal encoding.

Status

The canonical width mapping, Arrow and DuckDB imports, compatibility fixtures, integer Widen, and faster widening copy have merged. Decimal Widen and the reader changes remain open. Release 0.91.0 is still a draft as of 2026-11-02, so CD5 and CD6 remain gated on publishing its fixtures.

Motivation

Our canonical Decimal encoding has been incorrect for too long.

The main (but not only) issue is that the canonicalization abstraction in Vortex has been broken by Decimal, where we can have multiple (up to 6) different possible canonical encodings for the same data. This breaks the abstraction because we want a SINGLE canonical (normal form) of logical data.

For example, right now if I have a Decimal(9, 2) (which means at most 9 integer digits, max integer 999_999_999, max decimal 9_999_999.99), but my decimal array looks like [1.00, 2.00, 1.27] where all of the integers fit in an i8 integer, then the canonical encoding allows us to implicitly narrow the buffer to 8 bits.

This is incredibly wrong because it means for the same array, we can have 6 different canonical representations (i8, i16, i32, i64, i128, i256).

The logical/physical overloading and mismatches have caused a lot of problems (see issue #5820). We seemingly have stamped out most of the bugs, but that doesn't mean the current implementation is not hacky.

There are 2 things that need to be done in order to fix this:

The canonical decimal encoding for a given decimal dtype must have exactly 1 canonical encoding / normal form.
Narrowing the data buffer must be a physical encoding concern, done via a new Widen encoding.

Design

For 1), we simply need to choose what physical encoding an array of Decimal(p, s) canonicalizes to. The simplest and most elegant solution is to pick the smallest power-of-2 bit-width buffer whose integer representation can fit every possible value in the domain of Decimal(p, s).

As an example, Decimal(9, 2) has a max integer value of 999,999,999, which means the smallest bit width it can fit into is 32 bits (i32::MAX == 2,147,483,647), so we can use 32-bit width buffers as the canonical encoding. On the other hand, while some values of Decimal(10, 2) can fit in 32 bits, the largest values must be stored in 64 bits, so we use 64-bit width buffers.

Of course, the issue is that this can potentially blow up the amount of memory and storage we use for representing potentially very small decimal values. But given that Vortex deals with compression quite well (via physical encodings), we can easily solve this by introducing a Widen array.

See Tracking Issue: Add Widen encoding for integers and decimals #10733

Backcompat conversion

The planned reader converts a vortex.decimal node whose stored width differs from its dtype's canonical width through WidenArray::from_stored_decimal. The reader scans valid values for their minimum and maximum, then picks the smallest child precision that covers the values and the scale. The result is a Widen(Decimal) when the child has a narrower canonical width, or a canonical decimal array otherwise.

The stored metadata width identifies non-canonical nodes. Nodes already at the canonical width reuse their buffer without a scan or copy. The current writer plan also emits narrow vortex.decimal nodes for decimal Widen arrays, so this conversion covers ordinary IPC and writes without a compressor as well as older files.

The conversion can reuse a buffer only when the selected child precision has that canonical width. Decimal precision does not exactly match two's-complement widths, so some values fit the stored width but need a precision whose canonical width is larger.

Width	Largest precision for the width	Largest value of that precision	Largest value of the width
`i8`	2	99	127
`i16`	4	9,999	32,767
`i32`	9	999,999,999	2,147,483,647
`i64`	18	10^18 - 1	about 9.2 × 10^18
`i128`	38	10^38 - 1	about 1.7 × 10^38

For example, if an old array has type Decimal(9, s) but only contains integers up to 127, we currently store it as i8. The values 100 to 127 have 3 digits, so in the canonical decimal encoding world we ideally want to narrow it to Decimal(3, s). If we naively say "the largest decimal precision fitting i8 is 2, so we should use Decimal(2, s)", that would not cover the values 100-127 that we need. And since the canonical width of Decimal(3, s) is i16, the conversion copies the i8 values into a wider buffer.

To choose a valid child precision, the reader needs both extrema of the valid values, including negative values, and the scale. This does mean that there will be cases where we cannot narrow some old decimal arrays into a buffer that "fits" in this new world. When the needed precision has the same canonical width as the dtype, there is no Widen at all: an old Decimal(4, s) array with values up to 127 needs precision 3, and precisions 3 and 4 both use i16, so it becomes a plain Decimal(4, s) array at i16. See unresolved questions below.

Nodes stored wider than their dtype's canonical width, such as older Arrow Decimal128 imports at a low precision, use the same child selection and get a narrowing copy. Every narrowing copy writes zeros into null slots and fails on a valid value outside the target precision.

Serialization before `vortex.widen`

The current stack supersedes D26's serializer-only lowering with a non-executing array writer plugin. #11057 separates plugin read and write IDs (merged on 2026-11-02). #11002 closed without merging and its writer work moved into #10946, now based on develop. That PR registers a writer-only WidenPlugin whose only write ID is vortex.decimal.

The writer adapts a concrete decimal leaf to a narrow vortex.decimal node under the parent logical dtype, without executing values or validity. The node stores the child width, preserving narrow storage without introducing a new wire ID. Integer Widen, non-decimal leaves, and nonserializable validity require explicit caller preparation.

#10995 reads non-canonical decimal nodes through the shared conversion. Because the read plugin receives stored statistics only after it returns, each non-canonical read needs a range scan. This is an ordinary path for new IPC and writes without a compressor as well as older files. The future vortex.widen format and its edition remain separate work in #10733.

Steps

The work lands as an in-memory change and then a wire change, so that the edition change stays small. A step lands as its own PR only if it needs no temporary code. Strict width decoding comes first, because the detection of old arrays reads the width field, and an unknown width currently decodes as I8 without an error.

PR plan (agents keep this list up to date)

Each line is one PR of the implementation plan. The briefs describe each PR. When a PR opens, add its link. When it merges, tick its box. When its scope changes, add a short note here. Record a change of plan in the research branch only when it changes a brief or a decision.

Follow-up issues found while planning:

Decimal scalars panic on a stored value outside the precision #10749 Decimal scalars panic on a stored value outside the precision. Return an error for decimal scalars outside the precision #10767 remains open and fixes scalar_at and the min_max kernel. Decimal stats read from a file are not checked against the precision #10807 separately tracks cached decimal statistics that bypass that kernel. Other Scalar::decimal callers stay for later PRs
compat-gen skips old fixture versions in CI and stops at the first failure #10754 compat-gen skips old fixture versions in CI and stops at the first failure
Corrupt ALP-RD, RLE and RunEnd metadata can panic a file read #10756 Corrupt ALP-RD, RLE and RunEnd metadata can panic a file read. fix(alp): bounds-check ALP-RD stored metadata and left-parts codes #10760 fixes ALP-RD, merged on 2026-10-29. Return Serde errors for corrupt RLE and RunEnd metadata #10852 fixes RLE and RunEnd, open
Seven cast rules push nullability casts without trivially_cast_nullability #10750 Push-down cast rules without the nullability guard. Closed as not planned, because Document the nullability contract of cast rules #10798 documents them as valid

Related PRs outside this plan:

Bound decimal storage by precision #10305 and Add bounded decimal dispatch macros and migrate kernels #10314 bound decimal storage by precision and add bounded dispatch macros. Open. The plan does not build on them for now
perf(decimal-byte-parts): compare wide decimals through their parts #10146 compares wide decimals through their DecimalByteParts parts. Open
Skip codegen of unreachable int to decimal carrier arms #10735 removes unreachable decimal carrier arms from the integer to decimal cast. Merged on 2026-10-26
Establish widening cast benchmark baselines #11052 (widening cast benchmark baselines) and [EXPERIMENT] Keep ARM integer widening loops compact #11050 (ARM loop experiment) closed without merging. Neither is a dependency

Unresolved questions

Is it okay that we are not able to narrow every decimal array exactly how it was before? ~~Note that this only affects canonical decimal arrays, not encodings like DecimalByteParts.~~ Yes, see explanations above.
Should DecimalByteParts decoding find the max of a narrow most significant part and return a Widen over its buffer instead of widening it? ~~This only matters for files written before the compressor narrows the decimal type first, so the decision waits for benchmarks that show the cost of widening.~~ Yes. Widening every narrow MSP made decoding up to 11x slower in CodSpeed (Decode DecimalByteParts into the canonical array or a decimal Widen #10905), and every default-written decimal column whose values are small for its precision has a narrow MSP. CD5 takes the range from min_max on the MSP and returns a Widen that reuses the MSP buffer (LT2, folded into CD5).