[XiangpengHao]

Summary

DictStrategy closes a dictionary once its values reach 1 MiB before compression, and the default write strategy has no way to change that. High-cardinality string columns reach the cap every few tens of thousands of rows. In the file, each such column is laid out in runs of about one dictionary (its values and its codes), with other columns' segments in between. With CoalesceConfig::object_storage() (1 MiB gap, 16 MiB max), a scan of one of these columns merges across the gaps and reads the other columns' segments too.

This raises the default cap to 2 MiB and lets WriteStrategyBuilder set it. On ClickBench's hits_0 (1M rows), converted from Parquet with the default write strategy once with each cap:

Column	Dictionaries	Run length (p50)	Coalesced scan reads, × the column's bytes
URL	44 → 21	389 → 736 KiB	1.81× → 1.09×
Referer	34 → 17	440 → 836 KiB	1.51× → 1.00×
Title	27 → 12	346 → 659 KiB	1.80× → 1.41×
OriginalURL	21 → 11	331 → 608 KiB	1.48× → 1.12×

The file shrinks from 96.5 MB to 92.5 MB (−4.2%), and from 788 to 656 segments, since repeated strings are stored in fewer dictionaries. The scan figures are computed from the footer's segment map with the object-storage window, not measured from queries.

The cost is reads of a few rows: they fetch and decode a dictionary about twice as large. For URL, one row needs about 731 KiB of codes and values, up from 383 KiB. Writers who care more about point reads can keep 1 MiB with with_dict_max_bytes.

Changes

• DictLayoutConstraints::default() sets max_bytes to 2 MiB, up from 1 MiB. Its doc now says the cap is measured on the values before compression, and that the remaining rows start a new dictionary.
• WriteStrategyBuilder::with_dict_max_bytes sets the cap for the default write strategy, which used to always pass the default constraints.

API Changes

• New: WriteStrategyBuilder::with_dict_max_bytes.
• Files written with the default strategy get dictionaries of up to 2 MiB of values. Only string and binary columns change: primitive dictionaries reach the 65,535-entry limit before either cap.

No new tests. Builds, clippy and tests were not run locally.

[codspeed]

Merging this PR will improve performance by 23.13%

4 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ Unable to measure the execution of 2 benchmarks

These benchmarks measured no execution time, so their results are not comparable and count as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 22 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 40 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

⚡ 2 improved benchmarks
✅ 2406 untouched benchmarks
⏩ 12 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	CPU Simulation	test_repeated_scan_scalar_at[int]	318.8 µs	239.7 µs	+33.01%
⚡	Walltime	bitpack_blocked_compress_avx2	7.6 µs	6.7 µs	+13.98%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing XiangpengHao:dict-max-bytes (ee12d9a) with develop (91b38f4)}

Footnotes

1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩