dd: write complete blocks without buffering them first#13471
dd: write complete blocks without buffering them first#13471relative23 wants to merge 5 commits into
Conversation
Merging this PR will improve performance by ×7.5
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | dd_copy_separate_blocks |
44,075.9 µs | 970.1 µs | ×45 |
| ⚡ | Simulation | dd_copy_default |
13.6 ms | 11.1 ms | +22.41% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing relative23:dd-buffered-output-fast-path (ac19344) with main (4daa891)2
Footnotes
-
46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
main(e5fcabf) during the generation of this report, so 4daa891 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
|
GNU testsuite comparison: |
BufferedOutput::write_blocks copies the incoming bytes into its internal buffer, hands that buffer to the inner writer, clears it, and then re-buffers the trailing partial block -- even when nothing was pending and the incoming bytes already form complete blocks. That is the common case: the copy loop hands over a multiple of the output block size on every iteration, so each one paid for a full memcpy of the block. Take that case directly: with an empty internal buffer, write the complete blocks straight from the caller's slice and buffer only the remainder. The existing path still handles a pending partial block. Instruction counts (cachegrind, same toolchain and base): dd_copy_separate_blocks 55,916,646 -> 5,447,251 -90.26% dd_copy_default 36,615,676 -> 30,718,154 -16.11% dd_copy_with_seek 8,511,906 -> 8,488,175 -0.28% dd_copy_with_skip 8,412,120 -> 8,389,437 -0.27% dd_copy_4k_blocks 4,742,612 -> 4,730,524 -0.25% dd_copy_8k_blocks 4,408,987 -> 4,401,037 -0.18% dd_copy_partial 3,222,128 -> 3,220,704 -0.04% dd_copy_64k_blocks 5,107,029 -> 5,105,557 -0.03% dd_copy_1m_blocks 9,410,822 -> 9,410,932 +0.00% Peak heap is unchanged (DHAT, 236,549 -> 236,546 bytes).
e895791 to
3ff85c0
Compare
|
Wahou, impressive wins! Bravo |
3 participants
BufferedOutput::write_blockscopies the incoming bytes into its internal buffer, hands that buffer to the inner writer, clears it, and then re-buffers the trailing partial block — even when nothing was pending and the incoming bytes already form complete blocks. That is the common case: the copy loop hands over a multiple of the output block size on every iteration, so each iteration paid for a full memcpy of the block.Change
With an empty internal buffer, write the complete blocks straight from the caller's slice and buffer only the remainder. The existing path is untouched and still handles a pending partial block. Pure optimization — no behavioural change.
Measurements
Instruction counts via cachegrind on the CodSpeed simulation binaries (
cargo codspeed build -m simulation -p uu_dd), same toolchain, both built from this PR's base:maindd_copy_separate_blocksdd_copy_defaultdd_copy_with_seekdd_copy_with_skipdd_copy_4k_blocksdd_copy_8k_blocksdd_copy_partialdd_copy_64k_blocksdd_copy_1m_blocksThe two large wins are the configurations that actually use the buffered output path:
dd_copy_separate_blocks(ibs=8K obs=16K) anddd_copy_default(nobs, so output is buffered). The measurement is deterministic — repeated runs of the same binary vary by ~2 instructions out of 36M.Peak heap is unchanged: 236,549 → 236,546 bytes (DHAT,
ibs=8K obs=16K, both binaries built in the same target directory).Testing
cargo test -p uu_ddand the dd integration suite pass.mainover 17 configurations (oddibs/obs,ibs>obsandobs>ibs,bs,conv=sync/block/unblock/swab,count,count_bytes,skip/seek,obs=1M,ibs=1M obs=7): output and therecords in/records outlines are identical in every one.cargo fmt, the repository clippy invocation (util/run-clippy.py --features all --workspace) and cspell are clean.Note
This touches the same function as #13459, which fixes a correctness bug in the pending-partial-block path. The two are independent — this PR only adds the empty-buffer fast path and leaves the existing branch alone — but they overlap textually, so whichever lands second needs a trivial rebase.