fix: stream CLI output instead of line-buffering to memory by digizeph · Pull Request #80 · bgpkit/oneio

digizeph · 2026-07-23T22:55:58Z

Summary

The CLI wrapped the decompression reader in BufReader::lines() which reads until \n. For single-line files (e.g., the 2.2 GB decompressed as2rel-collector-latest.json.bz2), this forced decompression of the entire file into memory before any output.

Changes

Default mode: use io::copy() to stream decompressed bytes directly to stdout
--stats mode: byte-chunk scanner counts \n bytes and UTF-8 codepoints without ever building full-line String allocations. Properly handles multi-byte UTF-8 sequences split across chunk boundaries.

Before → After

Scenario	Before	After
Local `.bz2` → first byte	31.8s	0.008s
Remote URL → first byte	32.6s	0.11s
`--stats` (char+line count)	31.6s	31.1s (decompression-bound)

Testing

cargo fmt --check ✓
cargo clippy --all-features -- -D warnings ✓
cargo clippy --no-default-features ✓
cargo test --all-features — 88 passed, 0 failed, 16 ignored (S3)

The CLI wrapped the decompression reader in BufReader::lines() which reads until \n. For single-line files (e.g., 2.2 GB JSON), this forced decompression of the entire file into memory before any output. - Default mode: use io::copy() to stream bytes directly to stdout - --stats mode: byte-chunk scanner counts \n and UTF-8 codepoints without building full-line String allocations - Both paths now start streaming immediately (0.008s vs 31.8s for local 44 MB .bz2, 0.11s vs 32.6s for remote)

Copilot

Pull request overview

This PR improves the oneio CLI’s output path to avoid buffering entire “single-line” decompressed files into memory by streaming decompressed bytes directly to stdout, and reworks --stats to compute line/character counts via chunked scanning.

Changes:

Default CLI reader mode now streams decompressed bytes to stdout via std::io::copy() rather than iterating BufRead::lines().
--stats mode now scans byte chunks, counting \n bytes and UTF-8 codepoints without allocating full-line Strings.
Adds a changelog entry describing the CLI memory/performance fix.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/bin/oneio.rs	Replaces line-based iteration with streaming output; implements chunked UTF-8-aware stats scanning.
CHANGELOG.md	Documents the CLI streaming/stats change under `[Unreleased]`.

+    } else {
+        // Streaming mode: copy decompressed bytes directly to stdout
+        // without line-based buffering. Avoids buffering multi-GB single-line
+        // JSON files in memory.
+        let mut stdout = std::io::stdout();
+        if let Err(e) = std::io::copy(&mut reader, &mut stdout) {
+            if e.kind() != std::io::ErrorKind::BrokenPipe {
+                eprintln!("read error on {path}: {e}");
+                exit(1);
+            }
+        }
    }


+        let mut buf = [0u8; 65536];
+        let mut carry: Vec<u8> = Vec::new();
+        let mut last_byte = b'\n'; // treat empty file as ending with \n
+
+        loop {
+            let n = match reader.read(&mut buf) {
+                Ok(0) => break,
+                Ok(n) => n,
+                Err(e) => {
+                    eprintln!("read error on {path}: {e}");
+                    exit(1);
+                }
+            };
+
+            // Prepend carry bytes from the previous chunk (incomplete
+            // multi-byte UTF-8 sequence at end of last buffer).
+            let mut data: Vec<u8> = std::mem::take(&mut carry);
+            data.extend_from_slice(&buf[..n]);


…Vec allocation - --strict-utf8 now errors immediately in default streaming mode instead of being silently ignored - Stats mode reuses a persistent Vec<u8> for the combined chunk buffer instead of allocating a new Vec every iteration

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

src/bin/oneio.rs:386

Same as above: in the partial-valid UTF-8 branch, count_chars currently counts newline characters. To keep --stats output consistent with the previous line-based implementation, subtract the number of \n bytes from the char count for the validated prefix.

                                std::str::from_utf8_unchecked(&data[pos..pos + valid_up_to])
                            };
                            count_chars += valid.chars().count();
                            count_lines += valid.as_bytes().iter().filter(|&&b| b == b'\n').count();
                            if let Some(&b) = valid.as_bytes().last() {

src/bin/oneio.rs:449

io::copy can fail due to either read or write errors, but the current message always says "read error". This can mislead users when stdout fails (e.g., "No space left on device" when redirecting output).

        if let Err(e) = std::io::copy(&mut reader, &mut stdout) {
            if e.kind() != std::io::ErrorKind::BrokenPipe {
                eprintln!("read error on {path}: {e}");
                exit(1);

+                        count_chars += valid.chars().count();
+                        count_lines += valid.as_bytes().iter().filter(|&&b| b == b'\n').count();


+        if cli.strict_utf8 {
+            eprintln!("--strict-utf8 has no effect outside --stats mode; use --stats --strict-utf8 to validate UTF-8 content");
+            exit(1);
+        }


+                        if let Some(err_len) = e.error_len() {
+                            // Genuinely invalid UTF-8 sequence
+                            if cli.strict_utf8 {
+                                eprintln!("invalid UTF-8 at byte offset ~{} in {path}", pos);


…solute offset - count_chars now excludes newline bytes, matching BufReader::lines() semantics - --strict-utf8 in default mode warns to stderr instead of hard-exiting - invalid UTF-8 error reports approximate absolute file offset instead of chunk-relative position

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

+        // Match BufReader::lines() semantics: a non-empty file that
+        // doesn't end with \n still has one last (implicit) line.
+        let mut count_lines = count_newlines;
+        if last_byte != b'\n' {
+            count_lines += 1;
+        }
+
+        // Match BufReader::lines() semantics: newline characters are
+        // stripped from line strings, so old stats never counted them.
+        count_chars = count_chars.saturating_sub(count_newlines);
+


+                        let valid_up_to = e.valid_up_to();
+                        if valid_up_to > 0 {
+                            let valid = unsafe {
+                                std::str::from_utf8_unchecked(&data[pos..pos + valid_up_to])
+                            };
+                            count_chars += valid.chars().count();


+        let mut stdout = std::io::stdout();
+        if let Err(e) = std::io::copy(&mut reader, &mut stdout) {
+            if e.kind() != std::io::ErrorKind::BrokenPipe {
+                eprintln!("read error on {path}: {e}");
+                exit(1);
+            }
+        }


- Track CRLF pairs across chunk boundaries and subtract \r bytes from char count, matching BufReader::lines() which strips \r\n terminators - Replace unsafe from_utf8_unchecked with safe from_utf8().unwrap() - Lock stdout once in streaming mode instead of repeated per-chunk locks - Fix I/O error message to cover both read and write failures

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/bin/oneio.rs:487

--strict-utf8 is documented as “Fail on invalid UTF-8”, but in streaming (non---stats) mode it is silently ignored (only a warning is printed). This changes the flag’s contract and can lead to users believing output was validated when it wasn’t. Consider either implementing streaming UTF-8 validation, or failing fast when --strict-utf8 is used without --stats so the behavior is not silently ignored.

        if cli.strict_utf8 {
            eprintln!("warning: --strict-utf8 has no effect outside --stats mode; use --stats --strict-utf8 to validate UTF-8 content");
        }

+    if cli.stats {
+        let mut count_newlines = 0usize;
+        let mut count_crlf = 0usize;
+        let mut count_chars = 0usize;
+        let mut buf = [0u8; 65536];
+        let mut carry: Vec<u8> = Vec::new();
+        let mut combined: Vec<u8> = Vec::with_capacity(65536 + 4);
+        let mut last_byte = b'\n'; // treat empty file as ending with \n
+        let mut file_offset = 0usize; // approx absolute byte position
+        let mut prev_was_cr = false; // for CRLF tracking across chunks
+


 ## [Unreleased]

+### Fixed
+- CLI no longer buffers multi-GB single-line files into memory before output. Default mode now streams decompressed bytes directly to stdout via `io::copy` instead of line-by-line reading. `--stats` mode uses byte-chunk scanning to count lines and characters without building full-line `String` allocations.


+        let mut last_byte = b'\n'; // treat empty file as ending with \n
+        let mut file_offset = 0usize; // approx absolute byte position
+        let mut prev_was_cr = false; // for CRLF tracking across chunks


…avior - Extract --stats logic into compute_text_stats() helper that returns Result, improving testability for edge cases (CRLF across chunks, invalid sequences, incomplete UTF-8 at EOF) - Use u64 for file_offset to avoid overflow on 32-bit or large files - CHANGELOG now documents behavior change: default mode emits raw bytes with no UTF-8 decoding or line-based rewriting

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

                 {bytes}/{total_bytes} ({bytes_per_sec}, {eta})",
            )?
-            .progress_chars("#>-"),
+            .progress_chars(">-#"),


+/// Compute (lines, chars) from a byte stream using chunked scanning.
+///
+/// Matches `BufReader::lines()` semantics: line terminators (`\n` and
+/// `\r\n`) are excluded from the character count, and a non-empty file
+/// without a trailing newline still counts its last segment as a line.
+///
+/// When `strict_utf8` is true, returns `Err` on the first invalid UTF-8
+/// sequence or incomplete sequence at EOF.
+fn compute_text_stats(
+    reader: &mut (dyn Read + Send),
+    strict_utf8: bool,
+) -> Result<(usize, usize), String> {


- Fix indicatif progress_chars order: filled/current/remaining - Add #[cfg(test)] module with 9 tests covering LF, CRLF, no-trailing- newline, empty, single-line, strict-utf8, lossy, CRLF across chunk boundary, and bare CR edge cases

Copilot