Skip to content

fix: chunked file scan drops bytes - #404

Open
cx-jose-fernandes wants to merge 6 commits into
masterfrom
AST-191743-fix-chunked-file-scan-drops-bytes
Open

cx-jose-fernandes wants to merge 6 commits into
masterfrom
AST-191743-fix-chunked-file-scan-drops-bytes

Conversation

@cx-jose-fernandes

@cx-jose-fernandes cx-jose-fernandes commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Closes #

Proposed Changes

Fixes a bug in chunked file scanning (files larger than 1 MB) that skipped part of the file and reported wrong line numbers.

Root cause

ReadChunk read chunk size + peek size bytes with reader.Read, which consumes them. generateChunk can end the chunk earlier, at a blank line inside the peek window, but the leftover bytes were already consumed. They were never scanned and never counted in totalLines. As a result:

  • every finding after that point was reported on a line number that was too low by the number of dropped lines
  • secrets inside the dropped bytes were silently missed

Fix

  • ReadChunk now uses reader.Peek to look at the data without consuming it.
  • It then calls reader.Discard to consume only the bytes that went into the chunk, so the next call starts exactly where the previous chunk ended.
  • generateChunk, IChunk, the mock and engine.go are unchanged.

Whether the bug triggers depends on the exact bytes of the file (for example its line endings), because they decide where a blank line falls relative to the chunk boundaries.

Testing

  • Added TestReadChunkDoesNotDropBytes, which reads a whole input chunk by chunk and checks that nothing is lost and that the line count matches. It fails on the old code and passes with the fix.
  • Scanned the bug-attached file to make sure the issue no longer happens.

Jira: AST-191743

Checklist

  • I covered my changes with tests.
  • I Updated the documentation that is affected by my changes:
    • Change in the CLI arguments
    • Change in the configuration file

I submit this contribution under the Apache-2.0 license.

@cx-jose-fernandes
cx-jose-fernandes requested a review from a team November 17, 2026 11:49
@cx-jose-fernandes cx-jose-fernandes changed the title Ast 191743 fix chunked file scan drops bytes fix: chunked file scan drops bytes Nov 17, 2026
Comment thread engine/chunk/chunk.go Resolved
Comment thread engine/chunk/chunk.go Resolved
cx-jose-fernandes and others added 3 commits November 18, 2026 14:56
Co-authored-by: Diogo Rocha <104084969+cx-diogo-rocha@users.noreply.github.com>
….com:Checkmarx/2ms into AST-191743-fix-chunked-file-scan-drops-bytes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants