Repository navigation
fix: chunked file scan drops bytes - #404
Open
cx-jose-fernandes wants to merge 6 commits into
Open
cx-jose-fernandes wants to merge 6 commits into
cx-jose-fernandes wants to merge 6 commits into
Conversation
cx-diogo-rocha
requested changes
Nov 18, 2026
Co-authored-by: Diogo Rocha <104084969+cx-diogo-rocha@users.noreply.github.com>
….com:Checkmarx/2ms into AST-191743-fix-chunked-file-scan-drops-bytes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #
Proposed Changes
Fixes a bug in chunked file scanning (files larger than 1 MB) that skipped part of the file and reported wrong line numbers.
Root cause
ReadChunkreadchunk size + peek sizebytes withreader.Read, which consumes them.generateChunkcan end the chunk earlier, at a blank line inside the peek window, but the leftover bytes were already consumed. They were never scanned and never counted intotalLines. As a result:Fix
ReadChunknow usesreader.Peekto look at the data without consuming it.reader.Discardto consume only the bytes that went into the chunk, so the next call starts exactly where the previous chunk ended.generateChunk,IChunk, the mock andengine.goare unchanged.Whether the bug triggers depends on the exact bytes of the file (for example its line endings), because they decide where a blank line falls relative to the chunk boundaries.
Testing
TestReadChunkDoesNotDropBytes, which reads a whole input chunk by chunk and checks that nothing is lost and that the line count matches. It fails on the old code and passes with the fix.Jira: AST-191743
Checklist
I submit this contribution under the Apache-2.0 license.