Repository navigation
xdiff: reduce overhead in Myers diffs - #2311
sauravjoshi23 wants to merge 13 commits into
Conversation
Welcome to GitGitGadgetHi @sauravjoshi23, and welcome to GitGitGadget, the GitHub App to send patch series to the Git mailing list from GitHub Pull Requests. Please make sure that either:
You can CC potential reviewers by adding a footer to the PR description with the following syntax: NOTE: DO NOT copy/paste your CC list from a previous GGG PR's description, Also, it is a good idea to review the commit messages one last time, as the Git project expects them in a quite specific form:
It is in general a good idea to await the automated test ("Checks") in this Pull Request before contributing the patches, e.g. to avoid trivial issues such as unportable code. Contributing the patchesBefore you can contribute the patches, your GitHub username needs to be added to the list of permitted users. Any already-permitted user can do that, by adding a comment to your PR of the form Both the person who commented An alternative is the channel Once on the list of permitted usernames, you can contribute the patches to the Git mailing list by adding a PR comment If you want to see what email(s) would be sent for a After you submit, GitGitGadget will respond with another comment that contains the link to the cover letter mail in the Git mailing list archive. Please make sure to monitor the discussion in that thread and to address comments and suggestions (while the comments and suggestions will be mirrored into the PR by GitGitGadget, you will still want to reply via mail). If you do not want to subscribe to the Git mailing list just to be able to respond to a mail, you can download the mbox from the Git mailing list archive (click the curl -g --user "<EMailAddress>:<Password>" \
--url "imaps://imap.gmail.com/INBOX" -T /path/to/raw.txtTo iterate on your change, i.e. send a revised patch or patch series, you will first want to (force-)push to the same branch. You probably also want to modify your Pull Request description (or title). It is a good idea to summarize the revision by adding something like this to the cover letter (read: by editing the first comment on the PR, i.e. the PR description): To send a new iteration, just add another PR comment with the contents: Need help?New contributors who want advice are encouraged to join git-mentoring@googlegroups.com, where volunteers who regularly contribute to Git are willing to answer newbie questions, give advice, or otherwise provide mentoring to interested contributors. You must join in order to post or view messages, but anyone can join. You may also be able to find help in real time in the developer IRC channel, |
|
/allow |
|
User sauravjoshi23 is now allowed to use GitGitGadget. |
Cover unchanged-span boundaries, common records that also appear in the middle, and equal fingerprints with different line contents. Add a Myers shortstat performance case over 8,192 non-merge commits from the perf test repository. Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
Emitting records individually adds callback and buffer-walking overhead. Buffer complete records and flush before hunk headers and large records. Coalesce small prefixes, record contents and optional missing-newline markers into a single callback buffer. Preserve callback failures and the existing large-record fallback. Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
Each new line class is allocated from a chastore and chained into a hash bucket, and every lookup walks that chain through pointers. Keep the classes in one contiguous array and index them from a compact open-addressed table of 32-bit slots that grows when half full. For verbatim comparisons, check the length before comparing bytes, and compare eight bytes at a time. The table uses the high bits of the line hash, so mix the existing hash before lookup. The mix is bijective and does not change which lines are considered equal. Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
Hashing a line one byte at a time is a large part of record preparation for long files. When no whitespace option is in effect, hash each line by mixing eight-byte words and find the newline with a word-at-a-time check on little-endian GCC-compatible compilers, with a byte loop elsewhere. Equal byte sequences always hash equally, and a matching hash is still followed by a full comparison. Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
xdl_cleanup_records() allocates two zeroed action arrays and checks for an impossible state on every record. Allocate both arrays at once without clearing them, since every slot is written before it is read, and only run the "investigate" pass on a side that has such records. Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
The second file's record count is guessed from a small sample, so large files often grow the record array several times. Estimate it from the first file's line density when the two sizes are similar, refine it once the first file has been split, and when the array must grow, extrapolate from the bytes consumed so far. Size the class table from the first file's records. Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
If one input is empty, no line can match, yet both sides are still hashed and classified. Only split the non-empty side into records and mark every record as changed. Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
Records that are byte-identical at the start and end of both files are hashed and classified twice. When preparing the second file, find the common prefix and suffix with block comparisons and give those records the class already assigned for the first file, keeping the occurrence counts that cleanup relies on. Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
For a verbatim Myers diff, common leading and trailing lines are trimmed before the diff and never compared, but they are still classified. Split the common head and tail from the middle first and classify only the middle. Use a fingerprint filter to find common lines that might also appear in the middle; only those are classified, the rest get private class numbers. Fingerprints are only a negative filter: a possible match still goes through the normal hash and full comparison. Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
Every comparison in xdl_split() and xdl_recs_cmp() looks up a line's class through reference_index and then the record array. Copy the class of each line taking part in the search into a flat array once, and compare those arrays instead. The search, its recursion and its linear memory use are unchanged; the copies add one array per side. Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
After splitting a box, the second recursive call is the last thing xdl_recs_cmp() does. Handle the first half recursively as before, then continue with the second half in a loop instead of another recursive call. This is the same divide-and-conquer search in the same order; it only avoids one stack frame per split. Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
xdl_build_script() walks every line from the end, even long runs that are unchanged on both sides. When bool is one byte, check eight entries of changed[] at a time and skip a word that is all zero on both sides. Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
When a file has long unchanged stretches, xdl_change_compact() advances through empty groups one group_next() call at a time. Jump straight to the next changed line, and move the other file's group to the position the repeated group_next() calls would reach, counting unchanged lines eight at a time. Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
cbf48d2 to
4af06e0
Compare
|
/preview |
|
Preview email sent as pull.2311.git.1794621455.gitgitgadget@gmail.com |
This series reduces preparation, traversal and output overhead in
Myers-based diffs. Output and interfaces are unchanged.
The work is split into 13 small commits so each one can be reviewed
on its own. The Myers algorithm itself is unchanged; the commits only
change how its input is prepared and how its result is walked and
emitted.
Changes:
later commit is checked against them.
boundaries and missing-newline markers.
class table, hashing verbatim lines eight bytes at a time,
simpler cleanup_records() loops and better allocation estimates.
side is empty, and for common leading and trailing lines.
arrays, and turn the recursion on the second half of
xdl_recs_cmp() into a loop.
building the edit script and in xdl_change_compact().
Patience, histogram and merge algorithms are not changed.
Performance:
Myers
diff-tree --shortstatover the 8,192 most recent non-mergecommits reachable from v2.56.0 went from 5.926 s to 4.663 s median
wall time, a 1.275x speedup (95% CI 1.269-1.281x). This used 20
order-balanced pairs, warm caches and one pinned CPU on an Intel Xeon
Platinum 8581C, with both builds at GCC 15.2 -O3 -g against base
1891de4. Output was byte-identical on every run.
To reproduce, run with each binary in the same repository:
Testing:
the t40* diff tests, so the series is bisectable.
the native and portable hashing paths, matched the baseline.
Note: I used AI tools to understand the bottleneck, for research and
for code generation. I carefully reviewed, verified and ran the code
myself.