Skip to content

xdiff: reduce overhead in Myers diffs - #2311

Open
sauravjoshi23 wants to merge 13 commits into
gitgitgadget:masterfrom
sauravjoshi23:perf/myers-history-diff
Open

sauravjoshi23 wants to merge 13 commits into
gitgitgadget:masterfrom
sauravjoshi23:perf/myers-history-diff

Conversation

@sauravjoshi23

@sauravjoshi23 sauravjoshi23 commented Nov 12, 2026 •

Copy link
Copy Markdown

This series reduces preparation, traversal and output overhead in
Myers-based diffs. Output and interfaces are unchanged.

The work is split into 13 small commits so each one can be reviewed
on its own. The Myers algorithm itself is unchanged; the commits only
change how its input is prepared and how its result is walked and
emitted.

Changes:

  • Patch 1 adds regression and performance tests first, so every
    later commit is checked against them.
  • Patch 2 batches diff output callbacks while keeping hunk
    boundaries and missing-newline markers.
  • Patches 3-6 make line classification cheaper: an open-addressed
    class table, hashing verbatim lines eight bytes at a time,
    simpler cleanup_records() loops and better allocation estimates.
  • Patches 7-9 avoid classifying lines that do not need it: when one
    side is empty, and for common leading and trailing lines.
  • Patches 10-11 let the Myers search compare contiguous class
    arrays, and turn the recursion on the second half of
    xdl_recs_cmp() into a loop.
  • Patches 12-13 skip unchanged lines and empty groups faster when
    building the edit script and in xdl_change_compact().

Patience, histogram and merge algorithms are not changed.

Performance:

Myers diff-tree --shortstat over the 8,192 most recent non-merge
commits reachable from v2.56.0 went from 5.926 s to 4.663 s median
wall time, a 1.275x speedup (95% CI 1.269-1.281x). This used 20
order-balanced pairs, warm caches and one pinned CPU on an Intel Xeon
Platinum 8581C, with both builds at GCC 15.2 -O3 -g against base
1891de4. Output was byte-identical on every run.

To reproduce, run with each binary in the same repository:

git rev-list --no-merges --max-count=8192 v2.56.0 >commits
git diff-tree --stdin -r --root --no-renames --no-ext-diff \
    --no-textconv --no-color --diff-algorithm=myers --shortstat \
    <commits >/dev/null

Testing:

  • The full test suite passes on the final commit.
  • Every commit in the series builds with -Wall -Werror and passes
    the t40* diff tests, so the series is bisectable.
  • 3,996 command-line comparisons matched the unmodified baseline.
  • ASan/UBSan runs over 344 synthetic inputs in 12 modes, using both
    the native and portable hashing paths, matched the baseline.
  • The focused regression tests also pass with SHA-256 repositories.

Note: I used AI tools to understand the bottleneck, for research and
for code generation. I carefully reviewed, verified and ran the code
myself.

@gitgitgadget

gitgitgadget Bot commented Nov 12, 2026

Copy link
Copy Markdown

Welcome to GitGitGadget

Hi @sauravjoshi23, and welcome to GitGitGadget, the GitHub App to send patch series to the Git mailing list from GitHub Pull Requests.

Please make sure that either:

  • Your Pull Request has a good description, if it consists of multiple commits, as it will be used as cover letter.
  • Your Pull Request description is empty, if it consists of a single commit, as the commit message should be descriptive enough by itself.

You can CC potential reviewers by adding a footer to the PR description with the following syntax:

CC: Revi Ewer <revi.ewer@example.com>, Ill Takalook <ill.takalook@example.net>

NOTE: DO NOT copy/paste your CC list from a previous GGG PR's description,
because it will result in a malformed CC list on the mailing list. See
example.

Also, it is a good idea to review the commit messages one last time, as the Git project expects them in a quite specific form:

  • the lines should not exceed 76 columns,
  • the first line should be like a header and typically start with a prefix like "tests:" or "revisions:" to state which subsystem the change is about, and
  • the commit messages' body should be describing the "why?" of the change.
  • Finally, the commit messages should end in a Signed-off-by: line matching the commits' author.

It is in general a good idea to await the automated test ("Checks") in this Pull Request before contributing the patches, e.g. to avoid trivial issues such as unportable code.

Contributing the patches

Before you can contribute the patches, your GitHub username needs to be added to the list of permitted users. Any already-permitted user can do that, by adding a comment to your PR of the form /allow. A good way to find other contributors is to locate recent pull requests where someone has been /allowed:

Both the person who commented /allow and the PR author are able to /allow you.

An alternative is the channel #git-devel on the Libera Chat IRC network:

<newcontributor> I've just created my first PR, could someone please /allow me? https://github.com/gitgitgadget/git/pull/12345
<veteran> newcontributor: it is done
<newcontributor> thanks!

Once on the list of permitted usernames, you can contribute the patches to the Git mailing list by adding a PR comment /submit.

If you want to see what email(s) would be sent for a /submit request, add a PR comment /preview to have the email(s) sent to you. You must have a public GitHub email address for this. Note that any reviewers CC'd via the list in the PR description will not actually be sent emails.

After you submit, GitGitGadget will respond with another comment that contains the link to the cover letter mail in the Git mailing list archive. Please make sure to monitor the discussion in that thread and to address comments and suggestions (while the comments and suggestions will be mirrored into the PR by GitGitGadget, you will still want to reply via mail).

If you do not want to subscribe to the Git mailing list just to be able to respond to a mail, you can download the mbox from the Git mailing list archive (click the (raw) link), then import it into your mail program. If you use GMail, you can do this via:

curl -g --user "<EMailAddress>:<Password>" \
    --url "imaps://imap.gmail.com/INBOX" -T /path/to/raw.txt

To iterate on your change, i.e. send a revised patch or patch series, you will first want to (force-)push to the same branch. You probably also want to modify your Pull Request description (or title). It is a good idea to summarize the revision by adding something like this to the cover letter (read: by editing the first comment on the PR, i.e. the PR description):

Changes since v1:
- Fixed a typo in the commit message (found by ...)
- Added a code comment to ... as suggested by ...
...

To send a new iteration, just add another PR comment with the contents: /submit.

Need help?

New contributors who want advice are encouraged to join git-mentoring@googlegroups.com, where volunteers who regularly contribute to Git are willing to answer newbie questions, give advice, or otherwise provide mentoring to interested contributors. You must join in order to post or view messages, but anyone can join.

You may also be able to find help in real time in the developer IRC channel, #git-devel on Libera Chat. Remember that IRC does not support offline messaging, so if you send someone a private message and log out, they cannot respond to you. The scrollback of #git-devel is archived, though.

@dscho

dscho commented Nov 12, 2026

Copy link
Copy Markdown
Member

/allow

@gitgitgadget

gitgitgadget Bot commented Nov 12, 2026

Copy link
Copy Markdown

User sauravjoshi23 is now allowed to use GitGitGadget.

Cover unchanged-span boundaries, common records that also appear in the
middle, and equal fingerprints with different line contents.

Add a Myers shortstat performance case over 8,192 non-merge commits
from the perf test repository.

Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
Emitting records individually adds callback and buffer-walking overhead.
Buffer complete records and flush before hunk headers and large records.

Coalesce small prefixes, record contents and optional missing-newline
markers into a single callback buffer. Preserve callback failures and
the existing large-record fallback.

Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
Each new line class is allocated from a chastore and chained into a
hash bucket, and every lookup walks that chain through pointers.

Keep the classes in one contiguous array and index them from a compact
open-addressed table of 32-bit slots that grows when half full. For
verbatim comparisons, check the length before comparing bytes, and
compare eight bytes at a time.

The table uses the high bits of the line hash, so mix the existing
hash before lookup. The mix is bijective and does not change which
lines are considered equal.

Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
Hashing a line one byte at a time is a large part of record
preparation for long files.

When no whitespace option is in effect, hash each line by mixing
eight-byte words and find the newline with a word-at-a-time check on
little-endian GCC-compatible compilers, with a byte loop elsewhere.
Equal byte sequences always hash equally, and a matching hash is
still followed by a full comparison.

Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
xdl_cleanup_records() allocates two zeroed action arrays and checks
for an impossible state on every record.

Allocate both arrays at once without clearing them, since every slot
is written before it is read, and only run the "investigate" pass on
a side that has such records.

Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
The second file's record count is guessed from a small sample, so
large files often grow the record array several times.

Estimate it from the first file's line density when the two sizes
are similar, refine it once the first file has been split, and when
the array must grow, extrapolate from the bytes consumed so far.
Size the class table from the first file's records.

Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
If one input is empty, no line can match, yet both sides are still
hashed and classified.

Only split the non-empty side into records and mark every record as
changed.

Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
Records that are byte-identical at the start and end of both files are
hashed and classified twice.

When preparing the second file, find the common prefix and suffix
with block comparisons and give those records the class already
assigned for the first file, keeping the occurrence counts that
cleanup relies on.

Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
For a verbatim Myers diff, common leading and trailing lines are
trimmed before the diff and never compared, but they are still
classified.

Split the common head and tail from the middle first and classify
only the middle. Use a fingerprint filter to find common lines that
might also appear in the middle; only those are classified, the rest
get private class numbers. Fingerprints are only a negative filter:
a possible match still goes through the normal hash and full
comparison.

Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
Every comparison in xdl_split() and xdl_recs_cmp() looks up a line's
class through reference_index and then the record array.

Copy the class of each line taking part in the search into a flat
array once, and compare those arrays instead. The search, its
recursion and its linear memory use are unchanged; the copies add
one array per side.

Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
After splitting a box, the second recursive call is the last thing
xdl_recs_cmp() does.

Handle the first half recursively as before, then continue with the
second half in a loop instead of another recursive call. This is the
same divide-and-conquer search in the same order; it only avoids one
stack frame per split.

Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
xdl_build_script() walks every line from the end, even long runs
that are unchanged on both sides.

When bool is one byte, check eight entries of changed[] at a time and
skip a word that is all zero on both sides.

Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
When a file has long unchanged stretches, xdl_change_compact()
advances through empty groups one group_next() call at a time.

Jump straight to the next changed line, and move the other file's
group to the position the repeated group_next() calls would reach,
counting unchanged lines eight at a time.

Signed-off-by: Saurav Joshi <sauravjoshi2362000@gmail.com>
@sauravjoshi23
sauravjoshi23 force-pushed the perf/myers-history-diff branch from cbf48d2 to 4af06e0 Compare November 13, 2026 02:30
@sauravjoshi23

Copy link
Copy Markdown
Author

/preview

@gitgitgadget

gitgitgadget Bot commented Nov 14, 2026

Copy link
Copy Markdown

Preview email sent as pull.2311.git.1794621455.gitgitgadget@gmail.com

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants