connected: add incremental connectivity check - #2211
Open
spkrka wants to merge 4 commits into
Open
Conversation
Move the inline self-contained pack detection into a helper function. This makes check_connected() easier to follow and makes the detection logic available as a standalone helper. No functional change. Signed-off-by: Kristofer Karlsson <krka@spotify.com>
spkrka
marked this pull request as ready for review
August 28, 2026 10:15
spkrka
force-pushed
the
tree-diff-connectivity-v1-clean
branch
from
August 28, 2026 11:41
1f543de to
fd76802
Compare
The connectivity check walks objects reachable from incoming tips
using rev-list. mark_edges_uninteresting() prunes at commit
boundaries, but the traversal still visits the full tree closure
of each boundary commit to classify every entry. For repositories
with large trees, this is expensive.
Add an alternative that verifies incoming commits by diffing their
trees against parent trees, recursively descending only into entries
whose OIDs have changed. Use the Merkle tree property -- an unchanged
tree entry proves the entire subtree is intact -- to skip subtrees
that already exist in the repository. Cross-directory moves may be
reverified rather than recognized by the parent-tree comparison, but
correctness is preserved.
The algorithm works in three phases:
1. Collect and peel tips: consume the tip iterator, peel tags, and
verify non-commit objects immediately. Tips found in a
self-contained pack (verified by index-pack) are skipped.
2. Find boundary: feed commit tips to rev-list --stdin --not --all
to identify new commits not yet reachable from local refs.
3. Verify trees: walk new commits in topological order. For each
commit, diff its tree against parent trees. Unchanged entries
are skipped (Merkle property). The verified set persists across
commits so subtrees seen in earlier commits (change-then-revert,
subtree moves, merges) are not re-walked.
Gate the new algorithm behind transfer.connectivityCheck=incremental.
Fall back to rev-list for shallow fetches, partial clones, replacement
objects, and deepening fetches.
Signed-off-by: Kristofer Karlsson <krka@spotify.com>
Teach the incremental connectivity check to handle shallow fetches. Shallow commits are treated as traversal roots with no parents, matching the boundary semantics of rev-list. Add parse_shallow_file_gently() to parse the temporary shallow file without dying on errors, and thread the resulting oidset through verify_new_commits and verify_commit_tree. Pass --shallow-file to the boundary-finding rev-list so it respects the shallow grafts. Remove the shallow_file guard from incremental_check_applicable() so incremental mode is now used for shallow fetches when configured. Signed-off-by: Kristofer Karlsson <krka@spotify.com>
Teach the incremental connectivity check to handle partial clones where some objects are promised by a promisor remote but not present locally. When a tree or blob cannot be read, check whether it is a promisor object before reporting an error. Promisor objects are trusted and added to the verified set without fetching them. During tag peeling, a missing object that is a promisor object causes the tip to be silently skipped rather than treated as an error. Pass --exclude-promisor-objects to the boundary-finding rev-list so promisor commits do not pollute the set of commits to verify. Remove the repo_has_promisor_remote() guard from incremental_check_applicable() so incremental mode is now used for partial clones when configured. Signed-off-by: Kristofer Karlsson <krka@spotify.com>
spkrka
force-pushed
the
tree-diff-connectivity-v1-clean
branch
from
August 29, 2026 14:57
fd76802 to
68dbb9c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This series implements the tree verification optimization described in
my recent RFC [1], gated behind transfer.connectivityCheck=incremental.
The RFC received no replies; the patches here are hopefully a more
concrete way to evaluate the approach.
The current connectivity check uses a rev-list subprocess to perform
the object traversal. On repositories with large active trees this
can become expensive even for small fetches, because the tree/blob
closure at the connectivity boundary may be much larger than the
incoming change.
This series addresses the object-walk cost (step 2 from the RFC) by
introducing an incremental tree-diff approach. Incoming trees are
verified against their parents, recursively descending only into
entries whose OIDs have changed. Parent trees still need to be
scanned as comparison bases. The boundary search (step 1) continues
to use rev-list; optimizing that is a natural follow-up.
The approach uses the Merkle tree property: when a tree entry has the
same OID in both a new commit's tree and a trusted parent's tree, the
entire subtree is already verified. Only differing entries require
recursive verification. The verified set persists across commits, so
subtrees that have already been established as trusted can also be
reused across changes, reverts, moves, and merges.
Benchmarks on a large monorepo (~3M commits, 221K trees and 503K
blobs reachable from the tip). Scenario: 1 new commit, 1 file
changed, measured with hyperfine:
With ~10K local refs:
With 1 local ref:
In this benchmark the rev-list approach visits all 724K tree and blob
objects while classifying the connectivity boundary. The incremental
verifier walks one new tree and checks one blob; the corresponding
parent tree is scanned as the comparison base. Tree verification took
5 ms in both measurements.
With many local refs, the boundary search dominates the incremental
timings. Eliminating that subprocess and performing boundary discovery
in-process is the target for a follow-up.
On linux.git (~1.5M commits, 6.2K trees and 95K blobs reachable
from the tip, ~940 refs):
On git.git (~82K commits, 224 trees and 4851 blobs reachable
from the tip, ~7K refs):
The benefit scales with tree closure size. linux.git's 101K
tree+blob objects produce a clear 2.3x win; git.git's 5K objects
are too small for the difference to be measurable.
The implementation falls back to the rev-list path for deepening
fetches and repositories with active replacement objects.
One question around replacements: the existing rev-list connectivity
path follows replacement refs, while git prune explicitly disables
replacement refs before performing its reachability traversal. The
connectivity check ensures that refs do not point into incomplete
object graphs, while pruning ultimately operates on the underlying
object graph. It may therefore be worth discussing which replacement
semantics are intended here. I have left the existing behavior
unchanged in this series.
The series is structured as four commits:
connected: extract get_self_contained_pack() helper
Pure refactor: lifts the index-pack self-contained pack
optimization into a shared helper, used by both paths.
connected: add incremental connectivity check
Core implementation: config plumbing, three-phase algorithm
(collect tips, find boundary, verify new commits via
tree-diff), and tests.
connected: handle shallow fetches in incremental check
Adds shallow boundary support: commits listed in the
temporary shallow file are treated as roots with no parents,
receiving full closure verification.
connected: handle partial clones in incremental check
Adds promisor-remote support: missing objects that are
promisor objects are accepted, matching the existing
--exclude-promisor-objects semantics.
[1] https://lore.kernel.org/git/CAL71e4Nf=-zCrfN7ghEVGq11irajJhtdxYZgKe0Ycux0qs1ZvQ@mail.gmail.com/