Repository navigation
Batch Git object queries and validate ref names in Rust - #30
Merged
Merged
Conversation
Every list, fetch and push spawned git check-ref-format for each ref and git cat-file for each ref tip, verified ref and tag in a chain, often twice. With 50 branches and 1,000 annotated tags, one fetch started 8,260 processes and one single-ref push 13,365. - src/refname.rs implements git-check-ref-format(1) and is cross-checked against git on 55 cases. Manifest refs are validated once per accepted head instead of on every operation. - src/catfile.rs (moved from storage.rs) returns typed object types, and git::Objects keeps at most one cat-file --batch-check and one --batch process per operation. A git failure is an error, never "missing"; resolve_tag_chain no longer treats a failed lookup as a missing object. - Annotated tags are read through cat-file --batch with their headers parsed as bytes under a 64 KiB bound and the message skipped undecoded, so non-UTF-8 and very large messages are inspectable. Measured locally (Apple M1 Max): fetch 8,260 -> 1,061 processes and 440.7 s -> 79.5 s; one-ref push 13,365 -> 15 processes and 243.8 s -> 3.8 s. scripts/benchmark-git-processes.sh reproduces the measurement.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 5b of the refactoring plan (Opus 3.1; Astra Category 3 medium on tag parsing). It improves performance and the robustness of Git queries. Behaviour is unchanged except where noted below.
Measured (local only: aurum-work, Apple M1 Max)
Repository: 50 branches and 1,000 annotated tags. Both builds ran against copies of the same setup. Process counts come from
GIT_TRACE.Before, almost every spawn was a
cat-file(4,051 on fetch, 9,152 on push) or acheck-ref-format(3,151 and 4,201). What is left in fetch is 1,050index-packruns, one per generation's pack. That comes from creating one generation per pushed ref, which is step 6. To reproduce:scripts/benchmark-git-processes.sh SETUP_BIN MEASURE_BIN.... Setup takes about 16 minutes.Change
src/refname.rsimplements the rules in git-check-ref-format(1).ref_name_rules_match_git_check_ref_formatcompares 55 accept and reject cases againstgit check-ref-format, and all agree. Manifest refs are validated once per accepted head, in observe, import and preflight. Later passes only classify the namespace. The carrier-branch check uses the same function.CatFilemoves tosrc/catfile.rsand returns a typed object type.git::Objectskeeps at most onecat-file --batch-checkand one--batchprocess per operation. A git failure is an error, never "missing". This fixesresolve_tag_chain, which treated a failed lookup as a missing object.Tests
All three fail on the base branch:
git::tests::tags_with_non_utf8_or_large_messages_are_inspectable(a 16 MiB message, and a nested tag)git::tests::tag_headers_beyond_the_bound_are_rejectedgit::tests::a_failing_git_is_an_error_and_not_a_missing_objectcargo fmt --check,cargo clippy --all-targets -- -D warningsandcargo test --all-targetspass (193 tests).bash -npasses on the new script.~/.local/state/git-e2eeand~/.config/git-e2eedid not exist before or after.Small behaviour changes
objectheader is not 40 or 64 hex digits is rejected. Before, it was passed tocat-file -tas revision syntax.cat-fileanswers "ambiguous", that is now an error. Carrier IDs are full hashes, so this cannot happen in practice.verified_refsare only classified by namespace. They are never passed to git.Open
--batch-checkreports a corrupt loose object asmissing, so corruption still looks like absence. Telling them apart would need--batch-commandor fsck.cat-filestderr is discarded, as before.