Skip to content

Batch Git object queries and validate ref names in Rust - #30

Merged
hmirin merged 2 commits into
mainfrom
perf/batched-git-queries
Oct 29, 2026
Merged

hmirin merged 2 commits into
mainfrom
perf/batched-git-queries

Conversation

@hmirin

@hmirin hmirin commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Step 5b of the refactoring plan (Opus 3.1; Astra Category 3 medium on tag parsing). It improves performance and the robustness of Git queries. Behaviour is unchanged except where noted below.

Measured (local only: aurum-work, Apple M1 Max)

Repository: 50 branches and 1,000 annotated tags. Both builds ran against copies of the same setup. Process counts come from GIT_TRACE.

before after
fetch (list + fetch, fresh clone) 8,260 processes, 440.7 s 1,061 processes, 79.5 s
push one ref 13,365 processes, 243.8 s 15 processes, 3.8 s

Before, almost every spawn was a cat-file (4,051 on fetch, 9,152 on push) or a check-ref-format (3,151 and 4,201). What is left in fetch is 1,050 index-pack runs, one per generation's pack. That comes from creating one generation per pushed ref, which is step 6. To reproduce: scripts/benchmark-git-processes.sh SETUP_BIN MEASURE_BIN.... Setup takes about 16 minutes.

Change

  • Ref names are checked in Rust, once. src/refname.rs implements the rules in git-check-ref-format(1). ref_name_rules_match_git_check_ref_format compares 55 accept and reject cases against git check-ref-format, and all agree. Manifest refs are validated once per accepted head, in observe, import and preflight. Later passes only classify the namespace. The carrier-branch check uses the same function.
  • Object queries are batched and typed. CatFile moves to src/catfile.rs and returns a typed object type. git::Objects keeps at most one cat-file --batch-check and one --batch process per operation. A git failure is an error, never "missing". This fixes resolve_tag_chain, which treated a failed lookup as a missing object.
  • Tags are parsed as bytes. Annotated tag headers are parsed as bytes, bounded at 64 KiB. The message body is skipped without decoding, so a tag with a non-UTF-8 or very large message can be inspected.

Tests

All three fail on the base branch:

  • git::tests::tags_with_non_utf8_or_large_messages_are_inspectable (a 16 MiB message, and a nested tag)
  • git::tests::tag_headers_beyond_the_bound_are_rejected
  • git::tests::a_failing_git_is_an_error_and_not_a_missing_object

cargo fmt --check, cargo clippy --all-targets -- -D warnings and cargo test --all-targets pass (193 tests). bash -n passes on the new script. ~/.local/state/git-e2ee and ~/.config/git-e2ee did not exist before or after.

Small behaviour changes

  • An annotated tag whose object header is not 40 or 64 hex digits is rejected. Before, it was passed to cat-file -t as revision syntax.
  • If cat-file answers "ambiguous", that is now an error. Carrier IDs are full hashes, so this cannot happen in practice.
  • Ref names in the checkout's verified_refs are only classified by namespace. They are never passed to git.

Open

  • --batch-check reports a corrupt loose object as missing, so corruption still looks like absence. Telling them apart would need --batch-command or fsck.
  • cat-file stderr is discarded, as before.

Every list, fetch and push spawned git check-ref-format for each ref and
git cat-file for each ref tip, verified ref and tag in a chain, often twice.
With 50 branches and 1,000 annotated tags, one fetch started 8,260 processes
and one single-ref push 13,365.

- src/refname.rs implements git-check-ref-format(1) and is cross-checked
  against git on 55 cases. Manifest refs are validated once per accepted
  head instead of on every operation.
- src/catfile.rs (moved from storage.rs) returns typed object types, and
  git::Objects keeps at most one cat-file --batch-check and one --batch
  process per operation. A git failure is an error, never "missing";
  resolve_tag_chain no longer treats a failed lookup as a missing object.
- Annotated tags are read through cat-file --batch with their headers
  parsed as bytes under a 64 KiB bound and the message skipped undecoded,
  so non-UTF-8 and very large messages are inspectable.

Measured locally (Apple M1 Max): fetch 8,260 -> 1,061 processes and
440.7 s -> 79.5 s; one-ref push 13,365 -> 15 processes and 243.8 s -> 3.8 s.
scripts/benchmark-git-processes.sh reproduces the measurement.
@hmirin
hmirin merged commit 522eb15 into main Oct 29, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant