git-refit

Overhaul a git repository's history — purge unwanted files, strip Jupyter notebook outputs, move binaries to Git LFS — and verify the result. Like refitting a ship: same vessel, better condition.

git-refit REWRITES HISTORY: every commit SHA changes and collaborators must re-clone. It only ever operates on a fresh --mirror clone (the source is never modified) and never pushes anything.

Install

python3 -m venv .venv && .venv/bin/pip install .

Requires git-lfs on PATH. git >= 2.40 is recommended (diagnose degrades gracefully below that).

Usage

Write a config (see examples/migration.jsonc; your editor validates it via the $schema line against schema/git-refit.schema.json):

git-refit run migration.jsonc

This produces <output>/ (the rewritten mirror) and <output>.artifacts/:

artifact	contents
commit-map	every original commit SHA -> final SHA (all-zeros = deleted)
pruned-refs.bundle	pruned refs (e.g. Tuleap refs/tlpr) with their history
pruned-refs.json	which refs were bundled; refs bundles can't hold
report.json	sizes, stats, verification findings

Keep the artifacts: the commit-map is the only record linking old SHAs (in issues, PRs, docs) to the rewritten history. To restore pruned refs from the bundle into a repo:

git fetch pruned-refs.bundle '+refs/tlpr/*:refs/tlpr/*'

Other subcommands: publish (robust push to a remote), analyze (largest blobs), verify (re-run the checks), diagnose (why is this blob still raw?), force-convert (repair stragglers), schema (emit the config JSON Schema). All take -h/--help.

Shell completion (bash >= 4.4):

git-refit install-bash-completion

writes the completion script where the bash-completion framework auto-loads it (or falls back to a source line in ~/.bashrc); idempotent, takes effect in new shells. For zsh/fish, use click's built-in mechanism: eval "$(_GIT_REFIT_COMPLETE=zsh_source git-refit)" (or fish_source).

Gotchas

• LFS patterns are CASE-SENSITIVE: *.png does not match photo.PNG.
• A bare extension like .gz matches nothing; the config validator rejects it.
• notebooks.extra_keys must match your commit-time nbstripout filter, or cleaned history won't be byte-identical to freshly-committed notebooks.
• Refs outside refs/heads and refs/tags are pruned by default because git lfs migrate --everything does not traverse them (they'd keep raw blobs alive). They're archived to the bundle first. refs/replace grafts are instead baked into the rewritten history.

Publishing the result

git -C <output> remote add newremote <url>
git-refit publish <output> newremote

publish pushes with the sequence that survives fragile GitLab installations (proxy response timeouts, tight Gitaly storage, LFS pre-push hook stalls). By default it is non-destructive — create and fast-forward only, for a fresh empty remote. Overwriting a remote that already has history needs --force, which also enables the final stale-ref deletion pass:

1. git lfs push --all FIRST — no one can clone pointers whose content isn't uploaded, and the big bytes go to LFS storage before any large git transaction starts.
2. Branches, then tags, pushed in batches (--batch-size, default 30) so each server-side transaction stays small; a failed batch is retried ref-by-ref to isolate rejected refs.
3. With --force only: local refs/remotes/* tracking refs are deleted, then a final git push --mirror removes stale remote refs (--mirror inherently forces, so this pass never runs without --force).
4. The remote's heads/tags are verified against the local mirror.

All pushes use --no-verify: the git-lfs pre-push hook installed by git lfs migrate silently re-scans the entire history on every push (it looks like a hang on large repos), and step 1 already uploaded everything. On GitLab, unprotect branches/tags before publishing over an existing repo, and run housekeeping afterwards to reclaim space.

Troubleshooting large pushes over HTTPS: if the pack uploads to 100% and the push then dies with curl 18 transfer closed with outstanding read data remaining, a proxy/load balancer killed the connection while GitLab was silently processing the received pack (this scales with object and ref count, not upload speed — http.postBuffer will not help). First run git ls-remote — the push often completed server-side anyway. Otherwise push over SSH (no HTTP proxy timeout), or split the transaction: push master first, then refs/heads/*, then refs/tags/*, then a final --mirror to delete stale remote refs.

unable to migrate objects to permanent storage is server-side (Gitaly failed to move the quarantined pack into the repo — usually disk space or inodes on the storage node; check the gitaly log). Batching the push into smaller transactions (git for-each-ref --format='%(refname)' refs/heads | xargs -n 30 git push -f --no-verify <remote>) often gets through.

deny updating a hidden ref on the final --mirror push: successful partial pushes created remote-tracking refs (refs/remotes/<remote>/*) in the mirror, and --mirror then tries to push them. Delete them first: git for-each-ref --format='%(refname)' refs/remotes | xargs -r -n 1 git update-ref -d, then rerun the mirror push.

Development

.venv/bin/pip install -e '.[dev]'
.venv/bin/pytest              # tests
.venv/bin/ruff check .        # lint
.venv/bin/ruff format .       # format

CI (.gitlab-ci.yml) runs lint + format check + the test suite. See CLAUDE.md for the project's design rules.