Repository navigation
Conversation
spkrka
force-pushed
the
fast-import-big-file-threshold
branch
from
November 2, 2026 14:52
f4e187d to
3d54673
Compare
Since 7835ee7 (environment: move access to "core.bigFileThreshold" into repo settings, 2025-03-10), "git fast-import --big-file-threshold" stores its value with repo_settings_set_big_file_threshold(). That writes into the_repository->settings directly, without calling prepare_repo_settings() first. The first call to prepare_repo_settings() calls repo_settings_clear() to reset the whole struct. In fast-import, that first call happens when the pack code first accesses a pack, which is after the first blob has been written. From then on the threshold has reverted to the configured default (512 MiB), so later blobs above the requested threshold are read fully into memory and may be stored as deltas, instead of being streamed into the pack. Initialize the settings in the setter before storing the value, so that a later prepare_repo_settings() is a no-op and keeps it. The only precondition of prepare_repo_settings() is that the repository has a gitdir, and both callers of the setter (the command-line option and the "option" stream command) run in fast-import, which requires a repository. repo_settings_set_shared_repository() has the same ordering dependency, but its only callers either run during repository creation, where the value has already been written to the config by the time the settings are cleared, or restore the old value before any object access. Leave it alone for now: "git init" calls it before the configuration of the new repository has been written, so calling prepare_repo_settings() there would cache settings from an incomplete configuration. Add a test that imports three similar blobs above the threshold and checks that none of them was stored as a delta. Without this fix, only the first blob is streamed: the settings are first initialized while it is being written, which resets the threshold, so the second blob is stored whole and the third is stored as a delta against it. Signed-off-by: Kristofer Karlsson <krka@spotify.com>
spkrka
force-pushed
the
fast-import-big-file-threshold
branch
from
November 17, 2026 08:45
3d54673 to
ec709a1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
"git fast-import --big-file-threshold" stops taking effect after the
first blob: the value is stored in the repo settings before they are
initialized, and the first prepare_repo_settings() call then resets
it. Later blobs above the threshold are read fully into memory and
may be stored as deltas instead of being streamed into the pack.
This patch is deliberately a narrow fix. The underlying problem is
the lifecycle of the repo settings: prepare_repo_settings() starts by
clearing the whole struct, so any setter that runs before the first
prepare silently loses its value, and nothing catches that. Fields
are also initialized in different ways: some in prepare, some lazily
on first access with their own sentinels, and some through setters
like this one.
repo_settings_set_shared_repository() has the same ordering
dependency. I left it alone, because its callers either run during
repository creation, where the value has been written to the config by
the time the settings are cleared, or restore the old value before
any object access. The same fix would not work there either: "git
init" calls it before the configuration of the new repository has
been written, so preparing the settings at that point would cache
settings from an incomplete configuration.
A more complete fix would make the ordering explicit, for example by
preparing the settings as part of repository setup and making early
access a BUG(), or by keeping values from the command line apart from
the ones read from the config, so that preparing again does not drop
them. I have experimented a bit with both, but I think that is better
discussed as a separate topic, and I did not want to hold this fix
back for it.
I found this while profiling t5608, which is one of the slowest tests
in CI. The t5608 changes themselves have since landed as 89745e1
(t5608: assert that the packs involved really are that large,
2026-11-06).
cc: Patrick Steinhardt ps@pks.im