Skip to content

repack: write filtered-out objects as loose objects - #2314

Open
achakilam-nv wants to merge 2 commits into
gitgitgadget:masterfrom
achakilam-nv:filter-to-loose
Open

achakilam-nv wants to merge 2 commits into
gitgitgadget:masterfrom
achakilam-nv:filter-to-loose

Conversation

@achakilam-nv

Copy link
Copy Markdown

Some key-value object stores manage and retrieve Git objects
individually. The existing --filter-to option writes filtered-out
objects into packs, so using it with such stores requires unpacking
the resulting packs before importing the objects.

This series adds --filter-to-loose=dir to git repack, allowing
filtered-out objects to be written directly as loose objects into
another object directory.

With --filter-to-loose, repack streams the filtered pack into
unpack-objects, avoiding that extra step and the intermediate pack
file. For example:

git repack -a --filter=blob=1m --filter-to-loose=/path/to/objects

The destination is treated as a Git object directory.

The first patch adds the option for objects already stored in packs.
The second extends it to include loose objects excluded by the
filter, so the destination also receives those objects. Objects
already present in the destination are not rewritten. With -d,
the source copies of those loose objects are removed after all
objects have been written successfully.

cc: Martin Fick mfick@nvidia.com
cc: Christian Couder christian.couder@gmail.com

…objects

Some external object stores (like key-value) manage and retrieve git
objects individually. Although repack's --filter-to option can separate
objects from the main object store, it writes them into packs. Using
those objects individually then requires an additional unpacking step.

Add --filter-to-loose=<dir> to write filtered-out objects directly as
loose objects into <dir>, treating it as an object directory. Stream
the filtered pack into unpack-objects instead of writing an intermediate
pack file to disk.

The filtered pack is built with --stdin-packs, so this initially handles
only objects already stored in packs. Loose objects in the source
repository are left in place.

Signed-off-by: Adithya Chakilam <achakilam@nvidia.com>
--filter-to-loose only writes objects that were in an existing pack,
because the filtered pack is built with --stdin-packs. Loose objects
that the filter excludes stay behind in the source repository, so the
destination does not get the complete set of filtered-out objects.

Copy such loose objects to the destination as well. An object already
present in the destination is not rewritten. With -d, the source copies
are removed once everything is written.

Signed-off-by: Adithya Chakilam <achakilam@nvidia.com>
@achakilam-nv

Copy link
Copy Markdown
Author

/submit

@gitgitgadget

gitgitgadget Bot commented Nov 13, 2026

Copy link
Copy Markdown

Submitted as pull.2314.git.1794607911.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git/ pr-2314/achakilam-nv/filter-to-loose-v1

To fetch this version to local tag pr-2314/achakilam-nv/filter-to-loose-v1:

git fetch --no-tags https://github.com/gitgitgadget/git/ tag pr-2314/achakilam-nv/filter-to-loose-v1

@@ -12,7 +12,7 @@ SYNOPSIS
'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Junio C Hamano wrote on the Git mailing list (how to reply to this email):

"Adithya Chakilam via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Adithya Chakilam <achakilam@nvidia.com>
>
> Some external object stores (like key-value) manage and retrieve git
> objects individually. Although repack's --filter-to option can separate
> objects from the main object store, it writes them into packs. Using
> those objects individually then requires an additional unpacking step.
>
> Add --filter-to-loose=<dir> to write filtered-out objects directly as
> loose objects into <dir>, treating it as an object directory. Stream
> the filtered pack into unpack-objects instead of writing an intermediate
> pack file to disk.
>
> The filtered pack is built with --stdin-packs, so this initially handles
> only objects already stored in packs. Loose objects in the source
> repository are left in place.

I haven't decided whether this new feature makes sense or not, so I
won't comment on that aspect.  Take the following review comments
with that in mind (i.e., pretend as if I said "if we assume doing
this makes sense, then..." before every sentence).

> +--filter-to-loose=<dir>::
> +	Like `--filter-to`, but treat `<dir>` as an object directory
> +	and write the filtered out objects there as loose objects
> +	instead of as a pack. Cannot be combined with `--filter-to`
> +	or `--max-pack-size`. Objects already present in `<dir>` are
> +	not rewritten. All objects are written before any existing
> +	pack is removed.

We talk about "already present in ...", so presumably we require the
directory to exist before we run.  Have a test that ensures that
passing a non-existing directory will make the command error out
before doing any real work.

> diff --git a/builtin/repack.c b/builtin/repack.c
> index 21835453c6..140a992f1b 100644
> --- a/builtin/repack.c
> +++ b/builtin/repack.c
> @@ -41,7 +41,7 @@ static const char *const git_repack_usage[] = {
>  	N_("git repack [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]\n"
>  	   "[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]\n"
>  	   "[--write-midx[=<mode>]] [--name-hash-version=<n>] [--path-walk]\n"
> -	   "[--filter=<filter-spec>] [--drop-filtered [--dry-run]]"),
> +	   "[--filter=<filter-spec> [--filter-to-loose=<dir>]] [--drop-filtered [--dry-run]]"),

The above documentation and help text consistently tell us that the
value passed is a directory (do we use the abbreviated <dir> so
commonly?  If not, it would make sense to spell it out, i.e.,
<directory>, for clarity).

> +		OPT_STRING(0, "filter-to-loose", &filter_to_loose, N_("dir"),
> +			   N_("object directory to write filtered out objects into as loose objects")),

Here, too.  

This should use OPT_FILENAME so that a "git repack" started from a
subdirectory with a relative path to the external loose object store
will find the directory at the correct place, I think.  Have a test
to validate the combination to see that such a repack that was given
a relative path to a directory find ...

	mkdir extra-loose-objects &&
	mkdir sub/dir &&
	(
		cd sub/dir &&
		git repack --filter-to-loose=../../extra-loose-objects ...
	)

the directory correct at the right place.

> diff --git a/repack-filtered.c b/repack-filtered.c
> index 869b9fc6e3..51c9dfeb13 100644
> --- a/repack-filtered.c
> +++ b/repack-filtered.c
> @@ -29,10 +50,18 @@ int write_filtered_pack(const struct write_pack_opts *opts,
>  		strvec_pushf(&cmd.args, "--keep-pack=%s", item->string);
>  
>  	cmd.in = -1;
> +	if (opts->loose)
> +		cmd.out = unpack.in;
>  
>  	ret = start_command(&cmd);
> -	if (ret)
> +	if (ret) {
> +		if (opts->loose) {
> +			close(unpack.in);

Please double check that this is not closing the same thing twice.
IIRC, a failing start_command() would close all file descriptors
associated with cmd so you may have already closed unpack.in when
start_command() gave us a failure, which presumably closed cmd.out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant