In Gecko when cloning from Mercurial, we define a set of sparse profiles that then get passed on to robustcheckout.py which then uses them with Mercurial's sparse checkout extension. There's currently no equivalent supported for Git checkouts.
We'll need to add support for partial cloning in Git checkouts if we want to avoid regressing clone performance after switching those tasks.
Delivery Mechanism
We need a way to obtain the sparse profile in the task prior to the proper clone. There are a few options:
- Fetch just the sparse-profile blob prior to the proper clone. This is essentially what
robustcheckout does, but it's made efficient by the remotefilelog extension on HGMO. I'm unsure if we'd be able to do something similar with Github.
- Bake it into the task definition (e.g as an env var)
- Upload it as an artifact on the decision task and fetch it
I lean towards option 2 as it seems the simplest and fastest.
Some other implementation notes:
- We should only support native Git sparse profile formats (not the bespoke format in Gecko). For compatibility, we'll need a translation layer in
gecko_taskgraph to convert the build/sparse-profiles format to the Git one.
- We'll need to make sure checkout caches contain the sparse profile name in them so they don't get mixed.
- We should only use
no-cone mode for now. cone mode should be much faster, but it only works with directories which means won't be compatible with Gecko's existing profiles. In the future we should probably re-write these profiles so we can use cone mode, but that can be a follow-up. This can be achieved with something like:
git clone --filter=blob:none --no-checkout <url> <dir>
git -C <dir> sparse-checkout init --no-cone
git -C <dir> sparse-checkout set --stdin < patterns
git -C <dir> checkout <rev>
In Gecko when cloning from Mercurial, we define a set of sparse profiles that then get passed on to
robustcheckout.pywhich then uses them with Mercurial's sparse checkout extension. There's currently no equivalent supported for Git checkouts.We'll need to add support for partial cloning in Git checkouts if we want to avoid regressing clone performance after switching those tasks.
Delivery Mechanism
We need a way to obtain the sparse profile in the task prior to the proper clone. There are a few options:
robustcheckoutdoes, but it's made efficient by theremotefilelogextension on HGMO. I'm unsure if we'd be able to do something similar with Github.I lean towards option 2 as it seems the simplest and fastest.
Some other implementation notes:
gecko_taskgraphto convert thebuild/sparse-profilesformat to the Git one.no-conemode for now.conemode should be much faster, but it only works with directories which means won't be compatible with Gecko's existing profiles. In the future we should probably re-write these profiles so we can useconemode, but that can be a follow-up. This can be achieved with something like: