external_deps: add download action to download and extract the external deps tarball like CMake

[illwieckz]

Add download action to download and extract the external deps tarball like CMake.

Allows to do:

./build.sh macos-amd64-default download

It would download https://dl.unvanquished.net/deps/macos-amd64-default_11.tar.xz and extract it as external_deps/macos-amd64-default_11.

The reason why I'm doing this is that I'm upgrading the docker-based release scripts to use a newer Darling version (in hope I can install a newer XCode supporting arm64) and there is a regression in the latest version of Darling: everything works but the CMake builtin download. CMake probably uses a buggy library/feature that curl doesn't.

By downloading and extracting the deps before calling CMake, everything works.

Implementing it in external_deps/build.sh ensures the version, directory name, etc. are always correct.

[illwieckz]

@slipher any LGTM on that? 🙂️

[slipher]

Download caching and test packages don't go well together. Probably a test package archive shouldn't be cached, and definitely not if it can override the final one.

[illwieckz]

That's already a problem with the similar “download the preview deps archive” mechanism in the CMakeLists.txt.
I doubt this can be solved.

People having an already downloaded preview package in their folder are people testing preview packages on purpose, so they know about it and it's up to them to clean-up.

[slipher]

That's already a problem with the similar “download the preview deps archive” mechanism in the CMakeLists.txt. I doubt this can be solved.

Yes it's also bad how it works there. It could easily be solved by using a testing version number when you want to upload a test package for the CI, instead of using the same version number as the final package. Then there wouldn't be the annoying hazard for people who want to test the PR of needing to make a temporary external deps dir to avoid contamination.

Meanwhile for this PR testing packages could be not cached, or omitted altogether.

[illwieckz]

I also thought about the extra version suffix, but once the tested package is ready and verified, I just move it unmodified out of test/. We may just rename the archive and extract without the testing suffix, but then an already extracted deps would also prevent download.

The current situation looks good enough to me, as it allows us to guarantee that what is made a default download has been tested (not a repackage of it).

Having a preview package only matters to people testing their development.

In all cases, the matter of dealing with leftover preview packages is out of topic here. The purpose of this patch is to reproduce exactly what CMakeLists.txt does.

This PR doesn't write the package in download_cache/, it does like CMakeLists.txt: it just downloads the package in the default place and extracts it.

[illwieckz]

This PR doesn't write the package in download_cache/, it does like CMakeLists.txt: it just downloads the package in the default place and extracts it.

That also means testing packages should not be omitted. Otherwise, it would not be possible to test the release scripts with testing packages when using this alternate download method in release scripts.

[slipher]

This PR doesn't write the package in download_cache/, it does like CMakeLists.txt: it just downloads the package in the default place and extracts it.

Oh my mistake then. I thought it was using the regular download function that all the other ones use and would cache.
LGTM

[illwieckz]

Yes it uses the regular download function, but that function takes the complete download path (directory + file name) as first argument, and here it is not the download cache directory.

Thanks for the review.