Skip to content

python: add use_host_python option; use env in shebangs - #401

Open
katexochen wants to merge 2 commits into
amutable:mainfrom
katexochen:use-host-python
Open

katexochen wants to merge 2 commits into
amutable:mainfrom
katexochen:use-host-python

Conversation

@katexochen

Copy link
Copy Markdown
Member

No description provided.

Signed-off-by: Paul Meyer <katexochen0@gmail.com>
@martinpitt

martinpitt commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

The "env shebang" commit is fine of course. But the "use host python" is a rather big hack IMHO:

  • The action command line only has python3, so buck can't detect and rebuild things if the host interpreter changes. This breaks cache consistency, including the remote cache.
  • There's no check for a minimum version. The bootstrapping currently requires 3.14, or 3.12 with zstd.
  • Also, overlap: tine.python_interpreter already lets a project point the toolchain at a target of its own. The new flag skips the target and falls back to PATH.
  • This does not work where tine runs the bootstrapped python on the host, like for ty or PYTHON_TEST_INTERPRETER.
  • Host stuff like sitecustomize and PYTHONPATH leak into the build.

I assume this is about running tine in NixOS. Is it an option for you to use nix-ld ? (programs.nix-ld.enable). That's meant for exactly this situation.

An alternative might be to pin a musl build/loader in addition to Python, and invoke it as ld-musl-x86_64.so.1 bin/python3.14. That would actually improve hermeticity compared to today.

This is also undocumented, and untested. It could at least have a single test that enables this option, in a new isolation dir, and builds a small target. In umockdev I run CI in a nix container which is also reasonably cheap.

would be nice to have a hermetic way, but as long as we are downloading
dynamically linked stuff that must run on the host we need a way to work
around it.

Signed-off-by: Paul Meyer <katexochen0@gmail.com>
@katexochen

katexochen commented Nov 12, 2026 •

Copy link
Copy Markdown
Member Author
  • The action command line only has python3, so buck can't detect and rebuild things if the host interpreter changes. This breaks cache consistency, including the remote cache.
  • There's no check for a minimum version. The bootstrapping currently requires 3.14, or 3.12 with zstd.
  • Also, overlap: tine.python_interpreter already lets a project point the toolchain at a target of its own. The new flag skips the target and falls back to PATH.
  • This does not work where tine runs the bootstrapped python on the host, like for ty or PYTHON_TEST_INTERPRETER.
  • Host stuff like sitecustomize and PYTHONPATH leak into the build.

This is also undocumented, and untested. It could at least have a single test that enables this option, in a new isolation dir, and builds a small target.

Addressed.

An alternative might be to pin a musl build/loader in addition to Python, and invoke it as ld-musl-x86_64.so.1 bin/python3.14. That would actually improve hermeticity compared to today.

A musl Python doesn't help here. The drivers load glibc-built libraries into the interpreter with ctypes, such as the
box's libseccomp, libkmod and libalpm, and a musl interpreter can't load those. The static musl build can't load any
library at all, and the dynamic one needs /lib/ld-musl-x86_64.so.1, so it hits the same loader-path problem. The
host's python3 already works with the host's own libc and loader.

@martinpitt

Copy link
Copy Markdown
Member

OK, got it. Does nix-ld help?

@katexochen

Copy link
Copy Markdown
Member Author

OK, got it. Does nix-ld help?

I don't want to run that. It can't be scoped to a project but is system-wide configuration with massive impact.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants