feat: Python standalone images from python-build-standalone - #2154
feat: Python standalone images from python-build-standalone#2154reneleonhardt wants to merge 5 commits into
Conversation
Mirrors nodejs/ build; swaps Debian's python package for a python-build-standalone tarball (image base stays Debian). Versions 3.13.15, 3.14.7, 3.15.0rc1 x debian12+13 x amd64/arm64/s390x/riscv64(debian13). Signed-off-by: Rene Leonhardt <65483435+reneleonhardt@users.noreply.github.com> Co-authored-by: DeepSeek <noreply@deepseek.com>
README covers find_library/venv/compat; test harness renamed to stdlib_check.py. Signed-off-by: Rene Leonhardt <65483435+reneleonhardt@users.noreply.github.com> Co-authored-by: DeepSeek <noreply@deepseek.com>
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
|
At a highlevel:
|
- debian12 removed: new debian12 builds are being deprecated upstream; python/ is debian13-only now - 3.15rc1 removed: images are 3.13 and 3.14; matrix is general enough that 3.15 is added via config.bzl + python.bzl when stable - README/provenance/testdata updated; publish map picks up the new matrix automatically (root BUILD iterates config)
…on-archives) Reads the python matrix (minors + archs) from python/config.bzl and the current versions from //private/extensions:python.bzl, queries the latest PBS release (latest-release.json, API fallback), rewrites the extension's archive/version data, and bumps/creates testdata version strings. Idempotent (no-op when current). - knife.d/update_python_archives.sh (bash 3.2 compatible, gnu-sed style per repo convention) + knife cmd + README provenance + extension versions-comment + lockfile synced - hermetic smoke test //python:update_python_archives_test (no network, no fixtures): asserts the committed matrix, extension, and MODULE.bazel use_repo agree; the updater keeps its expected-minors line in sync Policy: the matrix is a contiguous support window anchored at its oldest minor - the updater fills every published stable minor above it (several at once), the maintainer drops the tail (EOL), minors are sorted+deduped internally, rc/a/b and freethreaded tarballs are excluded. Flags: --verbose (list found latest versions even when unchanged), --dry-run (print what would change, mutate nothing). Safety: a broken script must not clean up after itself - all verifiable failures happen BEFORE any mutation (config anchors checked, generated extension buildifier-fixed on a temp file, config edits and the MODULE.bazel use_repo line applied to temp copies and grep-verified, all only mv'd after verification). RED reports pre-flight (nothing mutated) vs mid-mutation (partial, git recovery hint incl. MODULE.bazel.lock) via a MUTATED_PARTIAL marker; GREEN reminds to sync the lockfile (bazel mod deps --lockfile_mode=update). A new minor now also updates MODULE.bazel's use_repo(py, ...) line - the new repos must be visible to the root module (previously the 3.15 image failed to analyze with 'No repository visible as @python315_arm64'). Verified: real no-op/verbose/dry-run; mocked PBS 3.15.0 (fill-top, tail-drop, unsorted/dup normalization, patch replace, smoke-test sync, use_repo update + target analysis, GREEN/RED paths incl. forced post-mutation failure); smoke test catches matrix/extension/use_repo drift.
Removed.
Explained above. PBS linux builds statically bundle zlib/bz2/sqlite3/OpenSSL/libffi/lzma/ncurses; the tarball binaries reference only glibc core libs, so the image adds zero python-specific deb packages. The 502-module stdlib import walk (ssl, sqlite3, zlib, ctypes, https) passes in-image with none; deliberate skips (GUI,
Done. Matrix is 3.13 + 3.14;
Included: |
|
I'm sorry I will get to this soon, lots of other things on my plate. |
|
No stress, my awesome contribution everyone's been waiting for isn't running away 😄 |
|
I'm no python expert, but the ldconfig seemed somewhat important to people. is CDLL sufficient in this case? are people not going to use "find_library()" ? |
|
I had a teammate recommend we keep the ldconfig binary and continue to generate the ldconfig.so.cache to allow find_library to continue to work |
Maintainer: 'Keep the ldconfig binary and continue to generate the
ldconfig.so.cache to allow find_library to continue to work'. python3/
parity, ported to python/:
- libc-bin added to the image (provides /sbin/ldconfig) via
PYTHON_PACKAGES in config.bzl
- per-arch ld.so.cache generated by running ldconfig inside the image
(python/ldconfig.bzl + ldconfig.sh, ported from python3; bazel run
//python:update_ldconfig) and shipped at /etc/ld.so.cache
- python/ldconfig.bzl: oci_load + do_load genrules + update_ldconfig +
check_ldconfig_{arch}_test + test suite (caches for amd64, arm64,
s390x, riscv64 generated and committed)
- find_library CST test added to testdata/python3.yaml
- check-ldconfig.yaml workflow extended to cover //python too
- README find_library section rewritten (works now; regenerate via
bazel run //python:update_ldconfig)
Verified: find_library('c'/'ssl'/'z') resolve in the image;
check_ldconfig tests 4/4 pass; CST (incl. find_library) + smoke test
pass.
be7c93f to
3c51b57
Compare
|
Good catch! Now python3/ parity.
Verified in the image: Scope note (deep-audited against the python3 image): PBS statically links zlib/bz2/sqlite3/OpenSSL/ffi/lzma/ncurses INTO libpython, so those modules work with no shared object to find - |
2 participants
Summary
New
python/package (mirrorsnodejs/) building distroless Python images from python-build-standalone (PBS) prebuilt tarballs instead of Debian-packaged python. The image base stays Debian (//ccprovides glibc); only the python distribution method changes.Matrix: 3.13.15 / 3.14.7 x debian13 x amd64 / arm64 / s390x / riscv64 x root/nonroot x normal/debug.
Tags:
python3.13-debian13,python3.14-debian13(+ per-arch and default-distro variants, nodejs publish pattern).Why PBS
knife update-python-archives.Platform exclusions (verified, not oversights)
/lib/ld-linux.so.3); the distroless armhf base ships/usr/lib/ld-linux-armhf.so.3- the binary cannot exec (confirmed in Docker). Removed./usr/lib/ld64.so.1,/usr/lib/ld-linux-riscv64-lp64d.so.1).Notes
find_library()works for system libs (c/ssl/z verified): image ships/sbin/ldconfig(libc-bin) + a per-archld.so.cache(python3/ parity, maintainer request); regenerated viabazel run //python:update_ldconfig, checked by//python:check_ldconfig_tests(4/4) + a CST test. Statically-linked PBS libs have nothing to find;python -m venvworks (ensurepip wheels bundled);cryptmoot (removed in 3.13).python3/(deb) untouched.Verification
bazelisk build //python/...green (debian13, 3.13+3.14, 4 archs).3.14.7 x86_64, ssl certs OK.//python:check_ldconfig_tests4/4 (amd64/arm64/s390x/riscv64); CST includes a find_library test; CDLL(soname) works (libc/libssl/libgcc/libpython).knife update-python-archives): no-op idempotent;--verbose/--dry-run; fills new stable minors, drops EOL tail, sorts+dedupes the matrix; hermetic smoke test//python:update_python_archives_test; pre-flight validation (RED mutates nothing; recovery via git, lockfile reset documented).kind(oci_image, deps(:sign_and_push)); CST taggedmanual+arch run viaattr(tags, "amd64", ...)— same rails as python3/;check-ldconfig.yamlextended to cover//python:check_ldconfig_tests.Checklist
bazelisk build //python/..., structure tests