Repository navigation
Implement OCI image support - #416
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
c518435 to
73075ee
Compare
| dirs.append(entry.path) | ||
| elif entry.is_symlink(): | ||
| target = os.readlink(entry.path) | ||
| if target.startswith("/"): |
There was a problem hiding this comment.
Only a target that starts with / is rewritten. On Linux .. stops at /, so a relative target with more .. than the link's depth is a legal way to reach the root: /a/l -> ../../../etc/hostname reads /etc/hostname. fixup_rootfs leaves that link unchanged, and followed natively in the attached volume it resolves outside rootfs/, to run/<pid>/etc/hostname, while resolve_in_root on the same link returns rootfs/etc/hostname. The hostile /etc link in scripts/ci/oci-smoke.sh has this shape and is sealed unchanged. Does elfuse clamp such a link when the guest follows it, or does it need the same rewrite as an absolute target?
There was a problem hiding this comment.
elfuse resolves link targets in the guest namespace, so .. stops at the guest root.
Under elfuse-oci, /a/l -> ../../../etc/hostname reads the image's file, and a link climbing to a host file gets ELOOP (also covered by test-sysroot-symlink-escape).
Only absolute targets need the rewrite, because elfuse follows those to the host.
73075ee to
a78445e
Compare
a78445e to
a4b6d18
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
xalestar
left a comment
xalestar
left a comment
There was a problem hiding this comment.
I read scripts/elfuse-oci.py, the smoke lane and the workflow change at a4b6d18, and ran the pure functions of the script against malformed inputs. One finding in the script, two in the lane, inline.
scripts/check-skill-refs.py changes its example path from cmd/elfuse-container to tools/elfuse-gen. That path is unrelated to cmd/oci, so it belongs in its own commit.
scripts/ci/ is a new directory, while the other scripts the workflow calls are in .ci/. .ci/oci-smoke.sh keeps them in one place.
a4b6d18 to
f9f7d2a
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
cmd/oci pulls images into an OCI layout that nothing in the tree reads: no command unpacks or runs what it stores. Delete it with go.mod, go.sum, mk/oci.mk, its Linux CI job, and docs/oci-images.md. It was the only Go code, so nothing left needs a Go toolchain. The references go with it: the README and docs/usage.md sections; the docs/testing.md row and section; the mk/oci.mk entry in proof-scope.py; the --user help clause and docs/usage.md sentence naming elfuse-oci as the resolver of symbolic names; a test comment naming its run command; the launch.c comment naming a planned OCI run helper; the image entrypoint, OCI layer, and container workload examples in test-launch-flags.sh, chown-overlay.c, and test-sysroot-tmp-remove.c; and the check-skill-refs.py test vector named after cmd/elfuse-container, an earlier OCI tool.
scripts/elfuse-oci.py pulls, runs, lists, and cleans linux/arm64 OCI images under elfuse. crane downloads an image and umoci unpacks it into a case-sensitive APFS volume, so names that differ only in case do not collide, and the volume is sealed read-only once per manifest digest. Each run attaches it with its own shadow file, which takes the run's writes. Its own errors and any OSError print as one line. A watcher process detaches a run's volume once elfuse exits. run execs elfuse only after the watcher waits on its pid, and fails if the watcher could not start. run requires the command to be a file in the image with an execute bit, a name with a slash included, because elfuse falls back to the host for a path the image lacks, and refuses an image whose Entrypoint, Cmd, or Env is not a list of strings. clean refuses an image that hdiutil lists without its device, as it does while attaching or detaching one. A pull or run whose volume is still attached after a forced detach leaves its directory for clean: removing it would delete the files on the volume and the file backing it. A pull gives the owner full access to a directory only while it adjusts the tree, then restores the image's mode. scripts/test-elfuse-oci.py holds table tests for the parts that need no image.
scripts/ci/oci-smoke.sh drives scripts/elfuse-oci.py under elfuse: the table tests in scripts/test-elfuse-oci.py, a cold pull that seals one 0400 image and a warm run that does not unpack again, list naming the sealed image, the guest exit status, a Debian tool pipeline, per-run isolation, a dynamic interpreter through --entrypoint, a run that does not see a concurrent run's files, clean refusing a non-store and a store in use, a pull without umoci failing with one line and leaving no build behind, an image whose /etc climbs to a host directory, the sealed image unchanged by the runs, and a final clean that empties the store. It pulls the library images through mirror.gcr.io, so it does not depend on Docker Hub's anonymous pull limit. The oci-smoke job builds crane and umoci from source with Go, because the runner user cannot write the Homebrew prefix. It then runs the lane on the self-hosted macOS runner against the elfuse binary build-macos uploads. Each runner gets its own store, since the runners on one host share $HOME.
docs/oci-images.md describes the store and its safeguards, how a pull seals one case-sensitive volume per manifest digest and what it changes in umoci's tree, how a run attaches that volume with a per-run shadow and why a read-only sysroot alone is not enough, what clean removes and refuses, what list shows, and how the table tests and the smoke lane validate it. docs/usage.md gains the commands, the run option, the store variable, and the local lane recipe, docs/testing.md the validation entry, and README.md the overview paragraph and link.
f9f7d2a to
c3c52b4
Compare
|
Thank @henrybear327 for contributing! |
3 participants
(Please review commit-by-commit to see the changes more clearly! This PR consists of removing all half-baked OCI go code from last iteration, and implementing the new support feature.)
Previous attempts focused on handrolling code around existing open source libraries instead of leveraging compiled open source tools directly, thus bloating the size dramatically.
In this latest attempt, crane and umoci are heavily relied upon as a tool directly, thus, containing the entire OCI support surface from the last attempt within very reviewable size.
Closes #31
Summary by cubic
Replaces the Go
elfuse-ocipull-only tool with a Python OCI front end that can actually run images under elfuse.scripts/elfuse-oci.pypulls images with crane, unpacks each image digest once with umoci onto a sealed read-only case-sensitive APFS volume, and runs it under elfuse with a per-run writable shadow so guest writes have somewhere to land. The Go binary, its build rules, and the Linux CI job are removed; anoci-smokelane on the self-hosted macOS runner covers the script end to end, andscripts/test-elfuse-oci.pyprovides table tests that need no image.Written for commit c3c52b4. Summary will update on new commits.