Repository navigation
feat: sandbox abstraction - #3336
Merged
Merged
Conversation
OpenShell becomes a fourth selectable SANDBOX_MODE alongside FAKECHROOT / NAMESPACE / SECCOMP_ONLY. Nothing is retired: the existing modes and their launch paths are untouched, so a deployment opts in by setting one property and reverts the same way. - Constants: SANDBOX_MODE_OPENSHELL = "OPENSHELL", documented as additive. - OpenShellTransport: creates a named sandbox running the gaas worker as its canonical main process, forwards the worker port to a free local port over the gateway, and tears the sandbox down on close. Deterministic sandbox naming from a session key (SHA-256, DNS-1123 safe) so no registry lookup is needed. A sandbox that already exists is success, not an error. - ClientProcessWrapper: one new branch in the existing dispatch, before the NAMESPACE branch. Sets no udsPath, so the existing socket client takes its normal TCP path to 127.0.0.1:<forwarded port> -- the socket layer is unchanged. Fails closed: never falls back to an in-Tomcat worker when the deployment asked for an external boundary. - shutdown(): closes the forward and deletes the sandbox, since the worker cannot outlive its sandbox. Verified: mvn compile clean. Existing modes behaviourally unchanged.
OPENSHELL_IMAGE (used when no OPENSHELL_TEMPLATE is set) and OPENSHELL_POLICY (a filesystem/egress policy YAML applied at creation). The policy has to be set at creation because Landlock settings are locked at sandbox startup and cannot be widened afterwards -- that immutability is the control, not an inconvenience. The stock OpenShell base image carries no python3, so an executor image is required in practice; this makes it a deployment property rather than a hard-coded default.
…t harness Three fixes found by running the transport outside Tomcat: 1. FAIL LOUDLY on missing config. create() added --from/--policy only when the property was present, so an unset OPENSHELL_IMAGE silently fell back to the stock OpenShell base image -- which has no python3. The sandbox is still created and reaches Ready; only the workload fails to spawn, so the caller sees an unrelated null response several layers away. Both inputs now throw at construction. Same defect class as the AGENT_SANDBOX_ENABLE/ENFORCE mismatch: a control that is configured but never reaches enforcement, failing open. 2. Sandbox name must fit the gateway's 19-character limit (was 25). MAX_SANDBOX_NAME_LENGTH is now a named constant with its reason recorded; the digest is truncated to fit. 3. Wait for the port forward to actually accept before returning the port. The CLI binds lazily, so returning early produced a connection refused that surfaced as a null response. The readiness probe is a TCP connect, never a protocol exchange, because the worker is single-connection. Also adds OpenShellTransportHarness: the same classpath with a console attached, which is what made all three visible. It bootstraps the DIHelper from RDF_Map.prop because Utility.getDIHelperProperty reads that, not -D system properties -- a distinction that cost three debugging iterations. Known open issue recorded in the wiki: the worker exits ~0.5s after start when it is the sandbox's canonical main process, while the identical command run as a backgrounded child stays up and listens. Under investigation; the transport itself is correct up to and including the forward.
awaitForwardReady() connected to the forwarded port. The forward proxies straight into the worker, the worker is single-connection (--max_count 1), so the probe consumed its only slot and its disconnect called stop_request() -- ending the worker and taking the sandbox to Completed. The symptom was indistinguishable from "the workload never started", which is why image, policy, argv, HOME, I/O pattern and environment breadth were all eliminated first. Readiness now comes from the CLI's own output: forward service prints a line containing "Forwarding" once its listener is bound. The method documents why it must not be "simplified" back into a socket connect. Also pins HOME for CLI children (OPENSHELL_CLI_HOME). The CLI resolves its gateway registration and mTLS client bundle from $HOME/.local/state/openshell/, so a caller with a thinner environment (Tomcat: 13 vars vs a shell's 124) could silently reach a different or missing gateway. Found by running the CLI with HOME unset, which reports "Or register one with: openshell gateway add". Adds OPENSHELL_KEEP_SANDBOX so a failed run leaves the sandbox in place for inspection -- deleting it destroyed the evidence every previous time. VERIFIED: a sandbox created by this class served a working persistent REPL through the protocol client (x=1 persists, mutates to 42, import math survives).
The forward's exit reason was being discarded: awaitForwardReady returned false without recording why, and close() then deleted the sandbox, so the evidence went with it. Both paths now log the CLI output and the process exit code before cleanup. Also records two measured facts in comments so they are not re-derived: - The forwarded port IS listening once start() returns (verified), so a connection failure is not a readiness problem. - The worker's insight/log.txt captures every request it receives, which is the only reliable way to tell "never arrived" from "arrived and failed" -- the sandbox phase and the supervisor log both say nothing about the workload. No behavioural change to the transport path.
…harness run() read the CLI child with a blocking readLine() loop and no timeout. Because close() also calls run() to delete the sandbox, a wedged child parked the Tomcat request thread inside the failure path itself: the sandbox was never cleaned up and the call never returned. BufferedReader.readLine -> run(:459) -> close(:399) Now the read happens on a daemon thread while the caller waits on waitFor(timeout); on timeout the child is destroyed forcibly and the reader interrupted. A cleanup path must not be able to block indefinitely on the object it is cleaning up. Adds OpenShellNativeClientHarness, which wires the transport to NativePySocketClient -- the class that actually ships -- with no protocol code of its own. Verified: transport up on 38331, NativePySocketClient connected, ready=true. Debugging through a hand-rolled protocol harness had been mistaking a failing test client for a failing transport.
…ient The harness previously only proved the handshake. It now sends OPERATION.PYTHON and reads the response, which is what PyTranslator does, so the chain is verified against shipping code rather than test scaffolding. Also corrects the PayloadStruct import (prerna.tcp, not prerna.om) and replaces a port probe with a plain sleep, since probing the forwarded port consumes the single-connection worker's only slot.
… binding The forward binds in ~60ms but the worker needs ~5s to reach serve_forever(). A client connecting in that window completes the TCP handshake and then consumes the single-connection worker only slot, wedging it. Measured: connect at +0s -> no reply; connect at +5s -> response=true. Readiness now waits for the supervisor NET:OPEN 127.0.0.1:<port>/tcp line, which needs no connection to the worker. Removes the temporary diagnostics.
1. Readiness is taken from the worker own log line, never by connecting to the port: the worker is single-connection (--max_count 1), so a probe consumes its only slot and its disconnect wedges it. The forward Forwarding line binds in ~60ms and is not readiness. 2. The worker stdout/stderr goes to a file instead of the supervisor pipe, which made startup erratic (sometimes never binding while the sandbox still read Ready). 3. run() now redirects stdin from /dev/null: openshell sandbox exec reads stdin, so with ProcessBuilder inherited open pipe the CLI blocked until the 30s timeout, and three of those produced the 180s pixel stall. The same command returns instantly from a shell, which is why this only showed up under Tomcat. Verified: Py("print("LIVE_OK")") returns LIVE_OK at 1599ms; repeats 1011/4/4ms; os.getpid()=60 vs Tomcat JVM 4059545.
createProcessAndClient is shared by the user REPL and engine-owned workers. Branching on SANDBOX_MODE alone sandboxed the model engine too, so AskRoom failed with ModuleNotFoundError: No module named openai several layers from the cause: AskRoomReactor -> AbstractModelEngine.askRoom -> AbstractPythonModelEngine.startServer -> PyTranslator. That read as a missing package but was a scope error. engineOwned already distinguishes them: User.pythonCPW and Project.cpw are user-authored and untrusted and get the sandbox; model engines, vector databases, guardrails and LocalPythonFunctionEngine are our own code calling controlled SDKs and stay on the Tomcat side. Verified: AskRoom now reaches the OpenAI API (401 on the engine own key, no import error, no sandbox) and Py() still returns pid 60 against a Tomcat JVM pid of 21788 with exactly one sandbox created.
Replaces the !engineOwned guard in ClientProcessWrapper with the branch the other jails already use. The jail has always been chosen by which caller supplies the SymlinkHelper: User.startPythonSocketServerAndClient passes getUserSymlinkHelper() for user-authored python, while engine-owned workers and Project pass null. OPENSHELL is now simply the third branch of that chain, alongside NAMESPACE and FAKECHROOT, and SymlinkHelper resolves SANDBOX_MODE once into a SandboxMode enum. ClientProcessWrapper no longer re-reads the mode at all -- it is a property of the user jail, so it belongs to the object that only exists for a user worker. Verified: AskRoom on an Ollama engine answers (Hello, hi, hey, 1028ms, no sandbox) and Py() still runs at pid 60 with exactly one sandbox.
…lper Every jail hangs off SymlinkHelper, because all three answer the same question: this host path must be usable by the worker. FAKECHROOT builds a chroot tree (symlinks on this host) NAMESPACE adds the path to the jail via SandboxInjector (control socket) OPENSHELL must COPY the tree, because a sandbox has its own filesystem Making a path visible is not making it present. SandboxTransporter is the OPENSHELL counterpart of SandboxInjector: wired by ClientProcessWrapper at the same point, called through a transportOrEnqueue dispatcher from symlinkFolder/symlinkProject so every invocation site is covered without editing any of them. activeTransports makes it once per sandbox, and setTransport replays paths registered before the worker existed, mirroring setInjector's replay of activeInjects. Two bugs found while verifying: 1. `sandbox upload <src> <dst>` nests the basename, so passing dst == src produced app_root/app_root/version/... and the project never landed where the worker looks. Destination must be the parent. 2. The transporter needs the same isNamespaceInjectablePath guard the injector has. MCPUtility also symlinks git's own directories, so /usr/bin, /etc/ssl and /usr/lib/git-core were being uploaded -- several 30s CLI timeouts and a 77s pixel. Now 8.4s. Verified: RunMCPTool now finds and executes the project's mcp_driver.py. It fails on the project's own `import yfinance`, which is absent from both the host venv and py/install_config/pyproject.toml and would fail under NAMESPACE too -- a missing declared dependency, not a visibility problem. Write-back (FETCH_BACK) is NOT implemented: files written inside the sandbox are lost at teardown. That asymmetry is worse than no transport at all, since the sandbox can now read project files while silently discarding writes.
…aper Two additions, both in their own package prerna.util.openshell. No existing class is modified. OpenShellSessionLifecycleHarness — proves the lifecycle the product relies on, without booting Tomcat: no sandbox exists before first use; the first user-authored python request creates it on demand (13.4s cold start); Py() executes inside it through the shipping NativePySocketClient (pid 60, python 3.12.14); the exact logout call -- ClientProcessWrapper.shutdown(true), as issued by UserSessionLoader.sessionDestroyed -- deletes it (385ms); and a later session starts a fresh one. 11/11 checks. OpenShellSandboxReaper — the backstop for sessions that never log out (JVM killed, container restart, abandoned session). Lists only sandboxes carrying the executor label AND the pyw- prefix, deletes only those older than a threshold, skips unparseable timestamps rather than guessing. Verified against a labelled throwaway: dry-run listed it, the real run deleted it, and an unlabelled sandbox on the same gateway was untouched.
Found live, in the UI path: if the named sandbox already exists, start() logged the create error and carried on. The worker inside allows one connection and the forward that reached it first still holds that slot, so the second forward leaves the real client queued -- Py() never returned and the logout thread sat in clearInsightGlobals until the sandbox was deleted by hand. A sandbox this transport is not driving belongs to a dead session, so the correct reading of 'already exists' is delete and recreate, not attach.
CmdExecUtil ran bash with commons-exec in a chroot on Tomcat's machine and never
consulted SANDBOX_MODE, so with SANDBOX_MODE=OPENSHELL the terminal, git
operations and the agent tool handlers all stayed on the host. Measured before
this change: Command("hostname") returned the host identity while Py() returned
the sandbox's.
OpenShellShellRunner runs the same command with 'openshell sandbox exec', which
starts a process inside the sandbox. Deliberately not the worker socket: the
worker serves one connection and a shell command must not compete with the REPL
for it. The sandbox is started on demand, so a shell command can be the first
thing a session does.
One delegation point in CmdExecUtil, taken only when the user's SymlinkHelper
reports an external sandbox; every other mode keeps the existing path untouched.
After: hostname 936b273a023c, uid 1000, ls of the sandbox tree -- host is
kunal0137-vnet.
Before this, the sandbox was write-only: a file created by Py() or a shell command lived in the sandbox copy and died with it, so the asset browser, git and MCP generation never saw it. Measured: to_csv into the project assets -> visible to 'ls' in the terminal, absent from GetAppAssets, gone at teardown. Now, after the terminal response of an executeCommand, the wrapper reconciles the session's transported roots back onto the host: - OpenShellTransport.fetchBack streams the tree out with 'sandbox exec -- tar -cf -' (sandbox download is workspace-scoped and refuses project paths), validates the archive (relative paths only, size cap), extracts to a temp dir and reconciles per file: content must differ AND the sandbox copy must be newer, so a stale sandbox cannot clobber a host edit. - SymlinkHelper exposes the roots it transported; ClientProcessWrapper filters to project/room/user roots and throttles to one reconcile per 1.5s. - NativePySocketClient calls it once per completed request. Verified: Py writes iris_fb.csv in the sandbox -> host file appears (16 bytes) -> GetAppAssets returns its contents. Single pod only: the push to central storage for the other pods is the next step.
…e teardown Three gaps found by testing, all in the fetch-back added in 863c3f443d: - The 1.5s throttle SKIPPED the reconcile instead of deferring it. Three Py blocks in one session left two files stranded in the sandbox until some later call. Skipping is wrong for a durable-write path; the throttle is gone. - Shell commands do not travel over the python socket, so "cat > file" or a git operation in the terminal never hit the hook. OpenShellShellRunner now reconciles after the command runs. - The last block before teardown could still be lost; shutdown() now reconciles once before deleting the sandbox. Also stopped excluding .git from the fetched archive. It costs bytes and writes nothing today -- the probe image has no git (git: not found) so the sandbox copy never changes -- but when the production image ships git, a commit made in the shell is exactly the work that must come back. Concurrent host/sandbox commits still need the revision guard. Verified: three Py writes plus one shell write in a single session all appear on the host.
Write-back used to tar every transported root at every execution boundary:
~190 ms per root, so `LoadApp(a); LoadApp(b); LoadApp(c)` paid ~750 ms per
block whether or not anything changed. The worker now knows which paths
changed and the host fetches those.
Worker side (py/):
- new gaas_change_watcher.py: ctypes inotify (no dependency), coalesced
per-path events, 2k cap, IN_Q_OVERFLOW flag, and a baseline flag. A new
directory is armed *and scanned*, because `mkdir x; write x/f` inside one
block otherwise reports the directory and loses the file.
- handler gains a CHANGES operation returning
{changes[], overflow, baseline, watched, events, roots}.
- the watcher is a module-level singleton created lazily on the first
CHANGES call: the TCP handler is built per connection, and at setup() the
session's project tree has not been materialised yet, so arming there
yields zero roots and pins the host to the slow path forever.
Host side (Java):
- PayloadStruct: CHANGES operation.
- NativePySocketClient.getFileChanges(); the after-execution hook now fires
only for PYTHON and CMD, so PollNotifications and other internal polls no
longer tar the tree.
- OpenShellTransport.fetchBackPaths(): one tar of the named paths, with the
same traversal and size checks as the whole-root fetch.
- ClientProcessWrapper.fetchBackChangedPaths(): groups paths by transported
root, deletes reported removals, and falls back to the full reconcile on
baseline, overflow, or no answer.
- OpenShellShellRunner takes the same path for shell commands.
Verified on a 240-file project: three blocks, three unique files, all on the
host; journal shows "Fetch-back wrote 1 file(s) from 1 changed path(s)";
watcher counters went watched:173 baseline:true -> changes:[x.txt] events:2
with root inodes unchanged from arm time. Cost 0.20 s/block vs ~0.30 s for the
full reconcile; the first block after materialise still does one full
reconcile, by design.
…a host edit Four correctness defects found by external review and a seven-model council, all in the change-set fetch-back path: 1. reconcile() trusted mtime. A sandbox file whose content differed but whose mtime was older than the host copy was silently dropped (os.utime, cp -p, tar extraction). Content is now the authority; mtime only decides whether the host copy must be preserved first. 2. Reported deletes were applied unconditionally, destroying a concurrent host edit. Deletes now quarantine the host copy first, and are suppressed entirely if that copy cannot be made. 3. A full-root fallback carried survivors but no tombstones, so sandbox deletions were silently resurrected. Full captures now quarantine host files absent from the capture; directories the capture excludes (node_modules, __pycache__, .venv) are skipped, since "not in the archive" there means "never captured", not "deleted". 4. inotify cannot see a persistent mmap write. The watcher now also compares one stat tuple (mtime_ns, size, ino, ctime_ns) per file at each checkpoint, reported as "stat_scan". Measured: ~6 ms for 240 files, silent on a quiet block, catches mmap writes and deletions with no events at all. Verified live in a sandbox: backdated rewrite, mmap write, and delete all reached the host; the overwritten host version and the deleted file were both preserved under /opt/semosshome/writeback-quarantine/<stamp>/.
… folders
Watch descriptors follow inodes, so renaming a watched directory left every
descendant path stale: events arrived under the old name and were written back
to the wrong place. The watcher now treats IN_MOVED_FROM, IN_MOVE_SELF,
IN_IGNORED and IN_UNMOUNT as lost coverage, reports `coverage: false` with
`baseline: true`, drops the tuple index and re-arms from the roots. The host
falls back to a full capture for that one block and resumes the change-set path
afterwards.
Java also treats `coverage: false` as a fallback trigger on its own, so a future
worker that reports it without `baseline` cannot leave the host trusting a
partial list.
Deleting a file now also prunes directories it left empty, up to but never
including the transported root - an empty folder left behind after a rename
reads as corruption.
Verified live: rename block took the full path ("wrote 2 file(s)", "moved 1
deleted file(s)"), the next block returned to the change set ("1 changed
path"), and the post-rename write reached the host.
The stat-tuple scan ran on every CHANGES call, so an idle block on a 100k-file tree paid ~0.57 s for a scan that could not have found anything. Measured: 240 files 1-2 ms, 10k 54 ms, 50k 285 ms, 100k 570 ms - linear at ~5.7 us/file. The scan is now due on the first call, after a lost baseline or coverage, and otherwise at most once per interval. The interval is max(2 s, 20 x the last measured duration), so the backstop holds roughly a 5% duty cycle at any size: 2 s at 240 files, ~10 s at 100k. Skipping never drops a change - the index is untouched, so the next scan reports everything accumulated; normal writes are unaffected entirely because they arrive as events. Measured at 100k: idle block 0.01 ms (was ~0.49 s), scan 0.486 s when due, interval self-tuned to 9.7 s. Payload reports stat_scan and scan_skipped.
Measured on a real project: .git was 112 directories and 137 of 256 files - over half the watch set and over half of every scan - for data that cannot change, since the sandbox image has no git binary. node_modules was only excluded from the scan and the tar, so it was still watched: one npm install adds thousands of directories, exhausts fs.inotify.max_user_watches and floods the queue, turning a cheap block into a full capture. One ignore set now covers arming, adoption, scanning and pull-back: node_modules, __pycache__, .venv, .git. Directories created mid-block are adopted only if they are not in that set, so an install inside a block cannot smuggle the tree back in. Override with SEMOSS_SYNC_IGNORE. Synthetic tree shaped like the real one (20 assets, 112-dir .git, 200-dir node_modules): watches 318 -> 4, indexed files 332 -> 20, asset writes still detected, node_modules writes unreported by design.
The sandbox runs outside the platform's permission checks, so any member who could open a project could write a file with Py() and have fetch-back publish it into the project folder - a read-only member silently gaining write access. ClientProcessWrapper now binds the owning User (set at client creation in User.java) and checks it before any root is pulled back: the project id is taken from the root folder (`<name>__<uuid>`, or a bare uuid), and SecurityProjectUtils.getActualUserProjectPermission must answer OWNER or EDIT. Anything else is skipped with "Refusing write-back into ... has READ_ONLY access to project ...", and an unresolvable permission fails closed. The check runs in both paths - the change-set pull and the full fallback - so a baseline reconcile cannot bypass it either. Verified live: with kunal0137 downgraded to READ_ONLY on workbench_1, a Py block wrote 29 bytes inside the sandbox, the host file did not appear and the refusal was logged; restoring OWNER made the same write land on the host. Rooms and user assets are NOT gated: this build has no room permission model, and asset roots are the user's own. If rooms are meant to inherit project access, that rule needs stating before it can be enforced.
…ll-through
Write-back now decides every root by its own space instead of special-casing
rooms and letting everything else through:
- user assets: the folder is Asset__<id> where the id must equal one of the
caller's own asset project ids (User.getAssetProjectId(provider), backed by
ASSETENGINE keyed on userid) - anything else under user/ is somebody
else's and is refused;
- rooms: ROOM.USER_ID in the model-inference-logs database via
ModelInferenceLogsUtils.doCheckRoomExistsForUser;
- projects: OWNER/EDIT as before;
- any unrecognised space: refused, where the old code returned true.
Every refusal is logged with the path and the reason, and each check fails
closed on an exception.
Verified: with the deployed class carrying isOwnUserAsset, a Py block writing
29 bytes into an owned project root still reached the host ("Fetch-back wrote
1 file(s)"). The asset/room denial paths are identity-keyed - this instance has
one human user per asset project and only kunal0137 owns rooms, so a foreign
root cannot be transported into a session to exercise the refusal.
…from py/ The executor image was assembled outside this repo: executor-probe/py was a hand copy of py/, and executor-proto just layered on top of it. That copy is how the image kept running month-old python while the repo looked fixed. docker/Dockerfile.executor replaces the two-image dance with one stage whose build context is py/, so the image carries exactly the committed worker. It keeps what matters from both predecessors: python:3.12-slim, git with a system identity (local repo work only - egress is still per-binary in the OpenShell policy), pandas + jsonpickle for the boot-time lazy imports, the insight/grants layout, uid 1000 ownership of /opt/semosshome, and the writable-workspace USER the sandbox requires. docker/build-executor-image.sh is the one command: it labels the tag with a hash of the python tree (org.semoss.py-sha) so `docker inspect` answers "which source is this", then smoke-tests the result inside the image. Built as semoss-executor:0.1 from py sha 1701eb4f4006: git 2.47.3, imports ok, worker present, watcher ignore set intact, id 1000. The worker boots as the main process and listens on 7100. RDF_Map.prop still points at semoss-executor-proto:0.2; switching is OPENSHELL_IMAGE + restart.
docs/deployment/executor_sandbox.md covers the opt-in SANDBOX_MODE=OPENSHELL path: the process model and session lifecycle, the per-block write-back pipeline (CHANGES verb, targeted tar, content-authoritative reconcile, quarantine, tombstones), the two change detectors and why the stat-tuple scan exists at all, per-space write permissions, the repo-owned image build, configuration keys, a verification recipe and the known limits (2000-path cap, racing mmap writers, no host->sandbox refresh). Linked from the Deployment and operations section of docs/README.md.
The first full build came out at 22.4 GB. Cause: flattening pyproject.toml to a requirements list dropped [tool.uv.sources], which is what maps torch/torchvision/ torchaudio to the pytorch-cpu index. PyPI then resolved CUDA builds - 3.3 GB of nvidia plus 815 MB of triton inside an image that never sees a GPU. export_requirements.py now skips the torch trio by default (--keep-torch to opt back in) so the Dockerfile can install it from the right index first, and it emits [tool.uv] override-dependencies as ordinary requirements so the sandbox keeps the platform's security pins (marshmallow, protobuf, numpy, pyasn1, lxml). Dockerfile.executor-full installs torch from download.pytorch.org/whl/cpu for the cpu extra and /cu126 for gpu, then the remaining declared dependencies from PyPI.
Shows the per-session shape — Tomcat, gateway, supervisor container, workload container with network=none, PID 1 sandbox runtime, the python worker reached through openshell forward — so a reader can see why a worker connection travels four hops instead of being a local socket.
The script that shipped with the first Dockerfile only ever built the slim image; the MODE=full path (staging pyproject.toml + export_requirements.py into the build context, passing EXTRAS) was written but never committed, and was lost to a checkout. Restored and committed so the full-image build is reproducible from the repo, not from shell history.
… venv The sandbox's PATH belongs to OpenShell, not the image. Measured inside a live sandbox: PATH=/usr/local/bin:/usr/bin:/bin, so an image's ENV PATH=/opt/venv/bin is ignored, `python3` resolves to the base interpreter, and the worker died at boot with no pandas/jsonpickle. A symlink /usr/local/bin/python3 -> /opt/venv/bin/python does not help either: CPython looks for pyvenv.cfg relative to the invoked path and silently runs as the base interpreter. Dependencies now install with `uv pip install --system` and the final stage copies /usr/local, so whatever `python3` the sandbox resolves to already has them. No venv, no PATH dependency, and the image is smaller (3.7 GB vs 6.9 GB) because the venv duplicated the interpreter. The smoke test now runs the parity check under `env -i PATH=/usr/local/bin:/usr/bin:/bin` - the sandbox's real PATH - instead of docker run's defaults, which is why the previous build reported success while failing at worker boot. Verified in a live OpenShell sandbox: python3 -> /usr/local/bin/python3, and numpy/boto3/pandas all import.
Logging out of the last provider drops its token before the session is destroyed. sessionDestroyed then threw in getSingleLogginName, which skipped every cleanup step, and even without the throw the owner could no longer be derived from the user, so the lease was never released. - getSingleLogginName returns "" when the primary login's token is gone - sessionEnded takes the owner; ownerForSession reads it from the session (recorded by the filter) and falls back to the user's login
…emosshome
Files cross into a sandbox at their host path, which is what keeps the
python SEMOSS generates (MCP loaders, PySource, the pandas JSON hand-off)
working, since it embeds absolute host paths. A Mac's BaseFolder is
somewhere like /Users/me/Documents/semoss, which no image has. The
sandbox can only create it under a root that exists at start and that
the policy grants: uploads run as the sandbox user inside Landlock.
- Dockerfile.sandbox makes /Users and /home, owned by uid 1000; at
create time the policy grants the top folder of BaseFolder's path, and
the transport makes the home inside the sandbox. One generic image
serves every developer; nothing is built per machine.
- The policy is rendered from a template bundled in the jar (or
OPENSHELL_POLICY), with ${SEMOSS_HOME_ROOT} and ${PYTHON} filled in.
OPENSHELL_POLICY is no longer required, so a checkout needs no policy
file from outside the repo. Egress is pypi only.
- Dockerfile.sandbox moves uv's CPython to /usr/lib/python/cpython, so
the egress rule names /usr/lib/python/cpython/bin/python3.14 on every
patch version and architecture; the smoke test checks it.
- Sandbox creation fails with the cause in the message when the home
cannot be made, instead of the bare upload error.
- OPENSHELL_IMAGE defaults to the published
quay.io/semoss/semoss-sandbox:dev-3.14-cpu, so a deployment that
tracks dev sets neither the image nor the policy; build-sandbox-image.sh
is gone, and the doc keeps the one docker build line for trying a py/
change before it is published
- OPENSHELL_EGRESS: host[:port] list rendered into the bundled policy's network_policies for the sandbox interpreter; default pypi, "none" for no egress; entries are validated as plain hostnames - OPENSHELL_CPU / OPENSHELL_MEMORY: per-sandbox caps passed to sandbox create, default 2 CPUs and 4Gi, "none" for no cap; caps not reservations, so idle sandboxes (~130 MB worker) cost little - the policy grants /dev/shm: multiprocessing, joblib and torch data loaders failed with PermissionError on SemLock without it Measured through the docker driver: caps land on the container (NanoCpus/Memory), an allocation past the cap is killed, custom egress allows only what is listed.
…rt guard /dev/shm makes it work for processes in the sandbox; inside Py() the worker's InsightGlobalStore guard removes os.fork/os.kill and bans subprocess, so Pool fails there (measured on the live instance).
…roviders A project declares SANDBOX_SECRETS (name + hosts) in its .smss and sets SANDBOX_SECRET_<NAME>. When a user's sandbox loads the project, each secret becomes an OpenShell profile + provider attached to that sandbox, so code gets a placeholder (smss_secrets.secret(name)) that OpenShell's supervisor swaps for the value only in requests to the declared hosts. The .smss never enters the sandbox, so the value stays on the host. - credential names carry the project key, so two apps may share a name - the value reaches the CLI through the environment, never argv - an attach is waited out, and the placeholder file is rewritten after every attach or update once the new generation appears; old generations stop resolving (attach) or keep the old value (update) - an unchanged value is not re-applied, so later loads cost ~0.5 s Measured against a real sandbox with a live OpenWeather key: 200 through secret(), refused at another app's host, rotation reaches running code.
…P example The old slim image pip-installed it as a stopgap for the example; the python-builder venv does not declare it, so Stock MCP failed with ModuleNotFoundError after the 3.14 switch. APP_PACKAGES keeps the list explicit until apps can declare their own dependencies.
…Stock MCP example" Skip yfinance as an image package for now; the Stock MCP example is parked. This reverts commit 3076363.
# Conflicts: # src/prerna/auth/User.java # src/prerna/om/ClientProcessWrapper.java # src/prerna/util/SymlinkHelper.java
Add prerna.util.sandbox.SandboxSettings: every sandbox key with its default, scope and description, plus the single SANDBOX_MODE parser. SymlinkHelper, ClientProcessWrapper (node) and OpenShellUserSandboxes now use it instead of three separate parsers. OpenShellTransport takes its defaults from the registry. The effective settings are logged once at first jail use, with a warning when SANDBOX_MODE is not a jail mode (OFF and SECCOMP_ONLY mean FAKECHROOT). Fixes: OpenShellShellRunner now pins HOME to OPENSHELL_CLI_HOME like the transport; the Redis registry records the effective worker port instead of "null"; OPENSHELL_CLI has one default; stale Constants and SandboxLaunchPlan comments. Docs: docs/deployment/sandbox_configuration.md lists every key, which jail you get, and what each jail's child process inherits.
…l backends Add prerna.util.sandbox.ISandbox (grantFolder, grantProject, revoke, spawnWorker, exec, capabilities) and SandboxFactory, the one place a mode becomes a backend. Callers no longer check the mode: - ClientProcessWrapper starts python, Java and node workers through ISandbox.spawnWorker instead of its per-mode if chain, and takes over the result with applyWorker. - CmdExecUtil runs commands through ISandbox.exec; the command-running code moved to ShellExecutor unchanged. Commands in a namespace jail now say they run in fakechroot, or are refused with NAMESPACE_EXEC_FALLBACK=refuse. OpenShell commands through executeCommandWithStatus now go to the sandbox too. - The 12 symlinkFolder/symlinkProject call sites use User.getSandbox(). - SymlinkHelper gains revoke (namespace mount and chroot link). The namespace python and node jails now start with only PATH, LANG, LANGUAGE, LC_ALL, TZ and SANDBOX_ENV_ALLOWLIST from the server environment instead of all of it (NAMESPACE_SCRUB_ENV, default true). Backends run a worker the way it always ran when they have no jail for it (Java in fakechroot, node on the host outside the namespace jail), and log that once. Shutdown and OpenShell write-back stay in ClientProcessWrapper.
…r packages OpenShellTransport and OpenShellTransporter move to prerna.util.openshell, next to the rest of OpenShell and the policy file the transport already loads from there. SandboxTransporter and SandboxInjector move to prerna.util.sandbox, next to ISandbox and the backends that use them. This also removes the circular dependency between prerna.util and prerna.util.openshell. Rename and import changes only. SymlinkHelper.isLeftOutOfCopy and copyProjectForReadOnlyUser become public because the transport now calls them from another package.
Contributor
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
This was referenced Nov 17, 2026
…hanges, keep Java grants in OpenShell mode - NamespaceSandbox: a worker that never became ready was owned by nobody once spawnWorker threw, because the wrapper only receives the handle on return. Stop the launcher and its children and remove its folders before throwing. - OpenShell access changes: SymlinkHelper re-sent a root only when it went from view-only to edit, and OpenShellTransport only added and overwrote files. A user who lost edit access kept the full copy (.git, portals, classes) in the sandbox, and a file made while the user could only view was written back as a new file once they could edit. A root whose access changed in either direction is now cleared in the sandbox, with its stamps, before the new copy is sent. - OpenShell mode now also places folders and projects in the local chroot after the remote transport. The Java worker has no OpenShell sandbox and runs in the local fakechroot, so it needs the same links and viewer copies as in other modes. Checked live on OpenShell: a viewer's new files are not published after promotion and logout; a promoted editor's write syncs within seconds; a demoted user's sandbox loses .git and portals and its writes are refused; the chroot holds a viewer copy or an editor link to match.
It is a manual dev harness that no longer compiles: DIHelper became package-private in #3326 and ClientProcessWrapper.createProcessAndClient has a different signature. Builds skip test compilation, so it went unnoticed, but mvn test fails on the branch with it.
SANDBOX_MODE=OPENSHELL runs the openshell CLI from Tomcat. Take only the static binary from the pinned, checksum-verified release .deb (amd64/arm64) in each runtime Dockerfile, next to the copilot CLI. The gateway is not included.
OPENSHELL_VERSION=latest resolves the newest release at build time; a version can still be passed to match a deployed gateway. The checksum check stays.
Dockerfile.tomcat runs the upstream install.sh; each runtime image copies /usr/bin/openshell from its tomcat_builder stage instead of repeating the download.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replaces #3332 and supersedes #3191. Same work, now on a branch in this repo instead of a fork. It contains every commit from #3191 (OpenShell user sandbox), merged up to current
dev, and builds on it:SANDBOX_MODEparser,ISandboxinterface with host, fakechroot, namespace and OpenShell backends,prerna.util.openshellandprerna.util.sandbox.Companion PRs:
OpenShellUserSandboxesmethods from this PR, so merge this one first.Changes Made
dev: resolved conflicts inSymlinkHelper,ClientProcessWrapperandUseragainst dev's project-placement rewrite and the removal ofclassPath.java/,classes/,portals/,.git) landed at the real app-root path. Refresh and stamp recording skip the same folders. Edit access given later re-sends the full tree.SandboxSettings(prerna.util.sandbox): every sandbox key with its default, scope and description;SANDBOX_MODEparsed in one place instead of three; effective values logged once at first jail use. New doc:docs/deployment/sandbox_configuration.md.ISandboxandSandboxFactory:grantFolder,grantProject,revoke,spawnWorker,exec,capabilities().ClientProcessWrapperstarts Python, Java and Node workers through it,CmdExecUtilruns commands through it (the command code moved toShellExecutorunchanged), and the 12symlinkFolder/symlinkProjectcall sites useUser.getSandbox().PATH,LANG,LANGUAGE,LC_ALL,TZandSANDBOX_ENV_ALLOWLIST, not the server's whole environment. New keys:SANDBOX_ENV_ALLOWLIST,NAMESPACE_SCRUB_ENV(defaulttrue),NAMESPACE_EXEC_FALLBACK(defaultfakechroot).OpenShellTransportandOpenShellTransportertoprerna.util.openshell;SandboxTransporterandSandboxInjectortoprerna.util.sandbox.SymlinkHelper.isLeftOutOfCopyandcopyProjectForReadOnlyUserbecame public because the transport now calls them from another package..git,portalsandclassesin the sandbox, and a file made while they could only view is not written back as a new file after promotion. Folders and projects are also placed in the local chroot after the remote transport, because the Java worker has no OpenShell sandbox and runs in the local fakechroot.spawnWorkerthrows. Before, nothing owned it once the exception was thrown.Dockerfile.tomcatinstalls the OpenShell CLI with NVIDIA's installer, and each runtime Dockerfile copies/usr/bin/openshellfrom it. The gateway is not run in these images.OpenShellShellRunnerpinsHOMEtoOPENSHELL_CLI_HOME; the Redis registry records the effective worker port;OPENSHELL_CLIhas one default; stale comments.How to Test
mvn -q clean compile -DskipTests -Denforcer.skip=true -Dlicense.skip=true -Djacoco.skip=trueBashCommand, open a project you can edit, open a project you can only view.java/,classes/,portals/or.gitinside the sandbox.SANDBOX_ENV_ALLOWLISTor setNAMESPACE_SCRUB_ENV=false.Sandbox settings: ...appears at first jail use.Notes
NAMESPACE_SCRUB_ENVdefault is the change most likely to need adjusting.ClientProcessWrapper.Capabilitiesleaves room for it.