Skip to content

feat: sandbox abstraction - #3336

Merged
kunal0137 merged 73 commits into
devfrom
feat/sandbox-abstraction
Nov 17, 2026
Merged

kunal0137 merged 73 commits into
devfrom
feat/sandbox-abstraction

Conversation

@kunal0137

@kunal0137 kunal0137 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Replaces #3332 and supersedes #3191. Same work, now on a branch in this repo instead of a fork. It contains every commit from #3191 (OpenShell user sandbox), merged up to current dev, and builds on it:

  1. the OpenShell view-only fix: a user who can only view a project gets the same read-only copy the other modes use,
  2. one list of every sandbox setting and one SANDBOX_MODE parser,
  3. an ISandbox interface with host, fakechroot, namespace and OpenShell backends,
  4. the OpenShell and sandbox helper classes moved into prerna.util.openshell and prerna.util.sandbox.

Companion PRs:

Changes Made

  • Merge with dev: resolved conflicts in SymlinkHelper, ClientProcessWrapper and User against dev's project-placement rewrite and the removal of classPath.
  • View-only OpenShell roots: the sandbox gets the read-only copy (no java/, classes/, portals/, .git) landed at the real app-root path. Refresh and stamp recording skip the same folders. Edit access given later re-sends the full tree.
  • SandboxSettings (prerna.util.sandbox): every sandbox key with its default, scope and description; SANDBOX_MODE parsed in one place instead of three; effective values logged once at first jail use. New doc: docs/deployment/sandbox_configuration.md.
  • ISandbox and SandboxFactory: grantFolder, grantProject, revoke, spawnWorker, exec, capabilities(). ClientProcessWrapper starts Python, Java and Node workers through it, CmdExecUtil runs commands through it (the command code moved to ShellExecutor unchanged), and the 12 symlinkFolder/symlinkProject call sites use User.getSandbox().
  • Namespace jail environment: Python and Node jails now start with only PATH, LANG, LANGUAGE, LC_ALL, TZ and SANDBOX_ENV_ALLOWLIST, not the server's whole environment. New keys: SANDBOX_ENV_ALLOWLIST, NAMESPACE_SCRUB_ENV (default true), NAMESPACE_EXEC_FALLBACK (default fakechroot).
  • Package moves (renames only): OpenShellTransport and OpenShellTransporter to prerna.util.openshell; SandboxTransporter and SandboxInjector to prerna.util.sandbox. SymlinkHelper.isLeftOutOfCopy and copyProjectForReadOnlyUser became public because the transport now calls them from another package.
  • Access changes in OpenShell mode: a root whose access changes in either direction (view to edit, or edit to view) is cleared in the sandbox, with its stamps, before the new copy is sent. A user who loses edit access no longer keeps .git, portals and classes in the sandbox, and a file made while they could only view is not written back as a new file after promotion. Folders and projects are also placed in the local chroot after the remote transport, because the Java worker has no OpenShell sandbox and runs in the local fakechroot.
  • Failed namespace starts: a worker that never became ready is now stopped and its folders removed before spawnWorker throws. Before, nothing owned it once the exception was thrown.
  • OpenShell CLI in the images: Dockerfile.tomcat installs the OpenShell CLI with NVIDIA's installer, and each runtime Dockerfile copies /usr/bin/openshell from it. The gateway is not run in these images.
  • Removed the OpenShell session lifecycle test harness.
  • Small fixes: OpenShellShellRunner pins HOME to OPENSHELL_CLI_HOME; the Redis registry records the effective worker port; OPENSHELL_CLI has one default; stale comments.

How to Test

  1. Compile: mvn -q clean compile -DskipTests -Denforcer.skip=true -Dlicense.skip=true -Djacoco.skip=true
  2. Per backend, drive Monolith + Tomcat: a Python cell, BashCommand, open a project you can edit, open a project you can only view.
  3. OpenShell: confirm a view-only project has no java/, classes/, portals/ or .git inside the sandbox.
  4. Namespace (Linux image): confirm the Python worker starts with the scrubbed environment and imports what it needs. If something is missing, add it to SANDBOX_ENV_ALLOWLIST or set NAMESPACE_SCRUB_ENV=false.
  5. Check the log line Sandbox settings: ... appears at first jail use.

Notes

  • The access-change and Java-worker behavior was checked live on OpenShell. The rest is compiled only so far. The fakechroot and namespace paths have not been run on Linux yet, so that smoke test is needed before merge. The NAMESPACE_SCRUB_ENV default is the change most likely to need adjusting.
  • Not in this PR: agent CLI env removals (needs Python client changes), a namespace jail for one-shot commands (they run in fakechroot, or are refused), Node jails outside namespace mode, and moving shutdown and OpenShell write-back out of ClientProcessWrapper.
  • Outside this PR: the OpenShell gateway, its certificates and any Kubernetes manifests live outside these repos. The images only get the CLI.
  • A nono backend is on hold; Capabilities leaves room for it.

OpenShell becomes a fourth selectable SANDBOX_MODE alongside FAKECHROOT /
NAMESPACE / SECCOMP_ONLY. Nothing is retired: the existing modes and their
launch paths are untouched, so a deployment opts in by setting one property
and reverts the same way.

- Constants: SANDBOX_MODE_OPENSHELL = "OPENSHELL", documented as additive.
- OpenShellTransport: creates a named sandbox running the gaas worker as its
  canonical main process, forwards the worker port to a free local port over
  the gateway, and tears the sandbox down on close. Deterministic sandbox
  naming from a session key (SHA-256, DNS-1123 safe) so no registry lookup is
  needed. A sandbox that already exists is success, not an error.
- ClientProcessWrapper: one new branch in the existing dispatch, before the
  NAMESPACE branch. Sets no udsPath, so the existing socket client takes its
  normal TCP path to 127.0.0.1:<forwarded port> -- the socket layer is
  unchanged. Fails closed: never falls back to an in-Tomcat worker when the
  deployment asked for an external boundary.
- shutdown(): closes the forward and deletes the sandbox, since the worker
  cannot outlive its sandbox.

Verified: mvn compile clean. Existing modes behaviourally unchanged.
OPENSHELL_IMAGE (used when no OPENSHELL_TEMPLATE is set) and OPENSHELL_POLICY
(a filesystem/egress policy YAML applied at creation). The policy has to be set
at creation because Landlock settings are locked at sandbox startup and cannot
be widened afterwards -- that immutability is the control, not an inconvenience.

The stock OpenShell base image carries no python3, so an executor image is
required in practice; this makes it a deployment property rather than a
hard-coded default.
…t harness

Three fixes found by running the transport outside Tomcat:

1. FAIL LOUDLY on missing config. create() added --from/--policy only when the
   property was present, so an unset OPENSHELL_IMAGE silently fell back to the
   stock OpenShell base image -- which has no python3. The sandbox is still
   created and reaches Ready; only the workload fails to spawn, so the caller
   sees an unrelated null response several layers away. Both inputs now throw at
   construction. Same defect class as the AGENT_SANDBOX_ENABLE/ENFORCE mismatch:
   a control that is configured but never reaches enforcement, failing open.

2. Sandbox name must fit the gateway's 19-character limit (was 25).
   MAX_SANDBOX_NAME_LENGTH is now a named constant with its reason recorded; the
   digest is truncated to fit.

3. Wait for the port forward to actually accept before returning the port. The
   CLI binds lazily, so returning early produced a connection refused that
   surfaced as a null response. The readiness probe is a TCP connect, never a
   protocol exchange, because the worker is single-connection.

Also adds OpenShellTransportHarness: the same classpath with a console attached,
which is what made all three visible. It bootstraps the DIHelper from RDF_Map.prop
because Utility.getDIHelperProperty reads that, not -D system properties -- a
distinction that cost three debugging iterations.

Known open issue recorded in the wiki: the worker exits ~0.5s after start when it
is the sandbox's canonical main process, while the identical command run as a
backgrounded child stays up and listens. Under investigation; the transport itself
is correct up to and including the forward.
awaitForwardReady() connected to the forwarded port. The forward proxies straight
into the worker, the worker is single-connection (--max_count 1), so the probe
consumed its only slot and its disconnect called stop_request() -- ending the
worker and taking the sandbox to Completed. The symptom was indistinguishable
from "the workload never started", which is why image, policy, argv, HOME, I/O
pattern and environment breadth were all eliminated first.

Readiness now comes from the CLI's own output: forward service prints a line
containing "Forwarding" once its listener is bound. The method documents why it
must not be "simplified" back into a socket connect.

Also pins HOME for CLI children (OPENSHELL_CLI_HOME). The CLI resolves its
gateway registration and mTLS client bundle from $HOME/.local/state/openshell/,
so a caller with a thinner environment (Tomcat: 13 vars vs a shell's 124) could
silently reach a different or missing gateway. Found by running the CLI with
HOME unset, which reports "Or register one with: openshell gateway add".

Adds OPENSHELL_KEEP_SANDBOX so a failed run leaves the sandbox in place for
inspection -- deleting it destroyed the evidence every previous time.

VERIFIED: a sandbox created by this class served a working persistent REPL
through the protocol client (x=1 persists, mutates to 42, import math survives).
The forward's exit reason was being discarded: awaitForwardReady returned false
without recording why, and close() then deleted the sandbox, so the evidence went
with it. Both paths now log the CLI output and the process exit code before
cleanup.

Also records two measured facts in comments so they are not re-derived:
 - The forwarded port IS listening once start() returns (verified), so a
   connection failure is not a readiness problem.
 - The worker's insight/log.txt captures every request it receives, which is the
   only reliable way to tell "never arrived" from "arrived and failed" -- the
   sandbox phase and the supervisor log both say nothing about the workload.

No behavioural change to the transport path.
…harness

run() read the CLI child with a blocking readLine() loop and no timeout. Because
close() also calls run() to delete the sandbox, a wedged child parked the Tomcat
request thread inside the failure path itself: the sandbox was never cleaned up
and the call never returned.
  BufferedReader.readLine -> run(:459) -> close(:399)

Now the read happens on a daemon thread while the caller waits on waitFor(timeout);
on timeout the child is destroyed forcibly and the reader interrupted. A cleanup
path must not be able to block indefinitely on the object it is cleaning up.

Adds OpenShellNativeClientHarness, which wires the transport to
NativePySocketClient -- the class that actually ships -- with no protocol code of
its own. Verified: transport up on 38331, NativePySocketClient connected,
ready=true. Debugging through a hand-rolled protocol harness had been mistaking a
failing test client for a failing transport.
…ient

The harness previously only proved the handshake. It now sends OPERATION.PYTHON and reads the response, which is what PyTranslator does, so the chain is verified against shipping code rather than test scaffolding. Also corrects the PayloadStruct import (prerna.tcp, not prerna.om) and replaces a port probe with a plain sleep, since probing the forwarded port consumes the single-connection worker's only slot.
… binding

The forward binds in ~60ms but the worker needs ~5s to reach serve_forever(). A client connecting in that window completes the TCP handshake and then consumes the single-connection worker only slot, wedging it. Measured: connect at +0s -> no reply; connect at +5s -> response=true. Readiness now waits for the supervisor NET:OPEN 127.0.0.1:<port>/tcp line, which needs no connection to the worker. Removes the temporary diagnostics.
1. Readiness is taken from the worker own log line, never by connecting to the port: the worker is single-connection (--max_count 1), so a probe consumes its only slot and its disconnect wedges it. The forward Forwarding line binds in ~60ms and is not readiness. 2. The worker stdout/stderr goes to a file instead of the supervisor pipe, which made startup erratic (sometimes never binding while the sandbox still read Ready). 3. run() now redirects stdin from /dev/null: openshell sandbox exec reads stdin, so with ProcessBuilder inherited open pipe the CLI blocked until the 30s timeout, and three of those produced the 180s pixel stall. The same command returns instantly from a shell, which is why this only showed up under Tomcat. Verified: Py("print("LIVE_OK")") returns LIVE_OK at 1599ms; repeats 1011/4/4ms; os.getpid()=60 vs Tomcat JVM 4059545.
createProcessAndClient is shared by the user REPL and engine-owned workers. Branching on SANDBOX_MODE alone sandboxed the model engine too, so AskRoom failed with ModuleNotFoundError: No module named openai several layers from the cause: AskRoomReactor -> AbstractModelEngine.askRoom -> AbstractPythonModelEngine.startServer -> PyTranslator. That read as a missing package but was a scope error. engineOwned already distinguishes them: User.pythonCPW and Project.cpw are user-authored and untrusted and get the sandbox; model engines, vector databases, guardrails and LocalPythonFunctionEngine are our own code calling controlled SDKs and stay on the Tomcat side. Verified: AskRoom now reaches the OpenAI API (401 on the engine own key, no import error, no sandbox) and Py() still returns pid 60 against a Tomcat JVM pid of 21788 with exactly one sandbox created.
Replaces the !engineOwned guard in ClientProcessWrapper with the branch the other jails already use. The jail has always been chosen by which caller supplies the SymlinkHelper: User.startPythonSocketServerAndClient passes getUserSymlinkHelper() for user-authored python, while engine-owned workers and Project pass null. OPENSHELL is now simply the third branch of that chain, alongside NAMESPACE and FAKECHROOT, and SymlinkHelper resolves SANDBOX_MODE once into a SandboxMode enum. ClientProcessWrapper no longer re-reads the mode at all -- it is a property of the user jail, so it belongs to the object that only exists for a user worker. Verified: AskRoom on an Ollama engine answers (Hello, hi, hey, 1028ms, no sandbox) and Py() still runs at pid 60 with exactly one sandbox.
…lper

Every jail hangs off SymlinkHelper, because all three answer the same
question: this host path must be usable by the worker.

  FAKECHROOT  builds a chroot tree (symlinks on this host)
  NAMESPACE   adds the path to the jail via SandboxInjector (control socket)
  OPENSHELL   must COPY the tree, because a sandbox has its own filesystem

Making a path visible is not making it present.

SandboxTransporter is the OPENSHELL counterpart of SandboxInjector: wired by
ClientProcessWrapper at the same point, called through a transportOrEnqueue
dispatcher from symlinkFolder/symlinkProject so every invocation site is
covered without editing any of them. activeTransports makes it once per
sandbox, and setTransport replays paths registered before the worker existed,
mirroring setInjector's replay of activeInjects.

Two bugs found while verifying:

1. `sandbox upload <src> <dst>` nests the basename, so passing dst == src
   produced app_root/app_root/version/... and the project never landed where
   the worker looks. Destination must be the parent.

2. The transporter needs the same isNamespaceInjectablePath guard the injector
   has. MCPUtility also symlinks git's own directories, so /usr/bin, /etc/ssl
   and /usr/lib/git-core were being uploaded -- several 30s CLI timeouts and a
   77s pixel. Now 8.4s.

Verified: RunMCPTool now finds and executes the project's mcp_driver.py. It
fails on the project's own `import yfinance`, which is absent from both the
host venv and py/install_config/pyproject.toml and would fail under NAMESPACE
too -- a missing declared dependency, not a visibility problem.

Write-back (FETCH_BACK) is NOT implemented: files written inside the sandbox
are lost at teardown. That asymmetry is worse than no transport at all, since
the sandbox can now read project files while silently discarding writes.
…aper

Two additions, both in their own package prerna.util.openshell. No existing
class is modified.

OpenShellSessionLifecycleHarness — proves the lifecycle the product relies on,
without booting Tomcat: no sandbox exists before first use; the first
user-authored python request creates it on demand (13.4s cold start); Py()
executes inside it through the shipping NativePySocketClient (pid 60, python
3.12.14); the exact logout call -- ClientProcessWrapper.shutdown(true), as
issued by UserSessionLoader.sessionDestroyed -- deletes it (385ms); and a later
session starts a fresh one. 11/11 checks.

OpenShellSandboxReaper — the backstop for sessions that never log out (JVM
killed, container restart, abandoned session). Lists only sandboxes carrying
the executor label AND the pyw- prefix, deletes only those older than a
threshold, skips unparseable timestamps rather than guessing. Verified against
a labelled throwaway: dry-run listed it, the real run deleted it, and an
unlabelled sandbox on the same gateway was untouched.
Found live, in the UI path: if the named sandbox already exists, start() logged
the create error and carried on. The worker inside allows one connection and the
forward that reached it first still holds that slot, so the second forward leaves
the real client queued -- Py() never returned and the logout thread sat in
clearInsightGlobals until the sandbox was deleted by hand.

A sandbox this transport is not driving belongs to a dead session, so the correct
reading of 'already exists' is delete and recreate, not attach.
CmdExecUtil ran bash with commons-exec in a chroot on Tomcat's machine and never
consulted SANDBOX_MODE, so with SANDBOX_MODE=OPENSHELL the terminal, git
operations and the agent tool handlers all stayed on the host. Measured before
this change: Command("hostname") returned the host identity while Py() returned
the sandbox's.

OpenShellShellRunner runs the same command with 'openshell sandbox exec', which
starts a process inside the sandbox. Deliberately not the worker socket: the
worker serves one connection and a shell command must not compete with the REPL
for it. The sandbox is started on demand, so a shell command can be the first
thing a session does.

One delegation point in CmdExecUtil, taken only when the user's SymlinkHelper
reports an external sandbox; every other mode keeps the existing path untouched.

After: hostname 936b273a023c, uid 1000, ls of the sandbox tree -- host is
kunal0137-vnet.
Before this, the sandbox was write-only: a file created by Py() or a shell
command lived in the sandbox copy and died with it, so the asset browser, git
and MCP generation never saw it. Measured: to_csv into the project assets ->
visible to 'ls' in the terminal, absent from GetAppAssets, gone at teardown.

Now, after the terminal response of an executeCommand, the wrapper reconciles
the session's transported roots back onto the host:

- OpenShellTransport.fetchBack streams the tree out with 'sandbox exec -- tar -cf -'
  (sandbox download is workspace-scoped and refuses project paths), validates the
  archive (relative paths only, size cap), extracts to a temp dir and reconciles
  per file: content must differ AND the sandbox copy must be newer, so a stale
  sandbox cannot clobber a host edit.
- SymlinkHelper exposes the roots it transported; ClientProcessWrapper filters to
  project/room/user roots and throttles to one reconcile per 1.5s.
- NativePySocketClient calls it once per completed request.

Verified: Py writes iris_fb.csv in the sandbox -> host file appears (16 bytes) ->
GetAppAssets returns its contents. Single pod only: the push to central storage
for the other pods is the next step.
…e teardown

Three gaps found by testing, all in the fetch-back added in 863c3f443d:

- The 1.5s throttle SKIPPED the reconcile instead of deferring it. Three Py
  blocks in one session left two files stranded in the sandbox until some later
  call. Skipping is wrong for a durable-write path; the throttle is gone.
- Shell commands do not travel over the python socket, so "cat > file" or a git
  operation in the terminal never hit the hook. OpenShellShellRunner now
  reconciles after the command runs.
- The last block before teardown could still be lost; shutdown() now reconciles
  once before deleting the sandbox.

Also stopped excluding .git from the fetched archive. It costs bytes and writes
nothing today -- the probe image has no git (git: not found) so the sandbox copy
never changes -- but when the production image ships git, a commit made in the
shell is exactly the work that must come back. Concurrent host/sandbox commits
still need the revision guard.

Verified: three Py writes plus one shell write in a single session all appear on
the host.
Write-back used to tar every transported root at every execution boundary:
~190 ms per root, so `LoadApp(a); LoadApp(b); LoadApp(c)` paid ~750 ms per
block whether or not anything changed. The worker now knows which paths
changed and the host fetches those.

Worker side (py/):
- new gaas_change_watcher.py: ctypes inotify (no dependency), coalesced
  per-path events, 2k cap, IN_Q_OVERFLOW flag, and a baseline flag. A new
  directory is armed *and scanned*, because `mkdir x; write x/f` inside one
  block otherwise reports the directory and loses the file.
- handler gains a CHANGES operation returning
  {changes[], overflow, baseline, watched, events, roots}.
- the watcher is a module-level singleton created lazily on the first
  CHANGES call: the TCP handler is built per connection, and at setup() the
  session's project tree has not been materialised yet, so arming there
  yields zero roots and pins the host to the slow path forever.

Host side (Java):
- PayloadStruct: CHANGES operation.
- NativePySocketClient.getFileChanges(); the after-execution hook now fires
  only for PYTHON and CMD, so PollNotifications and other internal polls no
  longer tar the tree.
- OpenShellTransport.fetchBackPaths(): one tar of the named paths, with the
  same traversal and size checks as the whole-root fetch.
- ClientProcessWrapper.fetchBackChangedPaths(): groups paths by transported
  root, deletes reported removals, and falls back to the full reconcile on
  baseline, overflow, or no answer.
- OpenShellShellRunner takes the same path for shell commands.

Verified on a 240-file project: three blocks, three unique files, all on the
host; journal shows "Fetch-back wrote 1 file(s) from 1 changed path(s)";
watcher counters went watched:173 baseline:true -> changes:[x.txt] events:2
with root inodes unchanged from arm time. Cost 0.20 s/block vs ~0.30 s for the
full reconcile; the first block after materialise still does one full
reconcile, by design.
…a host edit

Four correctness defects found by external review and a seven-model council,
all in the change-set fetch-back path:

1. reconcile() trusted mtime. A sandbox file whose content differed but whose
   mtime was older than the host copy was silently dropped (os.utime, cp -p,
   tar extraction). Content is now the authority; mtime only decides whether
   the host copy must be preserved first.

2. Reported deletes were applied unconditionally, destroying a concurrent
   host edit. Deletes now quarantine the host copy first, and are suppressed
   entirely if that copy cannot be made.

3. A full-root fallback carried survivors but no tombstones, so sandbox
   deletions were silently resurrected. Full captures now quarantine host
   files absent from the capture; directories the capture excludes
   (node_modules, __pycache__, .venv) are skipped, since "not in the archive"
   there means "never captured", not "deleted".

4. inotify cannot see a persistent mmap write. The watcher now also compares
   one stat tuple (mtime_ns, size, ino, ctime_ns) per file at each checkpoint,
   reported as "stat_scan". Measured: ~6 ms for 240 files, silent on a quiet
   block, catches mmap writes and deletions with no events at all.

Verified live in a sandbox: backdated rewrite, mmap write, and delete all
reached the host; the overwritten host version and the deleted file were both
preserved under /opt/semosshome/writeback-quarantine/<stamp>/.
… folders

Watch descriptors follow inodes, so renaming a watched directory left every
descendant path stale: events arrived under the old name and were written back
to the wrong place. The watcher now treats IN_MOVED_FROM, IN_MOVE_SELF,
IN_IGNORED and IN_UNMOUNT as lost coverage, reports `coverage: false` with
`baseline: true`, drops the tuple index and re-arms from the roots. The host
falls back to a full capture for that one block and resumes the change-set path
afterwards.

Java also treats `coverage: false` as a fallback trigger on its own, so a future
worker that reports it without `baseline` cannot leave the host trusting a
partial list.

Deleting a file now also prunes directories it left empty, up to but never
including the transported root - an empty folder left behind after a rename
reads as corruption.

Verified live: rename block took the full path ("wrote 2 file(s)", "moved 1
deleted file(s)"), the next block returned to the change set ("1 changed
path"), and the post-rename write reached the host.
The stat-tuple scan ran on every CHANGES call, so an idle block on a 100k-file
tree paid ~0.57 s for a scan that could not have found anything. Measured:
240 files 1-2 ms, 10k 54 ms, 50k 285 ms, 100k 570 ms - linear at ~5.7 us/file.

The scan is now due on the first call, after a lost baseline or coverage, and
otherwise at most once per interval. The interval is max(2 s, 20 x the last
measured duration), so the backstop holds roughly a 5% duty cycle at any size:
2 s at 240 files, ~10 s at 100k. Skipping never drops a change - the index is
untouched, so the next scan reports everything accumulated; normal writes are
unaffected entirely because they arrive as events.

Measured at 100k: idle block 0.01 ms (was ~0.49 s), scan 0.486 s when due,
interval self-tuned to 9.7 s. Payload reports stat_scan and scan_skipped.
Measured on a real project: .git was 112 directories and 137 of 256 files - over
half the watch set and over half of every scan - for data that cannot change,
since the sandbox image has no git binary. node_modules was only excluded from
the scan and the tar, so it was still watched: one npm install adds thousands of
directories, exhausts fs.inotify.max_user_watches and floods the queue, turning
a cheap block into a full capture.

One ignore set now covers arming, adoption, scanning and pull-back:
node_modules, __pycache__, .venv, .git. Directories created mid-block are
adopted only if they are not in that set, so an install inside a block cannot
smuggle the tree back in. Override with SEMOSS_SYNC_IGNORE.

Synthetic tree shaped like the real one (20 assets, 112-dir .git, 200-dir
node_modules): watches 318 -> 4, indexed files 332 -> 20, asset writes still
detected, node_modules writes unreported by design.
The sandbox runs outside the platform's permission checks, so any member who
could open a project could write a file with Py() and have fetch-back publish it
into the project folder - a read-only member silently gaining write access.

ClientProcessWrapper now binds the owning User (set at client creation in
User.java) and checks it before any root is pulled back: the project id is taken
from the root folder (`<name>__<uuid>`, or a bare uuid), and
SecurityProjectUtils.getActualUserProjectPermission must answer OWNER or EDIT.
Anything else is skipped with "Refusing write-back into ... has READ_ONLY access
to project ...", and an unresolvable permission fails closed.

The check runs in both paths - the change-set pull and the full fallback - so a
baseline reconcile cannot bypass it either.

Verified live: with kunal0137 downgraded to READ_ONLY on workbench_1, a Py block
wrote 29 bytes inside the sandbox, the host file did not appear and the refusal
was logged; restoring OWNER made the same write land on the host.

Rooms and user assets are NOT gated: this build has no room permission model,
and asset roots are the user's own. If rooms are meant to inherit project
access, that rule needs stating before it can be enforced.
…ll-through

Write-back now decides every root by its own space instead of special-casing
rooms and letting everything else through:

- user assets: the folder is Asset__<id> where the id must equal one of the
  caller's own asset project ids (User.getAssetProjectId(provider), backed by
  ASSETENGINE keyed on userid) - anything else under user/ is somebody
  else's and is refused;
- rooms: ROOM.USER_ID in the model-inference-logs database via
  ModelInferenceLogsUtils.doCheckRoomExistsForUser;
- projects: OWNER/EDIT as before;
- any unrecognised space: refused, where the old code returned true.

Every refusal is logged with the path and the reason, and each check fails
closed on an exception.

Verified: with the deployed class carrying isOwnUserAsset, a Py block writing
29 bytes into an owned project root still reached the host ("Fetch-back wrote
1 file(s)"). The asset/room denial paths are identity-keyed - this instance has
one human user per asset project and only kunal0137 owns rooms, so a foreign
root cannot be transported into a session to exercise the refusal.
…from py/

The executor image was assembled outside this repo: executor-probe/py was a hand
copy of py/, and executor-proto just layered on top of it. That copy is how the
image kept running month-old python while the repo looked fixed.

docker/Dockerfile.executor replaces the two-image dance with one stage whose build
context is py/, so the image carries exactly the committed worker. It keeps what
matters from both predecessors: python:3.12-slim, git with a system identity (local
repo work only - egress is still per-binary in the OpenShell policy), pandas +
jsonpickle for the boot-time lazy imports, the insight/grants layout, uid 1000
ownership of /opt/semosshome, and the writable-workspace USER the sandbox requires.

docker/build-executor-image.sh is the one command: it labels the tag with a hash of
the python tree (org.semoss.py-sha) so `docker inspect` answers "which source is
this", then smoke-tests the result inside the image.

Built as semoss-executor:0.1 from py sha 1701eb4f4006: git 2.47.3, imports ok, worker
present, watcher ignore set intact, id 1000. The worker boots as the main process and
listens on 7100. RDF_Map.prop still points at semoss-executor-proto:0.2; switching is
OPENSHELL_IMAGE + restart.
docs/deployment/executor_sandbox.md covers the opt-in SANDBOX_MODE=OPENSHELL path:
the process model and session lifecycle, the per-block write-back pipeline (CHANGES
verb, targeted tar, content-authoritative reconcile, quarantine, tombstones), the
two change detectors and why the stat-tuple scan exists at all, per-space write
permissions, the repo-owned image build, configuration keys, a verification recipe
and the known limits (2000-path cap, racing mmap writers, no host->sandbox refresh).

Linked from the Deployment and operations section of docs/README.md.
The first full build came out at 22.4 GB. Cause: flattening pyproject.toml to a
requirements list dropped [tool.uv.sources], which is what maps torch/torchvision/
torchaudio to the pytorch-cpu index. PyPI then resolved CUDA builds - 3.3 GB of
nvidia plus 815 MB of triton inside an image that never sees a GPU.

export_requirements.py now skips the torch trio by default (--keep-torch to opt back
in) so the Dockerfile can install it from the right index first, and it emits
[tool.uv] override-dependencies as ordinary requirements so the sandbox keeps the
platform's security pins (marshmallow, protobuf, numpy, pyasn1, lxml).

Dockerfile.executor-full installs torch from download.pytorch.org/whl/cpu for the cpu
extra and /cu126 for gpu, then the remaining declared dependencies from PyPI.
Shows the per-session shape — Tomcat, gateway, supervisor container, workload container with
network=none, PID 1 sandbox runtime, the python worker reached through openshell forward — so
a reader can see why a worker connection travels four hops instead of being a local socket.
The script that shipped with the first Dockerfile only ever built the slim image; the
MODE=full path (staging pyproject.toml + export_requirements.py into the build context,
passing EXTRAS) was written but never committed, and was lost to a checkout. Restored and
committed so the full-image build is reproducible from the repo, not from shell history.
… venv

The sandbox's PATH belongs to OpenShell, not the image. Measured inside a live
sandbox: PATH=/usr/local/bin:/usr/bin:/bin, so an image's ENV PATH=/opt/venv/bin is
ignored, `python3` resolves to the base interpreter, and the worker died at boot with
no pandas/jsonpickle. A symlink /usr/local/bin/python3 -> /opt/venv/bin/python does not
help either: CPython looks for pyvenv.cfg relative to the invoked path and silently
runs as the base interpreter.

Dependencies now install with `uv pip install --system` and the final stage copies
/usr/local, so whatever `python3` the sandbox resolves to already has them. No venv, no
PATH dependency, and the image is smaller (3.7 GB vs 6.9 GB) because the venv duplicated
the interpreter.

The smoke test now runs the parity check under `env -i PATH=/usr/local/bin:/usr/bin:/bin`
- the sandbox's real PATH - instead of docker run's defaults, which is why the previous
build reported success while failing at worker boot.

Verified in a live OpenShell sandbox: python3 -> /usr/local/bin/python3, and
numpy/boto3/pandas all import.
Logging out of the last provider drops its token before the session is
destroyed. sessionDestroyed then threw in getSingleLogginName, which
skipped every cleanup step, and even without the throw the owner could no
longer be derived from the user, so the lease was never released.

- getSingleLogginName returns "" when the primary login's token is gone
- sessionEnded takes the owner; ownerForSession reads it from the session
  (recorded by the filter) and falls back to the user's login
…emosshome

Files cross into a sandbox at their host path, which is what keeps the
python SEMOSS generates (MCP loaders, PySource, the pandas JSON hand-off)
working, since it embeds absolute host paths. A Mac's BaseFolder is
somewhere like /Users/me/Documents/semoss, which no image has. The
sandbox can only create it under a root that exists at start and that
the policy grants: uploads run as the sandbox user inside Landlock.

- Dockerfile.sandbox makes /Users and /home, owned by uid 1000; at
  create time the policy grants the top folder of BaseFolder's path, and
  the transport makes the home inside the sandbox. One generic image
  serves every developer; nothing is built per machine.
- The policy is rendered from a template bundled in the jar (or
  OPENSHELL_POLICY), with ${SEMOSS_HOME_ROOT} and ${PYTHON} filled in.
  OPENSHELL_POLICY is no longer required, so a checkout needs no policy
  file from outside the repo. Egress is pypi only.
- Dockerfile.sandbox moves uv's CPython to /usr/lib/python/cpython, so
  the egress rule names /usr/lib/python/cpython/bin/python3.14 on every
  patch version and architecture; the smoke test checks it.
- Sandbox creation fails with the cause in the message when the home
  cannot be made, instead of the bare upload error.
- OPENSHELL_IMAGE defaults to the published
  quay.io/semoss/semoss-sandbox:dev-3.14-cpu, so a deployment that
  tracks dev sets neither the image nor the policy; build-sandbox-image.sh
  is gone, and the doc keeps the one docker build line for trying a py/
  change before it is published
- OPENSHELL_EGRESS: host[:port] list rendered into the bundled policy's
  network_policies for the sandbox interpreter; default pypi, "none" for
  no egress; entries are validated as plain hostnames
- OPENSHELL_CPU / OPENSHELL_MEMORY: per-sandbox caps passed to sandbox
  create, default 2 CPUs and 4Gi, "none" for no cap; caps not
  reservations, so idle sandboxes (~130 MB worker) cost little
- the policy grants /dev/shm: multiprocessing, joblib and torch data
  loaders failed with PermissionError on SemLock without it

Measured through the docker driver: caps land on the container
(NanoCpus/Memory), an allocation past the cap is killed, custom egress
allows only what is listed.
…rt guard

/dev/shm makes it work for processes in the sandbox; inside Py() the
worker's InsightGlobalStore guard removes os.fork/os.kill and bans
subprocess, so Pool fails there (measured on the live instance).
…roviders

A project declares SANDBOX_SECRETS (name + hosts) in its .smss and sets
SANDBOX_SECRET_<NAME>. When a user's sandbox loads the project, each
secret becomes an OpenShell profile + provider attached to that sandbox,
so code gets a placeholder (smss_secrets.secret(name)) that OpenShell's
supervisor swaps for the value only in requests to the declared hosts.
The .smss never enters the sandbox, so the value stays on the host.

- credential names carry the project key, so two apps may share a name
- the value reaches the CLI through the environment, never argv
- an attach is waited out, and the placeholder file is rewritten after
  every attach or update once the new generation appears; old
  generations stop resolving (attach) or keep the old value (update)
- an unchanged value is not re-applied, so later loads cost ~0.5 s

Measured against a real sandbox with a live OpenWeather key: 200 through
secret(), refused at another app's host, rotation reaches running code.
…P example

The old slim image pip-installed it as a stopgap for the example; the
python-builder venv does not declare it, so Stock MCP failed with
ModuleNotFoundError after the 3.14 switch. APP_PACKAGES keeps the list
explicit until apps can declare their own dependencies.
…Stock MCP example"

Skip yfinance as an image package for now; the Stock MCP example is
parked. This reverts commit 3076363.
# Conflicts:
#	src/prerna/auth/User.java
#	src/prerna/om/ClientProcessWrapper.java
#	src/prerna/util/SymlinkHelper.java
Add prerna.util.sandbox.SandboxSettings: every sandbox key with its default,
scope and description, plus the single SANDBOX_MODE parser. SymlinkHelper,
ClientProcessWrapper (node) and OpenShellUserSandboxes now use it instead of
three separate parsers. OpenShellTransport takes its defaults from the registry.
The effective settings are logged once at first jail use, with a warning when
SANDBOX_MODE is not a jail mode (OFF and SECCOMP_ONLY mean FAKECHROOT).

Fixes: OpenShellShellRunner now pins HOME to OPENSHELL_CLI_HOME like the
transport; the Redis registry records the effective worker port instead of
"null"; OPENSHELL_CLI has one default; stale Constants and SandboxLaunchPlan
comments.

Docs: docs/deployment/sandbox_configuration.md lists every key, which jail
you get, and what each jail's child process inherits.
…l backends

Add prerna.util.sandbox.ISandbox (grantFolder, grantProject, revoke, spawnWorker,
exec, capabilities) and SandboxFactory, the one place a mode becomes a backend.
Callers no longer check the mode:

- ClientProcessWrapper starts python, Java and node workers through
  ISandbox.spawnWorker instead of its per-mode if chain, and takes over the
  result with applyWorker.
- CmdExecUtil runs commands through ISandbox.exec; the command-running code moved
  to ShellExecutor unchanged. Commands in a namespace jail now say they run in
  fakechroot, or are refused with NAMESPACE_EXEC_FALLBACK=refuse. OpenShell
  commands through executeCommandWithStatus now go to the sandbox too.
- The 12 symlinkFolder/symlinkProject call sites use User.getSandbox().
- SymlinkHelper gains revoke (namespace mount and chroot link).

The namespace python and node jails now start with only PATH, LANG, LANGUAGE,
LC_ALL, TZ and SANDBOX_ENV_ALLOWLIST from the server environment instead of all of
it (NAMESPACE_SCRUB_ENV, default true).

Backends run a worker the way it always ran when they have no jail for it (Java in
fakechroot, node on the host outside the namespace jail), and log that once.
Shutdown and OpenShell write-back stay in ClientProcessWrapper.
…r packages

OpenShellTransport and OpenShellTransporter move to prerna.util.openshell, next to
the rest of OpenShell and the policy file the transport already loads from there.
SandboxTransporter and SandboxInjector move to prerna.util.sandbox, next to ISandbox
and the backends that use them. This also removes the circular dependency between
prerna.util and prerna.util.openshell.

Rename and import changes only. SymlinkHelper.isLeftOutOfCopy and
copyProjectForReadOnlyUser become public because the transport now calls them from
another package.
@snyk-io

snyk-io Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

✅ Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
✅ Open Source Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

kunal0137 and others added 7 commits November 17, 2026 13:49
…hanges, keep Java grants in OpenShell mode

- NamespaceSandbox: a worker that never became ready was owned by nobody once
  spawnWorker threw, because the wrapper only receives the handle on return. Stop
  the launcher and its children and remove its folders before throwing.
- OpenShell access changes: SymlinkHelper re-sent a root only when it went from
  view-only to edit, and OpenShellTransport only added and overwrote files. A user
  who lost edit access kept the full copy (.git, portals, classes) in the sandbox,
  and a file made while the user could only view was written back as a new file
  once they could edit. A root whose access changed in either direction is now
  cleared in the sandbox, with its stamps, before the new copy is sent.
- OpenShell mode now also places folders and projects in the local chroot after
  the remote transport. The Java worker has no OpenShell sandbox and runs in the
  local fakechroot, so it needs the same links and viewer copies as in other modes.

Checked live on OpenShell: a viewer's new files are not published after promotion
and logout; a promoted editor's write syncs within seconds; a demoted user's
sandbox loses .git and portals and its writes are refused; the chroot holds a
viewer copy or an editor link to match.
It is a manual dev harness that no longer compiles: DIHelper became
package-private in #3326 and ClientProcessWrapper.createProcessAndClient has a
different signature. Builds skip test compilation, so it went unnoticed, but
mvn test fails on the branch with it.
SANDBOX_MODE=OPENSHELL runs the openshell CLI from Tomcat. Take only the static
binary from the pinned, checksum-verified release .deb (amd64/arm64) in each
runtime Dockerfile, next to the copilot CLI. The gateway is not included.
OPENSHELL_VERSION=latest resolves the newest release at build time; a version
can still be passed to match a deployed gateway. The checksum check stays.
Dockerfile.tomcat runs the upstream install.sh; each runtime image copies
/usr/bin/openshell from its tomcat_builder stage instead of repeating the
download.
@kunal0137
kunal0137 marked this pull request as ready for review November 17, 2026 20:40
@kunal0137
kunal0137 requested a review from a team as a code owner November 17, 2026 20:40
@kunal0137
kunal0137 merged commit f89080a into dev Nov 17, 2026
5 checks passed
@kunal0137
kunal0137 deleted the feat/sandbox-abstraction branch November 17, 2026 20:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants