How user code runs when SEMOSS is configured with SANDBOX_MODE=OPENSHELL, what the sandbox
can and cannot change on the host, and how write-back decides what to copy.
This is an opt-in execution path. With SANDBOX_MODE unset or set to any other value, the
platform uses the traditional local child process and none of this applies.
Py() and shell commands are arbitrary code. Running them as a child of the JVM means a
process that shares the host's filesystem, environment and network namespace. The OpenShell
mode moves that process into a separate container, spawned and reaped with the user's
session, and keeps the platform's files authoritative on the host.
Tomcat (host) ──CLI──► openshell gateway (host, :17670, mTLS)
│ │ provisions
│ ▼
│ supervisor container (uid nobody, trusted side)
│ │ control.sock (unix socket on a docker volume)
│ ▼
│ workload container (network=none)
│ PID 1 /.openshell/runtime/openshell-sandbox
│ └── python worker (loopback :7100)
│ ▲
└── openshell forward service --target-port 7100 --local <port> ─┘
(long-lived child of Tomcat; Java connects to 127.0.0.1:<port>)
Per session there are two containers: the workload (your image) and its supervisor (NVIDIA's image).
The supervisor is the trusted half of the boundary and is never inside the workload. The workload has no
network interface at all — its only egress is the channel to its supervisor — so a TCP connection to the
worker has to travel Tomcat → forward → gateway → supervisor → runtime → loopback.
| Piece | Runs where | Notes |
|---|---|---|
| Java (Tomcat) | host | owns the socket client, decides when to sync |
gaas_tcp_socket_server.py |
sandbox, main process | the persistent REPL; one per user, shared by all their sessions |
| shell commands | sandbox | started by OpenShellShellRunner, same write-back rules |
| user assets / projects / rooms | host | copied in at session start, written back at block boundaries |
The worker is stateful: variables survive between blocks, which is why it cannot be
restarted to pick up a change. It runs with --persist: a client leaving (disconnect,
stop, logout) does not end it, it serves up to OPENSHELL_WORKER_MAX_CLIENTS connections
at once, and it exits by itself only after OPENSHELL_IDLE_EXIT_MINUTES with no client.
Each server instance holds one connection per user, shared by that user's sessions there.
OpenShellUserSandboxes keeps a registry of owner → sandbox and owner → live sessions. It uses
Redis when one is configured (REDIS_ENABLED, or the cluster Redis), and an in-memory map
otherwise. The Redis keys follow the room-coordination layout: openshell:{<owner>}:sandbox,
:sessions and :lock, all with the {owner} hash tag so one user's keys share a Redis Cluster
slot, plus an openshell:owners index. The create lock is SET NX PX with a random token and is
released by a compare-and-delete script, as room locks are. The in-memory map is only correct for a single server instance,
and it forgets the sandbox when that instance restarts.
- Created lazily on the first
Py()or shell command of a user who has no running sandbox. Creation is serialised per user with a registry lock, so two instances do not both create one. Any later session, cookie-less API call or server instance of that user attaches to it. - Kept alive by a lease: every authenticated request refreshes the session's entry for the session timeout, at most once a minute. Browsing without python counts.
- Deleted when the user's last live session ends, by logout or timeout. A server stopping destroys every session, but that is not the user leaving: the leases stand, and the restarted or another instance reattaches with the worker's variables intact.
- Swept every five minutes on each instance. That removes sandboxes whose owner has no live lease (a session lost with its server), labelled sandboxes that are no longer running, and, with Redis, running user sandboxes no registry entry names.
Every PYTHON and CMD block starts with a refresh (host to sandbox) and ends with a write-back
(sandbox to host). The write-back steps, in order:
- Ask for the change set.
NativePySocketClientsends aCHANGESrequest on the existing socket. The worker answers{changes[], overflow, baseline, coverage, watched, events, stat_scan}. - Fail closed on doubt. If the answer is missing,
overflowis true,baselineis true, orcoverageis false, the host falls back to a full capture of each root instead of trusting a partial list. A change set the host drained but could not apply is owed as a full capture on the next block, because the worker has already forgotten those paths. - Pull only what changed. Changed paths are tarred inside the sandbox under one
sandbox execand streamed to a temporary file on the host, stopping at a 256 MB cap. The archive is refused if its listing fails or names an absolute path or a..segment. - Reconcile. Identical content is skipped. Otherwise each side is checked against the
stamp (size and mtime, in seconds) recorded at the last sync, when both copies matched:
uploads and captures preserve mtime, so a side that no longer matches is the side that
changed.
- Only the sandbox changed: the file is written back.
- Only the host changed (another user, a cluster pull, the UI): nothing is written. The next refresh brings the host copy in. This is what keeps a full capture from putting a stale copy back.
- Both changed: the last write wins. The losing copy is kept in quarantine (the
sandbox's under
<root>.sandbox/), and the write is skipped if the host copy cannot be preserved.
- Deletes. Only a path the sandbox is known to have held — seen in a full capture or written back since — is ever deleted on the host, so a file created and removed within one block cannot take an unrelated host file with it. A host file that changed since the sandbox had it is never deleted. Deletes are quarantine-then-delete, suppressed if the copy fails, and directories they empty are pruned up to (never including) the transported root.
- Publish. In cluster mode, a project root that was written back is pushed to cluster storage and announced on the cluster synchronizer, at most every ten seconds per root and once more on teardown. Servers that have the project loaded pull it, and their sandboxes refresh from that before their next block.
Every host-side write and delete is contained: a path that leaves the root, or crosses a symlink on the way, is refused. Without that, a link committed into a writable project could redirect a write or a delete into another space.
Inside .git, only git's data is written back — objects/ (not objects/info/), refs/,
logs/, HEAD, index, packed-refs and a few state files — so a commit made in the sandbox
reaches the host. config, hooks/ and info/ are never written back: host-side git runs or
obeys them, so a sandbox able to write them could execute commands on the server.
The refresh before each block walks every transported root on the host. Files whose stamp
moved since the last sync are copied in with a single sandbox upload, which merges into the
existing tree and keeps mtimes. Files removed on the host are removed in the sandbox. More
than ~2000 changes re-upload the whole root. Any sandbox change still owed from a failed
write-back is applied first, so a refresh never overwrites unsynced work.
Quarantine lives at $BASE/writeback-quarantine/<timestamp>/<root>/<relative>, outside
project/, room/ and user/, so it is never pushed to object storage.
A full capture carries survivors, not tombstones. A path the sandbox held before and the
capture no longer contains was deleted there; a host file the sandbox never held — added on the
host mid-session, or never uploaded — is left alone. If tar reports anything it could not
read (a directory the user made unreadable, say), the capture is partial: files are written
back and no deletion is inferred.
Two mechanisms, deliberately:
- inotify is the primary source. Watch descriptors are per directory, re-armed when a
directory is created, and scanned on adoption so
mkdir x; write x/finside one block does not lose the file. - A stat-tuple scan (
mtime_ns,size,inode,ctime) is the correctness floor under it, because a persistentmmapwrite emits no event at all. It runs at most once per interval —max(2 s, 20 × the last measured scan)— so an idle block costs nothing at any tree size. Skipping never drops a change: the index is untouched, so the next scan reports the backlog.
Coverage loss is a first-class outcome. IN_MOVED_FROM, IN_MOVE_SELF, IN_IGNORED and
IN_UNMOUNT all mean the path map may be stale; so does a watch that could not be added
(max_user_watches exhausted, an unreadable directory), a directory the scan could not list,
or a watcher thread that has stopped. In each case the watcher reports coverage: false, drops
the tuple index and re-arms, and the host takes one full capture for that block. A scan that
could not list part of the tree never reports the unlisted files as deleted.
Roots are re-evaluated on every CHANGES request: a root that appears later in the session (a
room created mid-session, say) is armed and forces one baseline capture.
Excluded from watching and scanning: node_modules, __pycache__, .venv, .git.
.git is a large share of a typical project's directories; sandbox commits still reach the
host through the full capture. node_modules is one install away from exhausting
fs.inotify.max_user_watches. Override with SEMOSS_SYNC_IGNORE.
The worker runs outside the platform's permission checks, so the host decides per root:
| Root | Rule |
|---|---|
project/<name>__<uuid> |
SecurityProjectUtils.userCanEditProject — the same check that grants the project read-write |
room/<roomId> |
ModelInferenceLogsUtils.doCheckRoomExistsForUser — ROOM.USER_ID in the model-inference-logs database |
user/Asset__<id> |
the id must equal one of the caller's User.getAssetProjectId(provider) values |
| anything else | refused |
Every refusal is logged with the path and the reason, and each check fails closed on error. A read-only member can still write inside the sandbox — that is a scratch area — but the file never reaches the host.
One image, docker/Dockerfile.sandbox, serves every sandbox. Like the Tomcat images, it is a
plain ubuntu:24.04 with /usr/lib/python copied in from the python-builder image
(quay.io/semoss/python-builder:3.14-cpu, built by docker/Dockerfile.python from
py/install_config/pyproject.toml with uv). A Py() block therefore gets the same Python and
the same packages inside the sandbox as outside it. On top of that the image adds git, a uid
10001 user and this repository's py/ as the worker.
CI publishes it as quay.io/semoss/semoss-sandbox from
.github/workflows/sandbox-builder.yml. It rebuilds on any py/ change on dev/main and
after every successful Python Builder run. Each architecture is smoke-tested before CI publishes
a rolling tag such as dev-3.14-cpu and a tag with the SEMOSS version and UTC publish date.
Snapshot tags look like dev-3.14-cpu-6.1.0-SNAPSHOT-2026-11-18. With RELEASE=TRUE, CI
publishes a dated release tag such as main-3.14-cpu-6.1.0-2026-11-18 and a release alias
such as 3.14-cpu-6.1.0. Publishing the same branch and version again on the same UTC day
updates that day's tag. Use the manifest digest in OPENSHELL_IMAGE to pin an exact image.
OPENSHELL_IMAGE defaults to quay.io/semoss/semoss-sandbox:dev-3.14-cpu, so a deployment or
a workstation that tracks dev sets nothing. To try a py/ change before CI publishes it:
docker build -f docker/Dockerfile.sandbox -t semoss-sandbox:dev py # context is py/
docker/sandbox-smoke-test.sh semoss-sandbox:devand set OPENSHELL_IMAGE semoss-sandbox:dev. The build context is py/ itself, so a local
build carries the checkout's worker, uncommitted edits included. CI runs the same
docker/sandbox-smoke-test.sh, which fails unless the worker tree in the image hashes to the
source py/, the venv python imports the worker under a bare PATH, the interpreter is at the
path the policy names, git is present, the image runs as uid 10001 and can write the directories
a sandbox writes.
The image runs as uid/gid 10001. That is the one value every driver agrees on: the Kubernetes driver
forces runAsUser/runAsGroup/fsGroup 10001 and ignores the image's USER, while Docker and
Podman run the image's USER, so declaring 10001 there gives a workstation and a cluster the same
identity. Every directory a sandbox writes must be owned by it; a directory owned by another uid
works on a workstation only if that uid happens to match, and fails on Kubernetes with
Permission denied.
The venv lives at /usr/lib/python/semossvenv and the worker is launched by absolute path, because
both the environment and the filesystem inside a sandbox belong to the runtime:
PATH: a workload starts with/usr/local/bin:/usr/bin:/bin, so an image'sENV PATHis never applied and a barepython3resolves to the base interpreter. Symlinkingpython3at the venv does not help either — CPython looks forpyvenv.cfgrelative to the invoked path and silently falls back to the base interpreter.- Filesystem: the policy allowlists
/usr,/lib,/etc,/bin,/sbin,/procread-only, plus/sandbox,/tmp,/opt/semosshomeand the platform's SEMOSS home read-write. A venv at/opt/venvis outside every one of those and dies withPermissionError: pyvenv.cfg;/usr/lib/pythonis inside/usr, which is exactly why upstream puts it there. - Egress:
OPENSHELL_EGRESSlists the hosts sandboxed code may reach (default pypi, sopip installworks). A network rule names the kernel-reported executable, so the list is granted to the interpreter's real path. uv's directory carries the patch version and the CPU (cpython-3.14.8-linux-aarch64-gnu), so the image moves it to/usr/lib/python/cpythonand leaves the old name as a symlink: the rule is/usr/lib/python/cpython/bin/python3.14on every build and architecture.
So OpenShellTransport resolves /usr/lib/python/semossvenv/bin/python3 (falling back to
command -v python3 for images without a venv) and creates the sandbox with
--env PATH=/usr/lib/python/semossvenv/bin:… so user code gets the same interpreter.
Smoke tests must reproduce both constraints, or they pass while the worker fails at boot.
At session start the launcher also uploads the host's /opt/semosshome/py over the image's
copy, but the worker's first imports happen before that upload, so the baked copy still has to
be current — a stale image shows up as code that ignores your latest change.
The policy is generated, not hand-kept. OpenShellTransport reads OPENSHELL_POLICY if it is
set, otherwise the template bundled in the jar
(src/prerna/util/openshell/sandbox-policy.yaml), replaces ${SEMOSS_HOME_ROOT} with the top folder of BaseFolder (see below)
and ${PYTHON} with OPENSHELL_PYTHON_BINARY, and creates the sandbox with the result (a file
in the temp directory named by its content). On a server the home is the image's own
/opt/semosshome, and that line is dropped. A custom policy may use the same
placeholders.
Set in RDF_Map.prop, applied to every sandbox when it is created (an existing user sandbox keeps
what it was created with until it is replaced):
OPENSHELL_EGRESS pypi.org, files.pythonhosted.org, query1.finance.yahoo.com:443
OPENSHELL_CPU 2
OPENSHELL_MEMORY 4Gi
OPENSHELL_EGRESSishost[:port]separated by commas or spaces, port 443 when omitted. Unset means pypi only;nonemeans no egress. Every host is granted to the sandbox interpreter; other programs (git,curl) get no network. Entries are validated as plain hostnames, so the setting cannot inject policy. It is rendered through the${NETWORK_POLICIES}placeholder, so a customOPENSHELL_POLICYwithout it keeps its own rules.OPENSHELL_CPUandOPENSHELL_MEMORYare caps, not reservations: an idle sandbox costs little (the worker is about 130 MB with pandas loaded), so the sum of the caps may exceed the host.noneremoves a cap. Defaults are 2 CPUs and 4 GiB.- Past the memory cap the kernel kills the largest process in the sandbox. A
Py()block runs in the worker, so an over-allocating block is expected to take the worker down and the user's next block to start a fresh python: variables lost, files kept. (Measured so far only for a separate process, which was killed while the sandbox stayed up.) - Measured through the docker driver: the caps land on the container
(
NanoCpus,Memory), and an allocation is killed at the memory cap. /dev/shmis granted so POSIX semaphores work for processes in the sandbox (a shell command running a script that usesmultiprocessing). InsidePy()itself, multiprocessing still fails: the worker's import guard (InsightGlobalStore) banssubprocessandsocketand deletesos.fork,os.kill,os.systemand theexec*/spawn*family. That guard predates OpenShell; whether to relax it now that the sandbox is the boundary is open.
A project can give its sandboxed code a secret the code can use but never read. In the project
.smss (project owners edit it):
SANDBOX_SECRETS [{"name":"OPENWEATHER_API_KEY","hosts":["api.openweathermap.org"]}]
SANDBOX_SECRET_OPENWEATHER_API_KEY <the value>
and in python:
from smss_secrets import secret
requests.get(url, params={"appid": secret("OPENWEATHER_API_KEY")})- The
.smssnever enters a sandbox, so the value stays on the host. Each secret becomes one OpenShell provider; code gets a placeholder, and OpenShell's supervisor swaps it for the value in requests to the declared hosts (headers and query string verified) and refuses it anywhere else (403 credential_endpoint_mismatch, logged as a HIGH finding). Any request method is allowed. HTTP(S) only: a database password cannot be brokered. - Brokered when a user's sandbox first loads the project (
OpenShellSecrets, called from the transporter for a project root). The first load waits for the attach to apply (~10 s). - Names are per project: two apps may both declare
OPENAI_API_KEY; with both loaded,secret(name, project=<id>)picks one. - A changed value is applied on the next load of the project (an edited
.smssneeds a project reload) and reaches code that is already running:secret()re-reads the placeholders, which OpenShell replaces on every update and attach. - The value property can later come from the secret store, which overlays the same properties.
- All code in one user's sandbox runs as that user, so any app the user has loaded can use the placeholders of another app they have loaded, at that app's hosts. Isolation is per user.
Files cross into a sandbox at the path they have on the host: a project uploaded from
/Users/me/Documents/semoss/project/x__id is at that same path inside the sandbox. That is what
keeps the python SEMOSS generates working unchanged: MCP tool loaders, PySource and the pandas
JSON hand-off all embed absolute host paths. Translating paths instead would mean finding and
rewriting every such string.
Nothing has to be built per developer. The home is taken from BaseFolder at runtime:
- the policy grants the top folder of its path (
/Usersfor/Users/me/Documents/semoss,/homefor/home/me/semoss), and - when the sandbox starts,
OpenShellTransportcreates the home under it as the sandbox user.
Both steps need that root to exist in the image when the sandbox starts, writable by uid 10001:
uploads run as the sandbox user inside the filesystem policy (measured: ssh tar extract exited with status 1 for a path the image did not have), and the policy can only grant a path that
exists. docker/Dockerfile.sandbox makes /Users and /home for this. A home under any other
root needs that root added the same way, for example:
FROM semoss-sandbox:dev
USER root
RUN mkdir -p /data && chown 10001:10001 /data
USER 10001:10001If the root is missing, sandbox creation fails with a message saying so, rather than with the upload error.
All sandbox settings, including the other modes, are listed in sandbox_configuration.md.
| Key | Meaning |
|---|---|
SANDBOX_MODE |
OPENSHELL enables this path (requires CHROOT_ENABLE=true, like every jail) |
OPENSHELL_IMAGE |
image a new sandbox starts from (default quay.io/semoss/semoss-sandbox:dev-3.14-cpu) |
OPENSHELL_TEMPLATE |
workload template to create from instead of an image |
OPENSHELL_POLICY |
policy template to render instead of the bundled one (see Policy); optional |
OPENSHELL_EGRESS |
hosts sandboxed code may reach, host[:port] list (default pypi; none for no egress) |
OPENSHELL_CPU |
CPU cap per sandbox, e.g. 2, 1.5, 500m (default 2; none for no cap) |
OPENSHELL_MEMORY |
memory cap per sandbox, e.g. 4Gi, 512Mi (default 4Gi; none for no cap) |
OPENSHELL_PYTHON_BINARY |
interpreter path the policy's egress rules name (default /usr/lib/python/cpython/bin/python3.14) |
OPENSHELL_CLI |
path to the openshell CLI (default openshell) |
OPENSHELL_CLI_HOME |
HOME for CLI children, where the gateway registration and mTLS bundle live |
OPENSHELL_LABEL |
label applied to every executor sandbox (default app=semoss-executor) |
OPENSHELL_WORKER_PORT |
worker port inside the sandbox (default 7100) |
OPENSHELL_READY_TIMEOUT_SECONDS |
cold-start budget (default 180) |
OPENSHELL_KEEP_SANDBOX |
true leaves sandboxes in place on teardown, for debugging |
OPENSHELL_IDLE_EXIT_MINUTES |
a user's worker exits after this long with no client connected (default 180; keep above the session timeout) |
OPENSHELL_WORKER_MAX_CLIENTS |
connections one user's worker serves at once (default 16) |
- The class actually deployed. Unpack
prerna/om/ClientProcessWrapper.classfrom the deployedsemoss-*-shaded-dependencies.jarand check it withjavap -pforfetchBackChangedPaths; a build log is not proof. - A write reaches the host. Write a file from
Py()in a loaded project and look forFetch-back wrotein the server log and the file in the project folder. - The permission gate holds. Give a test user read-only access to a project, write from
Py(), and confirm the host file never appears andRefusing write-backis logged.
- More than ~2000 changed paths in one block sets
overflowand forces a full capture. - A racing
mmapwriter cannot be captured without cooperation (msync) or interception this sandbox does not have. - Conflicts are decided by modification time, so a change made within the same second as the last sync, and leaving the size unchanged, is not detected.
- The refresh lists each transported root on every block; for very large roots that cost shows up as block latency.
- Background code that writes between blocks is not synced until the next block ends.
- Python variables are keyed by insight id, and insight ids live in one instance's memory. After a restart or a move to another instance the worker, its modules and its files are intact, but a new insight does not see the old insight's variables.
- One change watcher serves every connection to a worker. Two instances connected at once can consume each other's change sets. That is harmless while they share a disk; separate disks per instance need per-connection cursors.
- Rooms and assets are gated as described above, but the platform has no room-sharing model in this build; if one is added, the room branch must resolve the room to its project.