Repository navigation
Conversation
Phase 1 of the File DLP epic (#324), agent side, with no OS hooks yet, so nothing is enforced on files and device behaviour doesn't change. #326, contract (draft, agent side): - The bundle reads modules.files (channels, audit_only, limits, on_unscannable/on_timeout, egress apps, trusted devices/shares, messages) and the file fields of policy exceptions. - Both are decoded tolerantly: a files module or file exception the agent can't read is turned off and reported as a compile error, and the clipboard keeps working. source_apps keeps its old strictness. - Defaults: audit-only when audit_only is missing; fail open unless on_unscannable/on_timeout say "block"; limits clamped. - File channels don't count as enforced channels, so a files-only bundle leaves DLP idle instead of starting session helpers. The file engine is built lazily, so helpers never pay for it. - Events gain optional file/destination/process/scan fields; clipboard events keep their exact JSON. #327, internal/dlp/filescan: - The file type comes from the content. ASCII-only magic numbers ("BM", "MZ") are checked only after the text test, so a CSV starting "BMW" is still scanned. - Extractors: text in UTF-8, UTF-16 LE/BE with or without BOM, and Latin-1; OOXML (docx/xlsx/pptx text parts, cells and paragraphs kept apart, field codes and formulas skipped); ODF; RTF; zip archives, entry by entry. - The text is scanned in chunks that overlap. Cuts are made at a newline or space, and each chunk keeps only matches that start before the next chunk, so a match across a boundary is neither missed nor counted twice. - Limits: file size, text budget, deadline, archive depth/entries/total bytes/compression ratio. The zip central directory's entry count is checked before archive/zip allocates anything, and spill buffers grow with what is read, not with what an entry claims. - Explicit statuses: scanned, truncated, timeout, unscannable:<reason>. PDF and legacy OLE have no extractor yet (unsupported_type). Encrypted and IRM-protected OOXML are detected. - Microsoft sensitivity label names come from docProps/custom.xml. - Bounded LRU verdict cache keyed by file identity, size, times and policy version; timeouts aren't cached; invalidated on close-after-write. - Fuzz targets (FuzzScan, FuzzRTF) and benchmarks per format. #341: FileName gives an event nothing, the base name or the full path, by evidence mode. A structural test forbids filescan from logging, printing, networking or touching files beyond its spill file. Refs #324 #326 #327 #341 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion File DLP groundwork: modules.files contract, scan engine, privacy guard (#324)
- zip.go: sizes and offsets from zip headers go through toInt64, which refuses values over MaxInt64 instead of converting them. All limit checks are now int64 comparisons (G115 x3). - cfb.go: UTF-16 marker bytes are built with AppendUint16 (G115). - zip.go: the archive entry name is only a label in results, never a path anything is written to; annotated (G305 false positive). No behaviour change: the existing guards already rejected negative and oversized values; this makes it explicit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
filescan: resolve the gosec alerts from #364
Phase 2 of the File DLP epic (#324): #333 (fanotify), #328 (sandboxed scan workers) and the fail-safes of #329 that apply to audit-only. When a bundle enables file.removable or file.network_share, files written there are scanned and matches recorded as "monitored" events. Nothing is blocked. Design: fanotify notification events (FAN_CLOSE_WRITE on mount marks), so the kernel never waits for the agent; a slow or dead agent can't hang the machine. Blocking (permission events, timers, watchdog) is left for when audit data exists, as the epic's rollout rule says. - internal/dlp/scanworker: `sentinelgo dlp-scan` workers parse files, never the root service. The service passes the file's descriptor over a socket pair (SCM_RIGHTS); the worker never opens a path. As root, workers run as an unprivileged account (sentinelgo-scan, else nobody) in a new network namespace, falling back without one where the kernel refuses. Each worker sets no_new_privs and Landlock (no file access by path; no TCP from ABI 4) on every thread via AllThreadsSyscall, plus RLIMIT_FSIZE=0, NOFILE, AS, CORE and a Go memory limit. Every scan has a deadline: a stuck worker is killed and replaced. Five failures in ten minutes disable scanning (reported, not retried forever). Cancelling a scan's context ends its wait at once. - internal/dlp/filedlp: the Decider (OS-agnostic) applies the policies for the channel. It honours per-policy exceptions (apps, trusted devices, trusted shares, source apps), caches results by file identity and policy version, and records events with file, destination, process and scan details by evidence mode. Unscannable or timed-out files are recorded only when on_unscannable/on_timeout ask. - Linux enforcer: classifies mounts from mountinfo and sysfs. Removable means a USB/MMC bus or the removable flag, with dm/md followed to the devices under them (LUKS sticks). Network means cifs/smb3/nfs/sshfs, with the share host taken from the source (IPv6 aware). It marks them, re-marking on mount changes and policy changes. Bounded queue: a full queue drops and counts. Event fds are closed on every path. The agent's own writes are ignored. When the writer has already exited, the user comes from the file's owner. - Emergency switch: dlp-files-off in the config directory removes every mark within seconds, without the backend. - Manager: a files-only bundle enforces with a file hook and stays idle without one (every platform but Linux). The clipboard is unchanged. Status gains file channel capabilities and a files health block. Tests: decider unit tests; mount/sysfs classification against fixtures; worker pool tests running the test binary as a real sandboxed worker (scan, concurrency, stuck worker, crash loop, cancel, sandbox probe). Root-only end-to-end tests run real fanotify on a tmpfs "stick": an event, no event for clean files or while switched off, recovery, stop when the channel turns off, and stop during an in-flight scan. Refs #324 #328 #329 #333 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sts under -race - Pool.acquire: scans waiting for a worker were never woken when starting one failed (or a worker died), so they waited until their context ended, forever with context.Background. Slots freeing up now signal waiters, who retry. Regression test: 8 scans against a pool whose workers can't start all return. - Tests that run real workers skip in cgo builds (CI's -race step): the sandbox needs AllThreadsSyscall, which cgo builds lack, and a worker refuses to run unsandboxed. Releases are always CGO_ENABLED=0, and CI's regular test step still runs them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
File DLP on Linux: audit-only monitoring of USB and network shares (#324)
SonarCloud measured 58% coverage on #366's new code: CI isn't root, so the fanotify end-to-end tests skip there. Now covered without root: - Seams for fanotify_init, the mount marks and the mountinfo path. A test runs the real loops with a pipe as the "group" and writes kernel-format event records into it: marking (including a mark that fails), events mapped to a mount and decided, files elsewhere, directories, our own writes, overflow/unreadable counters, the emergency switch and stopping. - The "fanotify not permitted / unavailable" paths and their capability reasons. - The scan worker's serve loop in-process over a real socket pair (policy, scans with a passed descriptor, no policy, a bad policy, unknown messages, a clean exit), frame limits and the offsets cap. Fix found by the new test: after the monitor stopped, Health() returned nil when no worker had ever started, so the status lost its files block instead of reporting "stopped". It now remembers that a start was tried. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
filedlp: test the Linux monitor without root (coverage for #366)
#367's SonarCloud gate measured 46.7% on its new lines: mostly the real fanotify_init/fanotify_mark wrappers, which the loop tests replace with fakes. This test calls them for real. As a normal user (CI) the kernel refuses, which still runs every line and checks the failure mapping; as root it marks and unmarks a directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…alls filedlp: test the real fanotify calls (coverage for #367)
The fanotify end-to-end tests and the sandbox probe need root (a tmpfs that plays a USB stick, fanotify, workers dropping to nobody), so they skipped in CI and only ran on developer machines. Hosted runners allow sudo: a new step runs filedlp and scanworker as root, fails if those tests skipped instead of running, and adds their coverage to the Ubuntu report SonarCloud reads. Root gets its own build cache so later steps, run as the runner user, never meet root-owned cache files. Refs #340 #333 #328 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Workers fell back to nobody, which other software shares. The service now creates a sentinelgo-scan system account (no home, nologin shell, own group) the first time it starts workers as root, so every install path gets it: install.sh, -install, packages and devices that only self-update. If useradd is missing or fails (a directory-managed system), workers keep falling back to nobody, logged once. -uninstall and install.sh uninstall remove the account. useradd, userdel and nologin are resolved from fixed paths (binpath), never PATH. Refs #339 #328 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ci: run the file DLP end-to-end tests as root on Linux
…ents .doc, .xls and .ppt were reported unsupported_type, and files embedded in docx/xlsx/pptx weren't opened. - cfbfile.go: a bounded OLE compound file reader (header, FAT and DIFAT, directory tree, mini stream). Every sector number is range-checked and every chain bounded by the file's sector count; streams are capped at max_file_bytes. - .doc: text through Word's piece table (8-bit and UTF-16 pieces); field codes dropped and results kept, like instrText in docx; paragraph and cell marks as breaks; fEncrypted means password_protected. - .xls (BIFF8): the shared string table, including strings continued in CONTINUE records with their own 8/16-bit flag; LABEL and STRING cells; NUMBER, RK, MULRK and cached FORMULA values (whole numbers as digits, so a card number stored as a number is found); FILEPASS means password_protected. Excel 5/95 (Book) stays unsupported. - .ppt: TextChars and TextBytes atoms; CryptSession10 or EncryptedSummary means password_protected. - OLE packages (Package, Ole10Native) are unwrapped and scanned. - Encrypted OOXML is detected from the parsed directory; the stream-name search stays as the fallback for files that don't parse. - Embedded files (word/xl/ppt/embeddings) are scanned as entries. Ones with findings or protected by a password/RMS are recorded; unreadable objects (equations, drawings) aren't, so everyday documents don't become unscannable. Tests build real compound files (mini stream and regular sectors) for each format, encryption flags, packages, embeddings and malformed files (loops, bad sectors, oversized streams). Fuzz seeds now include compound files; 90 s of fuzzing found no crashers, and the slowest saved input scans in 0.2 ms. Refs #327 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DLP scan workers run as a dedicated sentinelgo-scan account
The backend's merged contract (Compliance-OS#562) lets a policy carry
file_filters {types, min_bytes, max_bytes}, and the agent ignored them, so a
policy meant for some file types applied to every file.
- Types are lowercase extensions or detected types: a policy matches a file
by its extension, its detected type (ooxml.xlsx) or the detected type's
last part (xlsx), so a renamed file is still caught by its content.
Archive entries are filtered on their own name and type.
- Sizes filter before the scan: a file outside every policy's size range is
not read at all.
- file_filters are decoded tolerantly: unreadable ones are reported as a
compile error and the policy applies to every file (a bad filter must not
silence a policy, nor reject the clipboard's bundle).
Refs #326
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # CHANGELOG.md
DLP: honour a file policy's file_filters (backend contract #326)
filescan: read legacy Office files and files embedded in Office documents
Uploads go with option C (#335): a local TLS-inspecting proxy. This is phase 1 from #344, the explicit proxy: a client pointed at it has its uploads to the hosts in scope decrypted and each uploaded file decided by the file policies (channel file.upload); everything else is tunnelled untouched. No OS redirect yet, and the service doesn't run it. - uploadproxy.CA: a per-device ECDSA CA, its key a root-only file written atomically, never leaving the device; renewed before expiry; MaxPathLenZero. 24-hour host certificates, cached. - uploadproxy.Proxy: CONNECT (HTTP/1.1 and HTTP/2 to the client and the server) and plain-HTTP proxying. Servers are verified with the system roots (a test forbids InsecureSkipVerify). It never chains to an environment proxy. A request on an intercepted connection can't be steered to another host (400). HostScope lists hosts and *.subdomains on 443/80. - uploadproxy.Inspector: the body goes to a root-only spill file up to max_file_bytes (taken from the same policy the scanner uses). Multipart file parts are decided one by one and plain fields together, other bodies whole, and the request is forwarded byte for byte. A body over the limit is decided as too large and the rest streams on unread. Scanning is by the sandboxed workers (the proxy only splits multipart parts). - filedlp.Decider.Decide returns a verdict. It blocks only when the hook can (the Linux monitor can't), the module isn't audit-only, and a matching policy says block (or on_unscannable/on_timeout do). The event action is then blocked; the user sees the block message with detector ids, never matches. Uploads skip the verdict cache (spill inodes are reused). Events carry destination.host. - sentinelgo dlp proxy -hosts …: runs it by hand for testing (loopback only, -ca-out to export the CA, events as JSON lines). Refs #344 #335 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Windows has no Unix modes, so the CA key file was readable by anyone the config directory let in (CI caught it: -rw-rw-rw-). The key's temp file now gets winsec's ACL (SYSTEM, Administrators, the service's account) before the key is written, and creating the CA fails if it can't be restricted. The spill directory gets the same ACL, inherited by its files. Mode checks in the tests are Unix-only; winsec has its own tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
File DLP uploads, phase 1: the upload inspection proxy (#344)
SonarCloud (S5144, SSRF) flagged that the proxy dials whatever a client asks for. A forward proxy must, but this one runs as root: a local client could use it to reach what it couldn't reach itself (root-only firewall rules, a link-local metadata service). A dialer Control hook now checks the address actually dialled (after DNS, so names and rebinding are covered) and refuses loopback, link-local, unspecified and multicast addresses, for tunnels and inspected requests alike. Private ranges stay allowed: intranet upload destinations are legitimate. Tests opt in to local targets (their sites run on 127.0.0.1). Refs #344 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
uploadproxy: never connect a client to loopback or link-local addresses
The bypass list as signed off on #338: the threat model first (accidental leaks, not a determined insider), where file DLP runs today, every vector with its status, the interim mitigations next to the rows that have them (MTP: Compliance-OS#576), and the re-review rule with the date of the last review. The admin tampering row is held back until the health alert (Compliance-OS#525) exists, as the sign-off asks. Refs #338 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs: publish what file DLP covers and what it doesn't (#338)
…bilities (#334) Backend Compliance-OS#570/#572 added modules.files.copy_correlation; this is the agent side, part 1. - policy.CopyCorrelation: scan_on_close, on_close_violation, process_taint, taint_window_seconds, os_copy_events, untrusted_devices. Decoded tolerantly; defaults keep today's behaviour; unreadable settings fall back to the defaults and are reported. - on_close_violation for files found to violate a block policy (or block fail modes) after they were written: record, quarantine or delete. A monitor policy never touches a file; the action happens once per file. Audit-only does nothing and records would_have. The upload proxy's audit-only blocks record would_have: blocked too. - Linux remediator: checks that the path still names the scanned file (device and inode) before moving or deleting it. Quarantine copies from the hook's descriptor into a root-only dlp-quarantine folder (1 GB cap, entries kept 30 days), then removes the original. Errors never carry the path. - events: actions quarantined and deleted; top-level would_have and correlation, as the backend's ingest reads them. - capabilities.copy_correlation in the status (an object inside capabilities); Linux reports scan_on_close and os_copy_events: unsupported:not_on_linux. scan_on_close: false stops the Linux monitor. - CI's root step must now also run TestEnforcer_QuarantineEndToEnd. Refs #334 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
untrusted_devices = read_only | block_writes on Linux: a removable drive whose device neither the files module nor a removable-channel policy trusts is remounted read-only, so nothing can be written to it. - A bind remount of that mount point only, keeping its flags (nosuid, nodev, noexec, atime) read with statfs: dropping nosuid on a stick would let its set-user-ID programs run with privilege. - One event per drive: blocked, scan.status not_scanned, no hash, the device's ids; monitored with would_have blocked while audit-only. - Fail open: writable again when the setting goes back to allow, the device becomes trusted, the emergency switch is on, or monitoring stops. An already read-only drive is left alone; a failed remount is logged once and never claimed as a block. - The mount loop now classifies once and serves both the marks (only with scan_on_close) and device control; the monitor runs for either. - capabilities.copy_correlation.untrusted_devices: supported on Linux. - CI's root step must also run TestEnforcer_UntrustedDeviceEndToEnd. Refs #334 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
File DLP copy correlation, part 1: settings, after-write action, capabilities (#334)
File DLP copy correlation, part 2: untrusted USB drives read-only (#334)
process_taint (Linux, off by default): a file written to a monitored drive
that can't be judged on its own (unreadable, or an archive) is judged by
the files its writer read just before.
- A second fanotify group gets FAN_OPEN notifications for the mount holding
the home folders, only while taint is on and a destination is watched; it
is opened the first time it's needed. Its own reader means a flood of
opens can only overflow its queue, never the destination events.
- Each open under /home costs a readlink and an fstat; the path and
identity go into an in-memory store (32 per process, 4096 processes,
taint_window_seconds). Nothing is scanned at open time.
- At a destination event that matched nothing and couldn't be judged on its
own, up to 8 sources (newest first, opened after the process started, so
a reused pid inherits nothing) are reopened with O_NOFOLLOW, checked to
be the same file, and scanned through the cache. A match replaces the
fail-mode events with correlated ones (correlation {method: taint,
source_sha256, source_name}, verdict_source correlation), and the
after-write action applies. A readable file that matched nothing is never
judged by its writer's earlier reads.
- capabilities.copy_correlation.process_taint: supported on Linux. Health
reports processes tracked and open events lost to overflow.
- Measured (privileged container, 100,000 opens while a drive is mounted):
the workload took 1.05 s instead of 0.83 s (~28%), the agent used ~5 us
of CPU per open, no events were lost.
- CI's root step must also run TestEnforcer_TaintEndToEnd.
Refs #334
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Windows can't rename a file that is open, so the test's destination file was never there and the fake scanner got nil. Write it under its final name, then open it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
File DLP copy correlation, part 3: process taint (#334)
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
This was referenced Nov 2, 2026
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Both came up on #379 (dev -> main). - CodeQL go/allocation-size-overflow (high), scanworker writeMsg: the frame buffer was sized 4+len(body) with no bound when sending, although the comment said frames were capped. writeMsg now refuses a frame over the protocol's largest limit (9 MB) before allocating. - SonarCloud S5144 (SSRF) in Proxy.connect: tunnels dialled the client's CONNECT target string. The connect-time guard (#374) already refused local addresses, but out of SonarCloud's sight. vetTarget now checks the port (1-65535), resolves the host itself, keeps only allowed addresses (allowedIP, shared with guardDial) and returns an IP literal, which is what gets dialled. guardDial still checks the dialled address. Refs #344 #328 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 of 3 tasks
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix the release PR's CodeQL and SonarCloud findings
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings
devtomain: the File DLP work of epic #324. That's 15 PRs, all merged intodevwith CI green.What changes on devices: nothing, until the backend sends a policy with a files module enabled. The backend contract was merged in BrainStation-23/Compliance-OS#562, #566 and #572. Clipboard DLP is unchanged.
File DLP (Linux; manual upload proxy on any OS)
modules.filescontract (decoded tolerantly), the file scan engine (internal/dlp/filescan), privacy guard; gosec fixesdlp-scanworkers (Landlock, no new privileges, no network, unprivileged); emergency off switch; tests runnable without rootsentinelgo-scanaccount for the workers.doc,.xls,.ppt) and files embedded in Office documentsfile_filters(types, sizes)sentinelgo dlp proxy); per-device CA; no connections to loopback or link-local addressesdocs/dlp-file-coverage.md)The CHANGELOG's
[Unreleased]section has the user-facing details.Known open points
sha256(files too large, timed out, or failed to scan), so those events are dropped server-side until it's relaxed.uploadproxy/proxy.go. It's mitigated in uploadproxy: never connect a client to loopback or link-local addresses #374 (guardDial), but the finding needs to be marked accepted in SonarCloud.Issues this completes
#326, #334 (Linux; macOS copy events come with #332), #335 (decided: option C, continued in #344), #338, #341 (agent side)
🤖 Generated with Claude Code