feat(pipelines): add pr_to_env + v2 iptables egress firewall - #85
adithya-s-k wants to merge 1 commit into
Conversation
Adds pr_to_env, the import-shape sibling of pr_runtime: instead of mining a
repo's PR history, it consumes an explicit list of PR URLs and produces one
Harbor task per URL (or fails closed with a per-URL reason). Same task
shape, same graded F2P/P2P verifier, same anti-contamination guards.
Why: dataset curation. When you know exactly which PRs you want in your
benchmark (e.g. code-health audit picks, deliberate difficulty distribution),
pr_runtime's mine-then-filter approach is the wrong tool. pr_to_env lets you
hand-pick and still get the full runtime verification + emit pipeline.
Ships:
- PrToEnvPipeline (registers as pipeline="pr_to_env", experimental=True)
- PrToEnvOptions with 13 knobs (url / urls_file, min_f2p, min_p2p,
oracle_gate, strict, etc.)
- Oracle-gate integrated in-loop: harbor run -a oracle after each emit;
envs scoring reward<1.0 are dropped and rmtree'd
- keepers.jsonl ledger writer (one JSONL line per URL with status/reward)
- Leak-strip v2 layered on pr_runtime._strip_info_leak:
- hard-strips bare short-SHAs (8-39 hex) that leak fix commit refs
- hard-strips pytest node-id paths that leak the grading target
- soft-flags basename/dirname mentions of touched files
- Dockerfile RUN-step gate #5 (pyproject.toml sanitize) — strips
bare [tool.pytest] when [tool.pytest.ini_options] also present
(pytest exit-4 usage error otherwise; hit repeatedly on peft PRs)
Egress firewall v2 (in _env_guard.py):
- Replaces v1 /etc/hosts blackhole with iptables OUTPUT DROP + IP
allow-list. v1 was bypassable via IP-direct fetches to
raw.githubusercontent.com; confirmed reward-hack cases with two agents.
- egress_firewall_entrypoint / _dockerfile_fragment / _compose helpers
- DEFAULT_ALLOW_HOSTS covers model APIs (Anthropic/OpenAI/HF Router) +
agent-installer sources (debian mirrors, PyPI, npm, GitHub Releases).
Fix-content hosts (github.com repo pages, raw.githubusercontent.com,
pypi wheels of target repos) stay refused.
- Requires CAP_NET_ADMIN in compose; falls back cleanly to /etc/hosts
guard with a stderr warning if iptables is missing.
Tests: 34/34 in test_pipeline_pr_to_env.py and test_env_guard.py green.
Contract test still green (new pipeline conforms to Pipeline Protocol).
Full suite 740/740 pass, ruff clean.
KNambiarDJsc
left a comment
There was a problem hiding this comment.
Reviewed and tested da423fa. This is a substantial, well-documented PR (RFC 0007 already exists on main for it) — went through the pipeline, the v2 firewall, and the CI blocker.
The one thing actually blocking CI is trivial. All three Lint (ruff) failures collapse to one: ruff format wants a single trailing blank line removed from the end of src/repo2rlenv/pipelines/_env_guard.py (ruff check itself is already clean). uv run ruff format src/repo2rlenv/pipelines/_env_guard.py fixes it in one commit — Tests (py3.12/13/14) are already green and have been since this opened.
Test suite: ran the full suite myself rather than trusting the PR description. Linux (WSL, matching CI's uv sync --group dev --all-extras --frozen): 739 passed, 6 skipped, 0 failed — matches your claimed 740/5 (the one extra skip in my run is just gh not authenticated in my sandbox). On Windows I saw 8 unrelated failures, all pre-existing Windows-portability gaps already tracked in #130 (bundle-mode checks, symlink privilege, etc.) — nothing here is this PR's fault, noting it only so it isn't mistaken for a regression.
Two things worth raising on the v2 firewall before merge, since _env_guard.py is this project's actual security boundary against reward-hacking (per CLAUDE.md's "the environment enforces, the prompt never asks" invariant), not just a nice-to-have:
- IPv6 is completely unfiltered.
egress_firewall_entrypoint()only ever callsiptables—ip6tablesis never touched anywhere in the fragment, andgetent ahostsv4(grade.py:205 in the diff) explicitly discards any AAAA records it resolves. The defaultOUTPUTpolicy on theip6tablestable is left at whatever the base image ships (normally ACCEPT), so on any container with an IPv6 route, an agent can reachraw.githubusercontent.com(or anything else) over its AAAA record and the entire v2 guard — including the exact IP-hardcoded-connection bypass this PR was written to close — is void. Docker's default bridge network has IPv6 disabled, which is probably why this hasn't shown up yet, but the PR targets 5 different sandbox providers (Modal/Daytona/E2B/Runloop, per the project's own docs) and some of those enable dual-stack by default. Given the docstring's claim ("Blocks all outbound traffic at the packet layer except a small allow-list") is currently only true for IPv4, I'd treat this as worth closing before relying on it, even if justip6tables -P OUTPUT DROPunconditionally (no allow-list needed there, since none ofDEFAULT_ALLOW_HOSTSneed to be reached over v6 if their v4 addresses are already allowed). - Port 53 is allowed to any destination, not just the container's resolver.
iptables -A OUTPUT -p udp/tcp --dport 53 -j ACCEPThas no-drestriction, so DNS-tunneling to an attacker-controlled external nameserver (a well-known exfil/infil channel, and squarely in the threat model this PR names — "DoH resolvers bypass it entirely") isn't actually closed, just narrowed to port 53 specifically. Restricting to the container's actual resolver (-d 127.0.0.11for Docker's embedded DNS, or read from/etc/resolv.confat entrypoint time) would close this the same way the rest of the allow-list does.
Neither of these undoes the real improvement here over v1 (IPv4 IP-hardcoded bypass is genuinely closed), and I don't have visibility into whether IPv6 is even reachable on the sandbox providers this runs against in practice — flagging them because this is the file the project explicitly trusts to make reward-hacking structurally impossible rather than just discouraged, so it seemed worth having on the record before merge rather than after.
pr_to_env.py's subprocess call (_run_oracle_gate) passes an argv list with no shell=True, so no injection concern there — spot-checked it since it's the one place PR-URL-derived input reaches a subprocess.
LGTM on the mechanical/functional side once the one-line format fix lands; the two firewall points above are for you and the maintainers to weigh, not blockers I'm asserting.
|
Closing this since PR-to-env is now covered by Tasksmith and the SWE-gen pipeline on main. Keeping the branch for reference; any remaining firewall improvements can come as a separate focused PR. |
Summary
pr_to_env— the import-shape sibling ofpr_runtime. Takes an explicit list of PR URLs (url=…orurls_file=…) and emits one Harbor task per URL, oracle-gated in-loop. Fills the dataset-curation gap where mine-then-filter is the wrong tool. Registered asexperimental=True._env_guard.py— replaces the bypassable/etc/hostsblackhole with iptablesOUTPUT DROP+ IP allow-list. Closes a confirmed reward-hack channel (opus + codex fetched fix content fromraw.githubusercontent.comvia IP-direct, sidestepping the v1 host guard).keepers.jsonlledger, leak-strip v2 (short-SHA + pytest node-id), pyproject.toml sanitize (Dockerfile RUN step for[tool.pytest]/[tool.pytest.ini_options]collisions on peft).test_pipeline_pr_to_env.py+test_env_guard.py; full suite 740/740 green, ruff clean, contract test conforms.Reference dataset built with this pipeline:
AdithyaSK/HF_ML_Bench_v0(16 oracle-verified tasks across 5 HF ML repos, with 4-agent eval matrix baked into the README).Closes #84
Test plan
uv run pytest -q→ 740 passed, 5 skippeduv run ruff check .→ cleanPipelineProtocolOut of scope
4 lower-frequency M3 gates still pending (parametrization suffix expansion, cross-repo dep pinning, salvage manifest tracking, LF-normalized content hash). Land as follow-ups — pipeline is honest about being
experimental = True.Version bump: not in this PR. Only after merge + review.