Skip to content

Relocate infra to $CODE/infra; keep $CODE as the workspace root#54

Open
krystophny wants to merge 8 commits into
mainfrom
chore/relocate-infra
Open

Relocate infra to $CODE/infra; keep $CODE as the workspace root#54
krystophny wants to merge 8 commits into
mainfrom
chore/relocate-infra

Conversation

@krystophny

@krystophny krystophny commented May 28, 2026

Copy link
Copy Markdown
Member

What

Make this repository the infra directory of a workspace instead of the
workspace root, and retire the umbrella integration CI that never passed.

$CODE/            workspace root (no .git, no pollution)
  infra/          this repo            -> $INFRA
  external/       prebuilt third-party libs, shared by all codes
  libneo/ SIMPLE/  code checkouts (own .git)
  • activate.sh/.fish export INFRA=$CODE/infra and derive CODE as its
    parent. Infra-private paths (scripts, .venv, modules) move to $INFRA.
  • $CODE stays the workspace root, so the codes keep resolving dependencies as
    $CODE/<name> (find_or_fetch) and prebuilt libraries as $CODE/external.
  • The two accidental gitlinks (SIMPLE, libneo) are dropped.
  • main.yml is removed. It built the unpinned tips of six code repos in one
    sequential job and never went green (0 of 94 runs). The replacement model is
    in docs/development-model.md: per-code CI at the source plus release-time
    reverse-dependency validation owned by the upstream.

Migration for users

Move your checkout one level down and repoint the activation line:

mkdir ~/code.new && mv ~/code ~/code.new/infra && mv ~/code.new ~/code
# move the sibling checkouts (libneo, SIMPLE, ...) and external/ up next to infra
# then in ~/.bashrc: source $HOME/code/infra/activate.sh

Verification

New activation resolves correctly. Simulated workspace with infra at <ws>/infra
and a libneo/ sibling:

INFRA = /tmp/ws/infra
CODE  = /tmp/ws
venv  = /tmp/ws/infra/.venv
INFRA/scripts on PATH: yes
CODE/libneo/build on lib path: yes
find_or_fetch contract [ -d $CODE/libneo ]: PASS

Before/after on the dependency-resolution contract, with infra at <ws>/infra:

BEFORE: CODE=<ws>/infra -> $CODE/libneo = <ws>/infra/libneo -> absent, refetched
AFTER : CODE=<ws>       -> $CODE/libneo = <ws>/libneo       -> present, local

Verified locally on macOS: variable resolution, venv activation, PATH and
library-path entries, set_branch. CI is no longer the umbrella build; that
workflow is removed rather than fixed.

Note

docs/development-model.md is proposed, not settled. It documents the layout,
the <DEP>_BRANCH convention, the release-branch flow, and YY.MINOR.PATCH
versioning for group discussion. setup.yml (the release-tarball builder) is
left in place but loses its only consumer with main.yml gone, so it needs a
separate decision.

The repository was the workspace root, so its .git tracked every sibling code
checkout as untracked noise and carried two accidental gitlinks (SIMPLE,
libneo) with no .gitmodules. Make it the infra directory of a workspace.

activate.sh, activate.fish and activate export INFRA (this repo) and derive
CODE as its parent. Infra-owned paths (scripts, .venv, external, modules,
images, local, lib, bin) move to $INFRA. $CODE stays the workspace root, so
the codes keep resolving dependencies as $CODE/<name> via find_or_fetch.
Sibling references ($CODE/libneo/build) and workspace navigation (cdcode,
vscode) stay on $CODE; set_branch reads the infra repo via $INFRA.

Drop the accidental SIMPLE and libneo gitlinks. Update setup.sh, the
devcontainer build and postCreateCommand.sh, the release CI in main.yml, the
STELLOPT and VMEC config paths, and the README to the new layout.
main.yml downloaded a pinned release tarball whose frozen activate.sh predates
the infra relocation, so $INFRA was unset and $INFRA/scripts/checkout_branch.sh
resolved to /scripts/... (exit 127). Check out the branch and rsync its
sources over the tarball, keeping the release's prebuilt .venv and external,
and relocate the venv shebangs to the new path.
activate.sh is meant for interactive shells; its add_to_path/add_to_library_path
helpers return the guard status, so under the runner's bash -e the first absent
dir ($INFRA/local/bin) aborted the step before the venv activated. Wrap the
source in set +e/set -e so activation completes while build commands stay strict.
The codes read prebuilt third-party libs from $CODE/external (e.g. NEO-2's
build links $CODE/external/fgsl-1.6.0/.libs/libfgsl.a). external is workspace-
shared like the code checkouts, not infra-private, so revert it from $INFRA to
$CODE and stop tracking it in the infra repo. setup.sh creates it; CI moves the
release's external to the workspace; CODE_ROOT (STELLOPT make configs) points at
$CODE. setup.yml release packaging flagged for tag-time validation.
@krystophny krystophny force-pushed the chore/relocate-infra branch from 9be1ed9 to a14b19c Compare May 28, 2026 19:38
main.yml built every downstream code on each push to infra and never passed
(0 of 94 runs): it built the unpinned tips of six repos in one sequential job
against a frozen release tarball. Remove it.

Add docs/development-model.md (proposed): workspace layout, the self-describing
dependency graph, the per-code + release-time reverse-dependency testing model,
the <DEP>_BRANCH convention, the release-branch flow, and YY.MINOR.PATCH
versioning. The genuine external -> $CODE fix from earlier commits stays; the
umbrella build it was chasing is gone.
docs/templates/downstream-integration.yml: per-consumer reverse-dep validation,
dispatched with a libneo ref, no commit. docs/templates/upstream-release.yml:
libneo release-branch + gate + bump-PR skeleton (untested, TODOs marked).
Referenced by the development-model design note.
Codes track the libneo release branch by default (committed LIBNEO_RELEASE);
tags are reproducibility snapshots. Rename the override to LIBNEO_REF (branch,
tag or commit), validated against the remote. Drop the $CODE local-path
shortcut: first-party deps are always fetched at the resolved ref. The release
gate dispatches each downstream's existing CI and opens auto-merge bump PRs to
the new release branch; templates reflect reuse of existing CI and point at the
canonical libneo release.yml.

@slopqueue slopqueue Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review verdict: Request changes

Summary: This PR relocates the infra repository from the workspace root ($CODE) to a subdirectory ($CODE/infra), keeping $CODE as the workspace root so codes continue to resolve dependencies as $CODE/<name>. The change is mostly mechanical: ~30 files are updated to distinguish infra-private paths ($INFRA: scripts, .venv, modules) from workspace-level paths ($CODE: external libs, code checkouts). Two accidental gitlinks (SIMPLE, libneo) are dropped, and the umbrella CI (main.yml, 0/94 green) is retired in favor of a proposed per-code CI model documented in docs/development-model.md. The design is sound and the $CODE/$INFRA split is consistently applied across shell scripts, Dockerfiles, and the devcontainer flow. However, the fish activation script has a regression that breaks CODE resolution on older fish, and one comment actively contradicts its code.

Findings:

  1. [blocker] activate.fish:20set -gx CODE (path resolve $INFRA/..) uses path resolve unconditionally, but lines 9-19 handle three cases for INFRA where path may not exist (fish < 3.5, no path builtin). On those systems INFRA is set via realpath or a cd/pwd fallback, but CODE tries path resolve, which fails — the command substitution returns an empty string, so CODE is set to "". Every subsequent $CODE reference breaks: add_to_library_path_fish $CODE/libneo/build is skipped (dir doesn't exist), CMAKE_INCLUDE_PATH gets a dangling $CODE/external/triangle, alias cdcode="cd $CODE" cds to $HOME. The old code on main set CODE through the same path/realpath/cd fallback as INFRA, so this is a regression. Fix: use the same fallback for CODE, e.g.:
if type -q path
    set -gx CODE (path resolve $INFRA/..)
else if type -q realpath
    set -gx CODE (realpath $INFRA/..)
else
    set -gx CODE (cd $INFRA/..; pwd)
end
  1. [major] scripts/setup/stellopt.sh:5 — Comment says "That tree lives in the infra repo, so point it at $INFRA" but the code on line 6 sets CODE_ROOT="$CODE". The external tree is at $CODE/external (workspace), not in the infra repo ($INFRA/external doesn't exist). The comment contradicts the code: the code is correct (STELLOPT is cloned into $CODE/external/STELLOPT and reads $(CODE_ROOT)/external/...), but a future developer following the comment would "fix" it to $INFRA and break STELLOPT. Fix: update the comment to say the tree lives in the workspace at $CODE/external.

  2. [minor] .github/workflows/setup.yml:84 — The TODO says "main.yml still consumes the pinned old release" but main.yml is deleted in this very PR (the umbrella CI being retired). The reference is stale. Fix: drop the main.yml clause or reword to "the pinned old release is still the latest tag; validate on a real tag before cutting the next one."

  3. [minor] No CI — gh pr checks reports no checks on this branch (confirmed: check-runs API returns total_count: 0). The deleted main.yml never passed (0/94 runs, per the PR body), and setup.yml runs only on version tags — it is not exercised by PR CI. The setup.yml changes (building external deps in the workspace parent, copying into the tarball) are explicitly marked unvalidated ("Validate on a real tag before cutting the next release"). The PR body claims local verification of variable resolution and venv activation, but no automated CI exercises the changed activation, setup, or devcontainer paths. This is a process risk, not a code defect, but it means the changed paths are unproven beyond manual testing.

Verdict: request_changes — the activate.fish CODE regression (finding 1) breaks fish activation on any system without the path builtin, a configuration the script explicitly supports via the INFRA fallback. The fix is a one-liner. Finding 2 should also be fixed before merge to avoid a misleading comment that could cause a future breakage.

@slopqueue slopqueue Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review verdict: Request changes

Summary: This PR relocates the infra repo from the workspace root ($CODE) to a subdirectory ($CODE/infra), keeping $CODE as the workspace parent. The path migration is mostly correct: infra-private paths (.venv, scripts, modules) move to $INFRA, while workspace-shared paths (external/, libneo/) stay under $CODE. The old broken umbrella CI (main.yml, 0/94 green runs) is deleted. However, the PR has no CI at all (0 check runs on the head SHA), a real code bug in activate.fish that silently breaks fish activation on systems without the path builtin, and a misleading comment in stellopt.sh that contradicts the code. The setup.yml release-tarball builder is explicitly untested (its own TODO admits this) and is missing libneo from the tarball. No behavioral tests verify the changed activation logic.

Findings:

  1. [blocker] .github/workflows/main.yml (deleted) — No CI exists on this PR. gh pr checks reports no checks; the head SHA has zero check runs (gh api .../check-runs returns total_count: 0). The old main.yml is deleted (justified: 0/94 green). No replacement CI is added. setup.yml runs only on version tags and its own TODO says "NOT exercised by PR CI... Validate on a real tag before cutting the next release." The PR body claims "Verified locally on macOS" but provides no automated evidence. Per review policy, do not approve on PR-body claims or CI that never exercises the changed path. Required fix: add a CI workflow (even a basic shellcheck / path-resolution smoke test on push and PR) that exercises the activation scripts and setup scripts on the changed code path, or block until setup.yml is validated on a tag.

  2. [major] activate.fish:20set -gx CODE (path resolve $INFRA/..) runs unconditionally after the if type -q path / else if realpath / else block. In the else if and else branches, path is not available (that's why those branches exist), so path resolve fails and CODE is set to an empty string. The old code set CODE correctly in each branch:

# old (main): each branch set CODE with the right tool
if type -q path
    set -gx CODE (path resolve $SCRIPT_DIR)
else if type -q realpath
    set -gx CODE (realpath $SCRIPT_DIR)
else
    ...
    set -gx CODE $PWD
end

With CODE empty, $CODE/libneo/build, $CODE/enzyme/current, and cd $CODE silently break on fish < 3.5 (no path builtin). Required fix: set CODE in each branch using the matching resolver, or use a universally available alternative like dirname $INFRA (since INFRA is already an absolute path in all three branches).

  1. [major] scripts/setup/stellopt.sh:4-6 — The new comment says "point it at $INFRA" but the code sets CODE_ROOT="$CODE":
# CODE_ROOT feeds the STELLOPT make configs, which read $(CODE_ROOT)/external/...
# That tree lives in the infra repo, so point it at $INFRA.
export CODE_ROOT="$CODE"

The code is correct ($CODE/external is where prebuilt deps live after relocation) but the comment is wrong: external/ is now workspace-shared at $CODE/external, not in the infra repo ($INFRA/external doesn't exist). Following the comment and changing CODE_ROOT to $INFRA would break STELLOPT's dependency resolution. Required fix: fix the comment to say "point it at $CODE (the workspace root, where external/ lives)" or remove it.

  1. [minor] .github/workflows/setup.yml:94-100 — The release tarball is missing libneo/. In the old layout, libneo was built in this repo and tar -czf /tmp/code.tar.gz . included it. Now libneo is in $CODE/libneo (workspace root, sibling of the checkout) and only external/ is copied into the tarball (cp -r "$CODE/external" ./external). The TODO admits setup.yml is untested but doesn't mention this gap. The devcontainer flow (devcontainer.sh) handles libneo separately (mv /workspaces/libneo /workspaces/code/libneo), so only the release-tarball path is affected. Recommended fix: either copy libneo/ into the tarball (cp -r "$CODE/libneo" ./libneo) or document that the new release tarball intentionally excludes libneo (codes fetch it via find_or_fetch).

  2. [minor] tests/ (whole directory) — No behavioral tests verify the activation path resolution. The PR changes activate.sh, activate.fish, setup.sh, util.sh, and 15+ setup scripts, but tests/ only contains code-specific tests (NEO-2, NEO-RT, libneo, conftest.py). A test that sources activate.sh in a mock workspace and asserts INFRA, CODE, PATH, LD_LIBRARY_PATH, and venv activation would catch the activate.fish bug (finding 2) and any future regressions. Recommended fix: add a shell test that exercises the activation contract in a temp workspace.

Verdict: request_changes — the PR removes all CI without a working replacement, has a real code bug in fish activation (CODE is empty when path is unavailable), and a comment that actively misleads about CODE_ROOT. The core relocation logic is sound, but these issues need to be fixed before merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant