Skip to content

feat: CPU steal-time context + CI real-run matrix - #5

Merged
BlueBirdBack merged 3 commits into
mainfrom
devin/ci-real-run
Oct 11, 2026
Merged

BlueBirdBack merged 3 commits into
mainfrom
devin/ci-real-run

Conversation

@BlueBirdBack

@BlueBirdBack BlueBirdBack commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • CPU steal-time context: new CPU steal (1s sample) informational metric read from /proc/stat (Linux only; skipped on macOS) so host contention on burstable/shared vCPUs is visible in output and result.json. Not scored.

  • Regression test: steal% parsing incl. the no-steal-column case. 24/24 tests pass, shellcheck clean.

  • Explicit install contract: new --install/INSTALL=1 flag (default stays on — primary use case is a fresh VPS missing deps) and the pre-install line now names the mechanism: via your system package manager with sudo: <tools> instead of a vague 'attempting install'.

CI upgrade (apply manually — PAT lacks workflow scope)

The workflow change adding a macOS+ubuntu matrix and a real end-to-end run (bash abs.sh --quick -n --no-network, asserting FULL) could not be pushed. Apply the attached diff to .github/workflows/test.yml:

 .github/workflows/test.yml	2026-10-11 03:24:10
+++ /tmp/test-new.yml	2026-10-11 03:24:19
@@ -9,13 +9,47 @@
 
 jobs:
   regression:
-    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest]
+    runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v5
-      - name: Check shell
+      - name: Check shell (Linux)
+        if: runner.os == 'Linux'
         run: |
           sudo apt-get update
           sudo apt-get install -y shellcheck
           shellcheck abs.sh
+      - name: Check shell (macOS)
+        if: runner.os == 'macOS'
+        run: |
+          brew install shellcheck
+          shellcheck abs.sh
       - name: Run regression suite
         run: python3 -m unittest -v tests/test_abs.py
+
+  end-to-end:
+    # Real benchmark run on real toolchains — the fake-tools suite cannot
+    # catch platform/engine issues (e.g. missing libaio on macOS).
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v5
+      - name: Install dependencies (Linux)
+        if: runner.os == 'Linux'
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y sysbench fio
+      - name: Install dependencies (macOS)
+        if: runner.os == 'macOS'
+        run: brew install sysbench fio
+      - name: Run ABS quick mode (score must be FULL)
+        run: |
+          set -o pipefail
+          bash abs.sh --quick -n --no-network --lang en --json-file /tmp/abs.json | tee /tmp/abs-out.txt
+          python3 -c 'import json; t=json.load(open("/tmp/abs.json"))["score"]["text"]; assert t.startswith("FULL"), t'

The patch's end-to-end job installs sysbench/fio via apt/brew and asserts score.text starts with FULL — if a future runner filesystem ever lacks O_DIRECT support this is the line to relax.

Devin session: https://app.devin.ai/sessions/c74df4e507504d63aa16cf898640ae58
Requested by: BlueBirdBack

BlueBirdBack and others added 3 commits October 11, 2026 03:24
- 'CPU steal (1s sample)' informational metric from /proc/stat (Linux;
  skipped on macOS) so host contention explains depressed CPU scores
  in output and JSON — not scored
- tests: steal% parsing regression incl. no-steal-column case
  (24/24 pass, shellcheck clean)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Keeps auto-install as the default (primary use case is a fresh VPS
missing deps), but makes the contract explicit: --install/INSTALL=1
documents opt-in, and the pre-install line now names the mechanism
('via system package manager with sudo: <tools>') instead of a vague
'attempting install'.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@BlueBirdBack
BlueBirdBack merged commit 15b2baf into main Oct 11, 2026
2 checks passed
@BlueBirdBack
BlueBirdBack deleted the devin/ci-real-run branch October 11, 2026 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant