Skip to content

fix: repair master after the 09-07 batch merge - #115

Open
Kartikey1306 wants to merge 2 commits into
embeddedos-org:masterfrom
Kartikey1306:fix/repair-master-after-batch-merge
Open

fix: repair master after the 09-07 batch merge#115
Kartikey1306 wants to merge 2 commits into
embeddedos-org:masterfrom
Kartikey1306:fix/repair-master-after-batch-merge

Conversation

@Kartikey1306

@Kartikey1306 Kartikey1306 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Problem

eBoot master has not configured, compiled, linked or passed its tests since the 09-07 batch merge (twenty PRs in ninety minutes, each replayed onto a master its author never saw). Every CI — eBoot, eBoot Build & Test, CodeQL and nightly run on master is red, and because eos/.github/workflows/eos-simulation.yml and eos's nightly Upstream drift build eBoot at master, eos is red on those jobs too, regardless of what an eos PR contains.

Verified against run 34177770756 (CI — eBoot on b429354) and 34177770734 (eBoot Build & Test), and reproduced locally on master before every fix below.

What broke, and which merge broke it

Stage Symptom Cause Fix
configure add_executable cannot create target "eboot_test_fdt_loader" #84 registered it; #85, stacked on #84, re-added it at a different anchor when replayed onto a master that already had #84 (my duplicate-insert) first block removed — the same hunk as #111, blob 11f2f3f
compile 'eos_sha512_ctx_t' has no member named 'count' ×5 #55 restored a core/sha512.c from 02b7dac that predates the bitlen[2]/buffer_len context (#69/#93) pre-#55 file restored, blob-identical (d9aa57c)
compile redefinition of 'scalarbase' #94 and #105 each repaired the Ed25519 verifier and each added an identical static scalarbase(); both merged one copy removed
compile redefinition of 'k_low_order' / 'messages' in test_ed25519.c same pair, same reason #105's copy removed; #94's stays because it also carries k_non_canonical[]
link _eos_boot_log_append, _eos_secure_boot, _eos_secure_boot_lock_debug undefined #55's source-list "correction" dropped core/boot_log.c, core/secure_boot.c, core/fdt_loader.c (added by #72/#84 after #55 was written) CMakeLists.txt blob-identical to pre-#55 (f8fe6eb)
compile eos_boot_log_get_head undeclared in test_boot_log.c #55 replaced its declaration with a second copy of eos_boot_log_read(); #91 had already fixed the prototype #55 meant to fix header restored (964ebb8)

Three suites then compiled and failed, because two merged PRs disagree on behaviour:

Guards from #95 that later merges walked back, invisible until now because configure failed first (observed locally: pytest tests/ on master-with-the-C-fixes was 5 failed / 73 passed):

One more, found on this PR's first run (run 34591438682): test_imgpack.py::test_out_of_range_version_is_rejected[-1.0.0] passes on Python 3.13+ and fails on ubuntu-22.04's 3.10, because the older argparse negative-number matcher reads -1.0.0 as an unknown option and rejects it before imgpack.py's own range check runs. The test now passes --version=X, which every interpreter reads as a value; verified the tool's message is what the test then sees.

CI plumbing on the same red runs:

  • eosim-sanity.yml: the install-validate job is written in bash ($(...), || { ... }, a heredoc, /tmp) but ran under PowerShell on the Windows legs, where SITE_PACKAGES=$(...) is an unknown command — and where the || { exit 1 } guard in "Verify installation" parses as an unexecuted script block and could never fail. defaults.run.shell: bash on the job. (Run 34561593562.)
  • scorecard.yml: ossf/scorecard-action@v2.4.0 pulls gcr.io/openssf/scorecard-action, and gcr.io now refuses with "This API method requires billing to be enabled". v2.4.3 pulls from ghcr.io; eos already pins it and its Scorecard job is green. (Run 34177770753.)

Test plan

Verified, macOS / clang:

cmake -DEBLDR_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Release   → 0 errors, 0 warnings
ctest --timeout 120                                        → 100% tests passed, 31/31
cmake -DEBLDR_BUILD_TESTS=ON -DEBLDR_SANITIZE=ON (ASan+UBSan) → 31/31
pytest tests/                                              → 78 passed

Not verified here: the STM32F4 / Cortex-M4 cross build (no arm-none-eabi toolchain on this machine) and the fuzz harness build (no libFuzzer runtime). Both failed on master for the compile errors above, which are gone; CI on this PR is the check.

Relationship to open PRs

Reported, not fixed here

core/keystore.c's compiled-in default_dev_key is described (in a #warning and in comments, including one in my own test_secure_boot_policy.c) as the RFC 8032 TEST 1 public key. It is not: it diverges from byte 21 on and does not decode to a point on the curve (verified by decoding y and checking x² against the curve equation). Nothing can ever verify against it, and since #104 that means firmware update refuses every image on a board without OTP. That is a defect in its own right, unrelated to the batch merge, and is why the tests here provision the key through simulated OTP rather than rely on the fallback. Separate PR to follow.

Twenty PRs were merged into master in ninety minutes on 09-07, each on the
base it was written against. Master has not configured, compiled, linked or
passed its tests since. Every nightly and every eos simulation run (which
builds eBoot master) has been red for the same reason.

Configure:
- tests/CMakeLists.txt registered eboot_test_fdt_loader twice. embeddedos-org#84 added it,
  and embeddedos-org#85 -- stacked on embeddedos-org#84 -- re-added it at a different anchor when it was
  replayed onto a master that already had embeddedos-org#84. Same hunk as embeddedos-org#111.

Compile (eboot_core):
- embeddedos-org#55 restored a core/sha512.c from 02b7dac that keeps its byte count in
  ctx->count; the eos_sha512_ctx_t master ships has bitlen[2]/buffer_len
  (embeddedos-org#69/embeddedos-org#93). Back to the pre-embeddedos-org#55 file, blob-identical (d9aa57c).
- embeddedos-org#55's source-list "correction" dropped core/boot_log.c, core/secure_boot.c
  and core/fdt_loader.c, which embeddedos-org#72/embeddedos-org#84 had added after embeddedos-org#55 was written.
  CMakeLists.txt is blob-identical to pre-embeddedos-org#55 again (f8fe6eb).
- embeddedos-org#55 replaced the eos_boot_log_get_head() declaration with a second copy of
  eos_boot_log_read(); embeddedos-org#91 had already fixed the prototype it meant to fix.
  Header restored (964ebb8).
- embeddedos-org#94 and embeddedos-org#105 each repaired the Ed25519 verifier and each added an identical
  static scalarbase(); both merged. One copy removed.
- The same pair each added k_low_order[]/messages[] to test_ed25519.c. The
  embeddedos-org#105 copy is removed; embeddedos-org#94's stays because it also carries k_non_canonical[].

Tests that stopped passing because two merged PRs disagree on behaviour:
- embeddedos-org#104 verifies the image signature at install unconditionally, before the
  anti-rollback check embeddedos-org#103 added, so embeddedos-org#103's unsigned images are refused as
  EOS_ERR_SIGNATURE before they can be refused as EOS_ERR_ANTI_ROLLBACK, and
  test_fw_transport's XMODEM install can no longer finalize. Both suites now
  stream genuinely signed images. eBoot has no Ed25519 signer in C, so
  tools/gen_fw_update_test_sigs.py signs the exact header prefixes those
  suites build under the RFC 8032 section 7.1 TEST 1 key and emits
  tests/vectors/fw_update_test_sigs.h; the suites serve that key from a
  simulated OTP slot 0. Negative control: one flipped signature byte fails
  test_write_streams_tlv_then_finalize_rejects_below_floor with
  EOS_ERR_SIGNATURE.
- embeddedos-org#103's step 5b reads the TLV counter through the HAL slot containing the
  image; test_secure_boot_policy (embeddedos-org#82) declared no slots, so eos_secure_boot()
  returned EOS_SBOOT_ERR_BAD_HEADER two steps before the one under test. The
  fixture now places its image in slot A.

Guards from embeddedos-org#95 that later merges walked back, never run until now because the
C configure step failed first:
- embeddedos-org#103 replayed the hand-written Valgrind foreach over the derived one.
  Restored foreach(TEST_NAME ${EBLDR_UNIT_TESTS}); eleven registered suites
  had no list(APPEND ...) and so no Valgrind run.
- Seven suites assign tests_run = <literal> and their TEST() does not count;
  four suites have no TEST() macro at all. Counted, and classified.
- embeddedos-org#101 added fuzz-build after embeddedos-org#90's gate; the gate did not wait for it.

CI plumbing:
- eosim-sanity.yml: the install-validate job is written in bash but ran under
  PowerShell on the Windows legs (no shell:), where SITE_PACKAGES=$(...) is an
  unknown command and `|| { exit 1 }` is an unexecuted script block.
- scorecard.yml: ossf/scorecard-action@v2.4.0 pulls gcr.io, which now demands
  GCP billing. v2.4.3 pulls ghcr.io; eos already pins it and is green.

Verified locally (macOS, clang): Release build clean, 31/31 ctest; the same
under -DEBLDR_SANITIZE=ON (ASan+UBSan); 78/78 pytest.

Not fixed here, reported separately: core/keystore.c's compiled-in
default_dev_key is described as the RFC 8032 TEST 1 public key but differs
from byte 21 on and is not a point on the curve, so nothing can verify
against it on any board without OTP. With embeddedos-org#104 that makes firmware update
refuse every image on such boards.
… -1.0.0 as an option

test_out_of_range_version_is_rejected[-1.0.0] passed locally (Python 3.14)
and failed in CI (ubuntu-22.04, Python 3.10) with argparse's own
"expected one argument": the older negative-number matcher does not accept
-1.0.0, so the token was read as an unknown option and imgpack.py's range
check -- the thing under test -- never ran. The joined form is unambiguous
on every interpreter and the test now reaches the tool's message.
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@srpatcha srpatcha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — eBoot#115 "fix: repair master after the 09-07 batch merge"

head: 4d8696c author: Kartikey1306 ci: pass (25 checks green, 0 red; only Create GitHub Release skipped)

Verdict: This repairs a master that does not configure, does not compile, does not
link and has no passing test suite. I reproduced the broken baseline and the repaired one
independently: clean origin/master fails to build; this head configures, builds with no
errors and passes 31/31 under ctest --no-tests=error. Every blob-identity claim in the
PR body checks out against git rev-parse. This should land before the other four open
eBoot PRs
— three of them are blocked behind exactly the breakage it fixes.

Findings

# Severity File:line Finding Recommended fix
1 Medium tests/unit/test_fw_update.c:156-181, tests/unit/test_fw_transport.c A behavioural arbitration is being settled inside a repair PR, with no ADR and no design text backing it. #104 (verify signature at install, before anti-rollback) and #103 (anti-rollback counter check) were both merged and contradict each other; this PR picks #104's ordering and rewrites #103's fixtures to sign their images. The call is right and the reasoning in the body is sound — a counter is only trustworthy after the prefix binding it is authenticated. But the master design does not actually say so for the update path: §8.1 orders "Verify Manifest → Verify Image → Check Version Policy" for boot, while §15's update flow is "Download → Verify → Install" and never places the anti-rollback counter check at all. So the repo now has an ordering rule that exists only in a PR body. .ai/architect.md asks that structural decisions be recorded, and this is one. Nothing to change in the code. Record the decision: an ADR under the §38 series (the natural slot is ADR-011, eOTA firmware/update contract) stating that anti-rollback evaluation happens strictly after signature verification in the install path. I have appended an architecture proposal for §15.1 to .ai/autoreview/proposals/2026-09.md so the design stops being silent on it.
2 Low core/sha512.c (eos_sha512_update) ctx->bitlen[0] += (uint64_t)copy >> 61; is always zero. copy is clamped to 128 - ctx->buffer_len, so copy <= 128 and copy >> 61 == 0 on every iteration. The real 128-bit carry is already handled correctly by the old_low comparison two lines above. The line reads as overflow handling and does nothing, which is the kind of thing a future reader preserves because they assume it matters. Not a defect — the digest is correct, confirmed by KAT below. Delete the line, or replace it with a comment noting that copy is block-bounded so the low word alone cannot overflow within one iteration. Since this file is a byte-exact restoration of a previously-reviewed blob, it is fine to leave for a follow-up rather than perturb the restoration.
3 Low tests/vectors/fw_update_test_sigs.h:1-8, tools/gen_fw_update_test_sigs.py The committed vectors carry a "do not edit / regenerate with …" header, but nothing verifies that the committed bytes are what the committed script produces. If someone edits the header by hand, or changes a field inside the signed prefix in build_image() without regenerating, the failure surfaces as an opaque EOS_ERR_SIGNATURE in an unrelated-looking test rather than as "your vectors are stale". The suites do fail closed, which is the important half. Add a Python test that runs tools/gen_fw_update_test_sigs.py and asserts its stdout equals tests/vectors/fw_update_test_sigs.h. requirements.txt:4 already pins cryptography>=41.0, and tests/unit/test_eos_sign_payload_offset.py:34 already has the "ImportError must fail the job" idiom to copy.

Three findings, all minor, on a 21-file 522-line repair. That is the honest count — I went
looking for more and did not find it.

Verification performed for this review

Detached scratch worktrees under .ai/autoreview/state/verify/. The user's eBoot
checkout was not touched; nothing was committed to a tracked branch or pushed.

Check Result
cmake -B build/host -DEBLDR_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Debug on this head PASS
cmake --build build/host --parallel 4 on this head PASS — no errors
ctest --test-dir build/host --output-on-failure --no-tests=error PASS — 100%, 31/31, 11.04s
Same three steps on clean origin/master FAIL — configure aborts on the duplicate eboot_test_fdt_loader; with that removed, compile fails core/sha512.c ×5 + core/ed25519_verify.c:303. Confirms the stated baseline.
test_ed25519 specifically (covers the restored SHA-512) PASS, 10.40s — RFC 8032 vectors accepted, test_sha512_known_answers, test_sha512_streaming_matches_one_shot
Blob claim core/sha512.c = d9aa57c CONFIRMEDgit rev-parse FETCH_HEAD:core/sha512.cd9aa57ccc57dd28aad380693b0c579e4a2b100be
Blob claim CMakeLists.txt = f8fe6eb CONFIRMED
Blob claim include/eos_boot_log.h = 964ebb8 CONFIRMED
Provenance of d9aa57c (was it really pre-#55?) CONFIRMED — that blob was core/sha512.c at abd4dab (#93, 2026-09-03); effa9a2 (#55, 2026-09-07) replaced it
#111 hunk equivalence claim CONFIRMED — the deletion hunk is identical; the files differ because #115 also fills in EBLDR_UNIT_TESTS. The body says "hunk", which is the accurate word.
pytest tests/ ("78 passed") NOT RUNNo module named pytest on this host. Unverified.

On the SHA-512 "rewrite". core/sha512.c shows as 211+/111− and the first instinct is
that someone hand-wrote a hash primitive into the TCB, which .ai/security.md §Cryptography
would treat harshly. It is not: the file is byte-identical to the blob that was in the tree
from #93 until #55 overwrote it with a version predating the bitlen[2]/buffer_len
context. I read it anyway. Init constants are the FIPS 180-4 SHA-512 IVs; the 128-bit length
is accumulated correctly with the carry into bitlen[0]; final() handles the i > 112
case (no room for the 16-byte length) by transforming a padded block first, and the i == 112
boundary lands exactly right; buffer_len can never exceed 127 on entry to final(), so the
0x80 write cannot overrun buffer[128]; the context is zeroized on completion. Together
with the passing KAT, streaming-vs-one-shot equivalence test, and RFC 8032 vectors — which
would fail on any deviation, since Ed25519 derives its challenge scalar from this hash — the
restoration is sound.

Was anything weakened? No, and I checked specifically because the diff deletes 40 lines
from tests/unit/test_ed25519.c and one line each from test_tlv_auth.c and others.
master declared k_low_order[8][32] and messages[] twice (test_ed25519.c:106 and
:266), which is itself a compile error; the PR removes the first, unused copy and keeps the
one that sits with k_non_canonical[]. All fourteen TEST() cases survive and all fourteen
are still invoked from main() — including test_ed25519_low_order_keys_rejected,
test_ed25519_non_canonical_encodings_rejected and
test_ed25519_low_order_R_with_a_valid_key_is_not_a_forgery, the three that .ai/security.md's
"reject invalid and low-order public keys explicitly" rule depends on. The tests_run = <literal>
deletions are the opposite of a weakening: they replace a hand-maintained count with
tests_run++ inside the TEST() macro, so a suite can no longer silently report 6/6
while running four.

The change also tightens three things worth naming: ci.yml:253 adds fuzz-build to
CI Gate's needs (the gate previously did not wait for it); eosim-sanity.yml pins
shell: bash on a job whose Windows legs ran bash syntax under PowerShell, where the
|| { exit 1 } guard parsed as an unexecuted script block and could never fail; and
tests/CMakeLists.txt:226 restores foreach(TEST_NAME ${EBLDR_UNIT_TESTS}) with fourteen
missing list(APPEND ...) entries filled in, which puts eleven previously-uncovered suites
back under Valgrind. All three are fail-closed repairs of checks that could not fail.

On the test signing key. tools/gen_fw_update_test_sigs.py embeds the RFC 8032 §7.1
TEST 1 secret scalar. That is a published constant in a public IETF document, not a secret,
and the script says so. Only the public half and the resulting signatures reach
tests/vectors/fw_update_test_sigs.h. Per .ai/security.md's "test keys must be
structurally incapable of signing a release artifact": the script is invoked by nothing in
.github/workflows/release.yml, which builds and signs through a separate path, and its
output is a C header consumed only by tests/. No exposure.

Architecture conformance

Conforms. §21: eBoot is Tier 1 Foundation; every file touched is inside the owning repo.
§5.1 dependency direction is preserved and in one place restored — the CMakeLists.txt
change adds core/boot_log.c, core/secure_boot.c and core/fdt_loader.c back to
eboot_core, which is the correct tier for them under .ai/architect.md's target shape
(core/ = shared boot logic). No edge points up a tier; nothing outside eBoot is referenced.
§5.1's "eBoot keeps the trusted computing base minimal and auditable" is directly served:
before this PR the secure-boot implementation was not compiled into the core library at
all
, so eos_secure_boot, eos_secure_boot_lock_debug and eos_boot_log_append were
undefined symbols — a TCB that does not link is not auditable. §8.1's boot ordering is
unchanged. The include/eos_boot_log.h edit looks at first like a removed public function,
but master declared eos_boot_log_read() twice (:61 and :72) and never declared
eos_boot_log_get_head(), which core/boot_log.c:57 defines; the PR replaces the redundant
declaration with the missing one. No public API is lost, so brief item 8 is not engaged.

The pattern behind this PR

Worth recording beyond this review: git log -- core/sha512.c shows five separate
"restore the build / repair merge artifacts" commits between 2026-08-27 and 2026-09-11
(9fbfc91, 45519d7, ffda321, effa9a2, a8743dd). The proximate cause named here —
twenty PRs merged in ninety minutes, each replayed onto a master its author never saw —
is a merge-process defect, not a code defect, and no amount of repair PRs will stop it
recurring. A required status check on master plus a merge queue that rebases and re-runs
before landing would. That is outside this PR's scope and is carried into the run summary.

Proposed changes

Merge order (each of the next three is blocked until this lands):
  1. #115  (this PR)
  2. #111  -> its tests/CMakeLists.txt hunk becomes empty; close it or let it rebase to nothing
  3. #109  -> rebase; resolve test_jump_app_bounds.c:main() by DROPPING `tests_run = 3;`
  4. #112, #116, #117 -> rebase and get their first real CI

Follow-ups, none blocking:
  - delete the dead `copy >> 61` term in core/sha512.c
  - add a test regenerating tests/vectors/fw_update_test_sigs.h and diffing it
  - ADR-011 addendum: anti-rollback is evaluated after signature verification

Not checked

  • pytest tests/ — NOT RUN. pytest is not installed on this host, so the body's
    "78 passed" is unverified here, as is test_imgpack.py's --version=X argparse fix and
    the four new NO_TEST_MACRO classifications in test_suite_bookkeeping.py. The reasoning
    for the argparse change is correct on its face (3.10's negative-number matcher treats
    -1.0.0 as an unknown option), but I did not run it on 3.10 to confirm.
  • Cross-compile (STM32F4, Cortex-M4) and the fuzz harness build — not run locally. CI
    reports both green on this head and I am taking that at face value; I did not read the logs.
  • ASan/UBSan — not run. The body claims 31/31 under -DEBLDR_SANITIZE=ON; I built Debug
    without sanitizers. Unverified.
  • The #104 vs #103 negative control. The body says flipping one signature byte makes
    test_write_streams_tlv_then_finalize_rejects_below_floor fail with EOS_ERR_SIGNATURE.
    I did not reproduce that mutation, so the claim that these tests would catch a signature
    regression is inferred from reading build_image(), not observed.
  • The gcr.io billing explanation for the scorecard-action bump. The version change is
    benign either way, but I did not verify the stated cause.
  • I did not audit the other ~18 PRs from the 09-07 batch for damage this PR did not find.
    The claim "this is everything that broke" is the author's; my evidence only supports
    "everything that breaks configure, compile, link and ctest on the host is now fixed."

Automated architecture review of 4d8696cbecbd — scheduled, model claude-opus-5, checked against the EmbeddedOS Master Design v2.0. Advisory only: this reviewer never approves, requests changes, or merges. Reply here to discuss or push back — a wrong finding is a bug worth reporting.

@srpatcha srpatcha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this — it is a careful repair and the description made it easy to check. What I verified locally on your branch (gcc, Ninja, Debug): configure OK, build clean apart from the intentional #warning in keystore.c, ctest 31/31, the 31 valgrind_* targets 31/31, pytest 78/78. I also confirmed the three "blob-identical" claims by hash, reproduced the master configure failure and the sha512.c/ed25519_verify.c:303 compile failures on #111's branch, and confirmed tools/gen_fw_update_test_sigs.py regenerates tests/vectors/fw_update_test_sigs.h byte-for-byte.

One non-blocking note: tests/unit/test_fw_update.c:174-181 — since the C fixture and the Python generator each build the signed prefix independently, a future edit to either silently breaks the other with EOS_ERR_SIGNATURE. A short comment above build_image() naming the regenerate command would help.

For the maintainers: #116 is stacked on this branch, and #111 is a strict subset of the tests/CMakeLists.txt hunk here. Approving — this should land first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants