Skip to content

Device harness: a physical iPhone and an iOS crash lab - #35

Merged
krassx merged 5 commits into
nextgenfrom
feat/ios-device-harness
Nov 18, 2026
Merged

krassx merged 5 commits into
nextgenfrom
feat/ios-device-harness

Conversation

@krassx

@krassx krassx commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #34.

  • verify.dart ios-device installs the signed release build on a physical iPhone with devicectl, launches it with its console attached (the child's exit proves the process ended), and runs the crash matrix (scenario 10) against the mock API bound to the Mac's LAN address (BUGSEE_VERIFY_MOCK_BIND, one address, never all interfaces; MOCK_BIND in mock_api.py, loopback by default).
  • The example's iOS crash lab (AppDelegate.swift, channel verify/crashlab): an uncaught Objective-C exception on the main thread and on a worker, and a Swift runtime trap; the kind table now says per platform where each kind exists, and a test compares it with the three apps' sources.
  • Not run to the end. Build, signing, install and launch work on an iPhone XS (console shows Bugsee IOS SDK ver:7.0.1 build:5c6b9d9db-8, before the 7.1.0 move), but the first attempt ended INVALID at "SDK launched": the phone says "The Internet connection appears to be offline", which means the Local Network prompt was not allowed (a script cannot answer it) or the phone is not on the Mac's network. The iOS expectations in the matrix are deliberately loose until a first real run shows what an iOS crash report holds (the Swift trap is expected as a signal 5 note, not a marker). Review fixes: every path stops the devicectl children and uninstalls, the mock host is passed explicitly (only this command leaves the loopback), the recovered report is accepted (200) so it cannot block the next launch. The mock on a LAN address has no authentication: run it on a network you trust.
  • Tests: flutter test 1389, analyze.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WeiJxNbNUdWru1zs68bYV8

@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 58 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f0f25a2a-9580-46ac-86df-b4d196ec2dca
Commits

Reviewing files that changed from the base of the PR and between 3a6b808 and 84da81e.

Files selected for processing: 12

  • doc/verification.md
  • example/integration_test/support/mock_api.py
  • example/ios/Runner/AppDelegate.swift
  • example/lib/crash_lab.dart
  • scripts/device/android_scenarios.dart
  • scripts/device/core/crash_kinds.dart
  • scripts/device/core/lan_host.dart
  • scripts/device/ios_physical.dart
  • scripts/device/mock.dart
  • scripts/device/toolchain.dart
  • scripts/device/verify.dart
  • test/device_harness_test.dart
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@claude

claude Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Claude finished @krassx's task in 1m 3s —— View job


Review of PR #35: physical iPhone harness and iOS crash lab

I read the full diff against feat/device-harness-physical and the code around it: Run, Child, mock_api.py, Info.plist and the kind table with its users. I did not run any tests, the analyzer or the harness. It needs a physical iPhone, which I don't have, and the PR notes the matrix itself has not run to the end either.

I posted 3 inline findings, none higher than P2.

Findings

Sev Where Summary
P2 scripts/device/ios_physical.dart:274 A failed guard leaks the devicectl --console children, the running app and the installed app.
P3 example/integration_test/support/mock_api.py:126-130 The LAN-bound mock is unauthenticated, and the "Loopback only" comment above the new one is now stale.
P3 scripts/device/mock.dart:29 BUGSEE_VERIFY_MOCK_BIND is read globally, so it also affects ios and android runs, and a hostname makes InternetAddress throw.

1. Overall risk: Low. This is dev-only tooling that CI does not run. Nothing in the shipped plugin changes, and the example app's crash lab is example code. crash_kinds.dart and the Android loop now filter by platforms, and the drift test maps each kind to the source that holds its marker. I found no regression in the Android or simulator paths.

2. Merge recommendation: Merge once the P2 cleanup is fixed, or accept it explicitly. This is a stacked PR on #34, and it is honest that the matrix has not run to the end.

3. Most important issues to fix

  • Stop the console children and uninstall in a finally so an INVALID run leaves nothing behind (P2).
  • Fix the stale and contradictory comment in mock_api.py, and say that the LAN bind is unauthenticated. Ideally add a per-run token and a body cap (P3).
  • Pass the mock host into Mock.start explicitly instead of reading the environment everywhere (P3).

4. Positives

  • It binds a single address, refuses the loopback for a phone, and requires --udid rather than picking a device.
  • The iOS expectations are deliberately loose until a first measured run, and the PR and doc/verification.md say plainly that the matrix has not run.
  • The kind table now records per-platform existence, and test/device_harness_test.dart checks each marker against the right app source.
  • The crash lab raises its exceptions from their own dispatch or thread, so the SDK sees an uncaught exception rather than one thrown inside the handler.
  • Info.plist already has NSLocalNetworkUsageDescription and NSAllowsLocalNetworking, so the offline error is probably the unanswered Local Network prompt, as the PR suspects.

Observation, not a finding: ffi-fpe raises SIGFPE via raise(), so it should still work on iOS arm64. A real arithmetic trap would not. Worth confirming on the first run.
· Branch: feat/ios-device-harness

Comment thread scripts/device/ios_physical.dart Outdated
first = false;
await delay(5);
await mock.command('crash ${k.name}');
final code = await c1.process.exitCode.timeout(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Medium — a failed guard leaks the console child, the running app and the installed app

c1 and c2 are devicectl ... --console child processes. They are only stopped on the happy path (c2.stop() at line 316). Every run.guard throws InvalidExperiment, and runScenario catches it. Two guards sit between a child's start and its stop:

  • SDK launched (line 238)
  • the process-ended check at line 279, which is a guard because it uses run.guard

When either throws, three things happen:

  • the devicectl child is never stopped, and Process.start children outlive exit();
  • the app keeps running on the phone with the console attached;
  • dev.uninstall() (line 321) is skipped, so the app stays installed.

The PR description says the first real run ended INVALID at "SDK launched", so this is the path that has already run. A re-run is partly protected by --terminate-existing and the uninstall at line 257. The orphaned devicectl processes pile up on the Mac, though, and a half-finished crash lab stays on the phone.

Also, on the -999 timeout at line 276 the app is by definition still alive, and c1 is not stopped before the guard fails.

Fix: track the live children in _Ctx and stop them, then uninstall, in a finally in crashMatrix (or in runIosDevice's finally).

Comment on lines +126 to +130
# Loopback only: the server stores whatever is PUT to it. The Android emulator reaches it as 10.0.2.2 and
# the iOS simulator as localhost, both of which arrive on the loopback interface of the host.
ThreadingHTTPServer(('127.0.0.1', int(sys.argv[1])), H).serve_forever()
# Loopback unless told otherwise: a PHYSICAL iPhone reaches the Mac only over the LAN, so the device
# harness may bind the one LAN address it prints (MOCK_BIND), never all interfaces.
ThreadingHTTPServer((os.environ.get('MOCK_BIND', '127.0.0.1'), int(sys.argv[1])), H).serve_forever()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 Low — the server is now on the LAN with no authentication, and the comment contradicts itself

The original reasoning was "loopback only: the server stores whatever is PUT to it". That reasoning is still in lines 126-127, directly above the new text, and now reads as false. MOCK_BIND is a single address, which is better than 0.0.0.0. Even so, any host on the same network can now:

  • POST /__cmd: queue commands for the verification app. crash, and net <url> (an arbitrary URL fetched from the phone), are both commands.
  • POST /__event: forge events.
  • PUT /upload/*: forge bundles. Each upload is stored with an unbounded content-length read into memory.
  • POST /__reset: wipe the evidence.

That can fake or destroy the evidence a verification run produces, and the harness trusts the mock as its source of truth. On an office or shared Wi-Fi this is a real exposure.

Fix: at minimum, drop the stale "Loopback only" sentences and say plainly that a LAN bind is unauthenticated. Better, require a per-run token header (generated by Mock.start, baked into the build as a --dart-define, checked in _do) and cap the body size.

Comment thread scripts/device/mock.dart Outdated
// readiness probe and the harness would talk to the wrong one.
try {
final probe = await ServerSocket.bind(InternetAddress.loopbackIPv4, port);
final probe = await ServerSocket.bind(InternetAddress(mockHost), port);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 Low — mockHost is read from the environment for every platform, not just ios-device

mockHost is the process-wide BUGSEE_VERIFY_MOCK_BIND value. It reaches Mock.start, the readiness probe and every Mock client call for the Android and simulator runs as well.

Suppose someone exports the variable for a phone run and later runs verify.dart ios or android in the same shell. The mock then binds the LAN address while those apps were built for localhost or 10.0.2.2. The run fails as a confusing INVALID, and the emulator and simulator runs are needlessly exposed on the LAN.

InternetAddress(mockHost) also throws a bare ArgumentError if a hostname rather than an IP is given.

Fix: pass the host into Mock.start explicitly (default loopback) and have only runIosDevice supply the LAN address. If a global is kept, validate it as an IP in one place and ignore it outside ios-device. Also, the header comment at line 3 now says "binds 127.0.0.1 only (one LAN address ...)", which reads oddly.

@claude

claude Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Claude finished @krassx's task in 51s —— View job


Re-review of PR #35 (push to feat/ios-device-harness)

  • Read the earlier review and the current diff against feat/device-harness-physical
  • Check whether the earlier findings are fixed
  • Look for new issues
  • Post inline findings and this summary

The head is 9f1d2f2, the same three commits I reviewed before. None of my three earlier findings is fixed in the code:

  • P2: scripts/device/ios_physical.dart still has no finally for the devicectl --console children or the uninstall.
  • P3: example/integration_test/support/mock_api.py still says "Loopback only" above the new MOCK_BIND text.
  • P3: mockHost in scripts/device/toolchain.dart is still read globally.

I am not repeating them. I read the code and did not run tests, the analyzer or the harness, because the harness needs a physical iPhone.

New findings (2 inline comments, both P3)

Sev Where Summary
P3 scripts/device/ios_physical.dart:278 Mock.start defaults to a 503 upload status. The recovered report stays pending, and per CLAUDE.md pending reports make iOS stop a later launch of the same install. Kind 2 onward could end INVALID at "SDK launched", with a message that blames the Local Network prompt.
P3 example/ios/Runner/AppDelegate.swift:43 fatalError("verify-swift-trap") is a trap, not an exception, so the report probably does not hold the marker. The run.expect on it will probably fail.

Summary

  1. Overall risk: Low. This is dev-only tooling that CI does not run, and the shipped plugin is untouched.
  2. Merge recommendation: Merge once the P2 cleanup is fixed or explicitly accepted. This is a stacked PR and it says plainly that the matrix has not run to the end.
  3. Most important issues to fix:
    • Stop the console children and uninstall in a finally (earlier P2).
    • Settle the upload status or per-kind reinstall question before the first full run, so later kinds do not fail on the pending-report rule (new P3).
    • Drop or soften the swift-trap marker expectation (new P3).
    • Fix the "Loopback only" comment and pass the mock host into Mock.start explicitly (earlier P3s).
  4. Positives:
    • It binds a single address, requires --udid, and refuses the loopback for a phone.
    • The kind table now records per-platform existence, and a test checks each marker against the right app source.
    • The crash lab raises its exceptions from their own dispatch or thread, so the SDK sees an uncaught exception.
      · Branch: feat/ios-device-harness

Comment thread scripts/device/ios_physical.dart Outdated
const Duration(seconds: 45),
onTimeout: () => -999,
);
final out = c1.output.toString();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 Low: a refused (503) recovered crash report may stay pending and stop the next kind's launch

Problem: Mock.start defaults to uploadStatus = 503, and runIosDevice does not change it. The recovered crash report that c2 captures is therefore refused by the mock and kept by the SDK. The loop then relaunches the same install for the next kind through launch(run, first: false), which guards on launched['result'] == 'launched'.

Impact: CLAUDE.md records that on iOS "pending reports make iOS stop a later launch of the same install". That is why the guidance says to run simctl uninstall between facade_test and launch_test. If it applies here, the matrix gets through the first kind and ends INVALID at "SDK launched" for every later kind, with an error message that blames the Local Network prompt.

Scenario: Kind 1 (ffi-segv) crashes and c2 recovers the report. The mock stores it and answers 503, so the SDK keeps it. Kind 2 launches c1 and the SDK reports already-running or a stopped result. The guard fails.

Fix: Either reinstall (uninstall, then install) per kind, or start the mock with uploadStatus: 200 for this scenario so the recovered report is accepted. If neither is needed, say so in a comment citing a measurement. The failure text in launch should also name the other cause (a pending report) so the Local Network hint does not mislead.

I did not run this on a device. It follows from the repo's own note about pending reports.

}
result(nil)
case "swiftTrap":
DispatchQueue.main.async { fatalError("verify-swift-trap") }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 Low: the swift-trap marker is unlikely to appear in the crash report, so a hard run.expect will probably fail

Problem: fatalError("verify-swift-trap") ends in a brk/EXC_BREAKPOINT trap (SIGTRAP). Swift prints the message to stderr (and to the system crash-info buffer), but it is not an exception name or reason. The SDK's native crash path normally records the signal and the stack. The matrix still requires text.contains('verify-swift-trap') in crash.json or the request, as a run.expect (ios_physical.dart:300-306).

Impact: objc-main and objc-thread carry their markers in an NSException reason, which the SDK does read. swift-trap is a different kind of crash, and its marker check will most likely fail by design. That makes the first real run report a failure for a kind that did crash and was recovered. The PR describes the iOS expectations as loose, and this is the one that is not.

Fix: For swift-trap, set marker: null in crash_kinds.dart, and note that the report only shows a trap. Alternatively, keep the marker as a run.note until a first run shows what the report holds. test/device_harness_test.dart requires a non-null marker for non-signal kinds (the else branch around line 810), so that test needs the matching change. A signal-style entry (signal: 5) fits better.

I did not run this on a device.

@krassx
krassx force-pushed the feat/device-harness-physical branch from c643937 to 1cc3a5a Compare November 17, 2026 10:28
@krassx
krassx force-pushed the feat/ios-device-harness branch from 9f1d2f2 to 3ab8049 Compare November 17, 2026 10:29
@claude

claude Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Claude finished @krassx's task in 1m 0s —— View job


Re-review of PR #35 (push of 3ab8049, "Review findings: ...")

I read the new commit and the code around it. I did not run tests, the analyzer or the harness, because the matrix needs a physical iPhone.

Earlier findings: all five look fixed in the code

  • P2, leaked devicectl children and the install: crashMatrix now wraps the loop in try/finally with _Ctx.cleanup(). It stops every child in _live and uninstalls the app.
  • P3, "Loopback only" comment: the comment in mock_api.py now says there is no authentication and a trusted network is needed. Request bodies are capped (MAX_BODY, 256 MB). A per-run token was not added. The unauthenticated LAN bind is documented and accepted, and a 256 MB cap is a weak limit against a hostile LAN host.
  • P3, mockHost read globally: Mock.start takes host explicitly (default loopback), and only runIosDevice passes the LAN address.
  • P3, 503 upload leaves a pending report: runIosDevice starts the mock with uploadStatus: 200. The launch failure text also names the pending-report cause now.
  • P3, swift-trap marker: the kind is now native with signal: 5 and no marker. The report's signal is only noted, not asserted. test/device_harness_test.dart fits that.

New findings (2 inline comments)

Sev Where Summary
P2 scripts/device/ios_physical.dart:316 The device matrix only notes the SDK banner and never guards it against native-versions.json. The simulator path does (exp.iosProblems, workbook 9.1.1).
P3 scripts/device/ios_physical.dart:94 "One address, never all interfaces" is not enforced: 0.0.0.0 and :: pass, as do 127.0.0.2 and ::1. IPv6 gives a malformed URL. Only the exact string 127.0.0.1 is refused.

The banner finding matters most. The pin moved to 7.1.0 (b65bdca06-9), yet the phone run in the PR text printed 7.0.1. A stale --reuse build would go through the whole matrix unnoticed, and the result would be recorded as 7.1.0 evidence.

Summary

  1. Overall risk: Low. This is dev-only tooling that CI does not run, and the shipped plugin is untouched.
  2. Merge recommendation: Merge after the banner guard is added. It is small, reuses exp.iosProblems, and CLAUDE.md requires the harness guards to stay on. The address check can be a follow-up.
  3. Most important issues to fix:
    • Guard the SDK banner and wrapper identity in the device matrix (P2).
    • Reject 0.0.0.0, ::, loopback and IPv6 for BUGSEE_VERIFY_MOCK_BIND, with a test (P3).
  4. Positives:
    • The cleanup now covers every guard failure, and c1 is registered before its guard.
    • Accepting the recovered report with 200 follows the repo's own pending-report rule, and the comment cites it.
    • The kind table's per-platform split is tested against each app's source.
    • The iOS expectations stay measured, not asserted, until a real report exists.
    • The docs say plainly that the matrix has not run to the end.
      · Branch: feat/ios-device-harness

);
final banner = parseIosBanner(out);
if (banner != null) {
seenBanner.add('${banner.version} ${banner.build}');

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Medium: the device matrix never guards which SDK build it ran against (workbook 9.1.1)

Problem: the banner is parsed here and only collected into seenBanner, then printed with run.note('console banners seen: ...'). ios_scenarios.dart:266-271 guards it with run.guard('build: SDK banner and wrapper identity', exp.iosProblems(banner, wrapper), ...), which compares the version and build to native-versions.json (ios.sdk 7.1.0, ios.sdkBuild b65bdca06-9). The device path has no such guard. CLAUDE.md says the harness guards "are on by default and must stay so".

Scenario: the PR description and doc/verification.md report the banner ver:7.0.1 build:5c6b9d9db-8 from the phone, while the pin has moved to 7.1.0 (native-versions.json provenance: "moved from 7.0.1 on 2026-11-17"). A stale --reuse build or a stale SwiftPM resolution would go through the whole matrix, and a PASS would be recorded for 7.0.1 as if it were the 7.1.0 evidence. The provenance text says the harness "matches it on the next run", and this path cannot do that.

Fix: after the first launch of the first kind, run run.guard with exp.iosProblems(banner, wrapper) (the wrapper log is in the same console output), or at least guard version and build. The iOS 7.1.0 matrix is the evidence this PR is meant to produce, so the guard should be there before the first full run.

return 64;
}
if (mockHost == '127.0.0.1') {
stderr.writeln(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 Low: "never all interfaces" is not enforced; only the exact string 127.0.0.1 is refused

Mock.start validates with InternetAddress.tryParse(host) == null, and this check rejects only mockHost == '127.0.0.1'. These values pass both:

  • BUGSEE_VERIFY_MOCK_BIND=0.0.0.0 (or ::) binds every interface. The docs and the PR text promise "one address, never all interfaces". The phone also cannot connect to 0.0.0.0, so the run would end INVALID at "SDK launched" after the build.
  • 127.0.0.2 or ::1 slip past the loopback refusal.
  • An IPv6 address passes validation, but http://$host:$port and --dart-define=BUGSEE_ENDPOINT=http://$host:$port are built without brackets, so the URL is malformed.

Fix: in runIosDevice (or in Mock.start), parse once and refuse isLoopback, 0.0.0.0, :: and non-IPv4 addresses. Unit-test it in test/device_harness_test.dart.

@krassx
krassx force-pushed the feat/device-harness-physical branch from 1cc3a5a to cec3fe7 Compare November 17, 2026 11:46
@krassx
krassx force-pushed the feat/ios-device-harness branch from 3ab8049 to 66fc3e0 Compare November 17, 2026 11:47
@claude

claude Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Claude finished @krassx's task in 43s —— View job


Re-review of PR #35 (push of 66fc3e0, "Review findings: the iOS device matrix guards the SDK banner and wrapper identity ...")

I read the new commit and the code it touches: ios_physical.dart, core/lan_host.dart, mock.dart, exp.iosProblems, parseWrapperLog and the new test. I did not run flutter test, the analyzer or the harness. The matrix needs a physical iPhone.

Earlier findings: both are fixed in the code

  • P2, no guard on the SDK build: crashMatrix now calls run.guard('build: SDK banner and wrapper identity', (await tc.expectation('release')).iosProblems(banner, wrapperLog), ...), the same call the simulator path uses. A banner that is missing or has the wrong version or build, or a wrapper that is not this one, makes the run INVALID. A stale --reuse build can no longer be recorded as 7.1.0 evidence.
  • P3, "one address, never all interfaces" not enforced: core/lan_host.dart adds bindHostProblem and lanHostProblem.
    • Mock.start refuses names, 0.0.0.0 and ::.
    • runIosDevice also refuses IPv6, loopback (127.0.0.2, ::1) and multicast.
    • test/device_harness_test.dart covers the cases I listed.

New findings

None. No inline comments this time.

Observation, not a finding

The new guard reads the banner and the wrapper registered / wrapper refined lines from the devicectl --console output. The PR text shows the banner reaching the console. I could not tell from the code whether the wrapper's NSLog lines (BGSFlutterWrapper.m:199) reach it too. On the simulator they come from the unified log, not from the console.

If they don't, the first full run ends INVALID at this guard with wrapper registered as null, after the first crash has already been induced. If that happens, the harness has a gap to fix, not an SDK finding. You could read devicectl device process launch --console output on the first run, or add devicectl device info logs or log collect, before trusting that guard's message.

Summary

  1. Overall risk: Low. This is dev-only tooling that CI does not run, and the shipped plugin is untouched.
  2. Merge recommendation: Ready to merge as a stacked PR. All findings from my three earlier reviews look addressed. The PR text and doc/verification.md still say plainly that the matrix has not run to the end, so the first real run should confirm the console carries the wrapper log lines.
  3. Most important issues to fix: None blocking. Check on the first run that the wrapper log reaches the console (see the observation).
  4. Positives:
    • The device matrix now enforces the pins, like every other harness path.
    • The host validation sits in one small file and has a table test.
    • Mock.start keeps loopback as its default.
    • The cleanup, the 200 upload status and the explicit mock host from the earlier fixes are all still in place.
      · Branch: feat/ios-device-harness

@krassx
krassx force-pushed the feat/device-harness-physical branch from cec3fe7 to 743eee5 Compare November 17, 2026 12:20
@krassx
krassx force-pushed the feat/ios-device-harness branch from 66fc3e0 to aeedf05 Compare November 17, 2026 12:21
@claude

claude Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Claude finished @krassx's task in 25s —— View job


Re-review of PR #35 (push to feat/ios-device-harness, head aeedf05)

The head's message matches the commit I reviewed last time as 66fc3e0 ("Review findings: the iOS device matrix guards the SDK banner and wrapper identity ..."). Its hash differs, so it may have been rebased. I did not diff the two. I read the current code of the touched files directly: ios_physical.dart, core/lan_host.dart, mock.dart, toolchain.dart, verify.dart, mock_api.py and the new test. I did not run flutter test, the analyzer or the harness. The matrix needs a physical iPhone.

Earlier findings: all still fixed in the code

  • Leaked devicectl children and install: crashMatrix wraps the loop in try/finally with cleanup().
  • Mock host: Mock.start takes host explicitly (default loopback). Only runIosDevice passes the LAN address, and it validates it with lanHostProblem. bindHostProblem in Mock.start refuses names, 0.0.0.0 and ::. The test covers the cases.
  • Pending report blocking later launches: the mock starts with uploadStatus: 200, and the launch failure text names the pending-report cause.
  • swift-trap marker: the kind carries a signal and no marker, and the report's signal is only noted.
  • Unauthenticated LAN bind: the comment in mock_api.py says so plainly, and bodies are capped at MAX_BODY.
  • SDK banner guard: run.guard('build: SDK banner and wrapper identity', ...iosProblems(banner, wrapperLog)) is in the loop, as on the simulator path.

New findings

None. No inline comments.

Carried-over observation, not a finding

The banner guard reads the SDK banner and the wrapper registered / wrapper refined lines from the devicectl --console output. The PR text shows the banner reaching the console. I could not tell from the code whether the wrapper's NSLog lines reach it too. If they don't, the first full run ends INVALID at that guard (wrapper registered as null) after the first crash has been induced. That would be a harness gap, not an SDK finding. Read the console output on the first real run before trusting that guard's message.

Summary

  1. Overall risk: Low. This is dev-only tooling that CI does not run, and the shipped plugin is untouched.
  2. Merge recommendation: Ready to merge as a stacked PR. The PR text and doc/verification.md still say the matrix has not run to the end.
  3. Most important issues to fix: None blocking. On the first real run, check that the wrapper log reaches the console.
  4. Positives:
    • The host validation sits in one small file, with a table test.
    • Cleanup now covers every guard failure.
    • The device matrix enforces the native pins like every other harness path.
    • The kind table's per-platform split is tested against each app's source.
    • The iOS expectations stay measured, not asserted, until a real report exists.
      · Branch: feat/ios-device-harness

@krassx
krassx force-pushed the feat/device-harness-physical branch from 743eee5 to 06d4fdf Compare November 17, 2026 12:52
@krassx
krassx force-pushed the feat/ios-device-harness branch from aeedf05 to db3fb37 Compare November 17, 2026 12:52
@claude

claude Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Claude finished @krassx's task in 26s —— View job


Re-review of PR #35 (push to feat/ios-device-harness, head db3fb37)

The head is db3fb37, with the same message as the commit I reviewed before. The earlier hashes (66fc3e0, aeedf05) don't exist in this checkout, so the branch was rebased. I couldn't diff against them. I re-read the current code of scripts/device/ios_physical.dart (the matrix, launch and cleanup paths) and scripts/device/core/lan_host.dart. I did not run flutter test, the analyzer or the harness. The matrix needs a physical iPhone.

Earlier findings: all still fixed in the code

  • Leaked devicectl children and install: the matrix loop is wrapped in try/finally with cleanup(). cleanup() stops every child in _live and uninstalls the app.
  • SDK banner and wrapper identity: run.guard('build: SDK banner and wrapper identity', ...iosProblems(banner, wrapperLog)) runs for each kind.
  • Bind address: bindHostProblem and lanHostProblem refuse names, 0.0.0.0, ::, IPv6, loopback and multicast.
  • Pending report blocking later launches: the mock starts with uploadStatus: 200. The launch failure text also names the pending-report cause.
  • swift-trap marker: the report's signal is only noted, not asserted.

New findings

None. No inline comments.

Carried-over observation, not a finding

The banner guard reads the SDK banner and the wrapper registered / wrapper refined lines from the devicectl --console output of c1. The PR text shows the banner reaching the console. The code doesn't show whether the wrapper's NSLog lines reach it too. If they don't, the first full run ends INVALID at that guard (wrapper registered as null), after the first crash has been induced. That would be a harness gap, not an SDK finding. On the first real run, read the console output before trusting that guard's message.

Summary

  1. Overall risk: Low. This is dev-only tooling that CI does not run, and the shipped plugin is untouched.
  2. Merge recommendation: Ready to merge as a stacked PR. The PR text and doc/verification.md still say the matrix has not run to the end.
  3. Most important issues to fix: None blocking. On the first real run, check that the wrapper log reaches the console.
  4. Positives:
    • Cleanup now covers every guard failure.
    • The device matrix enforces the native pins like every other harness path.
    • Host validation sits in one small file, with a table test.
    • The kind table's per-platform split is tested against each app's source.
    • The iOS expectations stay measured, not asserted, until a real report exists.
      · Branch: feat/ios-device-harness

@krassx
krassx force-pushed the feat/device-harness-physical branch from 06d4fdf to 55e123c Compare November 17, 2026 18:10
@krassx krassx closed this Nov 17, 2026
@krassx
krassx force-pushed the feat/ios-device-harness branch from db3fb37 to 55e123c Compare November 17, 2026 18:10
@krassx krassx reopened this Nov 17, 2026
@claude

claude Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Claude finished @krassx's task in 18s —— View job


Re-review of PR #35 (push to feat/ios-device-harness, head 376eb74)

The head's message is the same as the commit I reviewed last time ("Review findings: the iOS device matrix guards the SDK banner and wrapper identity ..."). Its hash differs again, so the branch was rebased. I re-read the current scripts/device/ios_physical.dart (launch, matrix and cleanup) and scripts/device/core/lan_host.dart. I did not run flutter test, the analyzer or the harness. The matrix needs a physical iPhone.

Earlier findings: all still fixed in the code

  • Leaked devicectl children and install: the matrix loop is in try/finally with cleanup(). It stops every child in _live, and launch registers the child before its guard.
  • SDK banner and wrapper identity: run.guard('build: SDK banner and wrapper identity', ...iosProblems(banner, wrapperLog)) runs for each kind.
  • Bind address: bindHostProblem and lanHostProblem refuse names, 0.0.0.0, ::, IPv6, loopback and multicast.
  • Pending report blocking later launches: the mock starts with uploadStatus: 200, and the SDK launched failure text names the pending-report cause.
  • swift-trap marker: the report's signal is only noted, not asserted.
  • Unauthenticated LAN bind: documented in mock_api.py, and request bodies are capped.

New findings

None. No inline comments.

Carried-over observation, not a finding

The banner guard reads the SDK banner and the wrapper registered / wrapper refined lines from c1's devicectl --console output, after the first crash has already been induced. The PR text shows the banner reaching the console. The code doesn't show whether the wrapper's NSLog lines reach it too. If they don't, the first full run ends INVALID at that guard (wrapper registered as null). That would be a harness gap, not an SDK finding. On the first real run, read the console output before trusting that guard's message.

Summary

  1. Overall risk: Low. This is dev-only tooling that CI does not run, and the shipped plugin is untouched.
  2. Merge recommendation: Ready to merge as a stacked PR. The PR text and doc/verification.md still say the matrix has not run to the end.
  3. Most important issues to fix: None blocking. On the first real run, check that the wrapper log reaches the console.
  4. Positives:
    • Cleanup covers every guard failure.
    • The device matrix enforces the native pins like every other harness path.
    • Host validation sits in one small file, with a table test.
    • The kind table's per-platform split is tested against each app's source.
    • The iOS expectations stay measured, not asserted, until a real report exists.
      · Branch: feat/ios-device-harness

krassx and others added 5 commits November 18, 2026 00:38
… lab

`verify.dart ios-device` installs the signed release build on a physical iPhone with
devicectl, launches it with its console attached (the child's exit is the proof the
process ended), and runs the crash matrix against the mock API bound to the Mac's
LAN address (BUGSEE_VERIFY_MOCK_BIND, one address, never all interfaces; MOCK_BIND
in mock_api.py, loopback by default). The first launch after an install waits for
the Local Network prompt, which a script cannot answer.

The example's iOS crash lab (AppDelegate.swift, channel verify/crashlab) adds an
uncaught Objective-C exception on the main thread and on a worker and a Swift
runtime trap to the shared kind table, which now says per kind where it exists.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WeiJxNbNUdWru1zs68bYV8
Change-Id: If8850e3e388377d3644035e484002e3db3af809f
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WeiJxNbNUdWru1zs68bYV8
Change-Id: I9c2d3d6ca0c72cc1fa19edc30e0901e2b92cf3af
…ndings

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WeiJxNbNUdWru1zs68bYV8
Change-Id: I76c41d77e0430116f1a939e0b153d6c6ebf9fe4f
…st explicitly, accepts the recovered report, expects a signal for a Swift trap

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WeiJxNbNUdWru1zs68bYV8
Change-Id: I233fd2d7d4bf43e94623e9219156c377a8aa891f
…per identity; the mock host is validated (never every interface, IPv4 for a phone)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WeiJxNbNUdWru1zs68bYV8
Change-Id: I9b4a47075f6bac37924ee7c4909fda0d5d390682
@krassx
krassx force-pushed the feat/device-harness-physical branch from d160c22 to 5b20c95 Compare November 17, 2026 19:38
@krassx
krassx force-pushed the feat/ios-device-harness branch from 376eb74 to 84da81e Compare November 17, 2026 19:38
@claude

claude Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Claude finished @krassx's task in 19s —— View job


Re-review of PR #35 (push to feat/ios-device-harness, head 84da81e)

I re-read the current scripts/device/ios_physical.dart (launch, matrix and cleanup) and scripts/device/core/lan_host.dart. I did not run flutter test, the analyzer or the harness. The matrix needs a physical iPhone. The commit hash differs from the one I reviewed last time, but the message is the same.

Earlier findings: all still fixed in the code

  • Leaked devicectl children and install: the matrix loop is in try/finally with cleanup(). launch registers each child in _live before its guard, so a failed SDK launched guard is covered.
  • SDK banner and wrapper identity: run.guard('build: SDK banner and wrapper identity', ...iosProblems(banner, wrapperLog)) runs for each kind.
  • Bind address: bindHostProblem and lanHostProblem refuse names, 0.0.0.0, ::, IPv6, loopback and multicast.
  • Pending report blocking later launches: the mock starts with uploadStatus: 200, and the SDK launched failure text names the pending-report cause.
  • swift-trap marker: the report's signal is only noted, not asserted.
  • Unauthenticated LAN bind: documented in mock_api.py, and request bodies are capped.

New findings

None. No inline comments.

Carried-over observation, not a finding

The banner guard reads the SDK banner and the wrapper registered / wrapper refined lines from c1's devicectl --console output, after the first crash has already been induced. The PR text shows the banner reaching the console. The code doesn't show whether the wrapper's NSLog lines reach it too. If they don't, the first full run ends INVALID at that guard (wrapper registered as null). That would be a harness gap, not an SDK finding. On the first real run, read the console output before trusting that guard's message.

Summary

  1. Overall risk: Low. This is dev-only tooling that CI does not run, and the shipped plugin is untouched.
  2. Merge recommendation: Ready to merge as a stacked PR. The PR text and doc/verification.md still say the matrix has not run to the end.
  3. Most important issues to fix: None blocking. On the first real run, check that the wrapper log reaches the console.
  4. Positives:
    • Cleanup covers every guard failure.
    • The device matrix enforces the native pins like every other harness path.
    • Host validation sits in one small file, with a table test.
    • The kind table's per-platform split is tested against each app's source.
    • The iOS expectations stay measured, not asserted, until a real report exists.
      · Branch: feat/ios-device-harness

@krassx
krassx changed the base branch from feat/device-harness-physical to nextgen November 18, 2026 05:24
@krassx
krassx merged commit cd96311 into nextgen Nov 18, 2026
11 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant