Repository navigation
Device harness: a physical iPhone and an iOS crash lab - #35
Conversation
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 58 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
|
Claude finished @krassx's task in 1m 3s —— View job Review of PR #35: physical iPhone harness and iOS crash labI read the full diff against I posted 3 inline findings, none higher than P2. Findings
1. Overall risk: Low. This is dev-only tooling that CI does not run. Nothing in the shipped plugin changes, and the example app's crash lab is example code. 2. Merge recommendation: Merge once the P2 cleanup is fixed, or accept it explicitly. This is a stacked PR on #34, and it is honest that the matrix has not run to the end. 3. Most important issues to fix
4. Positives
Observation, not a finding: |
| first = false; | ||
| await delay(5); | ||
| await mock.command('crash ${k.name}'); | ||
| final code = await c1.process.exitCode.timeout( |
There was a problem hiding this comment.
P2 Medium — a failed guard leaks the console child, the running app and the installed app
c1 and c2 are devicectl ... --console child processes. They are only stopped on the happy path (c2.stop() at line 316). Every run.guard throws InvalidExperiment, and runScenario catches it. Two guards sit between a child's start and its stop:
SDK launched(line 238)- the process-ended check at line 279, which is a guard because it uses
run.guard
When either throws, three things happen:
- the
devicectlchild is never stopped, andProcess.startchildren outliveexit(); - the app keeps running on the phone with the console attached;
dev.uninstall()(line 321) is skipped, so the app stays installed.
The PR description says the first real run ended INVALID at "SDK launched", so this is the path that has already run. A re-run is partly protected by --terminate-existing and the uninstall at line 257. The orphaned devicectl processes pile up on the Mac, though, and a half-finished crash lab stays on the phone.
Also, on the -999 timeout at line 276 the app is by definition still alive, and c1 is not stopped before the guard fails.
Fix: track the live children in _Ctx and stop them, then uninstall, in a finally in crashMatrix (or in runIosDevice's finally).
| # Loopback only: the server stores whatever is PUT to it. The Android emulator reaches it as 10.0.2.2 and | ||
| # the iOS simulator as localhost, both of which arrive on the loopback interface of the host. | ||
| ThreadingHTTPServer(('127.0.0.1', int(sys.argv[1])), H).serve_forever() | ||
| # Loopback unless told otherwise: a PHYSICAL iPhone reaches the Mac only over the LAN, so the device | ||
| # harness may bind the one LAN address it prints (MOCK_BIND), never all interfaces. | ||
| ThreadingHTTPServer((os.environ.get('MOCK_BIND', '127.0.0.1'), int(sys.argv[1])), H).serve_forever() |
There was a problem hiding this comment.
P3 Low — the server is now on the LAN with no authentication, and the comment contradicts itself
The original reasoning was "loopback only: the server stores whatever is PUT to it". That reasoning is still in lines 126-127, directly above the new text, and now reads as false. MOCK_BIND is a single address, which is better than 0.0.0.0. Even so, any host on the same network can now:
POST /__cmd: queue commands for the verification app.crash, andnet <url>(an arbitrary URL fetched from the phone), are both commands.POST /__event: forge events.PUT /upload/*: forge bundles. Each upload is stored with an unboundedcontent-lengthread into memory.POST /__reset: wipe the evidence.
That can fake or destroy the evidence a verification run produces, and the harness trusts the mock as its source of truth. On an office or shared Wi-Fi this is a real exposure.
Fix: at minimum, drop the stale "Loopback only" sentences and say plainly that a LAN bind is unauthenticated. Better, require a per-run token header (generated by Mock.start, baked into the build as a --dart-define, checked in _do) and cap the body size.
| // readiness probe and the harness would talk to the wrong one. | ||
| try { | ||
| final probe = await ServerSocket.bind(InternetAddress.loopbackIPv4, port); | ||
| final probe = await ServerSocket.bind(InternetAddress(mockHost), port); |
There was a problem hiding this comment.
P3 Low — mockHost is read from the environment for every platform, not just ios-device
mockHost is the process-wide BUGSEE_VERIFY_MOCK_BIND value. It reaches Mock.start, the readiness probe and every Mock client call for the Android and simulator runs as well.
Suppose someone exports the variable for a phone run and later runs verify.dart ios or android in the same shell. The mock then binds the LAN address while those apps were built for localhost or 10.0.2.2. The run fails as a confusing INVALID, and the emulator and simulator runs are needlessly exposed on the LAN.
InternetAddress(mockHost) also throws a bare ArgumentError if a hostname rather than an IP is given.
Fix: pass the host into Mock.start explicitly (default loopback) and have only runIosDevice supply the LAN address. If a global is kept, validate it as an IP in one place and ignore it outside ios-device. Also, the header comment at line 3 now says "binds 127.0.0.1 only (one LAN address ...)", which reads oddly.
adaf4e6 to
c643937
Compare
38e902f to
b4de464
Compare
|
Claude finished @krassx's task in 51s —— View job Re-review of PR #35 (push to
|
| Sev | Where | Summary |
|---|---|---|
| P3 | scripts/device/ios_physical.dart:278 |
Mock.start defaults to a 503 upload status. The recovered report stays pending, and per CLAUDE.md pending reports make iOS stop a later launch of the same install. Kind 2 onward could end INVALID at "SDK launched", with a message that blames the Local Network prompt. |
| P3 | example/ios/Runner/AppDelegate.swift:43 |
fatalError("verify-swift-trap") is a trap, not an exception, so the report probably does not hold the marker. The run.expect on it will probably fail. |
Summary
- Overall risk: Low. This is dev-only tooling that CI does not run, and the shipped plugin is untouched.
- Merge recommendation: Merge once the P2 cleanup is fixed or explicitly accepted. This is a stacked PR and it says plainly that the matrix has not run to the end.
- Most important issues to fix:
- Stop the console children and uninstall in a
finally(earlier P2). - Settle the upload status or per-kind reinstall question before the first full run, so later kinds do not fail on the pending-report rule (new P3).
- Drop or soften the
swift-trapmarker expectation (new P3). - Fix the "Loopback only" comment and pass the mock host into
Mock.startexplicitly (earlier P3s).
- Stop the console children and uninstall in a
- Positives:
- It binds a single address, requires
--udid, and refuses the loopback for a phone. - The kind table now records per-platform existence, and a test checks each marker against the right app source.
- The crash lab raises its exceptions from their own dispatch or thread, so the SDK sees an uncaught exception.
· Branch:feat/ios-device-harness
- It binds a single address, requires
| const Duration(seconds: 45), | ||
| onTimeout: () => -999, | ||
| ); | ||
| final out = c1.output.toString(); |
There was a problem hiding this comment.
P3 Low: a refused (503) recovered crash report may stay pending and stop the next kind's launch
Problem: Mock.start defaults to uploadStatus = 503, and runIosDevice does not change it. The recovered crash report that c2 captures is therefore refused by the mock and kept by the SDK. The loop then relaunches the same install for the next kind through launch(run, first: false), which guards on launched['result'] == 'launched'.
Impact: CLAUDE.md records that on iOS "pending reports make iOS stop a later launch of the same install". That is why the guidance says to run simctl uninstall between facade_test and launch_test. If it applies here, the matrix gets through the first kind and ends INVALID at "SDK launched" for every later kind, with an error message that blames the Local Network prompt.
Scenario: Kind 1 (ffi-segv) crashes and c2 recovers the report. The mock stores it and answers 503, so the SDK keeps it. Kind 2 launches c1 and the SDK reports already-running or a stopped result. The guard fails.
Fix: Either reinstall (uninstall, then install) per kind, or start the mock with uploadStatus: 200 for this scenario so the recovered report is accepted. If neither is needed, say so in a comment citing a measurement. The failure text in launch should also name the other cause (a pending report) so the Local Network hint does not mislead.
I did not run this on a device. It follows from the repo's own note about pending reports.
| } | ||
| result(nil) | ||
| case "swiftTrap": | ||
| DispatchQueue.main.async { fatalError("verify-swift-trap") } |
There was a problem hiding this comment.
P3 Low: the swift-trap marker is unlikely to appear in the crash report, so a hard run.expect will probably fail
Problem: fatalError("verify-swift-trap") ends in a brk/EXC_BREAKPOINT trap (SIGTRAP). Swift prints the message to stderr (and to the system crash-info buffer), but it is not an exception name or reason. The SDK's native crash path normally records the signal and the stack. The matrix still requires text.contains('verify-swift-trap') in crash.json or the request, as a run.expect (ios_physical.dart:300-306).
Impact: objc-main and objc-thread carry their markers in an NSException reason, which the SDK does read. swift-trap is a different kind of crash, and its marker check will most likely fail by design. That makes the first real run report a failure for a kind that did crash and was recovered. The PR describes the iOS expectations as loose, and this is the one that is not.
Fix: For swift-trap, set marker: null in crash_kinds.dart, and note that the report only shows a trap. Alternatively, keep the marker as a run.note until a first run shows what the report holds. test/device_harness_test.dart requires a non-null marker for non-signal kinds (the else branch around line 810), so that test needs the matching change. A signal-style entry (signal: 5) fits better.
I did not run this on a device.
c643937 to
1cc3a5a
Compare
9f1d2f2 to
3ab8049
Compare
|
Claude finished @krassx's task in 1m 0s —— View job Re-review of PR #35 (push of
|
| Sev | Where | Summary |
|---|---|---|
| P2 | scripts/device/ios_physical.dart:316 |
The device matrix only notes the SDK banner and never guards it against native-versions.json. The simulator path does (exp.iosProblems, workbook 9.1.1). |
| P3 | scripts/device/ios_physical.dart:94 |
"One address, never all interfaces" is not enforced: 0.0.0.0 and :: pass, as do 127.0.0.2 and ::1. IPv6 gives a malformed URL. Only the exact string 127.0.0.1 is refused. |
The banner finding matters most. The pin moved to 7.1.0 (b65bdca06-9), yet the phone run in the PR text printed 7.0.1. A stale --reuse build would go through the whole matrix unnoticed, and the result would be recorded as 7.1.0 evidence.
Summary
- Overall risk: Low. This is dev-only tooling that CI does not run, and the shipped plugin is untouched.
- Merge recommendation: Merge after the banner guard is added. It is small, reuses
exp.iosProblems, andCLAUDE.mdrequires the harness guards to stay on. The address check can be a follow-up. - Most important issues to fix:
- Guard the SDK banner and wrapper identity in the device matrix (P2).
- Reject
0.0.0.0,::, loopback and IPv6 forBUGSEE_VERIFY_MOCK_BIND, with a test (P3).
- Positives:
- The cleanup now covers every guard failure, and
c1is registered before its guard. - Accepting the recovered report with 200 follows the repo's own pending-report rule, and the comment cites it.
- The kind table's per-platform split is tested against each app's source.
- The iOS expectations stay measured, not asserted, until a real report exists.
- The docs say plainly that the matrix has not run to the end.
· Branch:feat/ios-device-harness
- The cleanup now covers every guard failure, and
| ); | ||
| final banner = parseIosBanner(out); | ||
| if (banner != null) { | ||
| seenBanner.add('${banner.version} ${banner.build}'); |
There was a problem hiding this comment.
P2 Medium: the device matrix never guards which SDK build it ran against (workbook 9.1.1)
Problem: the banner is parsed here and only collected into seenBanner, then printed with run.note('console banners seen: ...'). ios_scenarios.dart:266-271 guards it with run.guard('build: SDK banner and wrapper identity', exp.iosProblems(banner, wrapper), ...), which compares the version and build to native-versions.json (ios.sdk 7.1.0, ios.sdkBuild b65bdca06-9). The device path has no such guard. CLAUDE.md says the harness guards "are on by default and must stay so".
Scenario: the PR description and doc/verification.md report the banner ver:7.0.1 build:5c6b9d9db-8 from the phone, while the pin has moved to 7.1.0 (native-versions.json provenance: "moved from 7.0.1 on 2026-11-17"). A stale --reuse build or a stale SwiftPM resolution would go through the whole matrix, and a PASS would be recorded for 7.0.1 as if it were the 7.1.0 evidence. The provenance text says the harness "matches it on the next run", and this path cannot do that.
Fix: after the first launch of the first kind, run run.guard with exp.iosProblems(banner, wrapper) (the wrapper log is in the same console output), or at least guard version and build. The iOS 7.1.0 matrix is the evidence this PR is meant to produce, so the guard should be there before the first full run.
| return 64; | ||
| } | ||
| if (mockHost == '127.0.0.1') { | ||
| stderr.writeln( |
There was a problem hiding this comment.
P3 Low: "never all interfaces" is not enforced; only the exact string 127.0.0.1 is refused
Mock.start validates with InternetAddress.tryParse(host) == null, and this check rejects only mockHost == '127.0.0.1'. These values pass both:
BUGSEE_VERIFY_MOCK_BIND=0.0.0.0(or::) binds every interface. The docs and the PR text promise "one address, never all interfaces". The phone also cannot connect to0.0.0.0, so the run would end INVALID at "SDK launched" after the build.127.0.0.2or::1slip past the loopback refusal.- An IPv6 address passes validation, but
http://$host:$portand--dart-define=BUGSEE_ENDPOINT=http://$host:$portare built without brackets, so the URL is malformed.
Fix: in runIosDevice (or in Mock.start), parse once and refuse isLoopback, 0.0.0.0, :: and non-IPv4 addresses. Unit-test it in test/device_harness_test.dart.
1cc3a5a to
cec3fe7
Compare
3ab8049 to
66fc3e0
Compare
|
Claude finished @krassx's task in 43s —— View job Re-review of PR #35 (push of
|
cec3fe7 to
743eee5
Compare
66fc3e0 to
aeedf05
Compare
|
Claude finished @krassx's task in 25s —— View job Re-review of PR #35 (push to
|
743eee5 to
06d4fdf
Compare
aeedf05 to
db3fb37
Compare
|
Claude finished @krassx's task in 26s —— View job Re-review of PR #35 (push to
|
06d4fdf to
55e123c
Compare
db3fb37 to
55e123c
Compare
|
Claude finished @krassx's task in 18s —— View job Re-review of PR #35 (push to
|
… lab `verify.dart ios-device` installs the signed release build on a physical iPhone with devicectl, launches it with its console attached (the child's exit is the proof the process ended), and runs the crash matrix against the mock API bound to the Mac's LAN address (BUGSEE_VERIFY_MOCK_BIND, one address, never all interfaces; MOCK_BIND in mock_api.py, loopback by default). The first launch after an install waits for the Local Network prompt, which a script cannot answer. The example's iOS crash lab (AppDelegate.swift, channel verify/crashlab) adds an uncaught Objective-C exception on the main thread and on a worker and a Swift runtime trap to the shared kind table, which now says per kind where it exists. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WeiJxNbNUdWru1zs68bYV8 Change-Id: If8850e3e388377d3644035e484002e3db3af809f
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WeiJxNbNUdWru1zs68bYV8 Change-Id: I9c2d3d6ca0c72cc1fa19edc30e0901e2b92cf3af
…ndings Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WeiJxNbNUdWru1zs68bYV8 Change-Id: I76c41d77e0430116f1a939e0b153d6c6ebf9fe4f
…st explicitly, accepts the recovered report, expects a signal for a Swift trap Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WeiJxNbNUdWru1zs68bYV8 Change-Id: I233fd2d7d4bf43e94623e9219156c377a8aa891f
…per identity; the mock host is validated (never every interface, IPv4 for a phone) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WeiJxNbNUdWru1zs68bYV8 Change-Id: I9b4a47075f6bac37924ee7c4909fda0d5d390682
d160c22 to
5b20c95
Compare
376eb74 to
84da81e
Compare
|
Claude finished @krassx's task in 19s —— View job Re-review of PR #35 (push to
|
Stacked on #34.
verify.dart ios-deviceinstalls the signed release build on a physical iPhone withdevicectl, launches it with its console attached (the child's exit proves the process ended), and runs the crash matrix (scenario 10) against the mock API bound to the Mac's LAN address (BUGSEE_VERIFY_MOCK_BIND, one address, never all interfaces;MOCK_BINDinmock_api.py, loopback by default).AppDelegate.swift, channelverify/crashlab): an uncaught Objective-C exception on the main thread and on a worker, and a Swift runtime trap; the kind table now says per platform where each kind exists, and a test compares it with the three apps' sources.Bugsee IOS SDK ver:7.0.1 build:5c6b9d9db-8, before the 7.1.0 move), but the first attempt ended INVALID at "SDK launched": the phone says "The Internet connection appears to be offline", which means the Local Network prompt was not allowed (a script cannot answer it) or the phone is not on the Mac's network. The iOS expectations in the matrix are deliberately loose until a first real run shows what an iOS crash report holds (the Swift trap is expected as a signal 5 note, not a marker). Review fixes: every path stops thedevicectlchildren and uninstalls, the mock host is passed explicitly (only this command leaves the loopback), the recovered report is accepted (200) so it cannot block the next launch. The mock on a LAN address has no authentication: run it on a network you trust.flutter test1389, analyze.🤖 Generated with Claude Code
https://claude.ai/code/session_01WeiJxNbNUdWru1zs68bYV8