diff --git a/PLAN.md b/PLAN.md index 73a3b0f6..9527981b 100644 --- a/PLAN.md +++ b/PLAN.md @@ -87,7 +87,10 @@ Other gaps, none with an issue: a late completion re-persists) were not measured. * The Gradle plugin 4.0.6 failure arm was not run; `optimizeExtensionsLoading=false` and the feedback safety-net path were not run. * A real token and the production backend: the harness uses a loopback stand-in; server-side symbolication was run offline with the - worker's own tool. + backend's `dart_symbolize` (2026-11-18, `doc/exceptions.md`): it decodes the Android arm64 stack of a Dart 3.14 build only because the worker strips the + stack's header (with the header it crashes, as Flutter 3.50's `flutter symbolize` does: `_kDartSnapshotText`, no `_kDartIsolateSnapshotInstructions`); + its vendored `native_stack_traces` copy is from 2023 and needs a bump to 0.7.x to stop depending on that (owner: the backend; not changed here). A real iOS stack + was not decoded. * Secure views do not cover the first frame of a newly appearing region (documented on the widget); iOS verified in portrait only. Android startup: the SDK's `wrapper-hold` is on (found by the API audit of 2026-11-12: the wrapper never used it), so the first ~4 s of a session are not recorded and, while the hold is on, no secure view is recorded (a time cap ends it even if Dart has not published, so this is not a guarantee past the cap); measured on the Android phone in a profile build (hold released on the first frame), a debug build there reaches its first frame late (the 10 s cap released it). iOS has no equivalent. * The Android logcat echo of `print` is not governed by the Dart log filter (needs a native filter; documented). * `tvOS`, `visionOS`, the Android leak module, `x86_64` and `armeabi-v7a` ABIs, a second display, rotation and split screen were not covered. diff --git a/doc/exceptions.md b/doc/exceptions.md index 4c72ac09..8bcaadd6 100644 --- a/doc/exceptions.md +++ b/doc/exceptions.md @@ -213,7 +213,7 @@ It zips `build/symbols/*.symbols` with a manifest (`{'target':'flutter','items': `POST {endpoint}apps/{token}/symbols` (`{uuid, version, build}`), then `PUT`s the zip to the presigned URL it returns. Keep the symbols of every release you ship; the build id ties a stack to them. -**Verifying a build's symbols against a captured stack, offline** (the same tool the worker runs): +**Verifying a build's symbols against a captured stack, offline:** ```bash dart scripts/verify_symbolization.dart --symbols build/symbols/app.android-arm64.symbols \ @@ -221,14 +221,45 @@ dart scripts/verify_symbolization.dart --symbols build/symbols/app.android-arm64 [--tool /dist/Darwin-arm64/dart_symbolize] ``` +Without `--tool` the script decodes with `package:native_stack_traces` (0.7.1); with `--tool` it runs the backend's `dart_symbolize` on the +stack from the first `#00` with a leading empty line dropped and every line trimmed (the rule of the worker port in `test/exceptions_worker_contract_test.dart`). + +What the backend does (read from the code, 2026-11-18): the worker (`crash/managed/dart.py` and `symbolfiles/flutter.py` in the private +`bugsee-background-worker`) takes the build id from `build_id: '...'` in `originalStackTrace`, looks that id up among the symbols files the +customer uploaded, cuts everything before the first `#00` (the `***` header, `isolate_instructions` and all) and pipes the rest to +`dart_symbolize `. `dart_symbolize` is built from `dart-symbolize/` of `bugsee-symbolication-aux` (Dart SDK 2.19.6 pinned in its +`version.txt`; it vendors a 2023 copy of `native_stack_traces` with no `_kDartSnapshotText` support, and has Mach-O support). + Measured: a release arm64 build of a probe app (`--obfuscate --split-debug-info`) threw an error through `Bugsee.logException` and the root-zone hook on the emulator; the bundle was read; its `originalStackTrace` (vendored: `test/fixtures/exceptions/real_obfuscated_android_arm64_trace.txt`, the whole entry in `android_7_4_2_obfuscated_entry.json`) has build id `7a3ef930...`, which is inside that build's `.symbols` file, and both `dart_symbolize` (worker's tool) and `flutter symbolize` turned it into `level3 / level2 / level1 (obf_main.dart:12..14)`. -(Measured again at Flutter 3.50 / Android 7.6.0: `flutter symbolize` 3.50 fails on the symbols of a 3.50 build, "Cannot locate isolate instructions section in snapshot"; `package:native_stack_traces` 0.7.1, which the harness uses now, decodes them; the worker's `dart_symbolize` against Dart 3.14 snapshots is not verified.) The same script fails (`build id ... is another build's symbols`) with another build's symbols file. The symbols file + +Dart 3.14 (Flutter 3.50.0, Android SDK 7.6.0, measured 2026-11-18 on the stack of a real release build, its text vendored in +`test/fixtures/exceptions/real_obfuscated_android_arm64_dart_3_14_trace.txt`): the stack prints `_kDartSnapshotText+0x...` and +`vm_instructions: 0`, and the symbols file has the dynamic symbols `_kDartSnapshotText`, `_kDartSnapshotData` and `_kDartSnapshotBuildId` and no +`_kDartIsolateSnapshotInstructions` (`llvm-objdump --syms`). A decoder that reads the header looks for that symbol and fails: + +| Decoder | Stack with its header | Stack from `#00` (what the worker sends) | +| --- | --- | --- | +| `dart_symbolize` as the backend builds it (aux source, Dart 2.19.6) | FAILS (`Bad state: Future already completed` after the header lines) | symbolized (`Bugsee.logUnhandledException`, `_command`, `_poll` with file:line) | +| the same source built with Dart 3.14 | FAILS the same way | symbolized | +| `flutter symbolize` of Flutter 3.50.0 (`native_stack_traces` 0.6.1) | FAILS: "Cannot locate isolate instructions section in snapshot" | symbolized | +| `native_stack_traces` 0.7.1 (its 0.7.0 changelog: "Adjusts snapshot readers for the removal of the VM isolate." 0.7.0 itself was not run) | symbolized | symbolized | +| `dart_symbolize` with its vendored copy replaced by 0.7.1 | symbolized | symbolized | + +The "from `#00`" column was measured with both forms: the untrimmed text `symbolfiles/flutter.py` passes (a leading newline and the 4-space indent of each frame) and the trimmed lines; all three builds (Dart 2.19.6, Dart 3.14, 0.7.1 vendored) give the same frames for both. + +So the backend decodes a Dart 3.14 stack because it strips the header and the frames carry a `virt` address (the decoder falls back to it when it cannot +map the symbol name), not because it understands the new layout. That holds as long as the worker keeps stripping the header and the +frames keep their `virt` field; the build-id lookup is not affected (the `.note.gnu.build-id` section is still there). Not verified: the uploaded +symbols file reaching the worker through the production symbols endpoint (the 9.x `symbols.py` uploader is not in this repository), Flutter 3.41 on a device, and iOS (below). +The script also fails (`build id ... is another build's symbols`) with another build's symbols file. The symbols file is not committed (it holds the build machine's paths). +iOS (a Mach-O symbols file with an `LC_UUID`): what was and was not checked is in `doc/verification.md`, scenario 4, "Backend symbolizer against a Dart 3.14 build". + ## 6. Limits (known, not bugs) * iOS: the simulator files nothing; an unhandled error is a crash report delivered at the next launch, one at a time (source-read). diff --git a/doc/verification.md b/doc/verification.md index d1955b1a..c568af0d 100644 --- a/doc/verification.md +++ b/doc/verification.md @@ -376,9 +376,24 @@ harness itself, and neither is a product defect. (1) The first full run was INVA banner prints 10 (`2a88ea7fbf`); the build guard refused the runs, and the pin now comes from a run. (2) Scenario 4's offline symbolization check failed: Flutter 3.50's own `flutter symbolize` answers "Cannot locate isolate instructions section in snapshot" on the symbols of a 3.50 build, while `package:native_stack_traces` 0.7.1 decodes the same file and stack to the app frames (`Bugsee.logUnhandledException`, `_command`, `_poll`); the harness now uses -the package. The backend's `dart_symbolize` against Dart 3.14 snapshots was NOT run: it is the tool this check stands in for, and it is unverified. The iOS +the package. The backend's `dart_symbolize` was run on 2026-11-18 (see the next paragraph). The iOS 7.1.0 crash matrix was not run (the iPhone's Local Network prompt was not answered in time, see the Physical iPhone paragraph). +**Backend symbolizer against a Dart 3.14 build (2026-11-18, offline, no device):** the real `exception` entry and `app.android-arm64.symbols` of the Flutter 3.50.0 +release build above, run through `dart_symbolize` built from `bugsee-symbolication-aux` `dart-symbolize/` (commit `06db90f`, Dart SDK 2.19.6 as in its `version.txt`, +and also Dart 3.14). Fed what the worker feeds it (`symbolfiles/flutter.py` of `bugsee-background-worker` cuts everything before the first `#00`; untrimmed and trimmed lines both tried, same result), it +returns `Bugsee.logUnhandledException (.../bugsee.dart:638:30)`, `_command`, `_poll`. Fed the stack WITH its header it crashes (`Bad state: Future already +completed`), as `flutter symbolize` 3.50 (`native_stack_traces` 0.6.1) does ("Cannot locate isolate instructions section in snapshot"). Cause: Dart 3.14 +symbols have `_kDartSnapshotText`, not `_kDartIsolateSnapshotInstructions` (`llvm-objdump --syms`), which the header-based path of the older decoders needs; +the `native_stack_traces` 0.7.0 changelog reads "Adjusts snapshot readers for the removal of the VM isolate." (0.7.0 itself was not run; 0.6.1 fails, 0.7.1 works), and `dart_symbolize` with 0.7.1 in place of its vendored +copy decodes both forms. The backend works today only because it strips the header and the frames carry a `virt` address. The full table is in +`doc/exceptions.md`. The regression fixture is `test/fixtures/exceptions/real_obfuscated_android_arm64_dart_3_14_trace.txt`. +iOS: a Flutter 3.50 `--release --obfuscate --split-debug-info` build (`example`, `lib/verify_main.dart`, no codesign) writes `app.ios-arm64.symbols`, a Mach-O +file (`file`: "Mach-O 64-bit dSYM companion file arm64") with `LC_UUID` and the symbols `_kDartSnapshotText`, `_kDartSnapshotData` and `_kDartSnapshotBss`. +`dart_symbolize` (and 0.7.1) read it: a SYNTHETIC header-less stack (frames written by hand from the file's own symbol addresses, not captured on a phone) +came out as the right functions with file:line. Not verified: a real iOS stack (its header, whether its frames carry `virt`, the form of its `build_id`, +which should be the `LC_UUID` as 32 hex digits, which the worker looks up by that and by the upper-case dashed UUID), and the iOS upload path. + ## Findings ### Defects @@ -424,7 +439,8 @@ For follow-up (SDK or environment, not fixed here): * A handler that overruns the SDK's deadline, and a hot restart while a delivery is in flight. * The Gradle plugin 4.0.6 broken arm (only the fixed arm was run; the detection is unit-tested). * x86_64 and armeabi-v7a ABIs, a second display, rotation, split screen, a density change *during* a report. -* Server-side processing (symbolization through the real worker, real tokens): the mock API stands in for the backend. +* Server-side processing (real tokens, the symbols upload, the worker as a whole): the mock API stands in for the backend. Its `dart_symbolize` was run offline on + the Android arm64 Dart 3.14 artifacts only (see scenario 4); an iOS stack was not. * Dart `Isolate` errors and `reportAsCrash: true` in a release build. ## Extending it diff --git a/scripts/device/android_scenarios.dart b/scripts/device/android_scenarios.dart index 2a23adad..a3d6c0d0 100644 --- a/scripts/device/android_scenarios.dart +++ b/scripts/device/android_scenarios.dart @@ -937,7 +937,7 @@ class AndroidCtx { run.result.notCovered.addAll([ 'unhandled exception reported AS A CRASH by the zone (installErrorHandlers(reportAsCrash: true)) in a release build', 'errors thrown in a spawned isolate', - 'Dart stack symbolization through the real backend worker (the offline decoder, package:native_stack_traces, decodes the same format; the `flutter symbolize` of Flutter 3.50 does not decode 3.50 symbols; the dart_symbolize of the worker was not run)', + 'Dart stack symbolization through the real backend worker (production symbols upload and worker; offline, scripts/verify_symbolization.dart decodes with package:native_stack_traces, and with `--tool` the backend\'s dart_symbolize decodes a Dart 3.14 stack once the header is stripped as the worker does: doc/exceptions.md; `flutter symbolize` of Flutter 3.50 does not decode 3.50 symbols)', ]); }, ); diff --git a/scripts/verify_symbolization.dart b/scripts/verify_symbolization.dart index 85d2da6a..fc713df1 100644 --- a/scripts/verify_symbolization.dart +++ b/scripts/verify_symbolization.dart @@ -10,7 +10,12 @@ // printed in the stack's header must be inside the symbols file (that is how the backend finds the // symbols), and every `--expect` name must come out of the symbolizer. `--tool` is the worker's // `dart_symbolize` (stack on stdin, symbols file as the argument; dart-symbolize repository, `dist/`); -// without it `package:native_stack_traces` is used, which decodes the same format. +// without it `package:native_stack_traces` (0.7.1) is used, which decodes the same format. The worker +// strips everything before the first `#00` (header lines included, symbolfiles/flutter.py) before it runs +// `dart_symbolize`; with `--tool` the same is done here. That is not cosmetic: the dart_symbolize built from +// bugsee-symbolication-aux (a 2023 copy of native_stack_traces) crashes on a Dart 3.14 stack that still has +// its header (the header path looks for the `_kDartIsolateSnapshotInstructions` symbol, which Dart 3.14 symbols +// do not have: they carry `_kDartSnapshotText`), and decodes the stripped one by the `virt` address. // // The symbols file is NOT committed (it holds the build machine's paths): produce it with // `flutter build apk --release --obfuscate --split-debug-info=`. Exit code 0 when it checks out. @@ -71,9 +76,21 @@ Future main(List args) async { } stdout.writeln('build id $buildId is in the symbols file'); - // Without --tool: `package:native_stack_traces` (a dev dependency), the decoder `flutter symbolize` wraps. - // Flutter 3.50's own `flutter symbolize` fails on the symbols of a 3.50 build ("Cannot locate isolate - // instructions section in snapshot", measured), the package (0.7.1) decodes the same file. + // Without --tool: `package:native_stack_traces` (a dev dependency, 0.7.1). The decoder `flutter symbolize` wraps + // is 0.6.1 in Flutter 3.50 and fails on a 3.50 build's symbols with the header ("Cannot locate isolate + // instructions section in snapshot", measured); the 0.7.0 changelog reads + // "Adjusts snapshot readers for the removal of the VM isolate."; 0.7.1 decodes it (0.7.0 itself was not run). + // With --tool: the stack from the first `#00` on, as the worker port `obfuscatedFrameLines` + // (test/exceptions_worker_contract_test.dart) reads it: a leading empty line dropped, every line trimmed. + // (symbolfiles/flutter.py itself passes the untrimmed text; measured: both forms decode the same.) + if (tool != null) { + final m = RegExp(r'^(.*?)(\s*#00.*)', dotAll: true).firstMatch(stack); + if (m != null) { + var lines = const LineSplitter().convert(m.group(2)!); + if (lines.isNotEmpty && lines.first.isEmpty) lines = lines.sublist(1); + stack = [for (final l in lines) l.trim()].join('\n'); + } + } final process = await Process.start( tool ?? Platform.resolvedExecutable, tool != null diff --git a/test/exceptions_worker_contract_test.dart b/test/exceptions_worker_contract_test.dart index f01ce663..95c58e3d 100644 --- a/test/exceptions_worker_contract_test.dart +++ b/test/exceptions_worker_contract_test.dart @@ -158,6 +158,46 @@ void main() { }); }); + group('a stack from a real Dart 3.14 release build (Flutter 3.50.0, Android 7.6.0, arm64)', () { + // Dart 3.14 prints `_kDartSnapshotText+0x..` and `vm_instructions: 0` (one instructions section; + // older Dart printed `_kDartIsolateSnapshotInstructions+0x..`). The header must not be needed to + // decode it: the worker strips it (symbolfiles/flutter.py) and decodes by the `virt` address. + // Verified against the backend's dart_symbolize: doc/exceptions.md, "Verifying a build's symbols". + final stack = File( + 'test/fixtures/exceptions/real_obfuscated_android_arm64_dart_3_14_trace.txt', + ).readAsStringSync(); + + test('the worker finds the build id and the frames', () { + expect(buildIdFromStackTrace(stack), 'a579af9d76c8acd8528219258f174ae5'); + final frames = obfuscatedFrameLines(stack)!; + expect(frames.first, startsWith('#00 abs ')); + expect(frames.length, 4); + expect(frames.last, ''); + }); + + test( + 'every frame has the virtual address the header-less decode needs', + () { + final frames = obfuscatedFrameLines(stack)!; + final withAddress = RegExp( + r'^#\d+ abs [\da-f]+ virt [\da-f]+ _kDartSnapshotText\+0x[\da-f]+$', + ); + expect( + frames.where((f) => f.startsWith('#')).every(withAddress.hasMatch), + isTrue, + ); + }, + ); + + test('is sent untouched, header included', () { + final p = buildExceptionPayload( + StateError('x'), + StackTrace.fromString(stack), + ); + expect(parseReason(p.json)!['originalStackTrace'], stack); + }); + }); + group( 'an entry from a real obfuscated release build (Android 7.4.2, arm64)', () { diff --git a/test/fixtures/exceptions/real_obfuscated_android_arm64_dart_3_14_trace.txt b/test/fixtures/exceptions/real_obfuscated_android_arm64_dart_3_14_trace.txt new file mode 100644 index 00000000..676783c5 --- /dev/null +++ b/test/fixtures/exceptions/real_obfuscated_android_arm64_dart_3_14_trace.txt @@ -0,0 +1,10 @@ +*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** +pid: 23155, tid: 527685088504, name 1.ui +os: android arch: arm64 comp: yes sim: no +build_id: 'a579af9d76c8acd8528219258f174ae5' +isolate_dso_base: 78fa45f000, vm_dso_base: 0 +isolate_instructions: 78fa52f000, vm_instructions: 0 + #00 abs 00000078fa6493c7 virt 00000000001ea3c7 _kDartSnapshotText+0x11a3c7 + #01 abs 00000078fa7281b7 virt 00000000002c91b7 _kDartSnapshotText+0x1f91b7 + #02 abs 00000078fa727e57 virt 00000000002c8e57 _kDartSnapshotText+0x1f8e57 +