Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,10 @@ Other gaps, none with an issue:
a late completion re-persists) were not measured.
* The Gradle plugin 4.0.6 failure arm was not run; `optimizeExtensionsLoading=false` and the feedback safety-net path were not run.
* A real token and the production backend: the harness uses a loopback stand-in; server-side symbolication was run offline with the
worker's own tool.
backend's `dart_symbolize` (2026-11-18, `doc/exceptions.md`): it decodes the Android arm64 stack of a Dart 3.14 build only because the worker strips the
stack's header (with the header it crashes, as Flutter 3.50's `flutter symbolize` does: `_kDartSnapshotText`, no `_kDartIsolateSnapshotInstructions`);
its vendored `native_stack_traces` copy is from 2023 and needs a bump to 0.7.x to stop depending on that (owner: the backend; not changed here). A real iOS stack
was not decoded.
* Secure views do not cover the first frame of a newly appearing region (documented on the widget); iOS verified in portrait only. Android startup: the SDK's `wrapper-hold` is on (found by the API audit of 2026-11-12: the wrapper never used it), so the first ~4 s of a session are not recorded and, while the hold is on, no secure view is recorded (a time cap ends it even if Dart has not published, so this is not a guarantee past the cap); measured on the Android phone in a profile build (hold released on the first frame), a debug build there reaches its first frame late (the 10 s cap released it). iOS has no equivalent.
* The Android logcat echo of `print` is not governed by the Dart log filter (needs a native filter; documented).
* `tvOS`, `visionOS`, the Android leak module, `x86_64` and `armeabi-v7a` ABIs, a second display, rotation and split screen were not covered.
Expand Down
35 changes: 33 additions & 2 deletions doc/exceptions.md
Original file line number Diff line number Diff line change
Expand Up @@ -213,22 +213,53 @@ It zips `build/symbols/*.symbols` with a manifest (`{'target':'flutter','items':
`POST {endpoint}apps/{token}/symbols` (`{uuid, version, build}`), then `PUT`s the zip to the presigned URL it
returns. Keep the symbols of every release you ship; the build id ties a stack to them.

**Verifying a build's symbols against a captured stack, offline** (the same tool the worker runs):
**Verifying a build's symbols against a captured stack, offline:**

```bash
dart scripts/verify_symbolization.dart --symbols build/symbols/app.android-arm64.symbols \
--entry <crash.json exception entry or a trace.txt> --expect main,level1 \
[--tool <dart-symbolize>/dist/Darwin-arm64/dart_symbolize]
```

Without `--tool` the script decodes with `package:native_stack_traces` (0.7.1); with `--tool` it runs the backend's `dart_symbolize` on the
stack from the first `#00` with a leading empty line dropped and every line trimmed (the rule of the worker port in `test/exceptions_worker_contract_test.dart`).

What the backend does (read from the code, 2026-11-18): the worker (`crash/managed/dart.py` and `symbolfiles/flutter.py` in the private
`bugsee-background-worker`) takes the build id from `build_id: '...'` in `originalStackTrace`, looks that id up among the symbols files the
customer uploaded, cuts everything before the first `#00` (the `***` header, `isolate_instructions` and all) and pipes the rest to
`dart_symbolize <symbols file>`. `dart_symbolize` is built from `dart-symbolize/` of `bugsee-symbolication-aux` (Dart SDK 2.19.6 pinned in its
`version.txt`; it vendors a 2023 copy of `native_stack_traces` with no `_kDartSnapshotText` support, and has Mach-O support).

Measured: a release arm64 build of a probe app (`--obfuscate --split-debug-info`) threw an error through
`Bugsee.logException` and the root-zone hook on the emulator; the bundle was read; its `originalStackTrace` (vendored:
`test/fixtures/exceptions/real_obfuscated_android_arm64_trace.txt`, the whole entry in
`android_7_4_2_obfuscated_entry.json`) has build id `7a3ef930...`, which is inside that build's `.symbols` file, and
both `dart_symbolize` (worker's tool) and `flutter symbolize` turned it into `level3 / level2 / level1 (obf_main.dart:12..14)`.
(Measured again at Flutter 3.50 / Android 7.6.0: `flutter symbolize` 3.50 fails on the symbols of a 3.50 build, "Cannot locate isolate instructions section in snapshot"; `package:native_stack_traces` 0.7.1, which the harness uses now, decodes them; the worker's `dart_symbolize` against Dart 3.14 snapshots is not verified.) The same script fails (`build id ... is another build's symbols`) with another build's symbols file. The symbols file

Dart 3.14 (Flutter 3.50.0, Android SDK 7.6.0, measured 2026-11-18 on the stack of a real release build, its text vendored in
`test/fixtures/exceptions/real_obfuscated_android_arm64_dart_3_14_trace.txt`): the stack prints `_kDartSnapshotText+0x...` and
`vm_instructions: 0`, and the symbols file has the dynamic symbols `_kDartSnapshotText`, `_kDartSnapshotData` and `_kDartSnapshotBuildId` and no
`_kDartIsolateSnapshotInstructions` (`llvm-objdump --syms`). A decoder that reads the header looks for that symbol and fails:

| Decoder | Stack with its header | Stack from `#00` (what the worker sends) |
| --- | --- | --- |
| `dart_symbolize` as the backend builds it (aux source, Dart 2.19.6) | FAILS (`Bad state: Future already completed` after the header lines) | symbolized (`Bugsee.logUnhandledException`, `_command`, `_poll` with file:line) |
| the same source built with Dart 3.14 | FAILS the same way | symbolized |
| `flutter symbolize` of Flutter 3.50.0 (`native_stack_traces` 0.6.1) | FAILS: "Cannot locate isolate instructions section in snapshot" | symbolized |
| `native_stack_traces` 0.7.1 (its 0.7.0 changelog: "Adjusts snapshot readers for the removal of the VM isolate." 0.7.0 itself was not run) | symbolized | symbolized |
| `dart_symbolize` with its vendored copy replaced by 0.7.1 | symbolized | symbolized |

The "from `#00`" column was measured with both forms: the untrimmed text `symbolfiles/flutter.py` passes (a leading newline and the 4-space indent of each frame) and the trimmed lines; all three builds (Dart 2.19.6, Dart 3.14, 0.7.1 vendored) give the same frames for both.

So the backend decodes a Dart 3.14 stack because it strips the header and the frames carry a `virt` address (the decoder falls back to it when it cannot
map the symbol name), not because it understands the new layout. That holds as long as the worker keeps stripping the header and the
frames keep their `virt` field; the build-id lookup is not affected (the `.note.gnu.build-id` section is still there). Not verified: the uploaded
symbols file reaching the worker through the production symbols endpoint (the 9.x `symbols.py` uploader is not in this repository), Flutter 3.41 on a device, and iOS (below).
The script also fails (`build id ... is another build's symbols`) with another build's symbols file. The symbols file
is not committed (it holds the build machine's paths).

iOS (a Mach-O symbols file with an `LC_UUID`): what was and was not checked is in `doc/verification.md`, scenario 4, "Backend symbolizer against a Dart 3.14 build".

## 6. Limits (known, not bugs)

* iOS: the simulator files nothing; an unhandled error is a crash report delivered at the next launch, one at a time (source-read).
Expand Down
20 changes: 18 additions & 2 deletions doc/verification.md
Original file line number Diff line number Diff line change
Expand Up @@ -376,9 +376,24 @@ harness itself, and neither is a product defect. (1) The first full run was INVA
banner prints 10 (`2a88ea7fbf`); the build guard refused the runs, and the pin now comes from a run. (2) Scenario 4's offline symbolization check failed:
Flutter 3.50's own `flutter symbolize` answers "Cannot locate isolate instructions section in snapshot" on the symbols of a 3.50 build, while
`package:native_stack_traces` 0.7.1 decodes the same file and stack to the app frames (`Bugsee.logUnhandledException`, `_command`, `_poll`); the harness now uses
the package. The backend's `dart_symbolize` against Dart 3.14 snapshots was NOT run: it is the tool this check stands in for, and it is unverified. The iOS
the package. The backend's `dart_symbolize` was run on 2026-11-18 (see the next paragraph). The iOS
7.1.0 crash matrix was not run (the iPhone's Local Network prompt was not answered in time, see the Physical iPhone paragraph).

**Backend symbolizer against a Dart 3.14 build (2026-11-18, offline, no device):** the real `exception` entry and `app.android-arm64.symbols` of the Flutter 3.50.0
release build above, run through `dart_symbolize` built from `bugsee-symbolication-aux` `dart-symbolize/` (commit `06db90f`, Dart SDK 2.19.6 as in its `version.txt`,
and also Dart 3.14). Fed what the worker feeds it (`symbolfiles/flutter.py` of `bugsee-background-worker` cuts everything before the first `#00`; untrimmed and trimmed lines both tried, same result), it
returns `Bugsee.logUnhandledException (.../bugsee.dart:638:30)`, `_command`, `_poll`. Fed the stack WITH its header it crashes (`Bad state: Future already
completed`), as `flutter symbolize` 3.50 (`native_stack_traces` 0.6.1) does ("Cannot locate isolate instructions section in snapshot"). Cause: Dart 3.14
symbols have `_kDartSnapshotText`, not `_kDartIsolateSnapshotInstructions` (`llvm-objdump --syms`), which the header-based path of the older decoders needs;
the `native_stack_traces` 0.7.0 changelog reads "Adjusts snapshot readers for the removal of the VM isolate." (0.7.0 itself was not run; 0.6.1 fails, 0.7.1 works), and `dart_symbolize` with 0.7.1 in place of its vendored
copy decodes both forms. The backend works today only because it strips the header and the frames carry a `virt` address. The full table is in
`doc/exceptions.md`. The regression fixture is `test/fixtures/exceptions/real_obfuscated_android_arm64_dart_3_14_trace.txt`.
iOS: a Flutter 3.50 `--release --obfuscate --split-debug-info` build (`example`, `lib/verify_main.dart`, no codesign) writes `app.ios-arm64.symbols`, a Mach-O
file (`file`: "Mach-O 64-bit dSYM companion file arm64") with `LC_UUID` and the symbols `_kDartSnapshotText`, `_kDartSnapshotData` and `_kDartSnapshotBss`.
`dart_symbolize` (and 0.7.1) read it: a SYNTHETIC header-less stack (frames written by hand from the file's own symbol addresses, not captured on a phone)
came out as the right functions with file:line. Not verified: a real iOS stack (its header, whether its frames carry `virt`, the form of its `build_id`,
which should be the `LC_UUID` as 32 hex digits, which the worker looks up by that and by the upper-case dashed UUID), and the iOS upload path.

## Findings

### Defects
Expand Down Expand Up @@ -424,7 +439,8 @@ For follow-up (SDK or environment, not fixed here):
* A handler that overruns the SDK's deadline, and a hot restart while a delivery is in flight.
* The Gradle plugin 4.0.6 broken arm (only the fixed arm was run; the detection is unit-tested).
* x86_64 and armeabi-v7a ABIs, a second display, rotation, split screen, a density change *during* a report.
* Server-side processing (symbolization through the real worker, real tokens): the mock API stands in for the backend.
* Server-side processing (real tokens, the symbols upload, the worker as a whole): the mock API stands in for the backend. Its `dart_symbolize` was run offline on
the Android arm64 Dart 3.14 artifacts only (see scenario 4); an iOS stack was not.
* Dart `Isolate` errors and `reportAsCrash: true` in a release build.

## Extending it
Expand Down
2 changes: 1 addition & 1 deletion scripts/device/android_scenarios.dart
Original file line number Diff line number Diff line change
Expand Up @@ -937,7 +937,7 @@ class AndroidCtx {
run.result.notCovered.addAll([
'unhandled exception reported AS A CRASH by the zone (installErrorHandlers(reportAsCrash: true)) in a release build',
'errors thrown in a spawned isolate',
'Dart stack symbolization through the real backend worker (the offline decoder, package:native_stack_traces, decodes the same format; the `flutter symbolize` of Flutter 3.50 does not decode 3.50 symbols; the dart_symbolize of the worker was not run)',
'Dart stack symbolization through the real backend worker (production symbols upload and worker; offline, scripts/verify_symbolization.dart decodes with package:native_stack_traces, and with `--tool` the backend\'s dart_symbolize decodes a Dart 3.14 stack once the header is stripped as the worker does: doc/exceptions.md; `flutter symbolize` of Flutter 3.50 does not decode 3.50 symbols)',
]);
},
);
Expand Down
25 changes: 21 additions & 4 deletions scripts/verify_symbolization.dart
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,12 @@
// printed in the stack's header must be inside the symbols file (that is how the backend finds the
// symbols), and every `--expect` name must come out of the symbolizer. `--tool` is the worker's
// `dart_symbolize` (stack on stdin, symbols file as the argument; dart-symbolize repository, `dist/`);
// without it `package:native_stack_traces` is used, which decodes the same format.
// without it `package:native_stack_traces` (0.7.1) is used, which decodes the same format. The worker
// strips everything before the first `#00` (header lines included, symbolfiles/flutter.py) before it runs
// `dart_symbolize`; with `--tool` the same is done here. That is not cosmetic: the dart_symbolize built from
// bugsee-symbolication-aux (a 2023 copy of native_stack_traces) crashes on a Dart 3.14 stack that still has
// its header (the header path looks for the `_kDartIsolateSnapshotInstructions` symbol, which Dart 3.14 symbols
// do not have: they carry `_kDartSnapshotText`), and decodes the stripped one by the `virt` address.
//
// The symbols file is NOT committed (it holds the build machine's paths): produce it with
// `flutter build apk --release --obfuscate --split-debug-info=<dir>`. Exit code 0 when it checks out.
Expand Down Expand Up @@ -71,9 +76,21 @@ Future<void> main(List<String> args) async {
}
stdout.writeln('build id $buildId is in the symbols file');

// Without --tool: `package:native_stack_traces` (a dev dependency), the decoder `flutter symbolize` wraps.
// Flutter 3.50's own `flutter symbolize` fails on the symbols of a 3.50 build ("Cannot locate isolate
// instructions section in snapshot", measured), the package (0.7.1) decodes the same file.
// Without --tool: `package:native_stack_traces` (a dev dependency, 0.7.1). The decoder `flutter symbolize` wraps
// is 0.6.1 in Flutter 3.50 and fails on a 3.50 build's symbols with the header ("Cannot locate isolate
// instructions section in snapshot", measured); the 0.7.0 changelog reads
// "Adjusts snapshot readers for the removal of the VM isolate."; 0.7.1 decodes it (0.7.0 itself was not run).
// With --tool: the stack from the first `#00` on, as the worker port `obfuscatedFrameLines`
// (test/exceptions_worker_contract_test.dart) reads it: a leading empty line dropped, every line trimmed.
// (symbolfiles/flutter.py itself passes the untrimmed text; measured: both forms decode the same.)
if (tool != null) {
final m = RegExp(r'^(.*?)(\s*#00.*)', dotAll: true).firstMatch(stack);
if (m != null) {
var lines = const LineSplitter().convert(m.group(2)!);
if (lines.isNotEmpty && lines.first.isEmpty) lines = lines.sublist(1);
stack = [for (final l in lines) l.trim()].join('\n');
}
}
final process = await Process.start(
tool ?? Platform.resolvedExecutable,
tool != null
Expand Down
40 changes: 40 additions & 0 deletions test/exceptions_worker_contract_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,46 @@ void main() {
});
});

group('a stack from a real Dart 3.14 release build (Flutter 3.50.0, Android 7.6.0, arm64)', () {
// Dart 3.14 prints `_kDartSnapshotText+0x..` and `vm_instructions: 0` (one instructions section;
// older Dart printed `_kDartIsolateSnapshotInstructions+0x..`). The header must not be needed to
// decode it: the worker strips it (symbolfiles/flutter.py) and decodes by the `virt` address.
// Verified against the backend's dart_symbolize: doc/exceptions.md, "Verifying a build's symbols".
final stack = File(
'test/fixtures/exceptions/real_obfuscated_android_arm64_dart_3_14_trace.txt',
).readAsStringSync();

test('the worker finds the build id and the frames', () {
expect(buildIdFromStackTrace(stack), 'a579af9d76c8acd8528219258f174ae5');
final frames = obfuscatedFrameLines(stack)!;
expect(frames.first, startsWith('#00 abs '));
expect(frames.length, 4);
expect(frames.last, '<asynchronous suspension>');
});

test(
'every frame has the virtual address the header-less decode needs',
() {
final frames = obfuscatedFrameLines(stack)!;
final withAddress = RegExp(
r'^#\d+ abs [\da-f]+ virt [\da-f]+ _kDartSnapshotText\+0x[\da-f]+$',
);
expect(
frames.where((f) => f.startsWith('#')).every(withAddress.hasMatch),
isTrue,
);
},
);

test('is sent untouched, header included', () {
final p = buildExceptionPayload(
StateError('x'),
StackTrace.fromString(stack),
);
expect(parseReason(p.json)!['originalStackTrace'], stack);
});
});

group(
'an entry from a real obfuscated release build (Android 7.4.2, arm64)',
() {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 23155, tid: 527685088504, name 1.ui
os: android arch: arm64 comp: yes sim: no
build_id: 'a579af9d76c8acd8528219258f174ae5'
isolate_dso_base: 78fa45f000, vm_dso_base: 0
isolate_instructions: 78fa52f000, vm_instructions: 0
#00 abs 00000078fa6493c7 virt 00000000001ea3c7 _kDartSnapshotText+0x11a3c7
#01 abs 00000078fa7281b7 virt 00000000002c91b7 _kDartSnapshotText+0x1f91b7
#02 abs 00000078fa727e57 virt 00000000002c8e57 _kDartSnapshotText+0x1f8e57
<asynchronous suspension>
Loading