Skip to content

test(ios): establish snapshot convergence baselines and permanent evidence #2189

Description

@thymikee

Parent: #2188

Outcome

Check in the reproducible evidence needed to judge every later iOS snapshot-convergence change.
A third party must be able to produce current-HEAD local, proxy, compute, and package-size baselines
from documented commands without reconstructing artifacts from an investigation session.

Exact scope

  • Add benchmark definitions, a versioned raw-result schema, and one documented reproduction entry
    point under repository tooling rather than a runtime package.
  • Measure current runner-backed iOS Simulator snapshots in explicitly named states:
    • cold-cold: Simulator off, daemon off, no prepared runner/bridge cache;
    • cold: Simulator booted, daemon/helper processes absent;
    • warm: Simulator, app, daemon, runner, and target generation ready;
    • relaunch: same Simulator/tooling with a new app process/generation.
  • Include representative quiet, list, nested-scroll, alert/system-surface, and XCTest-stress screens.
  • Report daemon-resident request time and fresh-process CLI wall time separately.
  • Record direct-proxy baselines for persistent-client and fresh-process CLI execution at controlled
    approximately 0 ms, 20 ms, and 80 ms RTT. Record configured bandwidth/packet loss, response
    bytes, median, p95, failures, and raw samples.
  • Record current packed, clean-installed, and bundled npm sizes using the repository's existing size
    workflow.
  • Promote iOS: skip private-AX deep extension on observation-only captures (settle, stabilization, corroboration) #1626's deep-button false-failure into permanent, implementation-independent fixture
    evidence: a shallow surface observation must not prove that a full action had no effect when the
    changed descendant was omitted.
  • Link App-mount race on fast/cold opens: first capture can see an unreadable or empty tree (cross-platform) #1571 in the result schema and classify an unreadable/empty first tree separately from bridge,
    runner, timeout, and stale-generation failures.

Use at least 20 samples per warm cell and 10 per cold cell unless the checked-in harness documents a
larger statistically justified minimum. Store raw samples; summaries alone are insufficient.

Non-goals

Acceptance and evidence

  • One documented command reproduces each local measurement cell and writes schema-valid raw output.
  • One documented command reproduces each proxy cell without changing application semantics.
  • Repeated runs identify machine/toolchain, package revision, screen fixture, target generation,
    sample count, failures, and outliers.
  • The deep-button fixture is observed failing under the invalid shallow-corroboration rule and green
    under the current safe/full requirement. Record the exact planted-red command and failing assertion.
  • Package-size output reports packed, installed, and bundled values from the same revision.
  • Focused tooling/schema tests and pnpm check:affected --run pass. Native and proxy measurements
    are attached as evidence rather than represented by fixture-only tests.

Worker stop conditions

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions