Skip to content

Memory-bound rows flip their verdict between two valid runs of the same code (load 6-8) #20

Open

Description

@tannevaled

Two valid -perf.run reports of go-simd/floats, on the same code, disagree on the verdict of memory-bound rows by more than either report's interval. The verdict printed in perf-table.md (faster / tie / slower) is therefore not reproducible for those rows at loads between 6 and 8, although each run passes the gate of 8.

The two reports:

  • A: main's table, measured at 83132fd on 2026-11-15, load 6.5 to 7.96, 13 retries, refbench v0.4.1.
  • B: branch perf-remeasure-low-load (5edc8f2), measured at 9c45587 on 2026-11-16, load 6.0 to 7.94, 3 retries, refbench v0.4.1.

The code of the rows below is identical between the two commits.

row A: Go / numpy, ratio B: Go / numpy, ratio
Mul/1000000 3.309e5 [2.987e5, 3.342e5] / 2.291e5, 1.44 slower 2.016e5 [1.61e5, 2.167e5] / 2.0e5, 1.01 tie
Sub/1000000 2.021e5 / 1.951e5, 1.04 slower 1.76e5 / 1.999e5, 0.88 tie
Mul/100000 faster or tie 1.997e4 [1.905e4, 2.111e4] / 1.517e4, 1.32 slower
Add/100000 faster or tie 1.867e4 / 1.513e4, 1.23 slower
Sub/100000 faster or tie 1.883e4 / 1.5e4, 1.26 slower
CDotC/100000 faster or tie 1.746e4 / 1.457e4, 1.20 slower

Compute-bound rows (Dot, Sum, CosineSimilarity…) agree between A and B. The rows that flip move data at roughly memory bandwidth (1e5 to 1e6 float64, two or three operands). That makes them sensitive to what the ~25 other processes on the machine do to the memory bus and the shared caches, which the 1-minute load average does not measure.

Consequences:

  • A repository's table can turn a row "slower" without any code change. Under the fleet rule "slower = open defect", that opens a false defect; it can also hide a true one.
  • The per-row interval describes the variance inside one run, not between runs, so it cannot tell the two apart.

Possible directions (not a design):

  1. Record a between-run spread, for example by re-measuring each row in a second, separated pass and keeping both. Mark a row whose verdict differs between the two as unstable rather than faster or slower.
  2. A stricter gate for rows flagged memory-bound, for example by bytes moved per call.
  3. At minimum, print in the table that a tie band wider than the interval applies to these rows.

🤖 Generated with Claude Code

Prior art: priorart "run-to-run variance memory-bound rows flip verdict interval load gate" at 2026-11-16 08:28 CET reported prior art; checked: read the 16 entries priorart listed (#1, #3, #4, #5, #7, #9, #10, #15, #16, #17 and others): gating, retries, resume and references; none about the verdict varying between two valid runs

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions