Repository navigation
Iss1596 - #1597
Iss1596#1597tongtongcao wants to merge 2 commits into
Conversation
|
Would be good to see if that 1% average is much more in known bad places, p/theta/phi/vz/etc. |
yugotra
left a comment
There was a problem hiding this comment.
Review of coatjava PR #1597
PR: #1597 ("Iss1596", Tongtong Cao)
Reviewed head: 2202adea2 (commits 4bd012386 "fix bug for construction of SVT & BMTZ cross
combinations" and 2202adea2 "update setting for phiShift"), compared against the branch's own
parent e822218 (#1593). Only reconstruction/cvt/.../track/TrackSeederXY.java changes
(+137/−60). CI: all checks pass.
Date: 2026-11-16; revised after an independent audit.
Verdict
Request changes: retain the slot correction and address the runtime regression. The slot
collision is real. On RGK run 19331 the PR improves CTOF-correlated candidate coverage: the original
proxy gives +211 net tracks per 15k events; enforcing physical paddle extent and counting each
event/paddle once gives +193. An independent test without direction-cone matching gives +207
newly covered fired paddles. These support a positive data effect, but are not calibrated counts
of additional usable particles. Hit-based MC matching gives only +0.024 percentage points in
CTOF-qualified generated-signal efficiency, statistically consistent with zero.
The blocker is the severe per-event latency tail. On real RG-D runs 18528 and 18329, respectively
2 and 6 events per 10,000 exceed 2 minutes in CVTFP, against none without the PR; individual
events exceed 5 minutes. In the measured samples the dominant cost is
Seed.removeOverlappingSeeds, not enumeration and the first circle fits.
A tested per-event budget of 300 seeds before overlap removal retains nearly all measured data
coverage and removes the observed long tail, bringing mean CVTFP time to about 10–17% above base.
It is a useful mitigation, not a proven lossless selection: hit matching finds one lost signal
association in 30k MC events, although that uncapped track has poor kinematics. The prototype
patch is under "Requested change". For production, make the budget configurable and observable
and check the affected multi-track events before approval.
The track-count "gain" quoted in the PR (+2.7% tracks, +1% valid) should not be used as the figure
of merit. Most of the extra tracks are low-momentum, poor-χ² candidates (see below). The relevant
numbers are CTOF-correlated coverage on data and labeled-signal efficiency in MC, with their
selection and background limitations stated.
The bug, and the fix
findSeedCrossesFixedBin sorted crosses into slots by getRegion() - 1 for both SVT and
BMT-Z crosses. SVT region n and BMT-Z region n therefore shared a slot, and the slot walk
could put at most one of them into any seed. The PR separates them into six slots: SVT R1–3 and
BMT-Z R1–3.
It also replaces the walk with an exhaustive enumeration: every subset of ≥ 2 occupied slots per
φ bin, crossed with every cross in each slot. Two further parts of the PR are correct and needed:
- The String seed key. The old key
Σ id·10^indexstarts atindex = 2^s. With up to six
crosses the leading term is proportional to 10⁶⁴. Lower-order cross IDs can disappear below
double precision, so distinct seeds collide. This is a precision problem, not double overflow. phiShift {0,65,90}→{0,5}. With 36 bins of 10°, 65° gives the same bin edges as 5°
and 90° the same as 0°. The old third pass repeated the first, so the new setting is
equivalent in ideal bin partitions with one pass fewer. This does not imply identical traversal
order or floating-point boundary behavior.
Minor: the comment "Need at least three different layers" sits above the check
occupiedLayers.size() < NLAYERSMIN, and NLAYERSMIN = 2.
What the extra tracks are
These checks use 10k–40k events each of RG-D runs 19060, 18528 and 18329 and of MC, plus 15k events
of RGK run 19331.
Reproduction. Clean single-proton MC gives +18 tracks, against +20 on the PR slides' single-π⁺
sample. (All MC here is single protons: one clean sample and two with 50 nA of RGA or RGB background
merged in.) MC with 50 nA of RGA background gives +1.0%, against +1.3% on the slides, whose sample
also used 50 nA RGA background. RG-D data gives +1.7% to +3.3%.
Momentum. These per-track comparisons use 20k–40k events per sample. The track gain is strongly
non-uniform in p: χ²/ndf of a uniform gain is 100–320 for 8 dof. Most of the net gain is below 0.5
GeV: 83% of it in 19060, and more than all of it in 18528 and 18329. In the data that is +3% to +7%
of the tracks there, and +21% to +29% below 0.2 GeV (MC with RGA background: +3.8% and +14.5%).
Above 1 GeV the net change is −4% in 18528 and 18329. In the data runs θ and φ vary by a few percent
from bin to bin (χ²/ndf 29–72 for 10–11 dof), but the pattern differs from run to run, so there is
no consistent hot spot; in MC φ is flat (χ²/ndf 11/11). vz is consistent with flat in 19060 and
18528 (χ²/ndf 9/8 and 7/8). In 18329 (18/8) and the MC (22/6) the excess comes from individual bins
(one in 18329, three in MC), not a broad trend.
For a CTOF-required analysis, very low-momentum tracks are of limited use. The approximate
0.2 GeV reach threshold was supplied as an analysis premise, not measured here; actual reach
depends on transverse momentum, field, material and geometry. This review does not establish
that such tracks have no use in other analyses.
Quality. In the three data runs, new low-p tracks in directions with no baseline track have
median χ²/ndf 90–125, against 6–19 for tracks in directions both versions find. The change above 1
GeV is a swap of poor tracks. In runs 18528 and 18329, median χ²/ndf is 45–50 for the lost tracks
and 35–37 for the gained ones, against 11–13 for unchanged tracks.
MC truth. The two reconstructions were compared on the same 30k events of single-proton MC
with 50 nA of RGB background merged in, for pass 1. Hit-level association is available: CVTFP
writes BST::Clusters and BMT::Clusters. The first validator incorrectly hardcoded the pass-2
BSTRec::Clusters and BMTRec::Clusters names. Correcting those names in a scratch copy and
reading the original full reconstructions permits hit matching without rerunning reconstruction.
All 30k event keys align and no invalid truth references were found.
The hit matcher uses at least three signal hits, purity ≥ 0.5 and matching charge, assigns one
primary track per truth particle, and reports paired McNemar statistics:
| Hit-based result | base | PR |
|---|---|---|
| reconstructed tracks | 56,557 | 58,426 (+1,869) |
| efficiency, CTOF-qualified truth | 98.6455% | 98.6695% (+0.0240 pp, p = 0.55) |
| efficiency, all reconstructable truth | 98.1233% | 98.1667% (+0.0433 pp, p = 0.28) |
| unassociated tracks | 27,035 | 28,878 (+1,843) |
| duplicate rate | 0.0566% | 0.0702% |
The CTOF-qualified denominator is 29,162 truth particles, with 53 PR-only and 46 base-only
matches. Its CTOF qualification uses the existing truth-helix extrapolation plus active raw
paddles because this sample has no CTOF truth links. This remains distinct from calibrated
reconstructed CTOF matching.
For comparison, the original one-to-one kinematic matcher gives:
| base | PR | |
|---|---|---|
| reconstructed tracks | 56,557 | 58,426 (+1,869) |
| efficiency, CTOF-qualified truth | 96.26% | 96.32% (+0.06 pp, p = 0.16) |
| efficiency, all reconstructable truth | 95.38% | 95.47% (+0.09 pp, p = 0.04) |
| unassociated tracks | 27,933 | 29,774 (+1,841) |
| duplicate rate | 0.018% | 0.019% |
The original 98.5% figure is the ratio of net changes in unassociated tracks and total tracks,
not a particle-by-particle classification of every gained candidate. With overlaid background,
unassociated tracks may include real unlabeled background particles; they are not all fakes.
Tightening the kinematic windows to |Δp/p| < 0.25 and 3° gives a +0.093-point CTOF-qualified
change, versus +0.062 points with the original 0.5/5° windows. Both matchers support a small
generated-signal gain, while the absolute efficiencies depend on association criteria.
RGK data. This uses 15k events of run 19331, rgk_spring2024. There is no truth, so I used
the CTOF test: does a track that only one version finds point at a CTOF paddle that fired, and
that no track of the other version points at? A paddle is "fired" when its component appears in
both CTOF::adc and CTOF::tdc. Mean paddle occupancy is 6.1% (2.94 fired paddles per event
of 48); it is not the conditional accidental-match probability of this selected sample.
| found only by the PR | found only by base | net | |
|---|---|---|---|
| tracks on a fired, unclaimed CTOF paddle | 366 | 155 | +211 |
The raw net is +2.26% of base's 9,332 tracks on fired paddles. Its naive Poisson asymmetry is
211/√521 ≈ 9.24; that is not a background-calibrated significance for recovered particles.
The independent audit checked all event identities and the run-dependent CTOF geometry. The
paddle formula agrees with the 3.75° + 7.5°×(component−1) centers, but the original script does
not enforce finite paddle z extent or unique event/paddle counts. Those checks give:
| Data proxy selection | PR-only | base-only | net |
|---|---|---|---|
| Physical radius and paddle-dependent z extent | 351 | 151 | +200 |
| Above, unique event/paddle | 344 | 151 | +193 |
| Above, positive ADC and TDC on both ends | 282 | 122 | +160 |
Rotating the active-paddle map by each nonzero paddle offset gives a mean net +27.1 for the
physical, unique-paddle definition, versus +193 without rotation. Neighbor rotations can retain
real correlations and acceptance varies, so this is a sensitivity control, not a calibrated
subtraction. It demonstrates that symmetric counting does not guarantee cancellation of chance
matches when the eligible candidate populations differ (1,199 PR-only versus 739 base-only in
the original definition).
An independent metric discards the direction cone and compares the sets of physically reachable
fired paddles covered by the two reconstructions: 402 newly covered versus 195 lost, net +207.
These checks support additional CTOF-correlated coverage, while calibrated timing/z matching and
track-quality requirements are still needed to measure a gain in usable physical particles.
The original proxy on RGB MC gives +46; the hit-based truth result above is the appropriate
generated-signal efficiency check. No data/MC cause for their different gains was isolated.
The cost
| sample | CVTFP mean (ms/event), base → PR | median | p99 | events > 120 s per 10k |
|---|---|---|---|---|
| MC clean | 4.8 → 6.5 | 1.45× | 1.58× | 0 → 0 |
| MC 50 nA RGA | 8.4 → 22.4 | 1.40× | 1.70× | 0 → 0 (worst event 0.4 s → 88.6 s) |
| RG-D 19060 | 5.4 → 8.7 | 1.04× | 1.54× | 0 → 0 |
| RG-D 18528 | 18.8 → ≥ 88.7 | 1.12× | 1.71× | 0 → 2 |
| RG-D 18329 | 16.8 → ≥ 114 | 1.20× | 1.97× | 0 → 6 |
Events over 120 s were stopped by a watchdog and counted as 120 s, so those means are lower bounds.
In each row base and PR ran side by side, except the PR's events 5,296–9,999 of 18528, which were
timed in a later batch after a stalled job was restarted. The side-by-side run under "Alternatives
tested" confirms that row: ≥ 6.7× with a 300 s watchdog. The slowest events were confirmed one at a
time on an idle machine: 88.6 s, 31.4 s and 5.8 s, against 0.42 s, 0.14 s and 0.06 s. RG-D event
18528 #5669 exceeds a 300 s watchdog. It takes 15–28 s in base, depending on machine load, and is
base's slowest event too. An earlier run without a watchdog stalled for over 7 CPU-minutes at that
point in the file.
Mechanism. I measured this with counters added to local copies of both builds, on 7k–10k events
each of the RGA and RGB MC, RG-D 18528, and RGK 19331 (reconstructed with the MC variation).
- Candidates: the enumeration raises candidate seeds per event from 14–47 to 96–344 on
average, reaching 25k–354k in single events. - Enumeration time: the enumeration plus each candidate's first circle fit stays at
0.1–0.3% of CVTFP time, even at 354k candidates (191 ms). The later per-seed work in
TrackSeederXY.findSeed(refits,addMissingCrosses) is not in this counter. It falls under the
rest of CVTFP, which grows far less than overlap removal does. - Overlap removal: the survivors that reach
Seed.removeOverlappingSeedsgrow 1.5–1.9× on
average. The per-event maximum rises from 278–1,239 to 1,857–4,705 (4,705 is the event that timed
out). That step then takes 46–81% of the PR's CVTFP time. Its time scales with seeds-in as
n^1.6–n^2.1. In 18528 #6821, 206.6 s of 206.9 s is overlap removal. - Why it is expensive:
getOverlapRemovedSeeds1Passperforms an all-pairs overlap scan, with
an O(k²) cluster comparison per pair.removeCompleteOverlapsthen works through each seed's
overlap group, and passes repeat until nothing changes. Dense groups can make the worst-case
work worse than quadratic; n^1.6–n^2.1 describes the measured workloads.
Determinism. Each build was run twice on the same 4,000 events of 19060 and of the
RGA-background MC, and gave byte-identical tracks. The change of seed-map key from Double to
String introduced no run-to-run variation in those tests; this is not a cross-JVM ordering
guarantee.
Alternatives tested
Each alternative was built from the same sources and run with the same samples and metrics.
| variant | RGK 19331: net original CTOF proxy vs base | RG-D 18528: mean / worst event | RGK 19331: mean / worst event |
|---|---|---|---|
| base | — | 20.4 ms / 15.3 s | 12.1 ms / 5.5 s |
| PR | +211 | ≥ 136 ms (≥ 6.7×) / > 300 s | 48.9 ms (4.1×) / 162 s |
| 1: six slots, old walk, String key (minimal fix) | −174 | ≈ base* / 16 s* | ≈ base* / 9.9 s* |
| 2: PR enumeration, pairs and triples with ≥ 1 SVT slot | −357 | 2.9×* / 238 s* | 2.2×* / 118 s* |
| 3: PR + cap of 1,000 candidates per call | +211 | 4.1×* / > 300 s* | 3.9×* / > 300 s* |
| 4: PR + faster overlap removal (identical output) | +211 (same output as PR, not re-run) | ≥ 101 ms (≥ 4.9×) / > 300 s | 30.2 ms (2.5×) / 86 s |
| 5: PR + per-event limit of 300 seeds into overlap removal | +210 | 22.3 ms (1.10×) / 2.0 s | 14.1 ms (1.17×) / 2.5 s |
| 6: PR + per-event limit of 150 | +203 | 20.1 ms (0.99×) / 1.1 s | 13.0 ms (1.08×) / 1.6 s |
Cost columns, rows without *: base, PR and variants 4–6 were run side by side on the same events
(7k of 18528, 10k of 19331 with rgk_spring2024). There were 5 JVMs at a time and nothing else
running, with a 300 s watchdog, and means count stopped events as 300 s. Rows marked * come from an
earlier batch under different machine load, with 19331 reconstructed with the MC variation. Their
ratios are relative to base run in that same batch (base there: 37 ms on 18528 and 20 ms on 19331).
The conclusions drawn from them (lost tracks, or a remaining tail) do not depend on the batch. The
physics column used rgk_spring2024 throughout.
- These two cheaper alternatives lose coverage. Restoring the old walk (1) or bounding the
subset size (2) loses CTOF-correlated coverage relative to base, also in the independent
no-cone test (−185 and −340). Splitting the slots changes the old walk's candidate membership;
v1 does not preserve all base candidates and add corrected ones. These comparisons rule out
the tested variants, not every compatibility-pruned or otherwise bounded enumeration. - The tested per-call candidate cap leaves the tail.
TrackSeederXY.findSeedis called once
per RZ seed, so the excess builds up across calls (3). - Speeding up the overlap remover alone leaves the tail (4). Variant 4 replaces the all-pairs
overlap scan with a cluster→seed index, a set lookup inappendSeedList, and single evaluation of
the passes. The original evaluates each pass twice, once in thewhilecondition and again in the
body. Its output is byte-identical to the PR and it is 26–38% faster than the PR on average. It is
worth investigating separately. Output equivalence was tested on samples, not proven for all
inputs. What remains is most likely the per-groupremoveCompleteOverlaps, which
it leaves unchanged. - The per-event limit (5) retains nearly all measured coverage and removes the observed tail.
RGK original-proxy net is +210 against +211; physical unique-paddle net is +192 against +193;
independent no-cone coverage is +206 against +207. The kinematic signal efficiency is unchanged
at the displayed 96.32%, but hit matching finds one lost association: CTOF-qualified signal
changes from 28,774 to 28,773 of 29,162 (98.6695% → 98.6661%). That lost proton association has
Δp/p=+1.09 and χ²/NDF=15.8 under the uncapped PR; it does not demonstrate a loss of a useful
physics track. The hit-based unassociated count falls by 143, versus 144 with kinematic matching.
No event took over 10 s in the 16k post-warmup events timed side by side, against 11 for the PR
and 1 for base. The mean is about
10–17% above base and the median 8–16% above. The worst event (2.0–2.5 s) is far below base's own
(5.5–15 s). That is consistent with the baseline also having events with many seeds in overlap
removal: base's worst event on 18528 has 1,239. On 4,000 events of 19060 and 3,000 of the RGB MC,
the track set differs from the PR in only 1 and 4 events. That is consistent with the limit
binding only in the busiest events. No universal event-time bound follows: enumeration and
refitting occur before this cap.
Requested change
Address the overlap-removal tail with a tested event-level budget or an equivalently validated
algorithmic improvement. A minimal prototype, measured as variant 5, keeps at most
MAXSEEDSOVERLAP (300) seeds before Seed.removeOverlappingSeeds(seeds) in
TracksFromTargetRec.getSeeds. Its ordering agrees with Seed.compareTo method 0 for finite
values: more NDF first, then lower χ². The hardcoded prototype below is the tested patch;
configurability and diagnostics have not been implemented or benchmarked here.
--- a/reconstruction/cvt/src/main/java/org/jlab/rec/cvt/services/TracksFromTargetRec.java
+++ b/reconstruction/cvt/src/main/java/org/jlab/rec/cvt/services/TracksFromTargetRec.java
@@ -135,8 +135,17 @@
}
s.setKey(s.new Key(s));
}
- if(Constants.getInstance().removeOverlappingSeeds)
+ if(Constants.getInstance().removeOverlappingSeeds) {
+ // Bound the input of the overlap remover, whose cost grows ~quadratically with the number of seeds.
+ // Keep the best MAXSEEDSOVERLAP seeds by the ordering Seed.compareTo uses there (more NDF first, then
+ // lower chi2). Events below the limit are unaffected.
+ if (seeds.size() > Constants.MAXSEEDSOVERLAP) {
+ seeds.sort((a, b) -> a.getNDF() != b.getNDF() ? Double.compare(b.getNDF(), a.getNDF())
+ : Double.compare(a.getChi2(), b.getChi2()));
+ seeds.subList(Constants.MAXSEEDSOVERLAP, seeds.size()).clear();
+ }
Seed.removeOverlappingSeeds(seeds);
+ }
if(Constants.getInstance().seedingDebugMode) {
for(Seed s : seeds) {
--- a/reconstruction/cvt/src/main/java/org/jlab/rec/cvt/Constants.java
+++ b/reconstruction/cvt/src/main/java/org/jlab/rec/cvt/Constants.java
@@ -81,6 +81,7 @@
private static double RCUT = 120.0; // minimum radius of helix in mm
public static final double TANDIP = 2; // max value on dip angle
public static final double NDFCUT = 0; // minimum number of degres of freedom
+ public static final int MAXSEEDSOVERLAP = 300; // max seeds per event entering overlap removal
public static final double CHI2CUT = 10; // 50, minimum chi2 per degrees of freedom
public static final double CHI2CUTSSA = 10; // 50, minimum chi2 per degrees of freedom for SVTSTANDALONE (SSA)
public static final double DZCUTBUFFEESSA = 0; //dz cut additional contribution to account for SSA poorer resolutionThe patch applies to 2202adea2, builds, and reproduces variant 5 track for track on 3,000 RGB MC
events in the original test. The independent audit dry-ran it on copies of the reviewed source,
compiled both changed classes with Java 21 target compatibility, and verified that those sources
match v7. Events with ≤300 seeds do not enter its sorting/truncation branch.
Before adopting it as production policy:
- Make the budget configurable and record when it binds, how many seeds are discarded, and
their detector/sector coverage. - Validate cap-binding multi-track events and the downstream chain. Ranking before overlap
removal can fill the budget with alternatives to one particle and discard the only seed for
another; a synthetic overlap-group test demonstrates this possibility. The single-generated-
proton sample cannot establish multi-particle efficiency preservation. - Consider exact-duplicate removal or preserving distinct overlap components before truncation.
These are alternatives to investigate, not already validated replacements. - State the scope explicitly: this location affects other seeding paths entering
getSeeds,
including SVT-only and the old straight-track branch. It does not cover the debug-only
flagSeeds && !removeOverlappingSeedspath (TracksFromTargetRec:165). Moving the cap into
the shared overlap helper would widen its scope further and needs a separate decision.
The value 300 is an empirical operating point supported by these samples, not a universal
efficiency guarantee. For slot occupancies ni, enumeration in one bin/shift can still generate
product(1+ni) − 1 − sum(ni) combinations before the overlap budget is applied.
Optional, as a separate PR: improve overlap removal itself. Variant 4 is a measured starting
point with sample-level output equivalence, but the remaining cost is most likely in
removeCompleteOverlaps.
Limits of this review
- Only pass 1 (CVTFP) was run. CVTSP and the downstream chain (EB, vertexing, CTOF/CND matching)
were not tested. The PR slides show CVTSP time unchanged. - The data tests measure CTOF-correlated candidate/paddle coverage, not calibrated CTOF
association or validated particle kinematics. The original definition uses a 3° direction
cone; it gives +216, +211 and +208 with 2°, 3° and 5°. Physical-extent, unique-paddle,
two-ended and no-cone checks support a positive effect, with different net counts. - The corrected hit-based MC result uses pass-1 cluster banks, minimum three truth hits,
purity ≥0.5 and matching charge. The original kinematic result uses |Δp/p| < 0.5,
|Δθ|, |Δφ| < 5°, |Δvz| < 5 cm, one-to-one. Both are reported; their efficiencies differ. - Generated-signal association does not label overlaid background tracks. CTOF-qualified
truth uses helix extrapolation and active raw paddles in this sample, not CTOF truth links. - Timings are from a single Apple M5 Max machine. Each comparison is between versions run side by
side in the same batch, except the rows marked * and part of the 18528 row in the cost table,
both noted where they appear. Outliers were re-timed alone. Ratios are reliable to about ±1.5× on
single events. - The RGK data used is file 100 of run 19331, not the slides' file 00040.
Method
- Builds: the PR's two commits and their parent, built from source. The alternatives were built
from the same sources with only the change described. - Timing: each engine's
processDataEventwas timed per event in-process. Aggregate statistics
drop the first 500 events as warm-up, and single-event re-timings ran 300 warm-up events first.
Where a watchdog was used, it stopped any event exceeding the stated limit, recorded it, and
restarted after it. - Mechanism: counters were added to local copies of both builds. They record candidate seeds,
first fits and seeds entering overlap removal, and time the enumeration and overlap removal per
event. - Track comparisons: tracks from the two versions were paired per event, and a track counts as
covered in direction by both if the other version has one within 3°. This is an any-neighbor
direction criterion, not a one-to-one particle match. The independent CTOF coverage test
instead compares sets of covered fired paddles per event without a direction criterion. - Truth matching: the validator's three hardcoded cluster-bank selection sites were corrected
to use the pass-1 bank names, without changing the reconstruction files. Hit matching ran on all
30k full MC reconstructions. - Configuration: only MAGFIELDS and CVTFP ran. MC used variation
mc2026. RG-D used
rgd_fall2023with a CCDB timestamp of 07/24/2026 andbeamSpotConst = 2. RGK used
rgk_spring2024, except the mechanism runs and the cost rows marked *, which used the MC
variation as noted there. All other CVT settings were defaults.
Scripts and per-event outputs are available on request.
See slides for validation of the update:
Bug for construction of SVT & BMTZ cross combinations.pdf
The conclusions:
The update increases the number of reconstructed tracks by ~2.7%, with ~1% increase in valid tracks.
However, track finding becomes significantly more time-consuming, increasing the computational cost of the full reconstruction chain by ~24%.
So the question: Is the performance gain worth the additional computational cost?