Skip to content

Draw held-out eval targets from cli sessions only - #51

Open
devcodes9 wants to merge 1 commit into
mainfrom
fix/heldout-eval-cli-targets
Open

Draw held-out eval targets from cli sessions only#51
devcodes9 wants to merge 1 commit into
mainfrom
fix/heldout-eval-cli-targets

Conversation

@devcodes9

Copy link
Copy Markdown
Owner

Problem

rank_sessions sorts every auto session below every cli session, before score:

scored.sort(key=lambda x: (x[3][C_KIND] == "auto", -x[2], -x[0]))

So an auto session cannot reach rank 1 while any cli session matches the query at all.

heldout_queries() sampled targets from any session. On a corpus where most sessions are
SDK-spawned, most of the 300 held-out queries therefore asked the ranker to return something it
deliberately does not return. Split by target kind on my own corpus (63% auto):

target kind share of queries @1 MRR
cli 35% 0.519 0.643
auto 65% 0.000 0.005

That caps the headline held@1 at ~0.35 by construction, and it reported 0.170. Most of the
metric was measuring the demotion rule, not ranking quality, so a genuine ranking improvement
would barely move it.

It also made the number a corpus-shape artifact rather than a property of the code. Running the
same unmodified ranker against a second corpus produced independently on another machine, one
that happens to contain no SDK-spawned sessions at all, scored held@1 0.913 on the same
title-derived query construction. Size-matched subsampling ruled out corpus size as the cause.

Fix

Sample held-out targets from cli sessions only.

auto sessions are not a hard target, they are the wrong question. agsearch is for finding a
session you remember being in, and nobody remembers an SDK-spawned run they never saw. Demoting
them is correct product behaviour, so the eval should not be scoring the ranker on failing to
surface them.

They stay in rows and still compete as distractors. Removing them from the pool entirely moves
cli-target accuracy by +0.008 @1 / +0.005 MRR, so they cost essentially nothing and remain
searchable for the rare case you do want one back.

Result

              gold@1  gold-tol  goldMRR |   held@1  heldMRR
before         0.733     0.800    0.795 |    0.170    0.209
after          0.733     0.800    0.795 |    0.490    0.618

Gold set unchanged, as expected. The held-out number now has headroom and tracks something a
ranking change can actually move.

Test-harness only. No change to agsearch itself, no behaviour change. python3 -m unittest discover -s tests passes, 101 tests.

rank_sessions sorts every `auto` session below every `cli` one, so an `auto`
target cannot reach rank 1 while any `cli` session matches. heldout_queries
sampled any session, so on a corpus that is mostly `auto` most of the 300
queries asked for something the ranker deliberately does not return.

That is not a hard query, it is the wrong question: agsearch is for finding a
session you remember being in, and nobody remembers an SDK-spawned run they
never saw. The effect was a metric with almost no headroom, which could not
show a real ranking change.

Targets are now cli-only. `auto` sessions stay in the pool as distractors,
which costs ~0.008 @1 and keeps them searchable.

    held@1   0.170 -> 0.490
    heldMRR  0.209 -> 0.618

Gold set unchanged. Test-only; no behaviour change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant