Draw held-out eval targets from cli sessions only - #51
Open
devcodes9 wants to merge 1 commit into
Open
Conversation
rank_sessions sorts every `auto` session below every `cli` one, so an `auto` target cannot reach rank 1 while any `cli` session matches. heldout_queries sampled any session, so on a corpus that is mostly `auto` most of the 300 queries asked for something the ranker deliberately does not return. That is not a hard query, it is the wrong question: agsearch is for finding a session you remember being in, and nobody remembers an SDK-spawned run they never saw. The effect was a metric with almost no headroom, which could not show a real ranking change. Targets are now cli-only. `auto` sessions stay in the pool as distractors, which costs ~0.008 @1 and keeps them searchable. held@1 0.170 -> 0.490 heldMRR 0.209 -> 0.618 Gold set unchanged. Test-only; no behaviour change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
rank_sessionssorts everyautosession below everyclisession, before score:So an
autosession cannot reach rank 1 while anyclisession matches the query at all.heldout_queries()sampled targets from any session. On a corpus where most sessions areSDK-spawned, most of the 300 held-out queries therefore asked the ranker to return something it
deliberately does not return. Split by target kind on my own corpus (63%
auto):cliautoThat caps the headline
held@1at ~0.35 by construction, and it reported 0.170. Most of themetric was measuring the demotion rule, not ranking quality, so a genuine ranking improvement
would barely move it.
It also made the number a corpus-shape artifact rather than a property of the code. Running the
same unmodified ranker against a second corpus produced independently on another machine, one
that happens to contain no SDK-spawned sessions at all, scored
held@10.913 on the sametitle-derived query construction. Size-matched subsampling ruled out corpus size as the cause.
Fix
Sample held-out targets from
clisessions only.autosessions are not a hard target, they are the wrong question. agsearch is for finding asession you remember being in, and nobody remembers an SDK-spawned run they never saw. Demoting
them is correct product behaviour, so the eval should not be scoring the ranker on failing to
surface them.
They stay in
rowsand still compete as distractors. Removing them from the pool entirely movescli-target accuracy by +0.008 @1 / +0.005 MRR, so they cost essentially nothing and remainsearchable for the rare case you do want one back.
Result
Gold set unchanged, as expected. The held-out number now has headroom and tracks something a
ranking change can actually move.
Test-harness only. No change to
agsearchitself, no behaviour change.python3 -m unittest discover -s testspasses, 101 tests.