Skip to content

Draw held-out eval targets from cli sessions only #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
devcodes9 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Draw held-out eval targets from cli sessions only #51

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion tests/heldout_eval.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
require that session back. It covers 300 sessions instead of 15 and nobody tuned against it,
so a change that helps set 1 while flat or down on set 2 is a change fitted to the labels.

Set 2 draws its targets from `cli` sessions only. See heldout_queries().

The timing column matters as much as the quality columns: fzf re-ranks on every keystroke, so
a variant that buys accuracy at 5x the latency is not shippable.
"""
Expand All @@ -22,13 +24,24 @@ def load(path, name):
def heldout_queries(ag, rows, seed=7, n=300):
"""Pick content words from a session's own title. If you typed words from the title,
that session should come back. Titles are not what the ranker mainly scores (body is),
so this is a fair proxy, and it covers 300 sessions instead of 15."""
so this is a fair proxy, and it covers 300 sessions instead of 15.

Targets are `cli` sessions only. rank_sessions sorts every `auto` session below every
`cli` one, so an `auto` target cannot reach rank 1 while any `cli` session matches at
all. That is deliberate: agsearch is for finding a session you remember being in, and
nobody remembers an SDK-spawned run they never saw. Asking for one back is not a hard
query, it is the wrong question, and on a corpus that is mostly `auto` it caps @1 far
below 1.0 and leaves too little headroom for a real ranking change to show up.

`auto` sessions stay in `rows`, so they still compete as distractors."""
rnd = random.Random(seed)
pool = []
for r in rows:
title = r[ag.C_TITLE]
if title.startswith("<command-message>"): # slash-command invocations, not conversations
continue
if r[ag.C_KIND] == "auto": # never a valid target; see docstring
continue
words = [w for w in ag.parse_query(title)]
if len(words) < 2:
continue
Expand Down