Repository navigation
paper: rerank ceiling as headline; SweRankEmbed-Small short-list vote - #153
Merged
Merged
Conversation
…list vote - New Table (ceiling.tex, log sec. 8.20): every gold function in the engine's top 300 for 0.828 of Lite-274 (Verified 0.790; top 30 0.672 / 0.627), set against SweRank's 0.814 func Acc@5, explicitly as scale, not like-for-like; SweRank+ 0.854 and SWE-Adept 0.878 named as higher. - New Table (swerank.tex, log sec. 8.19, 8.21): SweRankEmbed-Small vote k=30 passes the pre-registered bar on all four halves; k=100 fails dev-rest; run once: Lite-274 func@5 0.460 -> 0.496 (+0.036 [0.011,0.062], p=0.013), Verified-458 0.428 -> 0.472 (+0.044, p=0.0001). jina vote stays the shippable row; its holdout gains are a lower bound (20 stored candidates). - Footnote + threat: lists regenerated with the release binary (299/300 identical file lists). - Abstract, intro, contributions, h2h, discussion, threats, conclusion, Table 1, cost table and REVIEWER-RISKS #1/#5 updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the measured results from contributions-log §8.19–§8.21 (#150, #152) into
paper/. This PR changes no benchmarks or code.New tables
tables/ceiling.tex, the rerank ceiling (§8.20). It shows the share of issues whose every gold function appears in the engine's first k definitions:The text compares this with SweRank's best function Acc@5 of 0.814, but only for scale. It says explicitly that a recall ceiling is not an accuracy. It also names two higher systems, SweRank+ at 0.854 and SWE-Adept at 0.878, so we do not overclaim. The ceiling is now in the abstract, the introduction and the contributions list (new item 3).
tables/swerank.tex, SweRankEmbed-Small as the short-list encoder (§8.19, §8.21).Other updates
REVIEWER-RISKS.md: P0-1: stable_project_id + .neuromesh/id #1 is rewritten around the ceiling and includes the counter-objection "recall@300 is not Acc@5". P0-5: stop workspace_root drift in reindex_incremental #5 is updated.python paper/check.pypasses: 39/39 citations and 25 labels. The paper has not been compiled, because there is no TeX on this machine.🤖 Generated with Claude Code