feat: Add pass@k with per-case CSV loading for NxM consolidation - #316
Draft
asamal4 wants to merge 1 commit into
Draft
feat: Add pass@k with per-case CSV loading for NxM consolidation#316asamal4 wants to merge 1 commit into
asamal4 wants to merge 1 commit into
Conversation
Contributor
WalkthroughThe behavioral pipeline now loads per-case results from detailed CSV files and computes an unbiased ChangesBehavioral pass@k aggregation
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant DetailedCSV
participant load_run_data
participant RunData
participant consolidate
participant pass_at_k
DetailedCSV->>load_run_data: read *_detailed.csv case rows
load_run_data->>RunData: store optional case_results
RunData->>consolidate: provide case results from multiple runs
consolidate->>pass_at_k: pass counts, totals, and k
pass_at_k-->>consolidate: return pass@k estimate
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lightspeed_evaluation/pipeline/behavioral/consolidation.py`:
- Around line 280-286: In the case-counting logic, replace the
colon-concatenated string assigned to key with the tuple of
case["conversation_group_id"], case["turn_id"], and case["metric_identifier"]
directly. Keep the existing case_total increment and pass/fail aggregation
unchanged so distinct identifiers cannot collide and corrupt pass_at_k.
In `@src/lightspeed_evaluation/pipeline/behavioral/loader.py`:
- Around line 92-95: Validate that DictReader.fieldnames contains
conversation_group_id, turn_id, and metric_identifier before iterating over rows
in the loader function containing this result mapping. If any required
identifier column is missing, log the invalid schema and return None; otherwise
preserve the existing row parsing behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 19e85eea-0156-4151-a309-0c7cc9470c5b
📒 Files selected for processing (6)
src/lightspeed_evaluation/pipeline/behavioral/consolidation.pysrc/lightspeed_evaluation/pipeline/behavioral/loader.pysrc/lightspeed_evaluation/pipeline/behavioral/statistics.pytests/unit/pipeline/behavioral/test_consolidation.pytests/unit/pipeline/behavioral/test_loader.pytests/unit/pipeline/behavioral/test_statistics.py
Comment on lines
+280
to
+286
| key = ( | ||
| f"{case['conversation_group_id']}:" | ||
| f"{case['turn_id']}:" | ||
| f"{case['metric_identifier']}" | ||
| ) | ||
| case_total[key] += 1 | ||
| if case["result"] == "PASS": |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Use a tuple for the case key.
If an identifier contains :, the string key can collide. For example, ("a:b", "c", "d") and ("a", "b:c", "d") both produce "a:b:c:d". Line 285 then combines unrelated case counts and corrupts pass_at_k.
Use the required identifier tuple directly as the dictionary key.
Proposed fix
-def _compute_pass_at_k(runs: list[RunData], k: int) -> Optional[float]:
+def _compute_pass_at_k(runs: list[RunData], k: int) -> Optional[float]:
"""Compute pass@k from per-case results across runs."""
- case_pass: dict[str, int] = defaultdict(int)
- case_total: dict[str, int] = defaultdict(int)
+ case_pass: dict[tuple[str, str, str], int] = defaultdict(int)
+ case_total: dict[tuple[str, str, str], int] = defaultdict(int)
...
- key = (
- f"{case['conversation_group_id']}:"
- f"{case['turn_id']}:"
- f"{case['metric_identifier']}"
+ key = (
+ case["conversation_group_id"],
+ case["turn_id"],
+ case["metric_identifier"],
)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| key = ( | |
| f"{case['conversation_group_id']}:" | |
| f"{case['turn_id']}:" | |
| f"{case['metric_identifier']}" | |
| ) | |
| case_total[key] += 1 | |
| if case["result"] == "PASS": | |
| key = ( | |
| case["conversation_group_id"], | |
| case["turn_id"], | |
| case["metric_identifier"], | |
| ) | |
| case_total[key] += 1 | |
| if case["result"] == "PASS": |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/lightspeed_evaluation/pipeline/behavioral/consolidation.py` around lines
280 - 286, In the case-counting logic, replace the colon-concatenated string
assigned to key with the tuple of case["conversation_group_id"],
case["turn_id"], and case["metric_identifier"] directly. Keep the existing
case_total increment and pass/fail aggregation unchanged so distinct identifiers
cannot collide and corrupt pass_at_k.
Comment on lines
+92
to
+95
| "conversation_group_id": row.get("conversation_group_id", ""), | ||
| "turn_id": row.get("turn_id", ""), | ||
| "metric_identifier": row.get("metric_identifier", ""), | ||
| "result": row.get("result", "ERROR"), |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Validate required case identifier columns.
If a detailed CSV lacks an identifier column, Lines 92-94 silently replace it with "". Consolidation then groups these rows as a real case and reports a distorted pass_at_k value.
Validate conversation_group_id, turn_id, and metric_identifier from DictReader.fieldnames. Log the invalid schema and return None before reading rows.
Proposed fix
cases: list[dict[str, str]] = []
with open(matches[0], encoding="utf-8") as f:
- for row in csv.DictReader(f):
+ reader = csv.DictReader(f)
+ required_columns = {
+ "conversation_group_id",
+ "turn_id",
+ "metric_identifier",
+ }
+ if not reader.fieldnames or not required_columns.issubset(reader.fieldnames):
+ logger.warning("Invalid detailed CSV schema in %s", matches[0])
+ return None
+ for row in reader:📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| "conversation_group_id": row.get("conversation_group_id", ""), | |
| "turn_id": row.get("turn_id", ""), | |
| "metric_identifier": row.get("metric_identifier", ""), | |
| "result": row.get("result", "ERROR"), | |
| cases: list[dict[str, str]] = [] | |
| with open(matches[0], encoding="utf-8") as f: | |
| reader = csv.DictReader(f) | |
| required_columns = { | |
| "conversation_group_id", | |
| "turn_id", | |
| "metric_identifier", | |
| } | |
| if not reader.fieldnames or not required_columns.issubset(reader.fieldnames): | |
| logger.warning("Invalid detailed CSV schema in %s", matches[0]) | |
| return None | |
| for row in reader: | |
| "conversation_group_id": row.get("conversation_group_id", ""), | |
| "turn_id": row.get("turn_id", ""), | |
| "metric_identifier": row.get("metric_identifier", ""), | |
| "result": row.get("result", "ERROR"), |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/lightspeed_evaluation/pipeline/behavioral/loader.py` around lines 92 -
95, Validate that DictReader.fieldnames contains conversation_group_id, turn_id,
and metric_identifier before iterating over rows in the loader function
containing this result mapping. If any required identifier column is missing,
log the invalid schema and return None; otherwise preserve the existing row
parsing behavior.
asamal4
marked this pull request as draft
August 12, 2026 06:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Type of change
Tools used to create PR
Identify any AI code assistants used in this PR (for transparency and review context)
Related Tickets & Documents
Checklist before requesting a review
Testing
Summary by CodeRabbit
New Features
Bug Fixes
Tests