Repository navigation
Conversation
The sandbox that runs generated solutions replaces sys.stdin with a plain
StringIO (testing_util.call_method) and sys.stdout with a plain StringIO
(testing_util.Capturing). Neither exposes the binary `.buffer` view that the
real standard streams have, so any solution using the common competitive
programming fast-IO idiom crashes:
data = sys.stdin.buffer.read().split()
sys.stdout.buffer.write(str(ans).encode())
AttributeError: '_io.StringIO' object has no attribute 'buffer'
The exception is swallowed and recorded as -1 (Runtime Error), so the
solution silently scores 0 and pass@1 is under-reported. How large the loss
is depends only on how often the evaluated model happens to use that idiom,
which is why it can go unnoticed for a long time.
Measured on LiveCodeBench v6 (release_version 2025-05, 175 problems) with a
model that uses `sys.stdin.buffer` in 68 of its 175 answers: 67 of the 74
failures were this AttributeError. After the fix 58 of them pass and pass@1
goes from 57.71 to 90.86; the 9 remaining failures are genuine (7 wrong
answers, 1 IndexError, 1 TLE).
Fix:
- `_StdinWithBuffer` backs `.buffer` with a BytesIO over the same input.
- `_StdoutWithBuffer._Buffer` decodes binary writes and forwards them into
the same StringIO, so `Capturing.__exit__`'s `getvalue()` still returns the
complete output. A separate BytesIO would have turned the Runtime Error
into an equally wrong empty-output Wrong Answer.
Adds 4 regression tests to tests/UT/datasets/livecodebench/, which fail
before this change and pass after it.
github-actions
Bot
added
no_issue
|
🚫 PR 合入质量检查未通过，PR 已被禁止合入。 未通过项：
请按上述提示整改后，重新推送（push）或评论本 PR 即可触发再次检查。 |
|
Closes #517 |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Three moderate issues remain regarding stream positioning, incremental UTF-8 decoding, and import-time exception handling.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (2)
What changed in this PR
Fixes LiveCodeBench mocked stdin/stdout streams to expose .buffer for binary fast I/O.
Changes:
- Adds binary stdin and stdout wrappers.
- Adds four regression tests for binary and mixed I/O.
- Preserves captured output through binary writes.
| File | Summary |
|---|---|
ais_bench/benchmark/datasets/livecodebench/testing_util.py |
Adds .buffer support. Findings: moderate shared-position issue (2 votes) and moderate split UTF-8 write issue (1 vote). |
tests/UT/datasets/livecodebench/test_livecodebench.py |
Adds regression coverage. Finding: moderate import-exception handling issue (3 votes). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| def __init__(self, data): | ||
| super().__init__(data) | ||
| self.buffer = io.BytesIO(data.encode()) |
| except ImportError: | ||
| raise unittest.SkipTest('testing_util not available') |
github-actions
Bot
added
review_opened
|
🚫 PR 合入质量检查未通过，PR 已被禁止合入。 未通过项：
请按上述提示整改后，重新推送（push）或评论本 PR 即可触发再次检查。 |
SJTUyh
left a comment
SJTUyh
left a comment
There was a problem hiding this comment.
[review] 动机核实结论：合理，且与上游问题完全对应。上游 LiveCodeBench 官方仓库确有同题 issue：LiveCodeBench/LiveCodeBench#160 "Code-generation judge rejects valid solutions using sys.stdin.buffer.readline() or sys.stdout.buffer"（https://github.com/LiveCodeBench/LiveCodeBench/issues/160，目前仍 open），正文明确指出 Capturing 把 sys.stdout 换成 StringIO 后没有 .buffer，sys.stdout.buffer.write(...) 会抛 AttributeError：并用一次完整 175 题 LCB v6 重判量化了影响（官方判 154/175，修好二进制 IO 后 159/175，涉及 abc393_e/abc398_f/arc191_a）。对应修复 PR 为 LiveCodeBench/LiveCodeBench#161 "Fix binary stdin and stdout handling in the code-generation judge"（open，尚未合并，同时修 stdin 游标与 stdout.buffer），其他 fork 也有同类改动（wasiahmad/livecodebench#1、RouteWorks/RouterArena#234）。我核对上游 main 的 lcb_runner/evaluation/testing_util.py，其 Capturing 仍是裸 StringIO，说明该 fix 上游尚未落地，本 PR 的方向正确。
需要说明的两点差异：1) 本 PR 标题写的是「stdin/stdout」，但改动只补了 stdout（testing_util.py 里 pre-PR 就有 BytesIO/TextIOWrapper 的 stdin mock，tests/UT/.../test_testing_util.py 里未被本 PR 修改的 test_call_method_supports_stdin_buffer 也已在验证 sys.stdin.buffer），所以新增的两个 stdin 用例只是既有行为的回归覆盖，并非本次修复对象；2) 上游 #161 还修了官方 MockBuffer.readline「连续调用永远返回第一行」的缺陷，本 PR 的新用例只做了单次 read/readline，覆盖不到这类问题，若本仓库的 stdin mock 与上游同源，建议补一条连续两次 readline 的用例。另外实现细节上还有返回值语义、非 UTF-8 字节、buffer 接口不完整三处小问题，见下。此外可留意：同类「mock 流缺少 .buffer」问题在 execute_utils.py 的 WriteOnlyStringIO（swallow_io）中也存在，若该路径也用于判 LCB 选手代码，可能残留误判（不在本 diff 内，故未给行号）。
| def write(self, data): | ||
| if isinstance(data, (bytes, bytearray)): | ||
| data = data.decode() | ||
| return self._owner.write(data) |
There was a problem hiding this comment.
[review] 返回值语义与真实 sys.stdout.buffer 不一致：真实 file/buffer 对象的 write() 返回「写入的字节数」，这里返回的是 StringIO.write 的「字符数」。对多字节 UTF-8 输出两者不等，若生成代码用 n = sys.stdout.buffer.write(data) 或断言写入长度（这类写法在竞赛代码里存在），仍会被判错，偏离「不再误判」的目标。建议在解码前记录字节长度并返回它，例如：raw = data if isinstance(data, (bytes, bytearray)) else data.encode(); self._owner.write(raw.decode()); return len(raw)。
|
|
||
| def write(self, data): | ||
| if isinstance(data, (bytes, bytearray)): | ||
| data = data.decode() |
There was a problem hiding this comment.
[review] data.decode() 使用默认 UTF-8 且无 errors 参数：真实 stdout.buffer 可写入任意字节，而这里遇到非 UTF-8 字节（如 latin-1、部分二进制串）会抛 UnicodeDecodeError；另外若一次多字节字符被拆成两次 write（分批 flush 的写法），第二次解码也会失败。这些异常仍会被 run_test 捕获成 Runtime Error，属于本次要修的问题的残留。建议改为 data.decode('utf-8', errors='replace')（或 'surrogateescape'），或按字节缓冲、只在 getvalue/比较阶段统一解码。
| for line in lines: | ||
| self.write(line) | ||
|
|
||
| def flush(self): |
There was a problem hiding this comment.
[review] _Buffer 只实现了 write/writelines/flush，缺少 close/isatty/fileno 等常见接口，且没有 getattr 兜底；生成代码里出现 sys.stdout.buffer.close()、sys.stdout.buffer.isatty()、fileno() 时依旧 AttributeError → 仍被记成 Runtime Error。建议补上 close()（no-op）、isatty() -> False，或加 __getattr__ 委托给 owner，使伪 buffer 更接近真实对象。顺带一提：同一 hunk 里的 self._stringio.close = lambda x: 1 需要一个位置参数，调用 sys.stdout.close()（真实 stdout 无参）会抛 TypeError，属既有代码，但既然在补 stdin/stdout 伪装，建议一并改成 lambda *a, **kw: 1。
| ' print(sum(map(int, data)))\n') | ||
| self.assertEqual(self._run(source, '1 2\n3\n'), '6') | ||
|
|
||
| def test_stdin_buffer_readline(self): |
There was a problem hiding this comment.
[review] 新增的 stdin 用例与既有覆盖重复且没有覆盖上游已知缺陷：call_method 在本 PR 前就已支持 sys.stdin.buffer（见同目录 test_testing_util.py 中未改动的 test_call_method_supports_stdin_buffer / test_call_method_shares_text_and_binary_stream_position），本次改动只作用于 stdout，所以 test_stdin_buffer_read / test_stdin_buffer_readline 属于既有行为的重复验证。更值得补的是上游 issue #160 指出的那个坑：官方 MockBuffer.readline() 永远返回第一行，连续两次 sys.stdin.buffer.readline() 会拿到同一行导致 Wrong Answer（如 abc391_f）。建议把这两个用例合并/精简，并新增一条连续两次 readline 应返回两行、以及 text(input()) 与 binary 读取共享位置的多行用例，才能真正防住这类误判。
The sandbox that runs generated solutions replaces sys.stdin with a plain StringIO (testing_util.call_method) and sys.stdout with a plain StringIO (testing_util.Capturing). Neither exposes the binary
.bufferview that the real standard streams have, so any solution using the common competitive programming fast-IO idiom crashes:The exception is swallowed and recorded as -1 (Runtime Error), so the solution silently scores 0 and pass@1 is under-reported. How large the loss is depends only on how often the evaluated model happens to use that idiom, which is why it can go unnoticed for a long time.
Measured on LiveCodeBench v6 (release_version 2025-05, 175 problems) with a model that uses
sys.stdin.bufferin 68 of its 175 answers: 67 of the 74 failures were this AttributeError. After the fix 58 of them pass and pass@1 goes from 57.71 to 90.86; the 9 remaining failures are genuine (7 wrong answers, 1 IndexError, 1 TLE).Fix:
_StdinWithBufferbacks.bufferwith a BytesIO over the same input._StdoutWithBuffer._Bufferdecodes binary writes and forwards them into the same StringIO, soCapturing.__exit__'sgetvalue()still returns the complete output. A separate BytesIO would have turned the Runtime Error into an equally wrong empty-output Wrong Answer.Adds 4 regression tests to tests/UT/datasets/livecodebench/, which fail before this change and pass after it.
Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。
PR Type / PR类型
Related Issue | 关联 Issue
Fixes #(issue ID / issue 编号) / Relates to #(issue ID / issue 编号)
🔍 Motivation / 变更动机
Please describe the motivation of this PR and the goal you want to achieve through this PR.
请描述您的拉取请求的动机和您希望通过此拉取请求实现的目标。
📝 Modification / 修改内容
Please briefly describe what modification is made in this PR.
请简要描述此拉取请求中进行的修改。
📐 Associated Test Results / 关联测试结果
Please provide links to the related test results, such as CI pipelines, test reports, etc.
请提供相关测试结果的链接，例如 CI 管道、测试报告等。
Does the modification introduce changes that break the backward compatibility of the downstream repositories? If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
是否引入了会破坏下游存储库向后兼容性的更改？如果是，请描述它如何破坏兼容性，以及下游项目应该如何修改其代码以保持与此 PR 的兼容性。
If the modification introduces performance degradation, please describe the impact of the performance degradation and the expected performance improvement.
如果引入了性能下降，请描述性能下降的影响和预期的性能改进。
🌟 Use cases (Optional) / 使用案例（可选）
If this PR introduces a new feature, it is better to list some use cases here and update the documentation.
如果此拉取请求引入了新功能，最好在此处列出一些用例并更新文档。
✅ Checklist / 检查列表
Before PR:
After PR:
👥 Collaboration Info / 协作信息
🌟 Useful CI Command / 实用的CI命令
/gemini review/gemini summary/gemini help/readthedocs build/pr_check