Add PrimerDesignOptimization benchmark for PCR primer design

[y-ji24]

Summary

This PR adds a new unified benchmark:

• Domain: Bioinformatics
• Task: PrimerDesignOptimization

Features

• Deterministic evaluator
• Configuration-driven scoring
• 10 hard validation gates
• 10 weighted quality metrics
• English and Chinese documentation
• Unified Frontier-Eval metadata
• Docker-compatible evaluation

Verification

Tested successfully:

python verification/evaluator.py scripts/init.py

python -m frontier_eval task=unified task.benchmark=Bioinformatics/PrimerDesignOptimization algorithm.iterations=0

Both commands execute successfully.

Notes

This benchmark evaluates PCR primer design under practical biological constraints using nearest-neighbor thermodynamic models.

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new bioinformatics benchmark for PCR Primer Design Optimization. It provides a framework to design and evaluate PCR primer pairs based on thermodynamic stability, structural constraints (hairpins, dimers), and sequence specificity.
• Modified File Structure & Modifications:
  • benchmarks/Bioinformatics/PrimerDesignOptimization/README.md & README_zh-CN.md: Comprehensive documentation in English and Chinese covering setup, scoring, and execution.
  • benchmarks/Bioinformatics/PrimerDesignOptimization/Task.md & Task_zh-CN.md: Detailed technical specifications of the primer design problem, including thermodynamic formulas and hard validation gates.
  • benchmarks/Bioinformatics/PrimerDesignOptimization/baseline/solution.py: Implementation of a Genetic Algorithm (GA) to solve the optimization problem.
  • benchmarks/Bioinformatics/PrimerDesignOptimization/verification/evaluator.py: (Referenced in diff) Script to validate submissions against 10 hard gates and calculate weighted quality scores.
  • benchmarks/Bioinformatics/PrimerDesignOptimization/frontier_eval/: Integration files for the frontier_eval framework, including evaluation commands and constraints.

2. AI Content Analysis

• Estimated AI Component: 75%
• Reasoning & Evidence:
  • Standard Boilerplate: The Genetic Algorithm implementation in solution.py (tournament selection, single-point crossover, mutation) follows a highly standard, textbook-style structure typical of AI-generated code.
  • Over-commenting: The docstrings and comments (e.g., nearest_neighbor_delta, _tournament_select) are extremely descriptive and follow a consistent, clean format often seen in LLM outputs.
  • Domain Knowledge Integration: While the thermodynamic parameters (SantaLucia 1998) are domain-specific, the implementation of the formulas is mathematically "perfect" and lacks the idiosyncratic "hacks" or logging often found in manually written research code.

3. Engineering & Economic Assessment

• Engineering Reality Check: This is a production-grade problem. Unlike "toy" examples that only check GC content, this benchmark incorporates the SantaLucia (1998) nearest-neighbor thermodynamic model, salt corrections (monovalent/divalent), and structural checks (hairpins/dimers). It handles realistic edge cases such as 3'-end GC clamps and homopolymer runs.
• Economic Value: Medium-High. Automated primer design is a staple in molecular biology. As a benchmark, it provides high value for evaluating LLMs' ability to handle multi-objective optimization with strict physical/chemical constraints, which is critical for AI-driven drug discovery and synthetic biology.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: Bioinformatics/PrimerDesignOptimization
  • Execution & Dependencies: The README.md clearly documents the installation of dependencies via verification/requirements.txt and provides explicit commands for running the baseline and evaluator.
• Documentation Quality: High. The documentation is bilingual and includes a clear file structure, scoring breakdown, and submission format.
  • Minor Issue: The English README.md contains a Byte Order Mark (BOM) or hidden character at the very beginning of the file (visible as #).
• Organizational Structure: Excellent. The project is logically partitioned into baseline, verification, scripts, and references, making it highly scalable for adding more complex DNA templates.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, API keys, or __pycache__ folders were detected in the diff.
• Absolute Paths: None detected. The code uses Path(__file__).resolve().parent for robust relative path handling in solution.py.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 引入了一个新的生物信息学基准测试：PCR 引物设计优化 (PCR Primer Design Optimization)。它提供了一个框架，用于根据热力学稳定性、结构约束（发夹结构、二聚体）和序列特异性来设计和评估 PCR 引物对。
• 修改的文件结构与变更摘要:
  • benchmarks/Bioinformatics/PrimerDesignOptimization/README.md & README_zh-CN.md: 中英文双语文档，涵盖安装、评分和运行指南。
  • benchmarks/Bioinformatics/PrimerDesignOptimization/Task.md & Task_zh-CN.md: 详细的任务技术规范，包括热力学公式和硬性验证门槛。
  • benchmarks/Bioinformatics/PrimerDesignOptimization/baseline/solution.py: 实现了一个基于遗传算法 (GA) 的基线解决方案。
  • benchmarks/Bioinformatics/PrimerDesignOptimization/verification/evaluator.py: (在 diff 中提及) 用于根据 10 个硬性门槛验证提交并计算加权质量得分的脚本。
  • benchmarks/Bioinformatics/PrimerDesignOptimization/frontier_eval/: frontier_eval 框架的集成文件，包含评估命令和约束条件。

2. AI 成分分析

• 预估 AI 含量: 75%
• 判断依据与证据:
  • 标准模板化: solution.py 中的遗传算法实现（锦标赛选择、单点交叉、变异）遵循高度标准的教科书式结构，这是 AI 生成代码的典型特征。
  • 过度注释: 函数文档字符串和注释（如 nearest_neighbor_delta, _tournament_select）描述极其详尽，且遵循 LLM 输出中常见的整洁格式。
  • 领域知识集成: 虽然热力学参数（SantaLucia 1998）具有专业性，但公式的实现数学上过于“完美”，缺乏手动编写研究代码中常见的临时调试逻辑或非标准处理。

3. 工程与经济评估

• 工程现实检验: 这是一个生产级别的问题。与仅检查 GC 含量的“玩具”示例不同，该基准测试结合了 SantaLucia (1998) 最邻近热力学模型、盐浓度校正（一价/二价阳离子）以及结构检查（发夹/二聚体）。它处理了诸如 3' 端 GC 夹和同聚物连续重复等现实边缘情况。
• 经济价值: 中高。自动引物设计是分子生物学的基础。作为基准测试，它为评估 LLM 处理具有严格物理/化学约束的多目标优化能力提供了极高价值，这对于 AI 驱动的药物研发和合成生物学至关重要。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是
  • task_name: Bioinformatics/PrimerDesignOptimization
  • 运行与依赖: README.md 清晰记录了通过 verification/requirements.txt 安装依赖的方法，并提供了运行基线和评估器的明确命令。
• 文档质量: 高。文档采用双语编写，包含清晰的文件结构、评分细则和提交格式。
  • 微小问题: 英文版 README.md 文件开头包含一个字节顺序标记 (BOM) 或隐藏字符（显示为 #）。
• 组织结构: 优秀。项目逻辑上划分为 baseline、verification、scripts 和 references，为添加更复杂的 DNA 模板提供了良好的可扩展性。

5. 安全与隐私检查

• 敏感文件: 未发现异常。Diff 中未检测到 .env、API 密钥或 __pycache__ 文件夹。
• 绝对路径: 未检测到。代码在 solution.py 中使用 Path(__file__).resolve().parent 进行健壮的相对路径处理。

[wrh-human]

Review — PrimerDesignOptimization (PR #84)

Thank you for contributing this benchmark! PCR primer design is a genuine bioinformatics problem, the thermodynamic model is well-chosen, and the hard gate system is comprehensive. After a line-by-line review of all code and files, several issues were identified that need to be addressed before merging.

---

Evaluation by dimension

1. Domain, Economic Value, and Frontier-Eng Fit ✅

PCR primer design is a critical step in real molecular biology workflows — primer quality directly affects PCR amplification specificity and success rate. The task uses the nearest-neighbor thermodynamic model (Breslauer parameters + SantaLucia salt correction), which is the industry standard for primer design. Optimizing primer design directly improves experimental success rates, giving the task clear economic value.

2. Not purely numerical ✅

The agent modifies the primer design algorithm logic (sequence generation, thermodynamic computation, constraint screening), not numerical parameters.

3. Search space ✅

Primer length is 18-25 bp, with a theoretical search space of 4^18 to 4^25 possibilities (approximately 6.9×10^10 to 1.1×10^15), making brute-force search infeasible.

4. Evaluator and engineering verification ✅

10 hard gates cover character validity, length, GC content, Tm, GC clamp, complementarity, hairpin structure, homopolymer runs, alignment, and product length. The thermodynamic computation is correctly implemented (nearest-neighbor model + SantaLucia salt correction). The evaluator runs the candidate in a subprocess with a timeout.

5. Constraint enforcement ✅

The 10 hard gates are checked incrementally in run_hard_gates(). Violating any single gate returns valid=False and final_score=0.0. Coverage is comprehensive.

6. Baseline experiment ❌ Needs improvement

baseline/solution.py (486 lines, GA algorithm) is included, but there is no baseline/result_log.json and no test file.

7. Scoring system

Each of the 10 quality metrics is normalized to [0, 1], with a weighted average clipped to [0, 1]. The weight distribution is reasonable (Tm highest at 1.0, product length lowest at 0.2).

---

Issues to address

Issue 1 (most critical): EVOLVE-BLOCK wraps the entire file — no read-only shell

In scripts/init.py, EVOLVE-BLOCK-START is at line 1 and EVOLVE-BLOCK-END is at line 459, wrapping the entire file — including import statements, the load_config() configuration loader, the compute_tm() thermodynamic computation function, and all other helper functions marked "DO NOT MODIFY". There is no read-only shell. grep -c "EVOLVE" on the evaluator returns 0 — it never validates EVOLVE-BLOCK boundaries.

The "DO NOT MODIFY" comments inside the file have no machine-enforceable mechanism — an agent could modify load_config() to return fake configuration values, or modify compute_tm() to return fake melting temperatures, thereby bypassing the hard gate constraint checks without detection by the evaluator.

Suggestion:

• Move read-only helper functions (config loader, thermodynamic computation, complementarity checks, etc.) outside the EVOLVE-BLOCK
• Restrict the EVOLVE-BLOCK to only wrap design_primers()
• Add EVOLVE-BLOCK boundary validation to the evaluator

Issue 2 (most critical): Single template only — no generalization test

Only one template (120bp synthetic DNA fragment) is provided. There are no hidden/validation templates. An agent could hardcode primers for this specific sequence without building a general primer design algorithm.

Suggestion: add at least 3-5 hidden test templates covering different sequence lengths, GC content ranges, and complexity levels.

Issue 3 (most critical): frontier_eval/run_eval.py path resolution error — cannot load the evaluator

run_eval.py line 66 locates the evaluator via Path(__file__).with_name("evaluator.py"), which resolves to frontier_eval/evaluator.py. However, the actual evaluator is at verification/evaluator.py. The file frontier_eval/evaluator.py does not exist, so spec_from_file_location() returns None and _load_local_evaluator() raises a RuntimeError, making unified evaluation completely non-functional.

Suggestion: change the path to Path(__file__).resolve().parent.parent / "verification" / "evaluator.py", or modify eval_command.txt to point directly to verification/evaluator.py (as other benchmarks do).

Issue 4: Baseline uses random without setting a random seed

The genetic algorithm in baseline/solution.py uses random.randint, random.random, random.choice, and random.randrange, but no random.seed() is set. Each run produces different primer results, making the baseline score non-reproducible.

Suggestion: add random.seed(42) at the entry point of design_primers() (or read the seed from primer_config.json), and add a determinism check in the evaluator for the candidate as well.

Issue 5: Missing baseline validation (no test file, no baseline run results)

There are no test files (test_evaluator.py) and no baseline/result_log.json. It is not possible to verify that the baseline's genetic algorithm can consistently produce primers that pass all 10 hard gates under the current configuration.

Suggestion: add test_evaluator.py (at minimum covering run_hard_gates() for each gate and an end-to-end test of the baseline), and create baseline/result_log.json recording the baseline run results.

Issue 6: Candidate subprocess lacks resource limits

subprocess.run() does not set a preexec_fn, and lacks RLIMIT_NPROC, RLIMIT_AS, RLIMIT_CPU, or other resource limits. The 120-second timeout is the only protection.

Suggestion: add preexec_fn with appropriate resource limits.

---

Non-blocking suggestions

1. Score clipping at [0, 1] compresses top-end discrimination: The final score is clipped to [0, 1]. The score gap between the baseline and excellent candidates may be compressed. Consider removing the cap or using a wider dynamic range.

2. 120-second timeout is generous: For a primer design task, consider reducing the timeout to 30-60 seconds.

3. Random module inconsistency between scripts/init.py and baseline/solution.py: init.py imports numpy, while solution.py uses random without importing numpy. Consider unifying the random source and maintaining import consistency between the two files.

4. Duplicate thermodynamic computation: Both scripts/init.py and verification/evaluator.py implement the full nearest-neighbor thermodynamic computation independently. While this is a reasonable requirement for evaluator independence, maintaining two copies of the same logic carries a risk of future divergence. Consider adding a comment explicitly noting that the two implementations must be kept in sync.

---

The direction — PCR primer design — is sound, and the hard gate system and thermodynamic model are well-designed. Issues 1 (EVOLVE-BLOCK wrapping the entire file), 2 (single template), and 3 (run_eval.py path resolution error breaking framework integration) are the highest-priority items, with Issue 3 being a bug that directly prevents unified evaluation from working. The review can proceed once the above issues are addressed.