Add FPGA Placement Optimization benchmark

[y-ji24]

Summary

This PR introduces the FPGA Placement Optimization benchmark.

The benchmark evaluates an agent's ability to design and improve placement algorithms through an editable artifact (scripts/init.py).

Design

• Editable artifact:
  scripts/init.py

• Baseline:
  reference only, not modified by agents

• Evaluation:
  candidate-generated solution.pl is evaluated independently

Benchmark Features

• Bookshelf format parsing
• Placement legality verification
• HPWL-based scoring
• Multiple benchmark support

Dataset

The default benchmark is bundled.

ISPD 2016 benchmarks are supported as optional external datasets.

Validation

Tested:

• default benchmark evaluation
• ISPD benchmark loading
• legality checks
• invalid benchmark handling

[github-actions]

🤖 AI Code Review (gemini-3-flash-preview)

🇬🇧 English Analysis

1. Executive Summary

• Core Purpose: This PR introduces a new evaluation metric ("Medal Score"), a lightweight benchmark subset (v1-lite), and comprehensive design/validation documentation for a new FPGA placement optimization task.
• Modified File Structure & Modifications:
  • .gitignore: Whitelisted leaderboard/*.csv and the new v1_lite.yaml configuration.
  • README.md & README_zh-CN.md: Updated with "News" regarding the Medal Score and v1-lite. Updated the leaderboard table to reflect these new metrics and future-dated (2026) project milestones.
  • benchmarks/FPGA/FPGAPlacementOptimization/ClockAwarePlacement_DesignValidation.md: A new, highly detailed audit of the aug-elfPlace baseline, identifying discrepancies between the "Clock-Aware" naming and actual code implementation (e.g., dead code in C++ parsers).
  • benchmarks/FPGA/FPGAPlacementOptimization/ClockAwarePlacement_Design_Report.md: A new architectural blueprint for the FPGA placement task, outlining scoring logic (HPWL-based), validation gates, and implementation roadmaps.

2. AI Content Analysis

• Estimated AI Component: 25%
• Reasoning & Evidence: The README updates and leaderboard data are clearly manual entries. The two new Markdown reports (DesignValidation.md and Design_Report.md) exhibit a highly structured, LLM-like formatting (tables, clear headers, pros/cons lists). However, the content is extremely domain-specific and references internal code locations (e.g., PlaceDB.cpp:805-813, lut_ff_legalization.py:240). This suggests an engineer used an AI to structure and draft findings based on a very thorough manual code audit.

3. Engineering & Economic Assessment

• Engineering Reality Check:
  • High Reality: The "Medal Score" addresses a sophisticated benchmarking problem: how to aggregate performance across diverse tasks without letting "long-tail" marginal gains in easy tasks skew the average.
  • Critical Audit: The DesignValidation.md is a standout piece of engineering work. It correctly identifies that the baseline lacks actual clock-aware optimization despite the naming, preventing the creation of a "toy" benchmark based on false assumptions.
• Economic Value: High.
  • Efficiency: The v1-lite subset significantly reduces the compute cost and time for developer iterations.
  • Technical Debt: The FPGA audit prevents the implementation of a flawed benchmark, saving future engineering hours that would have been spent debugging non-functional "clock" constraints.

4. Quality Assurance

• Verification & Testing:
  • frontier_eval Integration: Yes.
  • task_name: v1-lite (as a batch configuration) and ClockAwarePlacement (proposed in design docs).
  • Execution & Dependencies: The README explicitly provides the command to run the new v1-lite subset (frontier_eval/conf/batch/v1_lite.yaml). The design report correctly identifies the need for a pure-Python evaluator to avoid heavy C++/CUDA dependencies.
• Documentation Quality: Excellent. The design reports are thorough, and the README updates are clear. Note: The dates are set in June 2026, which appears to be a project roadmap convention rather than a typo.
• Organizational Structure: Logical. The design and validation documents are co-located with the specific benchmark they describe.

5. Security & Privacy Check

• Sensitive Files: Clean. No .env, .vscode, or cache files were included.
• Absolute Paths: None detected. The design report explicitly recommends using relative paths and pure-Python scoring to maintain portability.

---

🇨🇳 中文分析

1. 摘要

• 核心目的: 此 PR 引入了新的评分指标（“勋章分” Medal Score）、轻量级评测子集（v1-lite），以及针对新 FPGA 布局优化任务的详细设计与验证文档。
• 修改的文件结构与变更摘要:
  • .gitignore: 将 leaderboard/*.csv 和新的 v1_lite.yaml 配置列入白名单。
  • README.md & README_zh-CN.md: 更新了关于 Medal Score 和 v1-lite 的“新闻”动态。更新了排行榜表格以反映这些新指标，并标注了 2026 年的项目里程碑。
  • benchmarks/FPGA/FPGAPlacementOptimization/ClockAwarePlacement_DesignValidation.md: 一份全新的、极其详尽的 aug-elfPlace 基准线审计文档，指出了“时钟感知”命名与实际代码实现之间的差异（例如 C++ 解析器中的死代码）。
  • benchmarks/FPGA/FPGAPlacementOptimization/ClockAwarePlacement_Design_Report.md: FPGA 布局任务的新架构蓝图，概述了评分逻辑（基于 HPWL）、验证门槛和实施路线图。

2. AI 成分分析

• 预估 AI 含量: 25%
• 判断依据与证据: README 的更新和排行榜数据显然是手动录入的。两份新的 Markdown 报告（DesignValidation.md 和 Design_Report.md）表现出高度结构化、类似 LLM 的格式（表格、清晰的标题、优缺点列表）。然而，内容具有极强的领域专业性，并引用了内部代码位置（如 PlaceDB.cpp:805-813, lut_ff_legalization.py:240）。这表明工程师在进行了非常彻底的手动代码审计后，利用 AI 进行了结构化整理和草拟。

3. 工程与经济评估

• 工程现实检验:
  • 高度现实: “勋章分”解决了基准测试中的一个复杂问题：如何在不让简单任务中的“长尾”边际收益扭曲平均值的情况下，汇总不同任务的性能。
  • 批判性审计: DesignValidation.md 是工程实践中的亮点。它准确识别出基准线虽然命名为“时钟感知”，但实际上缺乏相关优化逻辑，从而避免了基于错误假设构建“玩具级”基准测试。
• 经济价值: 高。
  • 效率: v1-lite 子集显著降低了开发人员迭代的计算成本和时间。
  • 技术债务: FPGA 审计防止了缺陷基准测试的实施，节省了未来用于调试无效“时钟”约束的工程时间。

4. 质量保证

• 验证与测试:
  • frontier_eval 集成: 是。
  • task_name: v1-lite（作为批处理配置）和 ClockAwarePlacement（在设计文档中提出）。
  • 运行与依赖: README 明确提供了运行新 v1-lite 子集的命令（frontier_eval/conf/batch/v1_lite.yaml）。设计报告正确识别了对纯 Python 评估器的需求，以避免沉重的 C++/CUDA 依赖。
• 文档质量: 优秀。设计报告非常详尽，README 更新清晰。注：日期设定在 2026 年 6 月，这似乎是项目路线图的惯例而非拼写错误。
• 组织结构: 符合逻辑。设计和验证文档与它们所描述的特定基准测试存放在一起。

5. 安全与隐私检查

• 敏感文件: 未发现异常。未包含 .env、.vscode 或缓存文件。
• 绝对路径: 未检测到。设计报告明确建议使用相对路径和纯 Python 评分以保持可移植性。

[wrh-human]

Review — FPGAPlacementOptimization (PR #85)

Thank you for contributing this benchmark! FPGA placement optimization is a genuine EDA engineering problem, the HPWL scoring and three legality gates are well-designed, and the canonical reference implementation ensures evaluator independence. After a line-by-line review of all code and files, one issue was identified that needs to be addressed before merging.

---

Evaluation by dimension

1. Domain, Economic Value, and Frontier-Eng Fit ✅

FPGA placement optimization is a real electronic design automation (EDA) problem — logic cells (LUTs, FFs, DSPs, BRAMs) must be assigned to legal sites on the FPGA grid to minimize wirelength. Placement quality directly affects chip area, power consumption, and timing closure. The task uses the ISPD 2016 FPGA Placement Contest Bookshelf format, and HPWL is the industry-standard metric for placement quality. Economic value is clear.

2. Not purely numerical ✅

The agent modifies the placement algorithm logic in scripts/init.py (row-scan → heuristic/analytical/simulated annealing approaches), not numerical parameters.

3. Search space ✅

FPGA placement is an NP-hard combinatorial optimization problem. The example design has 3,336 instances (72 fixed IO, 3,264 movable). Brute-force search is infeasible.

4. Evaluator and engineering verification ✅

The evaluator runs the candidate in a subprocess (subprocess.run), then independently parses solution.pl and verifies legality and computes HPWL via canonical.py. All three legality gates are independently implemented in the canonical reference, independent of the candidate and not trusting candidate-reported values.

5. Constraint enforcement ✅

All three gates are independently implemented in check_site_type_compatibility(), check_site_capacity(), and check_carry_chain_integrity(). Violating any single gate returns valid=0.0.

6. Baseline experiment ✅

baseline/result_log.txt is included, documenting the fpga-example1 run (HPWL=210,721, all gates pass, runtime 0.19s).

7. Scoring system ✅

combined_score = -HPWL. HPWL is the industry-standard metric for placement quality. The scoring directly uses the raw wirelength — no clipping, no normalization. Discriminative power is clear and well-supported.

---

Issue to address

Issue 1: EVOLVE-BLOCK wraps the entire file — no read-only shell

In scripts/init.py, EVOLVE-BLOCK-START is at line 1 and EVOLVE-BLOCK-END is at line 314, wrapping the entire file — including imports, the main() function, argument parsing, and Bookshelf file I/O. There is no read-only shell. The evaluator contains no EVOLVE-related validation logic and never checks whether the candidate modified code outside the EVOLVE-BLOCK markers.

The "DO NOT MODIFY" comments inside the file have no machine-enforceable mechanism — if the agent modifies the CLI interface or the solution.pl output format, the evaluator may be unable to correctly parse the candidate's output.

Suggestion: move the read-only CLI interface, argument parsing, and Bookshelf file I/O logic outside the EVOLVE-BLOCK. Restrict the EVOLVE-BLOCK to only wrap the placement algorithm function, and add EVOLVE-BLOCK boundary validation to the evaluator.

---

FPGA placement optimization has the right direction, and the HPWL scoring and legality gates are well-designed. The review can proceed once the above issue is addressed.