Add new Frontier-CS 2.0 problem lsm_compaction_policy

[bochencs]

Summary

Adds a Frontier-CS 2.0 systems task on online LSM-tree compaction policy design. Recent systems research has revisited adaptive, workload-aware LSM compaction and tuning; this task tests whether agents can infer robust online policies from aggregate feedback under hidden workload shifts.

Type of Change

• New research problem
• New algorithmic problem
• New Frontier-CS 2.0 problem
• Bug fix
• Documentation update
• Other:

Testing

$ PYTHONPYCACHEPREFIX=/private/tmp/frontier-cs-pycache-post-amend python3 -m py_compile 2.0/problems/lsm_compaction_policy/evaluator.py 2.0/problems/lsm_compaction_policy/reference.py
# exit 0, no output

$ PYTHONPYCACHEPREFIX=/private/tmp/frontier-cs-pycache-post-amend python3 2.0/problems/lsm_compaction_policy/evaluator.py 2.0/problems/lsm_compaction_policy/reference.py
role=final; valid=1; cases=14; total_cost=52485852.352; invalid_actions=0; candidate_seconds=0.754678; score=10.000000
{
  "role": "final",
  "valid": 1,
  "cases": 14,
  "total_cost": 52485852.35237111,
  "invalid_actions": 0,
  "candidate_seconds": 0.7546780840493739
}
10.000000000000 10.000000000000

$ FRONTIER_SUBMISSION_ROLE=feedback PYTHONPYCACHEPREFIX=/private/tmp/frontier-cs-pycache-post-amend python3 2.0/problems/lsm_compaction_policy/evaluator.py 2.0/problems/lsm_compaction_policy/reference.py
role=feedback; valid=1; cases=6; total_cost=5191351.694; invalid_actions=0; candidate_seconds=0.254550; score=10.000000
{
  "role": "feedback",
  "valid": 1,
  "cases": 6,
  "total_cost": 5191351.6939448025,
  "invalid_actions": 0,
  "candidate_seconds": 0.25454999995417893
}
10.000000000000 10.000000000000

$ TMPDIR=/Users/r/github/.frontier-cs-validation-tmp uv run --no-sync python scripts/validate_problems.py --track 2.0 --problems lsm_compaction_policy --timeout 1200 --verbose
Validating 1 2.0 problem(s)

============================================================
Validating: lsm_compaction_policy
============================================================
  Reference: 2.0/problems/lsm_compaction_policy/reference.py
  Running evaluation...
  PASS: score = 10.0

============================================================
Summary
============================================================
  Passed: 1
  Failed: 0

All problems validated successfully!

GPT-5.5 difficulty check

Model: GPT-5.5, reasoning_effort=xhigh. All trials completed normally and were scored by the final verifier. No timeout, cancellation, exception, or invalid action occurred.

Trial A: role=final; valid=1; cases=14; invalid_actions=0; score=10.000000; reward=0.1000; total_cost=58,253,502.725; candidate_seconds=0.684643.
Trial B: role=final; valid=1; cases=14; invalid_actions=0; score=10.000000; reward=0.1000; total_cost=39,211,021.029; candidate_seconds=0.621834.
Trial C: role=final; valid=1; cases=14; invalid_actions=0; score=10.000000; reward=0.1000; total_cost=34,369,771.940; candidate_seconds=0.705880.

In Trial C, the model completed a bounded feedback search with valid submissions and improved feedback total_cost from 10,059,326.847 to 5,297,948.125, but the score remained 10.000000. This supports that the failure is due to task difficulty and hidden-family scoring, not invalid submissions or infrastructure timeout.

Checklist

• Code follows the project structure and conventions
• Self-review completed
• Documentation updated (if applicable)

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds the Frontier-CS 2.0 “LSM compaction policy” problem package, including documentation, a local evaluator/harness, and a reference policy implementation.

Changes:

• Introduce a baseline/reference policy (reference.py) and problem statement (readme).
• Add a full offline evaluator and sandbox runner (evaluator.py) plus a convenience entrypoint script (evaluate.sh).
• Add runtime/submission metadata for the task (config.yaml).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
2.0/problems/lsm_compaction_policy/reference.py	Adds a simple reference compaction policy used for baseline behavior.
2.0/problems/lsm_compaction_policy/readme	Documents the task, policy interface, observation schema, scoring, and restrictions.
2.0/problems/lsm_compaction_policy/evaluator.py	Implements the simulator, scoring, sandboxed policy runner, and case generation.
2.0/problems/lsm_compaction_policy/evaluate.sh	Adds a shell wrapper to run the evaluator against a solution path.
2.0/problems/lsm_compaction_policy/config.yaml	Declares environment/runtime and submission contract for the task.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[joyemang33]

Hi @bochencs, thanks for your contribution. Can you implement an LSM tree task in a database-native environment such as rocksDB? maybe refer our duckdb_e2e_optimization problems. I think RocksDB is where the most LSM tree research will implement their code

[bochencs]

in a database-native environment such as rocksDB?

@joyemang33 Thanks for the quick check! That make sense for me, let me work on it

[joyemang33]

something might also be useful: https://arxiv.org/pdf/2501.16759v3