Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

CompactionRL

Reinforcement Learning with Context Compaction for Long-Horizon Agents

Yujiang Li · Zhenyu Hou · Yi Jing · Jie Tang · Yuxiao Dong

arXiv paper Paper PDF Apache 2.0 license

Overview · Results · Code · Getting Started · Training · Evaluation · Citation

CompactionRL trains agents to solve tasks and summarize their own interaction history under a fixed context budget. When the context approaches its limit, the policy generates a summary and continues from the compressed state. Task actions and summary tokens learn together from the final task reward.

This repository brings together the agent framework, asynchronous RL training, and rollout buffer, with configurations for GLM-4.7-Flash and GLM-4.5-Air-SFT.

Overview

Context compaction and benchmark results

Context compaction extends the interaction horizon while bounding the working context.

Long agent trajectories accumulate tool outputs, reasoning and intermediate state. A useful summary must preserve the information needed by later actions, so compaction quality directly affects task success. CompactionRL makes that summary part of the trainable policy and includes it in the RL objective.

The method combines three elements:

  • Compaction-aware rollouts. The agent summarizes earlier history and resumes with the summary and recent tool-interaction pairs, allowing execution to continue within the context budget.
  • Joint execution and summary training. Single-rollout, critic-based PPO optimizes both kinds of generated tokens using the final task reward. Token loss normalization accounts for different execution and summary lengths.
  • Cross-segment credit assignment. A position correction to local GAE uses the number of trainable tokens in later segments to account for each segment's distance to the final outcome.

Method

CompactionRL rollout and training framework

Execution and summary segments share the final reward. Training retains the original generated tokens, log probabilities and loss masks across compaction.

For segment $s$, let $N_{>s}$ be the number of optimized tokens generated in later segments of the same rollout. The cross-segment correction is:

$$ \hat{A}_{s,i} = (\gamma\lambda_s)^{N_{>s}} A^{\mathrm{local}}_{s,i}. $$

This approximates trajectory-level credit assignment while keeping segments independently batchable. Summary tokens participate in the loss; observations and repeated context prefixes are masked. See the paper-to-code map and training semantics for segment construction, token counts and actor/critic updates.

Results

Pass@1 accuracy (%), mean ± standard deviation over three evaluation runs (Table 2):

Backbone Peak context Method SWE-bench Verified (200 tasks) Terminal-Bench 2.0
GLM-4.7-Flash (30B-A3B) 64k Base + inference compaction 50.5 ± 1.2 13.5 ± 0.9
GLM-4.7-Flash (30B-A3B) 64k CompactionRL 56.0 ± 0.4 20.2 ± 0.9
GLM-4.5-Air-SFT (106B-A12B) 80k Base + inference compaction 59.8 ± 1.9 21.3 ± 1.6
GLM-4.5-Air-SFT (106B-A12B) 80k CompactionRL 66.4 ± 0.7 26.2 ± 2.1

CompactionRL improves the two benchmarks by 5.5 / 6.7 points on GLM-4.7-Flash and 6.6 / 4.9 points on GLM-4.5-Air-SFT compared with their inference-compaction baselines. Compacted evaluations allow up to three compactions while retaining the stated peak context length.

Additional controlled comparisons from Table 2
Backbone Method Training compaction Inference compaction SWE-bench Verified (200 tasks) Terminal-Bench 2.0
GLM-4.7-Flash Base — No 47.3 ± 2.7 14.6 ± 1.6
GLM-4.7-Flash RL No No 52.5 ± 2.5 16.9 ± 2.4
GLM-4.7-Flash RL + inference compaction No Yes 48.2 ± 0.5 12.4 ± 0.9
GLM-4.7-Flash CompactionRL without summary loss Yes Yes 54.5 ± 1.2 12.4 ± 0.0
GLM-4.5-Air-SFT Base — No 57.8 ± 2.1 18.0 ± 2.8
GLM-4.5-Air-SFT RL No No 58.3 ± 0.6 20.2 ± 3.7
GLM-4.5-Air-SFT RL + inference compaction No Yes 62.5 ± 1.1 23.6 ± 0.9
GLM-4.5-Air-SFT CompactionRL without summary loss Yes Yes 64.5 ± 2.4 21.7 ± 2.8

These comparisons use the same model-specific peak context lengths as the main table. The two RL rows for each backbone use the same checkpoint. The no-summary-loss variant retains compaction during training and inference.

SWE-bench uses the fixed 200-instance subset from Appendix A.1, and Terminal-Bench uses the full 2.0 set.

Repository Layout

CompactionRL/
├── slime/                  # slime training framework with CompactionRL
│   ├── slime/              # PPO, GAE, model backends and buffer conversion
│   └── compactionrl/       # Training configs, scripts, patch, docs and tests
├── terminus/               # Harbor / Terminus-KIRA agent framework
│   ├── bin/                # TiTO Linux executable
│   ├── prompts/            # Summary, resume and offline judge prompts
│   ├── examples/           # File and compaction smoke tasks
│   └── scripts/            # Task preparation, evaluation and analysis
├── buffer/                 # SQLite buffer and rollout collector
│   ├── src/compactionrl/   # Buffer schema, service and storage helpers
│   └── data/manifests/     # Task-ID manifests
├── scripts/                # Unified training wrappers and storage setup
├── assets/                 # Paper overview and method figures
└── SOURCE_MANIFEST.json    # Source revisions and overlay scope
Component Guide Purpose
slime/ Training guide PPO, critic, cross-segment GAE, token-normalized loss and training configurations
terminus/ Agent guide Harbor / Terminus-KIRA, compaction, Docker task execution, evaluation and TiTO
buffer/ Buffer guide Rollout collection, token metadata and training-batch delivery

Source revisions and framework changes are recorded in SOURCE_MANIFEST.json and the slime overlay.

Getting Started

Requirements

  • Linux x86_64, Python 3.12, and glibc ≥ 2.34 for the bundled TiTO executable.
  • A compatible NVIDIA training environment with PyTorch, CUDA, Transformer Engine, SGLang, Megatron-LM and mbridge. See the environment guide for dependency versions and compatibility requirements.
  • Docker with the Compose plugin for task execution.
  • Local model weights and prepared benchmark task environments.

Installation

Run from the repository root in an existing compatible GPU environment. The current launcher requires the checkout, output/cache directories and Docker data root on a mounted data filesystem; see the deployment guide.

# Choose a short runtime path on a mounted data volume.
export COMPACTIONRL_WORK_DIR=/data/cr
source scripts/storage_env.sh

python3.12 -m venv --system-site-packages .venv
source .venv/bin/activate
pip install -e 'buffer[test]'
pip install -e terminus
pip install --no-deps -e slime

# Install the pinned public Megatron-LM checkout used by the Flash recipe.
bash slime/compactionrl/scripts/install_megatron.sh
export MEGATRON_PATH="$PWD/.deps/Megatron-LM"
export PYTHONPATH="$PWD/buffer/src:$PWD/slime:$PWD/terminus/src:$MEGATRON_PATH"

Download the official GLM-4.7-Flash HF checkpoint to a local directory before running the examples. The experiments use GLM-4.7-Flash (30B-A3B) and GLM-4.5-Air-SFT (106B-A12B); see models and data.

Training

Quick start: Flash smoke

Use the Flash checkpoint with model_type=glm4_moe_lite. The smoke recipe targets one 8 × 80 GB GPU node: four GPUs shared by actor/critic and four for rollout. It uses an 8k context, two bundled tasks and five update batches to check integration. CPU offload requires host RAM; check both the host and container memory limits before running.

Inspect the environment and generated commands first:

export MODEL_PATH=/absolute/path/to/GLM-4.7-Flash
python slime/compactionrl/scripts/preflight.py --model "$MODEL_PATH"
bash scripts/smoke_train_glm47_flash.sh --model "$MODEL_PATH" --dry-run
bash scripts/smoke_train_glm47_flash.sh --model "$MODEL_PATH" --validate-only

--dry-run prints the service and training commands. --validate-only checks model/task assets and training arguments. Once these checks pass, launch:

bash scripts/smoke_train_glm47_flash.sh \
  --model "$MODEL_PATH" --steps 5 --output "$PWD/runs/glm47-flash-smoke"

The launcher starts the SGLang router, Harbor, TiTO and buffer, then the slime training driver. The output directory contains service logs, TensorBoard events, the buffer database and actor/critic checkpoints. Use a new output directory for each invocation.

To continue a saved run for one further update batch:

bash scripts/smoke_train_glm47_flash.sh \
  --model "$MODEL_PATH" --resume "$PWD/runs/glm47-flash-smoke" \
  --steps 6 --output "$PWD/runs/glm47-flash-resume"

--steps specifies the total target, including batches completed before resuming. See the launch guide for ports, initialization, checkpoint continuation and resource settings.

Paper configurations

Configuration Purpose
Flash smoke Five-batch integration check, 8k context and one 8-GPU node
Flash paper 30B, 64k context, PPO and critic pretraining
Air paper 106B Air-SFT, 80k context, PPO and critic pretraining
Paper ablations Summary loss, cross-segment GAE, loss normalization, SUPO and 160k context comparisons

For Flash, prepare the paper training task list and Harbor-format environments, and use a compatible Ray cluster with this checkout, model and task directories available at the same absolute paths on all nodes. Inspect the plan first:

bash scripts/train_glm47_flash.sh \
  --config slime/compactionrl/configs/glm47_flash/paper.yaml \
  --model "$MODEL_PATH" \
  --tasks /absolute/path/to/paper-training-task-ids.json \
  --dataset /absolute/path/to/prepared-swedev-tasks \
  --ray-address "$HEAD_IP:6379" --host "$HEAD_IP" --bind-host 0.0.0.0 \
  --steps 230 --output "$PWD/runs/glm47-flash-paper" --dry-run

Use --validate-only in place of --dry-run to check local assets and training arguments. Remove that flag to launch once environment checks pass. Configure cluster parallelism for the available hardware as described in the launch guide.

The training wrapper defaults to the Flash paper configuration. Pass --tasks explicitly: the current launcher does not expand the YAML's ${TRAIN_TASK_MANIFEST} placeholder. In this implementation, 230 batches = 50 critic-only batches + 180 PPO batches. Air's configuration uses 150 = 50 + 100. The two critic passes per batch also apply during warmup; see training semantics for the distinction between batches and optimizer updates.

Evaluation

Use the fixed SWE-bench Verified manifest with locally prepared Harbor tasks and Docker images:

python terminus/scripts/prepare_tasks.py \
  --dataset /absolute/path/to/swe-verified-harbor \
  --manifest terminus/data/manifests/swe_verified_200.json \
  --output "$COMPACTIONRL_WORK_DIR/swe200.json"

This command validates existing tasks; it does not download or convert benchmark content. For Terminal-Bench 2.0, omit --manifest to select the full prepared local dataset and record its version.

Configure the inference, Harbor and TiTO services for the checkpoint being evaluated, then run:

export EVAL_MODEL_PATH=/absolute/path/to/evaluation-hf-checkpoint
python terminus/scripts/evaluate.py \
  --config slime/compactionrl/configs/glm47_flash/paper.yaml \
  --tasks "$COMPACTIONRL_WORK_DIR/swe200.json" --model "$EVAL_MODEL_PATH" \
  --runs 3 --output "$PWD/runs/evaluation"

The evaluation script calls existing services; it does not start them. Use the Air configuration for its 80k evaluation setting and the checkpoint matching the desired result row.

Every run is retained and aggregate Pass@1 statistics are written to summary.json. Infrastructure failures remain in the denominator and are recorded separately. See the evaluation protocol for timeout settings and result aggregation.

Documentation and Checks

Guide Contents
Training Smoke and paper launch, services, storage, resume and resource requirements
Paper-to-code map Algorithms, prompts and token metadata
slime overlay Source revisions, framework changes and patch reconstruction
Buffer protocol Segment construction, masks and batch delivery
Reproduction Training settings, data and evaluation protocol
Paper ablations Configuration overlays and inference settings

After installation, run the project CPU checks:

pytest -c pytest.ini -q
python -B scripts/check_release.py

Acknowledgments and License

CompactionRL builds on slime and Harbor, with the Terminus-KIRA agent scaffold. We thank their contributors for the training and execution frameworks.

Project code is licensed under Apache-2.0. Retained upstream notices and the TiTO distribution scope are described in THIRD_PARTY_NOTICES.md. Model checkpoints, benchmark content and container images retain their respective licenses.

Citation

If you use CompactionRL in your research, please cite our paper (PDF):

@misc{li2026compactionrl,
  title         = {CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents},
  author        = {Yujiang Li and Zhenyu Hou and Yi Jing and Jie Tang and Yuxiao Dong},
  year          = {2026},
  eprint        = {2607.05378},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2607.05378}
}

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages