Skip to content

Work.md

Latest commit

 

History

History
35 lines (27 loc) · 4.15 KB

Work.md

File metadata and controls

35 lines (27 loc) · 4.15 KB
Raw

Work

Identify who's coding agent you are by the harness.

Toni - you are pi

  1. Now Eval sweep report is in docs/experiments/eval-sweep-20260706.md; artifacts are synced to s3://code-jepa/runs/eval-sweep-20260706/.

  2. Next Next useful run: reference/ground-truth candidate reranking + semantic hard-mining; do not claim classic clone/retrieval SOTA from current checkpoints.

Notes

  • Tokenized cache: 59,404,781 examples, 28 segments, 7,335 token shards, 7,392 S3 objects, 138.2GB.
  • RTX PRO 4000 Blackwell 4-GPU profiling: 24GB-safe table improved 4-GPU scaling from 62.7% to 78.0%; Discoverer should use the H200 preset first.
  • Default trainer path is no-predictor Siamese: shared encoder + projection head; predictor only if later conditioned/ablated.
  • Current active S3 data: s3://code-jepa/data/codesearchnet/, s3://code-jepa/tokenizers/codesearchnet/, s3://code-jepa/tokenized/codesearchnet/bpe16k-buckets-128-256-512-1024-2048/, and s3://code-jepa/tokenized/codesearchnet-raw/bpe16k-code-doc-buckets-128-256-512-1024-2048/.
  • Raw CodeSearchNet bpe16k cache finished and uploaded: 1,781,835 code/doc pairs, 220 shards, 0.345GB, 6,973 examples truncated over 2048 tokens.
  • Merged upstream MLM-aux Siamese trainer with per-device pmap RNG state; do not reintroduce host-broadcast RNG keys.
  • POJ-104 checkpoint movement: pretrained pretrain-jax-siamese-177664 zero-shot h MAP@R was 8.97% valid / 5.30% test; after 2-epoch POJ fine-tune it was 11.72% valid / 7.19% test.
  • Two-head data uploaded: semantic s3://code-jepa/tokenized/posttrain-heads/poj104-semantic-pairs-16pos-16neg-bpe16k-buckets-256-512-1024-2048/ (1,040,000 train / 272,000 valid / 384,000 test pairs) and lexical s3://code-jepa/tokenized/posttrain-heads/codesearchnet-lexical-pairs-transform-v0-core-6lang-2shards-per-bucket-bpe16k-buckets-128-256-512-1024-2048/ (1,427,211 train pairs).
  • Two-head trainer uses frozen pretrained encoder + separate 4H SwiGLU MLP semantic/lexical heads; no SIGReg in v1. Later semantic hard-mining: same-problem pairs far under current h/head, different-problem pairs close under current h/head.
  • RunPod 2x A40 two-head profile: fixed 512-token lexical workload, per-device batch 128, bf16, compile excluded: 1 GPU 1,688.43 examples/s, 2 GPUs 3,254.69 examples/s, speedup 1.93x, scaling efficiency 96.38%.
  • Discoverer SLURM: posttrain-two-heads-multigpu.sh defaults to 2 GPUs/H200 and large semantic+lexical caches; profile-two-heads-posttrain-multigpu.sh defaults to a short 1-vs-2 GPU 512-token lexical scaling run.

Edit-conditioned latent JEPA (Phase 4) — ready to launch, 2026-07-07

  • New objective --objective edit_conditioned in train_latent_jepa_multigpu.py: LeJEPA foundation (masked-span latent + SIGReg at token-h/pool-h/z) + edit-conditioned next-latent prediction ([PRED_POS]/[PRED_NEG]/[FAM_*] condition tokens + changed-span markers; symmetric InfoNCE against in-batch pools with the mutated sibling as hard negative). lambda_pos=0, no EMA (Project.md doctrine).
  • Edit-family labels need NO re-tokenization: shards already store per-example transform ids; global remap registry at assets/edit_families/edit_families.json and s3://code-jepa/tokenized/codesearchnet/bpe16k-buckets-128-256-512-1024-2048/edit_families.json (34 families, 28 segments).
  • Launcher: slurm/pretrain-edit-conditioned-multigpu.sh; warm-start with INIT_FROM=<runs/pretrain-lejepa-code-178580/latest.pkl>. Kill gates at 20-30k steps: erank(h)>35, pair_cos<0.5, target_var flat, sep_acc>0.7, lang_knn>=0.85, edit_acc rising.
  • Crash-recovery plan + status: PLAN-edit-conditioned-jepa.md.
  • REMINDER for Vasko: upload pretrain-lejepa-code-178580 + its eval dir (RUN=pretrain-lejepa-code-178580 scripts/sync_to_s3.sh --checkpoints); the committed eval script used the stale python-only tokenizer — rerun eval with TOKENIZER=.../tokenizers/codesearchnet/bpe16k before quoting the POJ table.

Vasko - you are codex/claude code

Do not use deleted codesearchnet-python paths. Use multilingual s3://code-jepa/data/codesearchnet/, tokenizer s3://code-jepa/tokenizers/codesearchnet/bpe16k/, and tokenized cache s3://code-jepa/tokenized/codesearchnet/bpe16k-buckets-128-256-512-1024-2048/.