Symptom
If a Cursor transcript JSONL is rewritten in place by atomic rename (new file written, then renamed over the old path), every record already ingested is admitted again as a new message. Re-reading the same file stays idempotent; the duplication happens only when the file is replaced.
Evidence
Scratch tests on the real ingest_cursor_transcript_event path, with synthetic id-less Cursor JSONL and an isolated profile. Session message counts:
| Scenario |
Before rewrite |
After rewrite |
Expected |
| Same prefix, repeated identical lines, one line appended |
4 |
7 |
5 |
| Same prefix, distinct lines, one line appended |
5 |
11 |
6 |
| One existing line edited |
5 |
10 |
5 (or 5 plus a revision) |
The ingest refusal census stayed empty in all three, so nothing reports the duplication.
Controls: re-ingesting an unchanged file gives 0 new messages, and in-place appends ingest only the new line.
Expected
A rewrite that keeps existing records should be idempotent for those records, and admit only records that are new. The mechanism is not confirmed. The record identity itself is lossless (sha256 over session and raw JSON, plus a positional retry), so the likely cause is the replaced file being treated as a new source.
Found while investigating the observation_identity_collision refusals (#2682 follow-up); those turned out to be genuine duplicates of the same record in the top-level and subagent copies of a worker transcript.
Symptom
If a Cursor transcript JSONL is rewritten in place by atomic rename (new file written, then renamed over the old path), every record already ingested is admitted again as a new message. Re-reading the same file stays idempotent; the duplication happens only when the file is replaced.
Evidence
Scratch tests on the real
ingest_cursor_transcript_eventpath, with synthetic id-less Cursor JSONL and an isolated profile. Session message counts:The ingest refusal census stayed empty in all three, so nothing reports the duplication.
Controls: re-ingesting an unchanged file gives 0 new messages, and in-place appends ingest only the new line.
Expected
A rewrite that keeps existing records should be idempotent for those records, and admit only records that are new. The mechanism is not confirmed. The record identity itself is lossless (sha256 over session and raw JSON, plus a positional retry), so the likely cause is the replaced file being treated as a new source.
Found while investigating the
observation_identity_collisionrefusals (#2682 follow-up); those turned out to be genuine duplicates of the same record in the top-level and subagent copies of a worker transcript.