Repository navigation
fix: relay round 3, batch 2 (one-commit ingest, atomic backups, retry dedupe) - #46
Merged
Merged
Conversation
In _enqueue the EvidenceObject is committed with processing_state='pending', then a separate db.commit() creates the ProcessingJob. If the process dies or the second commit fails between the two commits, the evidence row exists forever in 'pending' with no ProcessingJob; the only retry path (/runtime/jobs/{id}/retry) requires a job, so the evidence is never processed.
Found by the GLM relay (memorygate__services__api_p0__fail__0), confirmed by GLM votes (round 2).
Regression test to add: Test that when auto_process is true, the evidence row and its ProcessingJob are committed atomically (e.g. patch the session so the single commit succeeds and verify both rows exist, or patch commit to fail and verify neither row persists).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
create_transcript commits the SessionTranscript, then commits the MemoryAudit in a second transaction. If the second commit fails (or the client times out and retries), the caller gets a 500 but the transcript row is already stored; a retry inserts a duplicate transcript since there is no idempotency key. Found by the GLM relay (memorygate__services__api_p0__fail__1), confirmed by GLM votes (round 2). Regression test to add: Test that calling POST /transcripts twice with the same agent_id and session_id returns the same transcript id without inserting a second row, and that a failure during the single commit rolls back both the SessionTranscript and its MemoryAudit entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
create_backup writes the JSON with a single write_bytes with no fsync; if the disk write or process fails partway, a truncated memorygate-*.json (whose filename already contains the full sha256 of the intended content) remains and list_backups/download_backup serve it as a restorable backup. Found by the GLM relay (memorygate__services__api_p0__fail__2), confirmed by GLM votes (round 2). Regression test to add: Simulate a crash mid-write (e.g., patch write_bytes/write to raise after partial bytes) and assert that list_backups() does not include any memorygate-*.json entry for the interrupted backup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An observation or entity is forgotten before any successful upsert (its Qdrant collection was never created because every earlier write failed); delete_observation_embedding/delete_entity_embedding call client.delete on a nonexistent collection, which raises, unlike delete_memory_embedding which guards with a get_collections check. The outbox row is retried with backoff forever and the forget receipt stays 'degraded'/'pending' permanently. Found by the GLM relay (memorygate__services__api_p1__fail__0), confirmed by GLM votes (round 2). Regression test to add: Test that delete_observation_embedding and delete_entity_embedding return without error when their Qdrant collections do not exist yet, mirroring the existing guard on delete_memory_embedding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
process_evidence commits after each stage. If a failure (e.g., vector/outbox or DB error) occurs at the memory or final stage, the episode and AnalysisObject rows from earlier stages are already committed, and the failed job is requeued (requeue/lease expiry). On retry, the analysis stage runs again and inserts a second AnalysisObject (and possibly a second local_ai_proposal) for the same evidence, with no dedup check like the episode and memory stages have. Found by the GLM relay (memorygate__services__api_p1__fail__2), confirmed by GLM votes (round 2). Regression test to add: A regression test should run process_evidence twice against the same evidence (simulating a retry after a failure at the memory/final stage) and assert only one deterministic_signal_extraction AnalysisObject and one local_ai_proposal AnalysisObject row exist for that evidence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reverted bd8bf5c: session_id defaults to "" and is not unique, so matching on (agent_id, session_id) would fold every session-less transcript into the first one, and scalar_one_or_none raises once two rows share a session. Kept with tidying: the analysis constructor sits inside its new if-block; os is imported at the top of backup_service. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The single-commit _enqueue built the ProcessingJob with evidence_id=row.id before the row was flushed, so the id was empty and /runtime/ingest returned 500 (10 tests). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Round 3 of the GLM bug-hunt relay, reviewed by Claude.
Kept:
pendingwith no job. Review fix: the row is flushed first so the job gets its id (the GLM version returned 500 on every ingest).Reverted in review:
Tests: services/api 406 passed; memory mutation drill 19/19 mutants caught. Ruff clean.
🤖 Generated with Claude Code