Commit ecf954f
fix(brain): unwedge suggestion approval and refresh knowledge counts live (#77)
Follow-up to #74. That PR made the failure *visible* (bulk decide now
returns `failures[]`, and the UI renders a partial/zero success as an
error) but the approval itself still threw — its diagnosis, a
`schema_documentation_source_check` missing `CODE_DERIVED`, did not
apply to the reporting install, where the CHECK was already correct and
the log said something else entirely.
## What was actually failing
```
WARN CodeScanService : bulk decide skipped e2011614-…: Query did not return a unique result: 2 results were returned
WARN CodeScanService : bulk decide skipped f3689e0e-…: Query did not return a unique result: 2 results were returned
```
`schema_documentation` carried no unique constraint on `(connection_id,
object_type, object_name, parent_object, source)` and
`CodeSuggestionApplier.approve` took no row lock, so **one bulk approve
submitted twice concurrently wrote 219 duplicate pairs** (pairs ~3s
apart; in every group the older row is an orphan and the newer one holds
the `applied_doc_id`). Every later approve landing on such a key threw
out of the `Optional`-returning upsert finder, and bulk-decide swallowed
it per item. The duplicate never self-heals, so **all 198 pending
SCHEMA_DOC suggestions were permanently unapprovable**.
## Fix
**1. Duplicate-tolerant upsert.** The three finders now return `List`,
so the compiler forces every caller to handle N matches.
`SchemaDocumentationDeduplicator` keeps the newest row, repoints any
`applied_doc_id` off the rows it deletes (a loose reference, not an FK —
a dangling value fails silently), and drops their RAG embeddings.
Applied at all four call sites, including `SchemaDriftListener`, which
would have thrown identically the first time one of 17 duplicated tables
was dropped.
**2. Data repair + constraint.**
`V116__dedupe_schema_documentation.sql`, applied by
`SchemaDocumentationDedupeInitializer` — there is no Flyway runtime
here, so a SQL file alone would never run. Idempotent: it returns before
touching a row once the index exists. `coalesce(parent_object,'')` in
the key because Postgres treats NULLs as distinct.
**3. The root cause.** `approve`/`reject` load the suggestion `FOR
UPDATE`, so the concurrent double-submit blocks and the second caller
sees `APPROVED`.
## Also fixed (reported after the first fix landed)
- **Counts stale until reload.** An approval also writes
`schema_documentation`, served by `brain/notes`, which backs the
Write-notes tab and its coverage counts. The decide hooks invalidated
only `codeScan` + `companyKnowledge`. `invalidateAfterDecision` now
covers `brain` and `schemaContext` too.
- **Newest notes sorted last.** `@PreUpdate` never fires on insert, so a
new note has a null `updatedAt`; sorting on it alone with nulls last
sent every brand-new note to the *bottom*. Now `COALESCE(updatedAt,
createdAt)`, matching `CompanyKnowledgeEntryRepository`.
- **Fresh approvals buried in the Approved view.** `listSuggestions`
sorted every status by confidence. PENDING stays confidence-first (it is
a work queue); decided statuses now sort by `decidedAt DESC NULLS LAST`.
## Two bugs in #74's own test tooling
- Both scripts hardcoded `sudo -u postgres psql`, which does not exist
on the Compose deployment `install.sh` produces — the verify command in
#74's description failed before testing anything. Now resolved through
`scripts/self-host/vaultdb.py`.
- **`e2e-review-approvals.py` step 10 rewrote every real `CODE_DERIVED`
row to `source='USER'` and never restored it.** Running it against a
live install silently relabelled 339 approved docs, and post-V116 it
would collide with the unique index. It now parks rows in a scratch
table and restores them with a verified count, and its cleanup deletes
the planted row only while nothing references it.
## Verification
Run against the live self-host stack, not just unit tests.
- Initializer on the affected install: `removed 219 duplicate rows, 219
orphaned embeddings`; 7068 → 6849 doc rows, 8190 → 7971 RAG rows, **0
duplicate groups, 0 dangling `applied_doc_id`**. Skipped cleanly on
restart.
- The two originally stuck suggestions: `{"requested": 2, "succeeded":
2, "failed": 0}`.
- Ordering, approving the three *lowest*-confidence items so the two
sorts disagree — all three land above older `conf=0.97` rows.
- `brain/notes` 4607 → 4608 on a first-time doc (the count that never
refreshed), and the just-approved `crm.customers` now sorts first.
- 21 backend unit tests green (4 new on collapse, 2 new on note
ordering). 184 related tests run; the 6 failures in
`TrainingServiceBusinessTermTest` / `BrainInitStageExecutorTest`
reproduce identically on pristine `main` and are unrelated.
- **E2E suite: 30/30 pass**, including duplicate collapse,
`applied_doc_id` repointing, the unique index rejecting a second row,
and two genuinely concurrent approves writing exactly one row.
```bash
python3 scripts/self-host/seed-review-suggestions.py <connectionId> --count 20
python3 scripts/self-host/e2e-review-approvals.py <connectionId>
# ✓ All review-approval edge cases passed
```
Note: the suite consumes its own fixtures, so re-run the seed before
each run.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_018mG2xj9gWJ8WzfDP2fDePP
Co-authored-by: Krishna Sasank Talasila <606482+geekypunk@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Venkat SF <venkatesh.sakamuri@stayflexi.com>1 parent f28cc95 commit ecf954f
20 files changed
Lines changed: 1028 additions & 83 deletions
File tree
- backend/src
- main
- java/com/dbaagent
- config
- repository
- service
- brain/core
- codescan
- resources/db/migration
- test/java/com/dbaagent/service
- brain/core
- codescan
- scripts/self-host
- src/lib/hooks/queries
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
296 | 296 | | |
297 | 297 | | |
298 | 298 | | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
299 | 309 | | |
300 | 310 | | |
301 | 311 | | |
| |||
384 | 394 | | |
385 | 395 | | |
386 | 396 | | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
387 | 444 | | |
388 | 445 | | |
389 | 446 | | |
| |||
Lines changed: 116 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
Lines changed: 30 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
7 | 12 | | |
8 | 13 | | |
| 14 | + | |
9 | 15 | | |
| 16 | + | |
10 | 17 | | |
11 | 18 | | |
12 | 19 | | |
13 | 20 | | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
14 | 44 | | |
15 | 45 | | |
16 | 46 | | |
| |||
Lines changed: 13 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
25 | 33 | | |
26 | 34 | | |
27 | 35 | | |
| |||
51 | 59 | | |
52 | 60 | | |
53 | 61 | | |
54 | | - | |
55 | | - | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
56 | 65 | | |
57 | 66 | | |
58 | 67 | | |
59 | | - | |
| 68 | + | |
60 | 69 | | |
61 | 70 | | |
62 | 71 | | |
| |||
Lines changed: 15 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
37 | 38 | | |
38 | 39 | | |
39 | 40 | | |
| |||
71 | 72 | | |
72 | 73 | | |
73 | 74 | | |
| 75 | + | |
74 | 76 | | |
75 | 77 | | |
76 | 78 | | |
| |||
80 | 82 | | |
81 | 83 | | |
82 | 84 | | |
| 85 | + | |
83 | 86 | | |
84 | 87 | | |
85 | 88 | | |
| |||
417 | 420 | | |
418 | 421 | | |
419 | 422 | | |
420 | | - | |
421 | | - | |
| 423 | + | |
| 424 | + | |
422 | 425 | | |
423 | | - | |
| 426 | + | |
| 427 | + | |
424 | 428 | | |
425 | | - | |
426 | | - | |
| 429 | + | |
| 430 | + | |
427 | 431 | | |
428 | 432 | | |
429 | 433 | | |
| |||
445 | 449 | | |
446 | 450 | | |
447 | 451 | | |
448 | | - | |
449 | | - | |
| 452 | + | |
| 453 | + | |
450 | 454 | | |
451 | | - | |
| 455 | + | |
| 456 | + | |
452 | 457 | | |
453 | | - | |
454 | | - | |
| 458 | + | |
| 459 | + | |
455 | 460 | | |
456 | 461 | | |
457 | 462 | | |
| |||
0 commit comments