feat(changelog): Lambda notes reconciler — per-target notes-{target}.json indexes - #3926
feat(changelog): Lambda notes reconciler — per-target notes-{target}.json indexes#3926Mpdreamz wants to merge 4 commits into
Conversation
|
Label error. Requires exactly 1 of: automation, breaking, bug, changelog:skip, chore, ci, dependencies, documentation, enhancement, feature, fix, redesign. Found: |
There was a problem hiding this comment.
Requesting changes for two correctness issues in the notes index reconciler: branch paths containing / are dropped during note discovery, and obsolete notes-{target}.json indexes are never removed when targets disappear.
What is this? | From workflow: PR Review
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| } | ||
| } | ||
|
|
||
| if (byTarget.Count == 0) |
There was a problem hiding this comment.
This early return leaves stale notes-{target}.json objects behind when targets disappear.
ReconcileRepoAsync only writes indexes for currently discovered targets. If the last note for a target is deleted/retargeted (or no notes remain), byTarget no longer contains that target and this code exits without removing the old index key. Consumers can then read permanently stale note paths.
The reconcile step should also remove obsolete notes-*.json keys under the repo prefix that are not in the newly computed target set (including the byTarget.Count == 0 case).
There was a problem hiding this comment.
Fixed in d2eca18 — added ListExistingNotesIndexes that lists all notes-*.json keys under the repo prefix, and DeleteStaleIndexes that removes any whose target is absent from the newly computed target set. The byTarget.Count == 0 early-return is replaced with a cleanup call so all existing indexes are removed when no notes remain. Two new tests: ReconcileRepo_StaleTargetRemoved_OldIndexDeleted and ReconcileRepo_NoNotes_DeletesAllExistingIndexes.
| { | ||
| var relativePath = obj.Key[notesScope.Prefix.Length..]; | ||
| // Accept only pool-relative paths: {branch}/note-{name}.yml (no further nesting) | ||
| var slash = relativePath.IndexOf('/'); |
There was a problem hiding this comment.
Branches with / in the name are currently filtered out, so their note files never make it into any notes-{target}.json index.
relativePath.IndexOf('/') splits at the first slash, then fileName becomes the remainder. For a valid branch path like feature/foo/note-a.yml, this yields foo/note-a.yml, and IsNoteFileName rejects it because it contains /.
That conflicts with the existing key model where branch names can contain / (multi-segment branch parts). This will cause silently missing notes for those branches.
Use the last slash to isolate the basename instead:
| var slash = relativePath.IndexOf('/'); | |
| var slash = relativePath.LastIndexOf('/'); | |
| if (slash <= 0) | |
| continue; | |
| var fileName = relativePath[(slash + 1)..]; |
There was a problem hiding this comment.
Fixed in d2eca18 — changed to LastIndexOf('/') so branch names containing / (e.g. feature/my-fix) are correctly split: only the last segment is checked against IsNoteFileName, and the full pool-relative path (including the branch prefix) is preserved. New test: ReconcileRepo_BranchWithSlashInName_IsIncludedInIndex.
…json indexes
Adds `NotesIndexReconciler` which lists `changelog/{org}/{repo}/` in the public
bucket, reads each `note-*.yml` to extract its `target:` values, and writes one
`notes-{target}.json` index per target. A note appearing at multiple targets
yields multiple index entries; a note on two branches yields two pool-relative
paths in the same index.
`ScrubberProcessor` now rejects client uploads of `notes-*.json` (reconciler-
owned) and triggers a notes-index reconcile whenever a `note-*.yml` event arrives.
`ChangelogScope.TryCreateNotes` and `ChangelogScopeKind.Notes` model the repo-level
scope. `NotesIndexJsonContext` provides AOT-compatible JSON serialization.
`NotesIndexReconciler` accepts an optional `sourceBucketName` so it reads note
YAML from the private bucket in test setups where the fake scrubber corrupts the
public copy; in production it falls back to the public bucket where the real
scrubber preserves `target:` and `products:` intact.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…s and stale index cleanup
- Use LastIndexOf('/') when splitting branch from filename so branch names
containing '/' (e.g. feature/my-fix) are not silently dropped
- After writing current indexes, list existing notes-*.json keys under the
repo prefix and delete any whose target is no longer in the computed set
- Handles the byTarget.Count == 0 case: all existing indexes are removed
- Four new tests cover branch-with-slash, stale-target removal, and the
no-notes cleanup path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
7563e90 to
d2eca18
Compare
There was a problem hiding this comment.
Requesting changes: the notes-index reconcile path still has correctness issues around concurrent writes/deletes and stale-index deletion failure handling that can leave published indexes inconsistent.
What is this? | From workflow: PR Review
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| }, ctx); | ||
| _logger.LogInformation("Removed stale notes index {Key}", key); | ||
| } | ||
| catch (Exception ex) when (ex is not OperationCanceledException) |
There was a problem hiding this comment.
Delete failures are swallowed here, so reconcile can report success while leaving stale notes-*.json behind.
Because ScrubberProcessor only marks messages failed when ReconcileRepoAsync throws, this catch-and-continue path means transient S3 delete failures (timeouts/5xx/throttling) are not retried via redelivery, and stale indexes can persist indefinitely if no later note event happens.
Please treat stale-index deletion as reconcile-critical (bounded retry + propagate failure if still failing), so message redelivery can heal the state.
| ctx.ThrowIfCancellationRequested(); | ||
| try | ||
| { | ||
| _ = await s3Client.PutObjectAsync(new PutObjectRequest |
There was a problem hiding this comment.
WriteIndexAsync does unconditional writes, and stale-index cleanup later does unconditional deletes based on an earlier listing snapshot. That makes concurrent reconciles for the same repo non-convergent.
Concrete failure case:
- Reconcile A lists notes/targets from older state.
- Reconcile B lists newer state, writes correct
notes-{target}.jsonset, and deletes stale ones. - Reconcile A then reaches this unconditional
PutObjectAsyncand laterDeleteObjectAsync, reintroducing stale indexes or deleting freshly-correct ones.
BundleRegistryReconciler and ShallowRegistryReconciler already use ETag-guarded conditional writes/deletes (IfMatch/IfNoneMatch) with retry-on-conflict to avoid this race. The notes reconciler needs the same optimistic concurrency strategy.
Summary
NotesIndexReconcilerthat listschangelog/{org}/{repo}/in the public bucket, reads eachnote-*.ymlto extracttarget:values, and writes onenotes-{target}.jsonindex per target (pool-relative paths, e.g.main/note-slow-rollover.yml)ChangelogScopeKind.NotesandChangelogScope.TryCreateNotesfor the repo-level scope (two-segmentchangelog/{org}/{repo}/)NotesIndexrecord with AOT-compatibleNotesIndexJsonContextScrubberProcessorrejects client uploads ofnotes-*.json(reconciler-owned) and triggers a notes-index reconcile onnote-*.ymleventsProgram.cswires upNotesIndexReconcilerwith defaults (reads from public bucket where real scrubber preservestarget:/products:); tests passsourceBucketName: PrivateBucketsince the fake scrubber would corrupt YAMLTest plan
dotnet test tests/Elastic.Changelog.Tests/— 952 passingNotesIndexReconcilerTests: single note, two targets, same name on two branches, no notes, note with no products, sorted pathsScrubberProcessorTests: client-uploaded notes index rejected with no public write; note file scrubbed and notes reconcile triggereddotnet build -c Release— 0 errorsPart of the two-anchor plan (#3923 step 5). Stacks on
fix/changelog-note-command.🤖 Generated with Claude Code