diff --git a/docs/cli/changelog/cmd-add.md b/docs/cli/changelog/cmd-add.md index 52e05d74cd..390f8c9646 100644 --- a/docs/cli/changelog/cmd-add.md +++ b/docs/cli/changelog/cmd-add.md @@ -76,6 +76,10 @@ For example: - `"cloud-serverless 2025-08-05"` - `"cloud-enterprise 4.0.3, cloud-hosted 2025-10-31"` +:::{note} +Specifying a version in the `--products` spec (the middle slot, for example `"elasticsearch 9.3.0 ga"`) is an error for `changelog add`. Entries derive their release line from their origin branch, not from a contributor-supplied version. To create an item that explicitly targets one or more versions — such as a known issue or a CVE — use [`changelog note`](/cli/changelog/note.md) instead, which accepts `versions` in place of a target. +::: + The `changelog add` command resolves product values in the following order: 1. The `--products` CLI option always takes priority. diff --git a/docs/cli/changelog/cmd-bundle-amend.md b/docs/cli/changelog/cmd-bundle-amend.md index cf32554833..78371adc72 100644 --- a/docs/cli/changelog/cmd-bundle-amend.md +++ b/docs/cli/changelog/cmd-bundle-amend.md @@ -3,6 +3,10 @@ Amend a bundle with additional or excluded changelog entries without modifying the parent bundle file. Amend bundles follow a specific naming convention: `{parent-bundle-name}.amend-{N}` plus the same `.yaml` or `.yml` extension as the parent, where `{N}` is a sequence number. +:::{note} +The suffix `.amend-notes` (for example `9.3.0.amend-notes.yaml`) is reserved for use by the changelog scrubber Lambda. The Lambda generates and manages these files automatically; you must not create, edit, or delete them manually. +::: + Specify at least one of `--add` or `--remove`. To create a bundle, use [](/cli/changelog/bundle.md). diff --git a/docs/cli/changelog/cmd-note.md b/docs/cli/changelog/cmd-note.md new file mode 100644 index 0000000000..f2b68c981e --- /dev/null +++ b/docs/cli/changelog/cmd-note.md @@ -0,0 +1,74 @@ +## Description + +Create a changelog note file for an item that applies to one or more specific release versions and has no associated pull request. +Notes are used for known issues, security advisories, and other items that are not tied to a single PR. +For details and examples, go to [](/data/release-notes/create.md). + +Note files are named `note-{slug}.yml` and are uploaded to the changelog pool like any other entry. +Each note declares `products[].versions` — the release versions it applies to — instead of deriving its release line from a branch. + +## Options + +: `--products` + Products and versions in the format `"product versions lifecycle, ..."` where `versions` is a `|`-separated list of release versions (for example, `"elasticsearch 9.3.0|9.4.0 ga"`). + Unlike `changelog add`, the middle slot is interpreted as a `|`-separated version list, not a single target. + The valid product identifiers are listed in [products.yml](https://github.com/elastic/docs-builder/blob/main/config/products.yml). + +: `--title` + A short, user-facing headline for the note (max 80 characters). Required. + +: `--type` + The type of change. For valid values, see [ChangelogEntryType.cs](https://github.com/elastic/docs-builder/blob/main/src/Elastic.Documentation/ChangelogEntryType.cs). Required. + +: `--description` + Additional information about the note (max 600 characters). Optional. + +: `--issues` + URLs of related issues. Optional citation field; does not determine note addressability. + +## Product and version format + +The `--products` option uses the same positional format as `changelog add`, but the middle slot is a version list: + +- `"elasticsearch 9.3.0 ga"` — one version +- `"elasticsearch 9.3.0|9.4.0|9.5.0 ga"` — multiple versions +- `"cloud-serverless 2025-08-05"` — date-based release, one version + +A note that spans products can declare each product separately: + +```sh +docs-builder changelog note \ + --title "Known issue with aggregations" \ + --type known-issue \ + --products "elasticsearch 9.3.0|9.4.0 ga" \ + --products "kibana 9.3.0|9.4.0 ga" +``` + +## Output + +The command writes a `note-{slug}.yml` file to the configured output directory. +The file contains `products[].versions` instead of `products[].target`: + +```yaml +title: Known issue with aggregations +type: known-issue +products: + - product: elasticsearch + versions: [9.3.0, 9.4.0] + lifecycle: ga +``` + +## Lifecycle after creation + +Notes are uploaded to `changelog/{org}/{repo}/{branch}/note-*.yml` in the private S3 bucket and go through the scrubber exactly like entries. +A Lambda-maintained index at `changelog/{org}/{repo}/notes-{version}.json` lists every note that applies to a given version. + +If the release bundle for that version has already shipped when a note is uploaded, the scrubber Lambda automatically generates an amend sidecar (`{bundle}.amend-notes.yaml`) so the note reaches CDN consumers without a manual rerun. + +## Configuration checks + +The same configuration-file checks that apply to `changelog add` apply here: +valid `products`, `lifecycles`, and `type` values are validated against `docs/changelog.yml` when it exists. + +Specifying a version target in `--products` for `changelog add` is an error; use `changelog note` instead. +Conversely, `--versions` has no meaning for `changelog add` — it is note-specific. diff --git a/docs/data/release-notes/_snippets/changelog-fields.md b/docs/data/release-notes/_snippets/changelog-fields.md index ebe6f6a057..74d92345e5 100644 --- a/docs/data/release-notes/_snippets/changelog-fields.md +++ b/docs/data/release-notes/_snippets/changelog-fields.md @@ -20,12 +20,11 @@ products: # filters, and categorization. # Refer to https://github.com/elastic/docs-builder/blob/main/config/products.yml for the acceptable values. - target: + versions: - # An optional string that facilitates pre-release doc previews. - # For products with version releases, it contains the target version number (V.R.M). - # For products with date releases, it contains the target release date - # or the date the PR was merged. + # Note files only — a required list of release versions this note applies to. + # This field is mandatory only when the changelog is a note (i.e. doesn't have a PR). + # Example: [9.3.0, 9.4.0] lifecycle: diff --git a/docs/development/changelog-bundle-registry.md b/docs/development/changelog-bundle-registry.md index fa04e2ccf1..7b217a28d7 100644 --- a/docs/development/changelog-bundle-registry.md +++ b/docs/development/changelog-bundle-registry.md @@ -68,6 +68,16 @@ narrowed reconciliation to the bundle tree): exclusively by the scrubber Lambda's `BundleRegistryReconciler`. This is the manifest the `{changelog}` directive and external CDN consumers enumerate, and the subject of the rest of this page. +- **Amend-notes sidecars** — `bundle/{product}/{parent}.amend-notes.yaml`, also **public bucket + only**, authored by the scrubber Lambda's `NoteAmendReconciler`. When a note is uploaded after + its release bundle has already shipped, the reconciler generates one aggregate sidecar per + published bundle that lists all such late notes. The Lambda rebuilds it from current state on + every reconcile, so redelivered events never produce duplicate amends. The `.amend-notes` suffix + is **reserved** — do not create files with that suffix manually; see + [](/cli/changelog/bundle-amend.md). +- **Notes index** — `changelog/{org}/{repo}/notes-{version}.json`, one per version, **public + bucket only**, produced by the scrubber Lambda's `NotesIndexReconciler`. See + [Notes-index format](#notes-index-format) below. - **Changelog-entry index** — `changelog/{org}/{repo}/{branch}/registry.json`, a **legacy client-authored pass-through**: the current `changelog upload` never writes one, but manifests written by older CLI versions are still mirrored verbatim from the private bucket, because @@ -110,6 +120,35 @@ for a product that was declared under `release_notes` but never published — th remove the declaration), while a manifest with an empty `bundles` list would read as a valid zero-bundle state. The reconciler deliberately restores the former. +## Notes-index format [notes-index-format] + +For each release version that has at least one note, the scrubber Lambda writes a notes index at +`changelog/{org}/{repo}/notes-{version}.json`. Its schema (`schema_version: 1`): + +```json +{ + "schema_version": 1, + "notes": [ + { "path": "main/note-esql-oom.yml", "bundle_seq": 2 }, + { "path": "main/note-cve-2026-1234.yml", "bundle_seq": 1 } + ] +} +``` + +| Field | Meaning | +|---|---| +| `schema_version` | Schema version. Currently `1`. | +| `notes[].path` | Pool-relative path of the note within `changelog/{org}/{repo}/`. The leading segment before the first `/` is the branch. | +| `notes[].bundle_seq` | Derived reporting field: `0` = no bundle published for this version yet, `1` = note shipped in the original bundle, `2` = note carried by the Lambda-generated `.amend-notes` sidecar. | + +`bundle_seq` is derived — it is never authored and never a latch. The Lambda recomputes it on every +reconcile by comparing the notes index against the set of entries in the published bundle and its +amend sidecars. + +A 404 on a notes index means "no notes published for this version". An empty `notes` array never +appears — the index is deleted rather than emptied, following the same +[absent ≠ empty](#absent-empty) rule as the bundle registry. + ## Shallow per-tree change maps [shallow-maps] Alongside the per-group manifests, the scrubber maintains one **shallow map per tree**, at the diff --git a/src/Elastic.Documentation.Configuration/ReleaseNotes/BundleAmendMerger.cs b/src/Elastic.Documentation.Configuration/ReleaseNotes/BundleAmendMerger.cs index 29e381758c..f7c32ec25d 100644 --- a/src/Elastic.Documentation.Configuration/ReleaseNotes/BundleAmendMerger.cs +++ b/src/Elastic.Documentation.Configuration/ReleaseNotes/BundleAmendMerger.cs @@ -13,7 +13,8 @@ namespace Elastic.Documentation.Configuration.ReleaseNotes; /// public static partial class BundleAmendMerger { - [GeneratedRegex(@"\.amend-(\d+)(\.ya?ml)$", RegexOptions.IgnoreCase)] + // Matches both numbered amends (.amend-1.yaml) and the reconciler-owned notes sidecar (.amend-notes.yaml). + [GeneratedRegex(@"\.amend-(\d+|notes)(\.ya?ml)$", RegexOptions.IgnoreCase)] private static partial Regex AmendFileRegex(); /// Whether a path is an amend sidecar ({name}.amend-{N}.yaml). diff --git a/src/Elastic.Documentation.Configuration/ReleaseNotes/CdnChangelogEntryFetcher.cs b/src/Elastic.Documentation.Configuration/ReleaseNotes/CdnChangelogEntryFetcher.cs index 175b0f9cf3..99d52fb4b2 100644 --- a/src/Elastic.Documentation.Configuration/ReleaseNotes/CdnChangelogEntryFetcher.cs +++ b/src/Elastic.Documentation.Configuration/ReleaseNotes/CdnChangelogEntryFetcher.cs @@ -429,14 +429,14 @@ private static Uri CombineSegments(Uri baseUri, IReadOnlyList segments) /// /// /// Fetches all note-*.yml entries for / at - /// from the CDN. Reads the notes-{target}.json index to enumerate + /// from the CDN. Reads the notes-{version}.json index to enumerate /// the pool-relative note paths; a missing index means no notes (not an error). A listed note that /// cannot be fetched is a hard error — the index is an authoritative promise that the note exists. /// /// CDN base URI. /// Repository org (e.g. elastic). /// Repository name (e.g. kibana). - /// Target version string (e.g. 9.0.0). + /// Release version string (e.g. 9.0.0). /// Called once per hard error; caller decides how to surface it. /// Cancellation token. /// The fetched note entries, keyed by pool-relative path (main/note-foo.yml). @@ -444,7 +444,7 @@ public async Task> FetchNotesAsync( Uri baseUri, string org, string repo, - string target, + string version, Action emitError, Cancel ctx) { @@ -454,21 +454,21 @@ public async Task> FetchNotesAsync( return []; } - var indexUri = CombineSegments(baseUri, ["changelog", org, repo, $"notes-{target}.json"]); + var indexUri = CombineSegments(baseUri, ["changelog", org, repo, $"notes-{version}.json"]); NotesIndex? index; try { var (notFound, content) = await FetchTextOrNotFoundAsync(indexUri, 1, ctx).ConfigureAwait(false); if (notFound) { - _logger.LogDebug("Notes index for {Org}/{Repo}@{Target} not found at {Uri}; no notes to bundle", org, repo, target, indexUri); + _logger.LogDebug("Notes index for {Org}/{Repo}@{Version} not found at {Uri}; no notes to bundle", org, repo, version, indexUri); return []; } index = JsonSerializer.Deserialize(content, NotesIndexJsonContext.Default.NotesIndex); } catch (Exception ex) when (ex is not OperationCanceledException) { - emitError($"Could not fetch notes index for {org}/{repo}@{target} from {indexUri}: {ex.Message}"); + emitError($"Could not fetch notes index for {org}/{repo}@{version} from {indexUri}: {ex.Message}"); return []; } @@ -477,15 +477,16 @@ public async Task> FetchNotesAsync( var repoLabel = $"{org}/{repo}"; var entries = new List(index.Notes.Count); - foreach (var poolRelativePath in index.Notes) + foreach (var noteEntry in index.Notes) { ctx.ThrowIfCancellationRequested(); // Pool-relative path is "{branch}/note-{name}.yml"; split on first '/' only. + var poolRelativePath = noteEntry.Path; var slash = poolRelativePath.IndexOf('/', StringComparison.Ordinal); if (slash <= 0 || slash == poolRelativePath.Length - 1) { - emitError($"Notes index for {repoLabel}@{target} lists an invalid pool-relative path '{poolRelativePath}'; expected {{branch}}/{{file}}."); + emitError($"Notes index for {repoLabel}@{version} lists an invalid pool-relative path '{poolRelativePath}'; expected {{branch}}/{{file}}."); return []; } var branch = poolRelativePath[..slash]; @@ -493,7 +494,7 @@ public async Task> FetchNotesAsync( if (!ChangelogKeys.IsValidBranch(branch)) { - emitError($"Notes index for {repoLabel}@{target} lists path '{poolRelativePath}' with an invalid branch segment."); + emitError($"Notes index for {repoLabel}@{version} lists path '{poolRelativePath}' with an invalid branch segment."); return []; } @@ -510,12 +511,12 @@ public async Task> FetchNotesAsync( // The notes index asserts this note exists — a miss is a real pipeline error. emitError( - $"Note '{poolRelativePath}' for {repoLabel}@{target} is listed in the notes index but could not be fetched from {noteUri}: {lastError}. " + + $"Note '{poolRelativePath}' for {repoLabel}@{version} is listed in the notes index but could not be fetched from {noteUri}: {lastError}. " + "Ensure the note was uploaded and scrubbed; if it persists check the changelog scrubber pipeline."); return []; } - _logger.LogInformation("Fetched {Count} note(s) for {Repo}@{Target} from {BaseUri}", entries.Count, repoLabel, target, baseUri); + _logger.LogInformation("Fetched {Count} note(s) for {Repo}@{Version} from {BaseUri}", entries.Count, repoLabel, version, baseUri); return entries; } diff --git a/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogEntry.cs b/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogEntry.cs index f0358a50aa..8d6cb40f12 100644 --- a/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogEntry.cs +++ b/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogEntry.cs @@ -51,6 +51,26 @@ public record ChangelogEntryDto public record ProductInfoDto { public string? Product { get; set; } + + /// + /// Obsolete — entries derive applicability from their origin branch; notes use . + /// Still deserialized for backward compatibility with already-published pool objects. + /// + [Obsolete("Entries derive applicability from their origin branch; notes use Versions.")] public string? Target { get; set; } + + /// + /// The releases this note applies to (note-only field). For entries this is always null or empty. + /// Expressed in the YAML as a sequence: + /// + /// versions: [9.3.0, 9.4.0, 9.5.0] + /// + /// or as a pipe-separated string in the --products CLI flag: + /// + /// --products 'elasticsearch 9.3.0|9.4.0|9.5.0 ga' + /// + /// + public List? Versions { get; set; } + public string? Lifecycle { get; set; } } diff --git a/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogKeys.cs b/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogKeys.cs index 159d3fda46..45f7a7db89 100644 --- a/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogKeys.cs +++ b/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogKeys.cs @@ -99,11 +99,17 @@ public static string ChangelogRegistryKey(string poolGroup) => $"{ChangelogPrefix}{poolGroup}/{RegistryFileName}"; /// - /// The notes-index key for one target within a repo: changelog/{org}/{repo}/notes-{target}.json. - /// Repo-level and branch-agnostic — all notes for a target, regardless of which branch they were authored on. + /// The notes-index key for one release version within a repo: changelog/{org}/{repo}/notes-{version}.json. + /// Repo-level and branch-agnostic — all notes for a version, regardless of which branch they were authored on. /// - public static string NotesIndexKey(string org, string repo, string target) => - $"{ChangelogPrefix}{org}/{repo}/notes-{target}.json"; + /// + /// The slug in the key is the release version (e.g. 9.3.0 or 2026-05-15). + /// Previously this parameter was named target to match the obsolete target: YAML field; + /// it was renamed to version when that field was replaced by versions: on notes. + /// The key layout (notes-{slug}.json) is unchanged — no migration is needed. + /// + public static string NotesIndexKey(string org, string repo, string version) => + $"{ChangelogPrefix}{org}/{repo}/notes-{version}.json"; /// /// The S3 prefix that covers all branches and notes indexes of one repo: changelog/{org}/{repo}/. diff --git a/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogRegistry.cs b/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogRegistry.cs index d7f15e0189..83f15414d4 100644 --- a/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogRegistry.cs +++ b/src/Elastic.Documentation.Configuration/ReleaseNotes/ChangelogRegistry.cs @@ -49,4 +49,4 @@ public sealed record ChangelogRegistryBundle [JsonSerializable(typeof(ChangelogRegistry))] [JsonSerializable(typeof(ChangelogRegistryBundle))] [JsonSerializable(typeof(Dictionary))] -internal sealed partial class ChangelogRegistryJsonContext : JsonSerializerContext; +public sealed partial class ChangelogRegistryJsonContext : JsonSerializerContext; diff --git a/src/Elastic.Documentation.Configuration/ReleaseNotes/NotesIndex.cs b/src/Elastic.Documentation.Configuration/ReleaseNotes/NotesIndex.cs index 8eaac0bcd7..c8abbd9d56 100644 --- a/src/Elastic.Documentation.Configuration/ReleaseNotes/NotesIndex.cs +++ b/src/Elastic.Documentation.Configuration/ReleaseNotes/NotesIndex.cs @@ -7,19 +7,62 @@ namespace Elastic.Documentation.Configuration.ReleaseNotes; /// -/// Notes index published at changelog/{org}/{repo}/notes-{target}.json. -/// Lists pool-relative paths of all note-*.yml fragments for one target, +/// One entry in a — a pool-relative path to a note-*.yml +/// file and a derived sequence number that records how many published bundle files (original + +/// amends) already include this note. +/// +/// +/// +/// The origin branch is the leading segment(s) of before the last / +/// (e.g. path[..path.LastIndexOf('/')]) and is not stored separately to avoid a second +/// source of truth that can disagree with the path. +/// +/// +/// bundle_seq values: +/// +/// 0 — no bundle is published for this version yet; the note is unreleased. +/// 1 — the note shipped in the original bundle. +/// 2 — the note was picked up by the reconciler-owned {parent}.amend-notes.yaml. +/// +/// The field is derived and updated on every reconcile pass; it is never authored manually. +/// +/// +public sealed record NoteIndexEntry +{ + /// Pool-relative path, e.g. main/note-slow-rollover.yml. + public required string Path { get; init; } + + /// + /// How many published bundle files for this version already contain this note. + /// 0 = unreleased, 1 = in original bundle, 2 = in reconciler amend sidecar. + /// Derived on every reconcile; never authored. + /// + public int BundleSeq { get; init; } +} + +/// +/// Notes index published at changelog/{org}/{repo}/notes-{version}.json. +/// Lists all note-*.yml fragments for one release version, /// across every branch of the repo. /// /// /// Contents are paths, not bodies — the note files remain the single source of truth. -/// A stale index can only omit or over-list, never serve stale prose. Bundling a target +/// A stale index can only omit or over-list, never serve stale prose. Bundling a version /// is therefore 1 GET for the index + one GET per listed note. /// public sealed record NotesIndex { - /// Pool-relative paths of notes for this target, e.g. ["main/note-slow-rollover.yml"]. - public required IReadOnlyList Notes { get; init; } + /// Schema version — bumped when consumers must change their parser. + public int SchemaVersion { get; init; } = CurrentSchemaVersion; + + /// Current schema version constant. + public const int CurrentSchemaVersion = 1; + + /// + /// Notes for this version. Each entry carries the pool-relative path, origin branch, + /// and a derived bundle_seq. + /// + public required IReadOnlyList Notes { get; init; } } [JsonSourceGenerationOptions( @@ -28,4 +71,5 @@ public sealed record NotesIndex DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull )] [JsonSerializable(typeof(NotesIndex))] +[JsonSerializable(typeof(NoteIndexEntry))] public sealed partial class NotesIndexJsonContext : JsonSerializerContext; diff --git a/src/Elastic.Documentation.Configuration/ReleaseNotes/ReleaseNotesSerialization.cs b/src/Elastic.Documentation.Configuration/ReleaseNotes/ReleaseNotesSerialization.cs index 0aae497c7f..f97edeeac6 100644 --- a/src/Elastic.Documentation.Configuration/ReleaseNotes/ReleaseNotesSerialization.cs +++ b/src/Elastic.Documentation.Configuration/ReleaseNotes/ReleaseNotesSerialization.cs @@ -187,12 +187,25 @@ private static string ToYamlDoubleQuotedString(string s) Link = entry.Link }; - private static ProductReference ToProductReference(ProductInfoDto dto) => new() + private static ProductReference ToProductReference(ProductInfoDto dto) { - ProductId = dto.Product ?? "", - Target = dto.Target, - Lifecycle = ParseLifecycle(dto.Lifecycle) - }; + // Read the new `versions` list; fall back to wrapping the legacy `target` string + // for backward compat with already-published pool objects. +#pragma warning disable CS0618 // reading obsolete Target for backward compat + IReadOnlyList versions = + dto.Versions is { Count: > 0 } + ? dto.Versions + : !string.IsNullOrWhiteSpace(dto.Target) + ? [dto.Target] + : []; + return new() + { + ProductId = dto.Product ?? "", + Versions = versions, + Lifecycle = ParseLifecycle(dto.Lifecycle) + }; +#pragma warning restore CS0618 + } private static Bundle ToBundle(BundleDto dto) => new() { @@ -311,12 +324,18 @@ private static ChangelogEntryDto ToDto(ChangelogEntry entry) }; } - private static ProductInfoDto ToDto(ProductReference product) => new() - { - Product = product.ProductId, - Target = product.Target, - Lifecycle = LifecycleToString(product.Lifecycle) - }; + private static ProductInfoDto ToDto(ProductReference product) => + // Never write `target` — it is obsolete. Write `versions` when populated (notes only). + // `target` is still *read* from existing pool objects (see ToProductReference), but never written. +#pragma warning disable CS0618 // deliberately not forwarding Target + new() + { + Product = product.ProductId, + Versions = product.Versions.Count > 0 ? product.Versions.ToList() : null, + Lifecycle = LifecycleToString(product.Lifecycle) + }; +#pragma warning restore CS0618 + private static BundleDto ToDto(Bundle bundle) => new() { diff --git a/src/Elastic.Documentation/ReleaseNotes/ProductReference.cs b/src/Elastic.Documentation/ReleaseNotes/ProductReference.cs index 37ba02dfa1..552dc27707 100644 --- a/src/Elastic.Documentation/ReleaseNotes/ProductReference.cs +++ b/src/Elastic.Documentation/ReleaseNotes/ProductReference.cs @@ -12,9 +12,21 @@ public record ProductReference /// The product identifier. public required string ProductId { get; init; } - /// Optional target version. + /// + /// Obsolete — entries derive applicability from their origin branch; notes use . + /// Kept for backward compatibility when reading already-published pool objects that still carry target. + /// + [Obsolete("Entries derive applicability from their origin branch; notes use Versions.")] public string? Target { get; init; } + /// + /// The releases this note applies to (note-only). Empty for entries. + /// Populated from or, for backward compatibility, + /// from a single-element list derived from when Versions + /// is absent on an already-published note. + /// + public IReadOnlyList Versions { get; init; } = []; + /// The lifecycle stage of the feature for this product. public Lifecycle? Lifecycle { get; init; } } diff --git a/src/infra/docs-lambda-changelog-scrubber/Program.cs b/src/infra/docs-lambda-changelog-scrubber/Program.cs index ff7d7c3c35..396f3ec243 100644 --- a/src/infra/docs-lambda-changelog-scrubber/Program.cs +++ b/src/infra/docs-lambda-changelog-scrubber/Program.cs @@ -57,7 +57,8 @@ async Task Handler(SQSEvent ev, ILambdaContext context) var reconciler = new BundleRegistryReconciler(logFactory, s3Client, publicBucketName, metrics: metrics); var shallowReconciler = new ShallowRegistryReconciler(logFactory, s3Client, publicBucketName, metrics: metrics); var notesReconciler = new NotesIndexReconciler(logFactory, s3Client, publicBucketName, metrics: metrics); - var processor = new ScrubberProcessor(logFactory, s3Client, publicBucketName, scrubber, reconciler, shallowReconciler, notesReconciler, metrics); + var noteAmendReconciler = new NoteAmendReconciler(logFactory, s3Client, publicBucketName, notesReconciler, metrics: metrics); + var processor = new ScrubberProcessor(logFactory, s3Client, publicBucketName, scrubber, reconciler, shallowReconciler, notesReconciler, noteAmendReconciler, metrics); var messages = ev.Records.Select(r => new ScrubberQueueMessage(r.MessageId, r.Body)).ToList(); var failedIds = await processor.ProcessAsync(messages, CancellationToken.None); diff --git a/src/services/Elastic.Changelog/Backfill/ChangelogBackfillService.cs b/src/services/Elastic.Changelog/Backfill/ChangelogBackfillService.cs index e89b6cc0e9..aba79ec74c 100644 --- a/src/services/Elastic.Changelog/Backfill/ChangelogBackfillService.cs +++ b/src/services/Elastic.Changelog/Backfill/ChangelogBackfillService.cs @@ -461,7 +461,10 @@ private static string SerializeEntry(BundledEntry bundled, BackfillScope scope, Areas = bundled.Areas, Prs = bundled.Prs, Issues = bundled.Issues, - Products = [new ProductReference { ProductId = scope.ProductId, Target = version }] + // `Target` is obsolete; backfill notes carry their version in `Versions`. +#pragma warning disable CS0618 // deliberately not populating the obsolete Target + Products = [new ProductReference { ProductId = scope.ProductId, Versions = [version] }] +#pragma warning restore CS0618 }; return ReleaseNotesSerialization.SerializeEntry(entry); } diff --git a/src/services/Elastic.Changelog/Bundling/AmendDocumentBuilder.cs b/src/services/Elastic.Changelog/Bundling/AmendDocumentBuilder.cs new file mode 100644 index 0000000000..f7de7060b2 --- /dev/null +++ b/src/services/Elastic.Changelog/Bundling/AmendDocumentBuilder.cs @@ -0,0 +1,29 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +using Elastic.Documentation.ReleaseNotes; + +namespace Elastic.Changelog.Bundling; + +/// +/// Builds an amend document without any filesystem dependency, +/// so both the CLI amend service and the Lambda's NoteAmendReconciler can reuse it. +/// +public static class AmendDocumentBuilder +{ + /// + /// Builds an amend bundle that copies the parent's products (so registry routing and + /// :version: selection work), records the supplied exclusions, and adds the supplied entries. + /// + public static Bundle Build( + IReadOnlyList parentProducts, + IReadOnlyList entriesToAdd, + IReadOnlyList exclusions) => + new() + { + Products = parentProducts, + ExcludeEntries = exclusions, + Entries = entriesToAdd + }; +} diff --git a/src/services/Elastic.Changelog/Bundling/BundleBuilder.cs b/src/services/Elastic.Changelog/Bundling/BundleBuilder.cs index 7e4b4e03c8..f8e08c79a8 100644 --- a/src/services/Elastic.Changelog/Bundling/BundleBuilder.cs +++ b/src/services/Elastic.Changelog/Bundling/BundleBuilder.cs @@ -94,7 +94,11 @@ private static List BuildProducts( continue; foreach (var product in entry.Data.Products) { - var version = product.Target ?? string.Empty; + // `Target` is obsolete; prefer the first entry of `Versions` (notes). + // For PR-anchored entries neither field is set → version is "". +#pragma warning disable CS0618 // reading obsolete Target for backward compat + var version = (product.Versions.Count > 0 ? product.Versions[0] : null) ?? product.Target ?? string.Empty; +#pragma warning restore CS0618 _ = productVersions.Add((product.ProductId, version, product.Lifecycle)); } } diff --git a/src/services/Elastic.Changelog/Bundling/ChangelogBundleAmendService.cs b/src/services/Elastic.Changelog/Bundling/ChangelogBundleAmendService.cs index b4c2a9aecb..08f6e94497 100644 --- a/src/services/Elastic.Changelog/Bundling/ChangelogBundleAmendService.cs +++ b/src/services/Elastic.Changelog/Bundling/ChangelogBundleAmendService.cs @@ -266,12 +266,7 @@ public async Task AmendBundle(IDiagnosticsCollector collector, AmendBundle // Copy the parent's complete products (target, repo, owner) so the amend is self-contained: // upload destination discovery, the registry's per-product target, and :version:-filtered // CDN fetches all derive from a bundle file's own products. - var amendBundle = new Bundle - { - Products = parentBundle.Products, - ExcludeEntries = excludeEntries, - Entries = entries - }; + var amendBundle = AmendDocumentBuilder.Build(parentBundle.Products, entries, excludeEntries); var bundleForWrite = amendBundle; if (entries.Count > 0 && linkAllowRepos != null) diff --git a/src/services/Elastic.Changelog/Bundling/ChangelogBundlingService.cs b/src/services/Elastic.Changelog/Bundling/ChangelogBundlingService.cs index 9f474b0278..7490c7c007 100644 --- a/src/services/Elastic.Changelog/Bundling/ChangelogBundlingService.cs +++ b/src/services/Elastic.Changelog/Bundling/ChangelogBundlingService.cs @@ -1366,7 +1366,10 @@ private static bool ValidateProfileOutputs(IDiagnosticsCollector collector, Chan if (noteEntries == null) return null; - foreach (var note in noteEntries) + // Backport collision: same leaf from different branches at the same version → prefer main/master. + var deduped = DeduplicateNotesByLeaf(noteEntries, noteTarget); + + foreach (var note in deduped) { if (seen.Add(note.Checksum)) combined.Add(note); @@ -1375,6 +1378,72 @@ private static bool ValidateProfileOutputs(IDiagnosticsCollector collector, Chan return combined; } + /// + /// Within a single version's note list, groups by leaf file name and resolves collisions from + /// backported notes on multiple branches. The main/master copy wins; when neither + /// branch is present the ordinal-first path is kept so the choice is deterministic. A warning is + /// logged for each discarded copy. + /// + private IReadOnlyList DeduplicateNotesByLeaf( + IReadOnlyList notes, + string version) + { + var byLeaf = new Dictionary>(StringComparer.OrdinalIgnoreCase); + foreach (var note in notes) + { + var leaf = NoteLeafName(note.FileName); + if (!byLeaf.TryGetValue(leaf, out var group)) + { + group = []; + byLeaf[leaf] = group; + } + group.Add(note); + } + + var result = new List(notes.Count); + foreach (var (leaf, group) in byLeaf) + { + if (group.Count == 1) + { + result.Add(group[0]); + continue; + } + + // Prefer main/master; fall back to ordinal-first for a deterministic pick. + var winner = group.FirstOrDefault(n => IsMainOrMasterBranch(NoteBranchOf(n.FileName))) + ?? group.OrderBy(n => n.FileName, StringComparer.Ordinal).First(); + + result.Add(winner); + + foreach (var discarded in group.Where(n => !ReferenceEquals(n, winner))) + { + _logger.LogWarning( + "Backport collision for '{Leaf}' at version {Version}: keeping '{Winner}', discarding '{Discarded}'", + leaf, version, winner.FileName, discarded.FileName); + } + } + + return result; + } + + private static string NoteLeafName(string fileName) + { + var normalized = fileName.Replace('\\', '/'); + var slash = normalized.LastIndexOf('/'); + return slash >= 0 ? normalized[(slash + 1)..] : normalized; + } + + private static string NoteBranchOf(string fileName) + { + var normalized = fileName.Replace('\\', '/'); + var slash = normalized.IndexOf('/'); + return slash > 0 ? normalized[..slash] : string.Empty; + } + + private static bool IsMainOrMasterBranch(string branch) => + string.Equals(branch, "main", StringComparison.OrdinalIgnoreCase) + || string.Equals(branch, "master", StringComparison.OrdinalIgnoreCase); + private async Task?> FetchCdnProbedEntriesAsync( IDiagnosticsCollector collector, string? org, diff --git a/src/services/Elastic.Changelog/Bundling/ChangelogEntryMatcher.cs b/src/services/Elastic.Changelog/Bundling/ChangelogEntryMatcher.cs index 4c8871b73c..8134937ffd 100644 --- a/src/services/Elastic.Changelog/Bundling/ChangelogEntryMatcher.cs +++ b/src/services/Elastic.Changelog/Bundling/ChangelogEntryMatcher.cs @@ -259,7 +259,26 @@ private static bool MatchesProductFilter( foreach (var changelogProduct in data.Products) { var productMatches = MatchesPattern(changelogProduct.Product, filter.ProductPattern); - var targetMatches = MatchesPattern(changelogProduct.Target, filter.TargetPattern); + + // Target filtering: null or "*" pattern matches everything. + // For notes (Versions list), any version matching the pattern counts. + // For legacy entries that still carry Target (read-side compat), fall back to that. + bool targetMatches; + if (filter.TargetPattern is null or "*") + { + targetMatches = true; + } + else if (changelogProduct.Versions is { Count: > 0 }) + { + targetMatches = changelogProduct.Versions.Any(v => MatchesPattern(v, filter.TargetPattern)); + } + else + { +#pragma warning disable CS0618 // reading obsolete Target for backward compat with legacy entries + targetMatches = MatchesPattern(changelogProduct.Target, filter.TargetPattern); +#pragma warning restore CS0618 + } + var lifecycleMatches = MatchesPattern(changelogProduct.Lifecycle, filter.LifecyclePattern); if (productMatches && targetMatches && lifecycleMatches) diff --git a/src/services/Elastic.Changelog/Creation/ChangelogCreationService.cs b/src/services/Elastic.Changelog/Creation/ChangelogCreationService.cs index 7a7b737f37..28a883a855 100644 --- a/src/services/Elastic.Changelog/Creation/ChangelogCreationService.cs +++ b/src/services/Elastic.Changelog/Creation/ChangelogCreationService.cs @@ -359,6 +359,10 @@ private async Task CreateSingleChangelogAsync( if (!_validator.ValidateRequiredFields(collector, input, prFetchFailed)) return false; + // Entries must not carry version targets; applicability comes from the origin branch + if (!_validator.ValidateNoVersionTarget(collector, input)) + return false; + // Validate against configuration if (!_validator.ValidateAgainstConfiguration(collector, input, config)) return false; @@ -456,6 +460,10 @@ private async Task CreateSingleChangelogFromIssueAsync( if (!_validator.ValidateRequiredFields(collector, input, issueResult.FetchFailed, fromIssue: true)) return false; + // Entries must not carry version targets; applicability comes from the origin branch + if (!_validator.ValidateNoVersionTarget(collector, input)) + return false; + if (!_validator.ValidateAgainstConfiguration(collector, input, config)) return false; diff --git a/src/services/Elastic.Changelog/Creation/ChangelogFileWriter.cs b/src/services/Elastic.Changelog/Creation/ChangelogFileWriter.cs index 1470858452..d340558ee3 100644 --- a/src/services/Elastic.Changelog/Creation/ChangelogFileWriter.cs +++ b/src/services/Elastic.Changelog/Creation/ChangelogFileWriter.cs @@ -281,8 +281,10 @@ private static string GenerateYaml(ChangelogEntry data, ChangelogConfiguration c # A required string with a valid product ID. # Valid values are defined in https://github.com/elastic/docs-builder/blob/main/config/products.yml # - # target: - # An optional string with the target version or date. + # versions: + # Note-only. A list of release versions this note applies to. + # Example: [9.3.0, 9.4.0] or [2026-05-15] + # For PR-anchored entries, leave this absent — the branch is the address. # # lifecycle: # An optional string for new features or enhancements that have a specific availability. diff --git a/src/services/Elastic.Changelog/Creation/CreateChangelogArgumentsValidator.cs b/src/services/Elastic.Changelog/Creation/CreateChangelogArgumentsValidator.cs index d0e9646916..76a2e3651c 100644 --- a/src/services/Elastic.Changelog/Creation/CreateChangelogArgumentsValidator.cs +++ b/src/services/Elastic.Changelog/Creation/CreateChangelogArgumentsValidator.cs @@ -121,15 +121,44 @@ public bool ValidateRequiredFields( return true; } + /// + /// Validates that every product in a changelog note has at least one concrete version + /// in its list. The target field is obsolete and + /// must not be used; the CLI now parses the middle positional slot as a pipe-separated version + /// list into . + /// public bool ValidateNoteProducts(IDiagnosticsCollector collector, CreateChangelogArguments input) { foreach (var product in input.Products) { - if (string.IsNullOrWhiteSpace(product.Target) || product.Target == "*") + if (product.Versions.Count == 0) + { + collector.EmitError(string.Empty, + $"Product '{product.Product}' must have at least one specific version for 'changelog note'. " + + "Use --products 'product version[|version2...] [lifecycle]' with a concrete version " + + "(for example, 'elasticsearch 9.3.0|9.4.0 ga' or 'cloud-serverless 2026-05-15')."); + return false; + } + } + return true; + } + + /// + /// Validates that no product in a changelog add entry carries a specific version target. + /// The target field is obsolete for entries; applicability is expressed through the origin + /// branch. Wildcards (*) and absent values are still accepted for bundle filter profiles. + /// + public bool ValidateNoVersionTarget(IDiagnosticsCollector collector, CreateChangelogArguments input) + { + foreach (var product in input.Products) + { + if (product.Versions.Count > 0) { collector.EmitError(string.Empty, - $"Product '{product.Product}' must have a specific target for 'changelog note'. " + - "Use --products 'product target lifecycle' with a concrete target value (for example, '9.2.0' or '2026-05-15')."); + $"Product '{product.Product}' specifies version(s) '{string.Join("|", product.Versions)}', " + + "but changelog entries do not carry version applicability — the origin branch is the address. " + + "For PR-less items that apply to specific release versions (known issues, security advisories) " + + "use 'changelog note' instead."); return false; } } diff --git a/src/services/Elastic.Changelog/GithubRelease/GitHubReleaseChangelogService.cs b/src/services/Elastic.Changelog/GithubRelease/GitHubReleaseChangelogService.cs index 84e3ec382a..7b3e331932 100644 --- a/src/services/Elastic.Changelog/GithubRelease/GitHubReleaseChangelogService.cs +++ b/src/services/Elastic.Changelog/GithubRelease/GitHubReleaseChangelogService.cs @@ -364,7 +364,12 @@ private async Task ProcessPrReference( Products = [new ProductReference { ProductId = context.ProductInfo.Product ?? "", - Target = context.ProductInfo.Target, + // `Target` is obsolete; carry forward via `Versions` for compat with existing pool objects. +#pragma warning disable CS0618 // reading obsolete Target for backward compat + Versions = context.ProductInfo.Versions is { Count: > 0 } + ? context.ProductInfo.Versions + : context.ProductInfo.Target is not null ? [context.ProductInfo.Target] : [], +#pragma warning restore CS0618 Lifecycle = !string.IsNullOrWhiteSpace(context.ProductInfo.Lifecycle) ? (LifecycleExtensions.TryParse(context.ProductInfo.Lifecycle, out var lc, ignoreCase: true, allowMatchingMetadataAttribute: true) ? lc : null) : null diff --git a/src/services/Elastic.Changelog/ProductArgument.cs b/src/services/Elastic.Changelog/ProductArgument.cs index c1ac97cf9f..2cd176755b 100644 --- a/src/services/Elastic.Changelog/ProductArgument.cs +++ b/src/services/Elastic.Changelog/ProductArgument.cs @@ -16,21 +16,39 @@ public record ProductArgument /// Product ID or wildcard pattern. public string? Product { get; init; } - /// Target version or wildcard pattern. + /// + /// Raw middle slot of the CLI positional spec. For bundle-filter commands this is a wildcard + /// (*) or a single version value. For changelog note it may be a pipe-separated + /// list of versions (9.3.0|9.4.0|9.5.0); use for the parsed form. + /// public string? Target { get; init; } + /// + /// Parsed version list derived from by splitting on |. + /// Empty when is null, empty, or the wildcard *. + /// Used by changelog note validation and . + /// + public IReadOnlyList Versions => + string.IsNullOrWhiteSpace(Target) || Target == "*" + ? [] + : Target.Split('|', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries) + .Where(v => v.Length > 0 && v != "*") + .ToList(); + /// Lifecycle string or wildcard pattern. public string? Lifecycle { get; init; } /// /// Converts this ProductArgument to a ProductReference domain type. + /// The middle CLI slot () is parsed via into the + /// domain's list; + /// is never populated — the entry/note schema no longer writes it. /// - public ProductReference ToProductReference() => new() - { - ProductId = Product ?? "", - Target = Target, - Lifecycle = ParseLifecycle(Lifecycle) - }; + public ProductReference ToProductReference() => +#pragma warning disable CS0618 // Target is intentionally not forwarded — entries write Versions + new() { ProductId = Product ?? "", Versions = Versions, Lifecycle = ParseLifecycle(Lifecycle) }; +#pragma warning restore CS0618 + /// /// Converts this ProductArgument to a BundledProduct domain type. diff --git a/src/services/Elastic.Changelog/Reconciliation/NoteAmendReconciler.cs b/src/services/Elastic.Changelog/Reconciliation/NoteAmendReconciler.cs new file mode 100644 index 0000000000..f812fa2939 --- /dev/null +++ b/src/services/Elastic.Changelog/Reconciliation/NoteAmendReconciler.cs @@ -0,0 +1,503 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +using System.Net; +using System.Text.Json; +using Amazon.S3; +using Amazon.S3.Model; +using Elastic.Changelog.Bundling; +using Elastic.Documentation.Configuration.ReleaseNotes; +using Elastic.Documentation.ReleaseNotes; +using Microsoft.Extensions.Logging; + +namespace Elastic.Changelog.Reconciliation; + +/// +/// Compares each note in the per-version notes indexes against published bundles for the same +/// version, and either creates or deletes a reconciler-owned amend sidecar +/// ({parent}.amend-notes.yaml) that carries notes that arrived after the release shipped. +/// Also updates each bundle_seq in the notes index: 0 = no bundle yet, 1 = shipped in the +/// original bundle or a human amend, 2 = carried by the reconciler amend sidecar. +/// +/// +/// +/// This reconciler is idempotent: the sidecar is rebuilt from current state on every pass, +/// so a redelivered or out-of-order S3 event cannot produce a duplicate amend. +/// +/// +/// Matching notes against bundle entries uses the leaf file name (case-insensitive), not +/// the checksum. The checksum is unreliable for identity because the scrubber re-serializes content +/// when it strips private references, so a public pool object's hash differs from the one a +/// locally-bundled entry recorded. A missing file: block on an entry means the shipped +/// status is unknown (hand-authored bundles); those versions are skipped to avoid false positives. +/// +/// +public sealed class NoteAmendReconciler( + ILoggerFactory logFactory, + IAmazonS3 s3Client, + string publicBucketName, + NotesIndexReconciler notesIndexReconciler, + TimeSpan? retryBaseDelay = null, + ReconcileMetrics? metrics = null +) +{ + private const int MaxParallelWrites = 4; + + private readonly ILogger _logger = logFactory.CreateLogger(); + private readonly ReconcileMetrics _metrics = metrics ?? new ReconcileMetrics(); + private readonly TimeSpan _retryBaseDelay = retryBaseDelay ?? TimeSpan.FromMilliseconds(200); + + /// + /// For the given repository scope, scans every product's bundle registry to determine which + /// notes have shipped and which are late, writes or deletes the reconciler-owned amend sidecars, + /// and re-writes the notes indexes with correct bundle_seq values. + /// + /// The notes scope for this repo. + /// Output of . + /// Cancellation token. + public async Task ReconcileAsync( + ChangelogScope notesScope, + IReadOnlyDictionary> notesByVersion, + Cancel ctx) + { + if (notesByVersion.Count == 0) + return; + + var groupParts = notesScope.Group.Split('/'); + var (org, repo) = (groupParts[0], groupParts[1]); + + // Track bundle_seq for each (version → path → seq). Default 0 = unreleased. + var seqMap = new Dictionary>(StringComparer.Ordinal); + foreach (var (version, notes) in notesByVersion) + seqMap[version] = notes.ToDictionary(n => n.Path, _ => 0, StringComparer.Ordinal); + + // List all product names from the bundle tree. + var products = await ListBundleProductsAsync(ctx); + _logger.LogDebug("NoteAmendReconciler: scanning {Count} bundle product(s) for repo {Org}/{Repo}", products.Count, org, repo); + + foreach (var product in products) + { + ctx.ThrowIfCancellationRequested(); + await ProcessProductAsync(org, repo, product, notesByVersion, seqMap, ctx); + } + + // Re-write notes indexes with the updated bundle_seq values. + await Parallel.ForEachAsync( + notesByVersion, + new ParallelOptions { MaxDegreeOfParallelism = MaxParallelWrites, CancellationToken = ctx }, + async (kvp, ct) => + { + var (version, notes) = kvp; + var seqs = seqMap[version]; + var updatedEntries = notes + .Select(n => n with { BundleSeq = seqs.TryGetValue(n.Path, out var s) ? s : 0 }) + .OrderBy(n => n.Path, StringComparer.Ordinal) + .ToList(); + var indexKey = ChangelogKeys.NotesIndexKey(org, repo, version); + await notesIndexReconciler.WriteIndexAsync(indexKey, updatedEntries, ct); + }); + } + + // ----------------------------------------------------------------------------------------- + // Product scanning + // ----------------------------------------------------------------------------------------- + + private async Task> ListBundleProductsAsync(Cancel ctx) + { + var products = new List(); + var request = new ListObjectsV2Request + { + BucketName = publicBucketName, + Prefix = ChangelogKeys.BundlePrefix, + Delimiter = "/" + }; + + ListObjectsV2Response response; + do + { + response = await s3Client.ListObjectsV2Async(request, ctx); + foreach (var prefix in response.CommonPrefixes ?? []) + { + // CommonPrefix is like "bundle/elasticsearch/" — strip the outer segments. + var inner = prefix[ChangelogKeys.BundlePrefix.Length..]; + var product = inner.TrimEnd('/'); + if (!string.IsNullOrEmpty(product)) + products.Add(product); + } + request.ContinuationToken = response.NextContinuationToken; + } while (response.IsTruncated == true); + + return products; + } + + private async Task ProcessProductAsync( + string org, + string repo, + string product, + IReadOnlyDictionary> notesByVersion, + Dictionary> seqMap, + Cancel ctx) + { + // Read this product's bundle registry. + var registryKey = ChangelogKeys.BundleRegistryKey(product); + ChangelogRegistry? registry; + try + { + using var response = await s3Client.GetObjectAsync(new GetObjectRequest + { + BucketName = publicBucketName, + Key = registryKey + }, ctx); + await using var stream = response.ResponseStream; + registry = await JsonSerializer.DeserializeAsync( + stream, ChangelogRegistryJsonContext.Default.ChangelogRegistry, ctx); + } + catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.NotFound) + { + return; + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + _logger.LogWarning(ex, "Could not read bundle registry for product {Product}; skipping", product); + return; + } + + if (registry is null || registry.Bundles.Count == 0) + return; + + // For each version that has notes, look for a matching parent bundle. + foreach (var (version, notes) in notesByVersion) + { + ctx.ThrowIfCancellationRequested(); + + // Parent bundle: not an amend file, target matches the version. + var parentBundle = registry.Bundles + .FirstOrDefault(b => + !string.IsNullOrEmpty(b.File) + && !BundleAmendMerger.IsAmendFile(b.File) + && ChangelogVersionMatch.Matches(version, b.Target, b.File)); + + if (parentBundle is null) + continue; // No bundle yet → every note stays at bundle_seq 0. + + await ProcessVersionBundleAsync(org, repo, product, parentBundle, registry, version, notes, seqMap[version], ctx); + } + } + + private async Task ProcessVersionBundleAsync( + string org, + string repo, + string product, + ChangelogRegistryBundle parentRegistryBundle, + ChangelogRegistry registry, + string version, + IReadOnlyList notes, + Dictionary seqByPath, + Cancel ctx) + { + var parentFile = parentRegistryBundle.File!; + var parentKey = $"{ChangelogKeys.BundlePrefix}{product}/{parentFile}"; + + var parent = await TryReadBundleAsync(parentKey, ctx); + if (parent is null) + return; + + // A bundle with no file-annotated entries (hand-authored / legacy) has no reliable + // shipped set — skip to avoid false positives. + var parentHasFileAnnotations = parent.Entries.Any(e => !string.IsNullOrEmpty(e.File?.Name)); + if (parent.Entries.Count > 0 && !parentHasFileAnnotations) + { + _logger.LogDebug( + "Parent bundle {Key} has no file annotations; skipping amend-notes reconcile for version {Version}", + parentKey, version); + return; + } + + // Read existing numeric amend bundles (in order) to compute the full merged set. + var numericAmends = registry.Bundles + .Where(b => + !string.IsNullOrEmpty(b.File) + && BundleAmendMerger.IsAmendFile(b.File) + && BundleAmendMerger.GetAmendFileNumber(b.File) > 0 + && string.Equals( + BundleAmendMerger.GetParentBundlePath(b.File), + parentFile, + StringComparison.OrdinalIgnoreCase)) + .OrderBy(b => BundleAmendMerger.GetAmendFileNumber(b.File!)) + .ToList(); + + var amendBundles = new List(numericAmends.Count); + foreach (var amend in numericAmends) + { + var bundle = await TryReadBundleAsync($"{ChangelogKeys.BundlePrefix}{product}/{amend.File}", ctx); + if (bundle is not null) + amendBundles.Add(bundle); + } + + // Shipped set = parent entries merged with all numeric amends. + var mergedEntries = BundleAmendMerger.MergeEntries(parent.Entries, amendBundles); + var shippedLeaves = new HashSet(StringComparer.OrdinalIgnoreCase); + foreach (var entry in mergedEntries) + { + var leaf = LeafName(entry.File?.Name); + if (!string.IsNullOrEmpty(leaf)) + _ = shippedLeaves.Add(leaf); + } + + // Classify each note. + var lateNotes = new List(); + foreach (var note in notes) + { + var leaf = LeafName(note.Path); + if (leaf is not null && shippedLeaves.Contains(leaf)) + seqByPath[note.Path] = 1; // shipped in original bundle or a human amend + else + lateNotes.Add(note); + } + + // Amend-notes sidecar key. + var parentStem = Path.GetFileNameWithoutExtension(parentFile); + var parentExt = Path.GetExtension(parentFile); + var amendNotesFile = $"{parentStem}.amend-notes{parentExt}"; + var amendNotesKey = $"{ChangelogKeys.BundlePrefix}{product}/{amendNotesFile}"; + + if (lateNotes.Count > 0) + { + // Fetch each late note's content from the pool to build BundledEntry records. + var lateEntries = await FetchLateNoteEntriesAsync(org, repo, lateNotes, ctx); + if (lateEntries.Count > 0) + { + var amendBundle = AmendDocumentBuilder.Build(parent.Products, lateEntries, []); + var newJson = ReleaseNotesSerialization.SerializeBundle(amendBundle); + await WriteAmendNotesAsync(amendNotesKey, newJson, ctx); + + foreach (var note in lateNotes) + seqByPath[note.Path] = 2; // carried by the reconciler amend + } + } + else + { + // All notes are shipped — delete the sidecar if it exists. + await DeleteAmendNotesIfExistsAsync(amendNotesKey, ctx); + } + } + + // ----------------------------------------------------------------------------------------- + // Note content fetching + // ----------------------------------------------------------------------------------------- + + private async Task> FetchLateNoteEntriesAsync( + string org, + string repo, + IReadOnlyList lateNotes, + Cancel ctx) + { + var entries = new List(lateNotes.Count); + foreach (var note in lateNotes) + { + ctx.ThrowIfCancellationRequested(); + var key = $"changelog/{org}/{repo}/{note.Path}"; + try + { + using var response = await s3Client.GetObjectAsync(new GetObjectRequest + { + BucketName = publicBucketName, + Key = key + }, ctx); + + await using var stream = response.ResponseStream; + using var reader = new StreamReader(stream); + var yaml = await reader.ReadToEndAsync(ctx); + + var normalized = ReleaseNotesSerialization.NormalizeYaml(yaml); + var dto = ReleaseNotesSerialization.GetEntryDeserializer().Deserialize(normalized); + var entry = ReleaseNotesSerialization.ConvertEntry(dto); + var checksum = ChangelogBundlingService.ComputeSha1(yaml); + + entries.Add(entry.ToBundledEntry() with + { + File = new BundledFile + { + Name = note.Path, + Checksum = checksum + } + }); + } + catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.NotFound) + { + _logger.LogWarning("Late note {Key} not found in pool; skipping", key); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + _logger.LogWarning(ex, "Could not read late note {Key}; skipping", key); + } + } + return entries; + } + + // ----------------------------------------------------------------------------------------- + // Conditional S3 write / delete + // ----------------------------------------------------------------------------------------- + + private async Task WriteAmendNotesAsync(string key, string newJson, Cancel ctx) + { + const int maxAttempts = 5; + for (var attempt = 1; attempt <= maxAttempts; attempt++) + { + ctx.ThrowIfCancellationRequested(); + try + { + // Read current ETag for conditional PUT. + string? currentETag = null; + try + { + var head = await s3Client.GetObjectMetadataAsync(new GetObjectMetadataRequest + { + BucketName = publicBucketName, + Key = key + }, ctx); + currentETag = head.ETag; + } + catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.NotFound) { } + + // Skip when content is unchanged. + if (currentETag is not null) + { + try + { + using var existing = await s3Client.GetObjectAsync(new GetObjectRequest + { + BucketName = publicBucketName, + Key = key + }, ctx); + await using var existStream = existing.ResponseStream; + using var existReader = new StreamReader(existStream); + var existingJson = await existReader.ReadToEndAsync(ctx); + if (existingJson == newJson) + { + _logger.LogDebug("Amend-notes {Key} is unchanged; skipping write", key); + return; + } + } + catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.NotFound) + { + currentETag = null; + } + } + + var putRequest = new PutObjectRequest + { + BucketName = publicBucketName, + Key = key, + ContentBody = newJson, + ContentType = "application/yaml" + }; + if (currentETag is not null) + putRequest.IfMatch = currentETag.Trim('"'); + else + putRequest.IfNoneMatch = "*"; + + _ = await s3Client.PutObjectAsync(putRequest, ctx); + _metrics.IncrementRegistryWrites(); + _logger.LogInformation("Wrote amend-notes sidecar {Key}", key); + return; + } + catch (AmazonS3Exception ex) when (ex.StatusCode is HttpStatusCode.PreconditionFailed || (int)ex.StatusCode == 409) + { + if (attempt >= maxAttempts) + { + _logger.LogError("Amend-notes write {Key} failed after {Max} conditional conflicts", key, maxAttempts); + throw; + } + var jitter = TimeSpan.FromMilliseconds(Random.Shared.NextDouble() * 100); + var delay = (_retryBaseDelay * attempt) + jitter; + await Task.Delay(delay, ctx); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + if (attempt >= maxAttempts) + throw; + await Task.Delay(_retryBaseDelay * attempt, ctx); + _logger.LogDebug(ex, "Amend-notes write {Key} failed (attempt {A}/{Max}); retrying", key, attempt, maxAttempts); + } + } + } + + private async Task DeleteAmendNotesIfExistsAsync(string key, Cancel ctx) + { + try + { + var head = await s3Client.GetObjectMetadataAsync(new GetObjectMetadataRequest + { + BucketName = publicBucketName, + Key = key + }, ctx); + var etag = head.ETag; + + _ = await s3Client.DeleteObjectAsync(new DeleteObjectRequest + { + BucketName = publicBucketName, + Key = key, + IfMatch = etag + }, ctx); + _logger.LogInformation("Deleted stale amend-notes sidecar {Key}", key); + } + catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.NotFound) + { + // Nothing to delete — this is the expected steady state when all notes are shipped. + } + catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.PreconditionFailed) + { + // Another reconciler deleted or replaced it concurrently — safe to ignore. + _logger.LogDebug("Amend-notes {Key} was updated concurrently; delete skipped", key); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + _logger.LogWarning(ex, "Could not delete stale amend-notes sidecar {Key}; will retry on next reconcile", key); + } + } + + // ----------------------------------------------------------------------------------------- + // S3 bundle reading + // ----------------------------------------------------------------------------------------- + + private async Task TryReadBundleAsync(string key, Cancel ctx) + { + try + { + using var response = await s3Client.GetObjectAsync(new GetObjectRequest + { + BucketName = publicBucketName, + Key = key + }, ctx); + await using var stream = response.ResponseStream; + using var reader = new StreamReader(stream); + var yaml = await reader.ReadToEndAsync(ctx); + return ReleaseNotesSerialization.DeserializeBundle(yaml); + } + catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.NotFound) + { + _logger.LogDebug("Bundle {Key} not found; skipping", key); + return null; + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + _logger.LogWarning(ex, "Could not read bundle {Key}; skipping", key); + return null; + } + } + + // ----------------------------------------------------------------------------------------- + // Helpers + // ----------------------------------------------------------------------------------------- + + private static string? LeafName(string? path) + { + if (string.IsNullOrEmpty(path)) + return null; + var normalized = path.Replace('\\', '/'); + var slash = normalized.LastIndexOf('/'); + return slash >= 0 ? normalized[(slash + 1)..] : normalized; + } +} diff --git a/src/services/Elastic.Changelog/Reconciliation/NotesIndexReconciler.cs b/src/services/Elastic.Changelog/Reconciliation/NotesIndexReconciler.cs index ad533dd846..11d5529f9f 100644 --- a/src/services/Elastic.Changelog/Reconciliation/NotesIndexReconciler.cs +++ b/src/services/Elastic.Changelog/Reconciliation/NotesIndexReconciler.cs @@ -12,14 +12,17 @@ namespace Elastic.Changelog.Reconciliation; /// -/// Rebuilds the per-target notes-{target}.json indexes for one repository by listing +/// Rebuilds the per-version notes-{version}.json indexes for one repository by listing /// all note-*.yml objects under changelog/{org}/{repo}/, reading each to extract -/// its target: values, and writing the affected indexes atomically. +/// its versions: values (falling back to the legacy target: field for backward +/// compatibility), and writing the affected indexes atomically with conditional S3 writes. /// /// -/// A note may list products at multiple targets, so one note can appear in several indexes. -/// The index stores pool-relative paths ({branch}/note-{name}.yml) so the same file -/// name on two branches yields two distinct entries in the same index. +/// A note may declare multiple versions, so one note can appear in several indexes. The index +/// stores pool-relative paths ({branch}/note-{name}.yml) so the same filename on two +/// branches yields two distinct entries in the same index. The bundle_seq field on each +/// entry is filled by in a subsequent pass; this reconciler +/// sets it to 0 for all entries (no bundle awareness here, keeping concerns separated). /// public sealed class NotesIndexReconciler( ILoggerFactory logFactory, @@ -39,11 +42,17 @@ public sealed class NotesIndexReconciler( private readonly string _sourceBucketName = sourceBucketName ?? publicBucketName; /// - /// Rebuilds all notes-{target}.json indexes for the given repository scope. + /// Rebuilds all notes-{version}.json indexes for the given repository scope. /// All currently published note-*.yml files across every branch are listed and - /// read to derive the target grouping; every affected index is then (re)written. + /// read to derive the version grouping; every affected index is then (re)written. /// - public async Task ReconcileRepoAsync(ChangelogScope notesScope, Cancel ctx) + /// + /// A map of version → list of NoteIndexEntry for all versions found. + /// Returns an empty dictionary when no notes exist. Consumed by + /// in the same SQS-batch pass to compute bundle_seq and publish amend sidecars. + /// + public async Task>> ReconcileRepoAsync( + ChangelogScope notesScope, Cancel ctx) { if (notesScope.Kind != ChangelogScopeKind.Notes) throw new ArgumentException($"Notes reconcile requires a Notes scope; got '{notesScope}'.", nameof(notesScope)); @@ -54,19 +63,23 @@ public async Task ReconcileRepoAsync(ChangelogScope notesScope, Cancel ctx) var noteObjects = await ListNoteFiles(notesScope, ctx); _logger.LogDebug("Found {Count} note file(s) for {Repo}", noteObjects.Count, notesScope.Group); - // Read each note to extract its targets. - // Using List per target; duplicates are removed at write time via Distinct(). - var byTarget = new Dictionary>(StringComparer.Ordinal); + // Read each note to extract its versions. + // byVersion: version slug → list of NoteIndexEntry (bundle_seq defaulted to 0; filled by NoteAmendReconciler) + var byVersion = new Dictionary>(StringComparer.Ordinal); foreach (var obj in noteObjects) { ctx.ThrowIfCancellationRequested(); var poolRelativePath = obj.Key[notesScope.Prefix.Length..]; - var targets = await ExtractTargetsAsync(obj.Key, ctx); - foreach (var target in targets) + + var versions = await ExtractVersionsAsync(obj.Key, ctx); + foreach (var version in versions) { - if (!byTarget.TryGetValue(target, out var paths)) - byTarget[target] = paths = []; - paths.Add(poolRelativePath); + if (!byVersion.TryGetValue(version, out var entries)) + byVersion[version] = entries = []; + + // Deduplicate by path within the same version. + if (!entries.Any(e => e.Path == poolRelativePath)) + entries.Add(new NoteIndexEntry { Path = poolRelativePath, BundleSeq = 0 }); } } @@ -76,33 +89,42 @@ public async Task ReconcileRepoAsync(ChangelogScope notesScope, Cancel ctx) // List existing notes-*.json indexes so we can remove obsolete ones. var existingIndexKeys = await ListExistingNotesIndexes(notesScope, ctx); - if (byTarget.Count == 0) + if (byVersion.Count == 0) { - _logger.LogDebug("No targets found for repo {Repo}; removing any stale indexes", notesScope.Group); + _logger.LogDebug("No versions found for repo {Repo}; removing any stale indexes", notesScope.Group); await DeleteStaleIndexes(existingIndexKeys, [], org, repo, ctx); - return; + return new Dictionary>(); } - // Write one index per target. DeleteStaleIndexes runs even if some writes fail — stale - // deletion is safe because we only remove targets absent from byTarget.Keys, which is - // independent of whether the new writes succeeded. + // Write one index per version. bundle_seq values default to 0 here; NoteAmendReconciler updates them. + // DeleteStaleIndexes runs even if some writes fail — stale deletion is safe because we only + // remove versions absent from byVersion.Keys, which is independent of write success. + var written = new Dictionary>(StringComparer.Ordinal); try { await Parallel.ForEachAsync( - byTarget, + byVersion, new ParallelOptions { MaxDegreeOfParallelism = MaxParallelReads, CancellationToken = ctx }, async (kvp, ct) => { - var (target, paths) = kvp; - var indexKey = ChangelogKeys.NotesIndexKey(org, repo, target); - await WriteIndexAsync(indexKey, [.. paths.Distinct(StringComparer.Ordinal).Order(StringComparer.Ordinal)], ct); + var (version, entries) = kvp; + var indexKey = ChangelogKeys.NotesIndexKey(org, repo, version); + var sortedEntries = entries + .DistinctBy(e => e.Path, StringComparer.Ordinal) + .OrderBy(e => e.Path, StringComparer.Ordinal) + .ToList(); + await WriteIndexAsync(indexKey, sortedEntries, ct); + lock (written) + written[version] = sortedEntries; }); } finally { - // Remove indexes whose targets are no longer present. - await DeleteStaleIndexes(existingIndexKeys, byTarget.Keys.ToHashSet(StringComparer.Ordinal), org, repo, ctx); + // Remove indexes whose versions are no longer present. + await DeleteStaleIndexes(existingIndexKeys, byVersion.Keys.ToHashSet(StringComparer.Ordinal), org, repo, ctx); } + + return written; } private async Task> ListExistingNotesIndexes(ChangelogScope notesScope, Cancel ctx) @@ -131,7 +153,7 @@ private async Task> ListExistingNotesIndexes(ChangelogScop private async Task DeleteStaleIndexes( IReadOnlyList existingKeys, - HashSet currentTargets, + HashSet currentVersions, string org, string repo, Cancel ctx) @@ -141,11 +163,11 @@ private async Task DeleteStaleIndexes( foreach (var key in existingKeys) { - // Extract the target slug from the key to check if it's still needed. + // Extract the version slug from the key to check if it's still needed. if (!key.StartsWith(notesKeyPrefix, StringComparison.Ordinal)) continue; - var targetSlug = key[notesKeyPrefix.Length..^".json".Length]; - if (currentTargets.Contains(targetSlug)) + var versionSlug = key[notesKeyPrefix.Length..^".json".Length]; + if (currentVersions.Contains(versionSlug)) continue; try @@ -202,7 +224,12 @@ private static bool IsNoteFileName(string fileName) => && (fileName.EndsWith(".yml", StringComparison.OrdinalIgnoreCase) || fileName.EndsWith(".yaml", StringComparison.OrdinalIgnoreCase)) && !fileName.Contains('/', StringComparison.Ordinal); - private async Task> ExtractTargetsAsync(string key, Cancel ctx) + /// + /// Reads a note file and returns all version slugs it should be indexed under. + /// Prefers products[].versions; falls back to the legacy products[].target + /// for already-published notes that pre-date the versions: field. + /// + private async Task> ExtractVersionsAsync(string key, Cancel ctx) { try { @@ -223,14 +250,29 @@ private async Task> ExtractTargetsAsync(string key, Cancel return []; var valid = new List(); - foreach (var target in dto.Products.Select(p => p.Target).Where(t => !string.IsNullOrWhiteSpace(t)).Distinct(StringComparer.Ordinal)) + foreach (var productInfo in dto.Products) { - if (target!.Contains('/', StringComparison.Ordinal)) + // Prefer the new `versions` list; fall back to the legacy `target` field for compat. +#pragma warning disable CS0618 // reading obsolete Target for backward compat + IEnumerable rawVersions = + productInfo.Versions is { Count: > 0 } + ? productInfo.Versions + : productInfo.Target is not null + ? [productInfo.Target] + : []; +#pragma warning restore CS0618 + + foreach (var raw in rawVersions.Where(v => !string.IsNullOrWhiteSpace(v))) { - _logger.LogWarning("Note {Key} has target '{Target}' containing '/'; skipping — targets must be single path segments", key, target); - continue; + var v = raw!.Trim(); + if (v.Contains('/', StringComparison.Ordinal)) + { + _logger.LogWarning("Note {Key} has version '{Version}' containing '/'; skipping — versions must be single path segments", key, v); + continue; + } + if (!valid.Contains(v, StringComparer.Ordinal)) + valid.Add(v); } - valid.Add(target); } return valid; } @@ -241,33 +283,107 @@ private async Task> ExtractTargetsAsync(string key, Cancel } catch (Exception ex) when (ex is not OperationCanceledException) { - _logger.LogWarning(ex, "Could not read targets from note {Key}; skipping", key); + _logger.LogWarning(ex, "Could not read versions from note {Key}; skipping", key); return []; } } - private async Task WriteIndexAsync(string key, IReadOnlyList paths, Cancel ctx) + /// + /// Writes the notes index with conditional S3 writes (If-Match / If-None-Match) to guard against + /// concurrent reconcile races, mirroring the pattern used by . + /// + /// + /// have their bundle_seq already set by the caller + /// (0 from this reconciler; updated values from ). + /// + public async Task WriteIndexAsync(string key, IReadOnlyList entries, Cancel ctx) { - var index = new NotesIndex { Notes = paths }; - var json = JsonSerializer.Serialize(index, NotesIndexJsonContext.Default.NotesIndex); + var index = new NotesIndex + { + SchemaVersion = NotesIndex.CurrentSchemaVersion, + Notes = entries + }; + var newJson = JsonSerializer.Serialize(index, NotesIndexJsonContext.Default.NotesIndex); for (var attempt = 1; attempt <= MaxWriteAttempts; attempt++) { ctx.ThrowIfCancellationRequested(); try { - _ = await s3Client.PutObjectAsync(new PutObjectRequest + // Read current ETag so we can do a conditional PUT. + string? currentETag = null; + try + { + var head = await s3Client.GetObjectMetadataAsync(new GetObjectMetadataRequest + { + BucketName = publicBucketName, + Key = key + }, ctx); + currentETag = head.ETag; + } + catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.NotFound) + { + // Key does not exist yet — conditional create. + } + + // Skip write when content is unchanged (content equality, not ETag). + if (currentETag != null) + { + try + { + var existing = await s3Client.GetObjectAsync(new GetObjectRequest + { + BucketName = publicBucketName, + Key = key + }, ctx); + await using var existStream = existing.ResponseStream; + using var existReader = new StreamReader(existStream); + var existingJson = await existReader.ReadToEndAsync(ctx); + if (existingJson == newJson) + { + _logger.LogDebug("Notes index {Key} is unchanged; skipping write", key); + return; + } + } + catch (AmazonS3Exception ex) when (ex.StatusCode == HttpStatusCode.NotFound) + { + currentETag = null; // lost a race — treat as not-found + } + } + + var putRequest = new PutObjectRequest { BucketName = publicBucketName, Key = key, - ContentBody = json, + ContentBody = newJson, ContentType = "application/json" - }, ctx); + }; + + // Conditional write: update matches ETag; create uses If-None-Match. + if (currentETag != null) + putRequest.Headers["If-Match"] = currentETag; + else + putRequest.Headers["If-None-Match"] = "*"; + + _ = await s3Client.PutObjectAsync(putRequest, ctx); _metrics.IncrementRegistryWrites(); - _logger.LogInformation("Wrote notes index {Key} with {Count} path(s)", key, paths.Count); + _logger.LogInformation("Wrote notes index {Key} with {Count} entry(ies)", key, entries.Count); return; } + catch (AmazonS3Exception ex) when (ex.StatusCode is HttpStatusCode.PreconditionFailed || (int)ex.StatusCode == 409) + { + // Conditional write lost — another reconciler won the race. Retry after jittered delay. + if (attempt >= MaxWriteAttempts) + { + _logger.LogError("Notes index write {Key} failed after {Max} conditional-write conflicts", key, MaxWriteAttempts); + throw; + } + var jitter = TimeSpan.FromMilliseconds(Random.Shared.NextDouble() * 100); + var delay = (_retryBaseDelay * attempt) + jitter; + _logger.LogDebug("Notes index {Key} conditional write conflict (attempt {A}/{Max}); retrying in {Delay}", key, attempt, MaxWriteAttempts, delay); + await Task.Delay(delay, ctx); + } catch (Exception ex) when (ex is not OperationCanceledException) { if (attempt >= MaxWriteAttempts) diff --git a/src/services/Elastic.Changelog/Scrubbing/ScrubberProcessor.cs b/src/services/Elastic.Changelog/Scrubbing/ScrubberProcessor.cs index 1c4c5bcfcb..dbb66550e6 100644 --- a/src/services/Elastic.Changelog/Scrubbing/ScrubberProcessor.cs +++ b/src/services/Elastic.Changelog/Scrubbing/ScrubberProcessor.cs @@ -34,6 +34,7 @@ public sealed class ScrubberProcessor( BundleRegistryReconciler reconciler, ShallowRegistryReconciler shallowReconciler, NotesIndexReconciler notesReconciler, + NoteAmendReconciler noteAmendReconciler, ReconcileMetrics? metrics = null ) { @@ -134,7 +135,8 @@ public async Task> ProcessAsync(IReadOnlyList p.Contains("/pull/958")); } diff --git a/tests/Elastic.Changelog.Tests/Changelogs/BundleAmendMergerTests.cs b/tests/Elastic.Changelog.Tests/Changelogs/BundleAmendMergerTests.cs index 8061ee47d9..d9fbd459ff 100644 --- a/tests/Elastic.Changelog.Tests/Changelogs/BundleAmendMergerTests.cs +++ b/tests/Elastic.Changelog.Tests/Changelogs/BundleAmendMergerTests.cs @@ -58,6 +58,8 @@ public void MergeEntries_AppliesAmendsInOrder() [InlineData("repo-9.3.0.amend-12.yml", "repo-9.3.0.yml")] [InlineData("cloud-2025-11.AMEND-2.YAML", "cloud-2025-11.YAML")] [InlineData("/releases/9.3.0.amend-1.yaml", "/releases/9.3.0.yaml")] + [InlineData("elasticsearch-9.3.0.amend-notes.yaml", "elasticsearch-9.3.0.yaml")] + [InlineData("cloud-2025-11.amend-notes.yml", "cloud-2025-11.yml")] public void GetParentBundlePath_AmendFile_StripsAmendSuffix(string amendPath, string expectedParent) => BundleAmendMerger.GetParentBundlePath(amendPath).Should().Be(expectedParent); diff --git a/tests/Elastic.Changelog.Tests/Changelogs/BundleCdnSourcingTests.cs b/tests/Elastic.Changelog.Tests/Changelogs/BundleCdnSourcingTests.cs index 5baba8c2f6..44f170c2d4 100644 --- a/tests/Elastic.Changelog.Tests/Changelogs/BundleCdnSourcingTests.cs +++ b/tests/Elastic.Changelog.Tests/Changelogs/BundleCdnSourcingTests.cs @@ -47,6 +47,10 @@ public class BundleCdnSourcingTests(ITestOutputHelper output) : ChangelogTestBas // language=yaml private static string MarkerFor(int parentPr) => $"link: \"{parentPr}\"\n"; + private const string RegistryJson = + /*lang=json,strict*/ """{ "schema_version": 1, "product": "elasticsearch", "bundles": [ { "file": "1-alpha.yaml" }, { "file": "2-bravo.yaml" } ] }"""; + + private static StubHandler ProbeHandler() => new(req => { var path = req.RequestUri!.AbsolutePath; @@ -497,6 +501,181 @@ public async Task ProfileGitHubRelease_ScopesByOutputProductsAndFiltersByRelease bundle.Should().NotContain("Bravo"); } + // language=yaml + private const string NoteKnownIssueMain = """ + title: Known issue on main + type: known-issue + products: + - product: elasticsearch + versions: + - 9.3.0 + lifecycle: ga + """; + + // language=yaml + private const string NoteKnownIssue94 = """ + title: Known issue on 9.4 branch (backport) + type: known-issue + products: + - product: elasticsearch + versions: + - 9.3.0 + lifecycle: ga + """; + + // language=yaml + private const string NoteKnownIssueFeature = """ + title: Known issue on feature branch + type: known-issue + products: + - product: elasticsearch + versions: + - 9.3.0 + lifecycle: ga + """; + + [Fact] + public async Task BackportCollision_MainBranchWins_WarnAndKeepMain() + { + // notes-9.3.0.json lists both main/note-known-issue.yml and 9.4/note-known-issue.yml. + // The fetcher requests note-known-issue.yml twice (same leaf URL), returning main content first + // and 9.4-branch content second. The backport rule must keep the main copy and warn about the 9.4 copy. + var callCount = new Dictionary(StringComparer.Ordinal); + var handler = new StubHandler(req => + { + var path = req.RequestUri!.AbsolutePath; + if (path.EndsWith("/registry.json", StringComparison.Ordinal)) + return Json(RegistryJson); + if (path.EndsWith("notes-9.3.0.json", StringComparison.Ordinal)) + return Json(/*lang=json,strict*/ """{"schema_version":1,"notes":[{"path":"main/note-known-issue.yml","bundle_seq":0},{"path":"9.4/note-known-issue.yml","bundle_seq":0}]}"""); + if (path.EndsWith("note-known-issue.yml", StringComparison.Ordinal)) + { + callCount.TryGetValue(path, out var n); + callCount[path] = n + 1; + // First fetch → main content; second fetch → 9.4-branch content (simulates differing backport content). + return n == 0 ? Yaml(NoteKnownIssueMain) : Yaml(NoteKnownIssue94); + } + if (path.EndsWith("1-alpha.yaml", StringComparison.Ordinal)) + return Yaml(EntryAlpha); + if (path.EndsWith("2-bravo.yaml", StringComparison.Ordinal)) + return Yaml(EntryBravo); + return new HttpResponseMessage(HttpStatusCode.NotFound); + }); + + var fetcher = new CdnChangelogEntryFetcher(new TestLoggerFactory(Output), handler, sleep: (_, _) => Task.CompletedTask); + var service = new ChangelogBundlingService(LoggerFactory, FileSystem, null, null, fetcher); + var output = OutputPath(); + + var input = new BundleChangelogsArguments + { + Prs = ["https://github.com/elastic/elasticsearch/pull/100"], + Output = output, + Repo = "elasticsearch", + OutputProducts = [new ProductArgument { Product = "elasticsearch", Target = "9.3.0", Lifecycle = "ga" }] + }; + + var result = await service.BundleChangelogs(Collector, input, TestContext.Current.CancellationToken); + + result.Should().BeTrue($"Errors: {string.Join("; ", Collector.Diagnostics.Where(d => d.Severity == Severity.Error).Select(d => d.Message))}"); + + var bundle = await FileSystem.File.ReadAllTextAsync(output, TestContext.Current.CancellationToken); + // Only the main-branch note title should appear once. + bundle.Should().Contain("Known issue on main"); + bundle.Should().NotContain("Known issue on 9.4 branch"); + } + + [Fact] + public async Task BackportCollision_NoMainOrMaster_KeepsOrdinalFirst() + { + // When neither branch is main/master, the alphabetically-first path wins. + // "9.4/note-known-issue.yml" < "feature/note-known-issue.yml" lexicographically. + var callCount = new Dictionary(StringComparer.Ordinal); + var handler = new StubHandler(req => + { + var path = req.RequestUri!.AbsolutePath; + if (path.EndsWith("/registry.json", StringComparison.Ordinal)) + return Json(RegistryJson); + if (path.EndsWith("notes-9.3.0.json", StringComparison.Ordinal)) + return Json(/*lang=json,strict*/ """{"schema_version":1,"notes":[{"path":"9.4/note-known-issue.yml","bundle_seq":0},{"path":"feature/note-known-issue.yml","bundle_seq":0}]}"""); + if (path.EndsWith("note-known-issue.yml", StringComparison.Ordinal)) + { + callCount.TryGetValue(path, out var n); + callCount[path] = n + 1; + return n == 0 ? Yaml(NoteKnownIssue94) : Yaml(NoteKnownIssueFeature); + } + if (path.EndsWith("1-alpha.yaml", StringComparison.Ordinal)) + return Yaml(EntryAlpha); + if (path.EndsWith("2-bravo.yaml", StringComparison.Ordinal)) + return Yaml(EntryBravo); + return new HttpResponseMessage(HttpStatusCode.NotFound); + }); + + var fetcher = new CdnChangelogEntryFetcher(new TestLoggerFactory(Output), handler, sleep: (_, _) => Task.CompletedTask); + var service = new ChangelogBundlingService(LoggerFactory, FileSystem, null, null, fetcher); + var output = OutputPath(); + + var input = new BundleChangelogsArguments + { + Prs = ["https://github.com/elastic/elasticsearch/pull/100"], + Output = output, + Repo = "elasticsearch", + OutputProducts = [new ProductArgument { Product = "elasticsearch", Target = "9.3.0", Lifecycle = "ga" }] + }; + + var result = await service.BundleChangelogs(Collector, input, TestContext.Current.CancellationToken); + + result.Should().BeTrue($"Errors: {string.Join("; ", Collector.Diagnostics.Where(d => d.Severity == Severity.Error).Select(d => d.Message))}"); + + var bundle = await FileSystem.File.ReadAllTextAsync(output, TestContext.Current.CancellationToken); + // "9.4/note-known-issue.yml" sorts before "feature/note-known-issue.yml". + bundle.Should().Contain("Known issue on 9.4 branch"); + bundle.Should().NotContain("Known issue on feature branch"); + } + + [Fact] + public async Task BackportCollision_IdenticalContent_ChecksumDedupHandlesIt() + { + // Same leaf on two branches, identical content → same checksum → existing checksum dedup + // removes the duplicate; the backport rule emits no warning since DeduplicateNotesByLeaf + // sees two entries but the second is absorbed by seen.Add(checksum) before reaching combined. + var handler = new StubHandler(req => + { + var path = req.RequestUri!.AbsolutePath; + if (path.EndsWith("/registry.json", StringComparison.Ordinal)) + return Json(RegistryJson); + if (path.EndsWith("notes-9.3.0.json", StringComparison.Ordinal)) + return Json(/*lang=json,strict*/ """{"schema_version":1,"notes":[{"path":"main/note-known-issue.yml","bundle_seq":0},{"path":"9.4/note-known-issue.yml","bundle_seq":0}]}"""); + if (path.EndsWith("note-known-issue.yml", StringComparison.Ordinal)) + return Yaml(NoteKnownIssueMain); // identical for both branch fetches + if (path.EndsWith("1-alpha.yaml", StringComparison.Ordinal)) + return Yaml(EntryAlpha); + if (path.EndsWith("2-bravo.yaml", StringComparison.Ordinal)) + return Yaml(EntryBravo); + return new HttpResponseMessage(HttpStatusCode.NotFound); + }); + + var fetcher = new CdnChangelogEntryFetcher(new TestLoggerFactory(Output), handler, sleep: (_, _) => Task.CompletedTask); + var service = new ChangelogBundlingService(LoggerFactory, FileSystem, null, null, fetcher); + var output = OutputPath(); + + var input = new BundleChangelogsArguments + { + Prs = ["https://github.com/elastic/elasticsearch/pull/100"], + Output = output, + Repo = "elasticsearch", + OutputProducts = [new ProductArgument { Product = "elasticsearch", Target = "9.3.0", Lifecycle = "ga" }] + }; + + var result = await service.BundleChangelogs(Collector, input, TestContext.Current.CancellationToken); + + result.Should().BeTrue($"Errors: {string.Join("; ", Collector.Diagnostics.Where(d => d.Severity == Severity.Error).Select(d => d.Message))}"); + + var bundle = await FileSystem.File.ReadAllTextAsync(output, TestContext.Current.CancellationToken); + bundle.Should().Contain("Known issue on main"); + // Note appears exactly once (not duplicated). + bundle.Split("Known issue on main", StringSplitOptions.None).Length.Should().Be(2, "note must appear exactly once"); + } + private static HttpResponseMessage Json(string body) => new(HttpStatusCode.OK) { Content = new StringContent(body, System.Text.Encoding.UTF8, "application/json") }; diff --git a/tests/Elastic.Changelog.Tests/Changelogs/Create/AddReportOptionTests.cs b/tests/Elastic.Changelog.Tests/Changelogs/Create/AddReportOptionTests.cs index f5f74ca3ea..0c4212a347 100644 --- a/tests/Elastic.Changelog.Tests/Changelogs/Create/AddReportOptionTests.cs +++ b/tests/Elastic.Changelog.Tests/Changelogs/Create/AddReportOptionTests.cs @@ -70,7 +70,7 @@ public async Task CreateChangelog_FromPromotionReportHtmlFile_CreatesOneYamlPerP var input = new CreateChangelogArguments { Prs = prUrls, - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), UsePrNumber = true diff --git a/tests/Elastic.Changelog.Tests/Changelogs/Create/BasicInputTests.cs b/tests/Elastic.Changelog.Tests/Changelogs/Create/BasicInputTests.cs index 855824b96b..9630570e39 100644 --- a/tests/Elastic.Changelog.Tests/Changelogs/Create/BasicInputTests.cs +++ b/tests/Elastic.Changelog.Tests/Changelogs/Create/BasicInputTests.cs @@ -19,7 +19,7 @@ public async Task CreateChangelog_WithBasicInput_CreatesValidYamlFile() { Title = "Add new search feature", Type = "feature", - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Description = "This is a new search feature", Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], Output = CreateOutputDirectory() @@ -49,7 +49,7 @@ public async Task CreateChangelog_WithBasicInput_CreatesValidYamlFile() yamlContent.Should().Contain("title: Add new search feature"); yamlContent.Should().Contain("type: feature"); yamlContent.Should().Contain("product: elasticsearch"); - yamlContent.Should().Contain("target: 9.2.0"); + // entries no longer carry a target/version — the origin branch is the address yamlContent.Should().Contain("lifecycle: ga"); yamlContent.Should().Contain("description: This is a new search feature"); } @@ -66,8 +66,8 @@ public async Task CreateChangelog_WithMultipleProducts_CreatesValidYaml() Type = "feature", Products = [ - new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }, - new ProductArgument { Product = "kibana", Target = "9.2.0", Lifecycle = "ga" } + new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }, + new ProductArgument { Product = "kibana", Lifecycle = "ga" } ], Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], Output = CreateOutputDirectory() @@ -111,7 +111,7 @@ public async Task CreateChangelog_WithBreakingChangeAndSubtype_CreatesValidYaml( Title = "Breaking API change", Type = "breaking-change", Subtype = "api", - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Impact = "API clients will need to update", Action = "Update your API client code", Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], diff --git a/tests/Elastic.Changelog.Tests/Changelogs/Create/BlockingLabelTests.cs b/tests/Elastic.Changelog.Tests/Changelogs/Create/BlockingLabelTests.cs index c5be473172..a082580f57 100644 --- a/tests/Elastic.Changelog.Tests/Changelogs/Create/BlockingLabelTests.cs +++ b/tests/Elastic.Changelog.Tests/Changelogs/Create/BlockingLabelTests.cs @@ -53,7 +53,7 @@ public async Task CreateChangelog_WithBlockingLabel_SkipsChangelogCreation() var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/1234"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory() }; @@ -117,7 +117,7 @@ public async Task CreateChangelog_WithBlockingLabelForSpecificProduct_OnlyBlocks Prs = ["https://github.com/elastic/elasticsearch/pull/1234"], Products = [ - new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }, + new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }, new ProductArgument { Product = "cloud-serverless", Target = "2025-08-05" } ], Config = configPath, @@ -183,7 +183,7 @@ public async Task CreateChangelog_WithCommaSeparatedProductIdsInAddBlockers_Expa Prs = ["https://github.com/elastic/elasticsearch/pull/1234"], Products = [ - new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }, + new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }, new ProductArgument { Product = "cloud-serverless", Target = "2025-08-05" } ], Config = configPath, diff --git a/tests/Elastic.Changelog.Tests/Changelogs/Create/FlagsAndFeaturesTests.cs b/tests/Elastic.Changelog.Tests/Changelogs/Create/FlagsAndFeaturesTests.cs index 081a7775d4..c569115441 100644 --- a/tests/Elastic.Changelog.Tests/Changelogs/Create/FlagsAndFeaturesTests.cs +++ b/tests/Elastic.Changelog.Tests/Changelogs/Create/FlagsAndFeaturesTests.cs @@ -19,7 +19,7 @@ public async Task CreateChangelog_WithHighlightFlag_CreatesValidYaml() { Title = "Important feature", Type = "feature", - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Highlight = true, Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], Output = CreateOutputDirectory() @@ -57,7 +57,7 @@ public async Task CreateChangelog_WithFeatureId_CreatesValidYaml() { Title = "New feature with flag", Type = "feature", - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], FeatureId = "feature:new-search-api", Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], Output = CreateOutputDirectory() @@ -95,7 +95,7 @@ public async Task CreateChangelog_WithIssues_CreatesValidYaml() { Title = "Fix multiple issues", Type = "bug-fix", - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Issues = [ "https://github.com/elastic/elasticsearch/issues/123", diff --git a/tests/Elastic.Changelog.Tests/Changelogs/Create/LabelMappingTests.cs b/tests/Elastic.Changelog.Tests/Changelogs/Create/LabelMappingTests.cs index 10864275ca..eab40b1ecf 100644 --- a/tests/Elastic.Changelog.Tests/Changelogs/Create/LabelMappingTests.cs +++ b/tests/Elastic.Changelog.Tests/Changelogs/Create/LabelMappingTests.cs @@ -51,7 +51,7 @@ public async Task CreateChangelog_WithPrOptionAndLabelMapping_MapsLabelsToType() var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Config = configPath, Output = CreateOutputDirectory() }; @@ -340,7 +340,7 @@ public async Task CreateChangelog_WithLabelProductMapping_ExplicitProductsOverri { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], // Explicit product takes precedence over label mapping - Products = [new ProductArgument { Product = "kibana", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "kibana" }], Config = configPath, Output = CreateOutputDirectory() }; @@ -404,7 +404,7 @@ public async Task CreateChangelog_WithPrOptionAndAreaMapping_MapsLabelsToAreas() var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Config = configPath, Output = CreateOutputDirectory() }; @@ -495,7 +495,7 @@ public async Task CreateChangelog_WithAreaNameContainingCommas_PreservesAreaName var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Config = configPath, Output = CreateOutputDirectory() }; @@ -691,7 +691,7 @@ public async Task CreateChangelog_WithLabelFeatureMapping_DerivesFeatureIdFromLa var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Config = configPath, Output = CreateOutputDirectory() }; @@ -748,7 +748,7 @@ public async Task CreateChangelog_WithExplicitFeatureId_IgnoresLabelMapping() var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], FeatureId = "feature:cli-override", Config = configPath, Output = CreateOutputDirectory() @@ -803,7 +803,7 @@ public async Task CreateChangelog_WithMultipleFeatureLabelMatches_WarnsAndUsesFi var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Config = configPath, Output = CreateOutputDirectory() }; @@ -857,7 +857,7 @@ public async Task CreateChangelog_WithNoMatchingFeatureLabels_OmitsFeatureId() var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Config = configPath, Output = CreateOutputDirectory() }; diff --git a/tests/Elastic.Changelog.Tests/Changelogs/Create/NoteCreationTests.cs b/tests/Elastic.Changelog.Tests/Changelogs/Create/NoteCreationTests.cs index a85d81c2ee..118b803db5 100644 --- a/tests/Elastic.Changelog.Tests/Changelogs/Create/NoteCreationTests.cs +++ b/tests/Elastic.Changelog.Tests/Changelogs/Create/NoteCreationTests.cs @@ -58,7 +58,7 @@ public async Task CreateNote_ProductWithoutTarget_ReturnsError() result.Should().BeFalse(); Collector.Diagnostics.Should().Contain(d => - d.Severity == Severity.Error && d.Message.Contains("elasticsearch") && d.Message.Contains("target")); + d.Severity == Severity.Error && d.Message.Contains("elasticsearch") && d.Message.Contains("version")); } [Fact] @@ -79,7 +79,7 @@ public async Task CreateNote_EmptyTarget_ReturnsError() result.Should().BeFalse(); Collector.Diagnostics.Should().Contain(d => - d.Severity == Severity.Error && d.Message.Contains("elasticsearch") && d.Message.Contains("target")); + d.Severity == Severity.Error && d.Message.Contains("elasticsearch") && d.Message.Contains("version")); } [Fact] diff --git a/tests/Elastic.Changelog.Tests/Changelogs/Create/PrFetchFailureTests.cs b/tests/Elastic.Changelog.Tests/Changelogs/Create/PrFetchFailureTests.cs index 61d30d49c3..2db2b67a4f 100644 --- a/tests/Elastic.Changelog.Tests/Changelogs/Create/PrFetchFailureTests.cs +++ b/tests/Elastic.Changelog.Tests/Changelogs/Create/PrFetchFailureTests.cs @@ -22,7 +22,7 @@ public async Task CreateChangelog_WithPrOptionAndTitleAndType_SkipsApiFetch() Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], Title = "Manual title provided", Type = "feature", - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Output = CreateOutputDirectory() }; @@ -68,7 +68,7 @@ public async Task CreateChangelog_WithPrOptionButPrFetchFails_WithoutTitleAndTyp var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Output = CreateOutputDirectory() }; @@ -120,7 +120,7 @@ public async Task CreateChangelog_WithMultiplePrsButPrFetchFails_GeneratesBasicC Prs = ["https://github.com/elastic/elasticsearch/pull/12345", "https://github.com/elastic/elasticsearch/pull/67890"], Title = "Shared title", Type = "bug-fix", - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Output = CreateOutputDirectory() }; @@ -170,7 +170,7 @@ public async Task CreateChangelog_WithMultiplePrsFetchFails_EmitsAggregateWarnin var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345", "https://github.com/elastic/elasticsearch/pull/67890"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Output = CreateOutputDirectory() }; @@ -202,7 +202,7 @@ public async Task CreateChangelog_WithMultiplePrsFetchFailsAndStrictFetch_EmitsE var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345", "https://github.com/elastic/elasticsearch/pull/67890"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], StrictFetch = true, Output = CreateOutputDirectory() }; @@ -235,7 +235,7 @@ public async Task CreateChangelog_WithMultipleIssuesFetchFailsAndStrictFetch_Emi var input = new CreateChangelogArguments { Issues = ["https://github.com/elastic/elasticsearch/issues/12345", "https://github.com/elastic/elasticsearch/issues/67890"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], StrictFetch = true, Output = CreateOutputDirectory() }; @@ -269,7 +269,7 @@ public async Task CreateChangelog_WithSinglePrFetchFailsAndStrictFetch_EmitsErro var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], StrictFetch = true, Output = CreateOutputDirectory() }; diff --git a/tests/Elastic.Changelog.Tests/Changelogs/Create/PrIntegrationTests.cs b/tests/Elastic.Changelog.Tests/Changelogs/Create/PrIntegrationTests.cs index 7c55fc5a94..5ee21fc177 100644 --- a/tests/Elastic.Changelog.Tests/Changelogs/Create/PrIntegrationTests.cs +++ b/tests/Elastic.Changelog.Tests/Changelogs/Create/PrIntegrationTests.cs @@ -49,7 +49,7 @@ public async Task CreateChangelog_WithPrOption_FetchesPrInfoAndDerivesTitle() var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory() }; @@ -119,7 +119,7 @@ public async Task CreateChangelog_WithUsePrNumber_CreatesFileWithPrNumberAsFilen var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/140034"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), UsePrNumber = true @@ -198,7 +198,7 @@ public async Task CreateChangelog_WithMultiplePrsAndUsePrNumber_CreatesOneFilePe var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/1234", "https://github.com/elastic/elasticsearch/pull/5678"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), UsePrNumber = true @@ -255,7 +255,7 @@ public async Task CreateChangelog_WithBothIssuesAndPrs_UsesPrNumberForFilename() { Issues = ["https://github.com/elastic/kibana/issues/233425"], Prs = ["https://github.com/elastic/kibana/pull/250840"], - Products = [new ProductArgument { Product = "kibana", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "kibana", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), Title = "Release notes test", @@ -286,7 +286,7 @@ public async Task CreateChangelog_WithPrNumberAndOwnerRepo_SkipsApiFetchWhenTitl Repo = "elasticsearch", Title = "Update documentation", Type = "docs", - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Output = CreateOutputDirectory() }; @@ -352,7 +352,7 @@ public async Task CreateChangelog_WithMultiplePrs_CreatesOneFilePerPr() var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/1234", "https://github.com/elastic/elasticsearch/pull/5678"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory() }; @@ -461,7 +461,7 @@ public async Task CreateChangelog_WithPrsFromFile_ProcessesAllPrsFromFile() var input = new CreateChangelogArguments { Prs = parsedPrs, // PRs read from file - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory() }; @@ -565,7 +565,7 @@ public async Task CreateChangelog_WithMixedPrsFromFileAndCommaSeparated_Processe var input = new CreateChangelogArguments { Prs = allPrs.ToArray(), // Mixed PRs from comma-separated and file - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory() }; @@ -639,7 +639,7 @@ public async Task CreateChangelog_WithBareNumberPrAndOwnerRepo_WritesFullUrlInto Prs = ["155500"], Owner = "elastic", Repo = "cloud", - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), UsePrNumber = true, diff --git a/tests/Elastic.Changelog.Tests/Changelogs/Create/ReleaseNoteExtractionTests.cs b/tests/Elastic.Changelog.Tests/Changelogs/Create/ReleaseNoteExtractionTests.cs index 900013ac71..3a8d35a13b 100644 --- a/tests/Elastic.Changelog.Tests/Changelogs/Create/ReleaseNoteExtractionTests.cs +++ b/tests/Elastic.Changelog.Tests/Changelogs/Create/ReleaseNoteExtractionTests.cs @@ -51,7 +51,7 @@ public async Task CreateChangelog_WithExtractReleaseNotes_ShortReleaseNote_UsesP var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), ExtractReleaseNotes = true @@ -115,7 +115,7 @@ public async Task CreateChangelog_WithExtractReleaseNotes_LongReleaseNote_UsesAs var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), ExtractReleaseNotes = true @@ -179,7 +179,7 @@ public async Task CreateChangelog_WithExtractReleaseNotes_MultiLineReleaseNote_U var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), ExtractReleaseNotes = true @@ -243,7 +243,7 @@ public async Task CreateChangelog_WithExtractReleaseNotes_NoReleaseNote_UsesPrTi var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), ExtractReleaseNotes = true @@ -311,7 +311,7 @@ public async Task CreateChangelog_WithExtractReleaseNotes_ExplicitTitle_TakesPre var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), ExtractReleaseNotes = true, @@ -376,7 +376,7 @@ public async Task CreateChangelog_WithExtractReleaseNotes_ExplicitDescription_Ta var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), ExtractReleaseNotes = true, @@ -443,7 +443,7 @@ public async Task CreateChangelog_WhenExtractNotSpecifiedByCli_UsesConfigExtract var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), ExtractReleaseNotes = null // CLI did not specify; config default applies @@ -515,7 +515,7 @@ public async Task CreateChangelog_InCI_ExtractionEnabledByConfig_PreservesCIDesc var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), ExtractReleaseNotes = null @@ -582,7 +582,7 @@ public async Task CreateChangelog_InCI_ExtractionDisabledByCli_ClearsCIDescripti var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), ExtractReleaseNotes = false @@ -669,7 +669,7 @@ public async Task CreateChangelog_InCI_MultiplePrs_ExtractionDisabled_ClearsCIDe "https://github.com/elastic/elasticsearch/pull/100", "https://github.com/elastic/elasticsearch/pull/200" ], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = output, ExtractReleaseNotes = null, @@ -743,7 +743,7 @@ public async Task CreateChangelog_InCI_ExtractionDisabledByConfig_ClearsCIDescri var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), ExtractReleaseNotes = null @@ -813,7 +813,7 @@ public async Task CreateChangelog_InCI_CliEnablesExtraction_OverridesConfigFalse var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), ExtractReleaseNotes = true @@ -879,7 +879,7 @@ public async Task CreateChangelog_InCI_ExtractionDisabled_ExplicitCliDescription var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), ExtractReleaseNotes = false, diff --git a/tests/Elastic.Changelog.Tests/Changelogs/Create/TitleProcessingTests.cs b/tests/Elastic.Changelog.Tests/Changelogs/Create/TitleProcessingTests.cs index e25be7667a..c5fc697b8c 100644 --- a/tests/Elastic.Changelog.Tests/Changelogs/Create/TitleProcessingTests.cs +++ b/tests/Elastic.Changelog.Tests/Changelogs/Create/TitleProcessingTests.cs @@ -48,7 +48,7 @@ public async Task CreateChangelog_WithStripTitlePrefix_RemovesSquareBracketsAndC var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), StripTitlePrefix = true @@ -110,7 +110,7 @@ public async Task CreateChangelog_WithStripTitlePrefix_RemovesSquareBracketsWith var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), StripTitlePrefix = true @@ -171,7 +171,7 @@ public async Task CreateChangelog_WithStripTitlePrefix_RemovesMultipleSquareBrac var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), StripTitlePrefix = true @@ -231,7 +231,7 @@ public async Task CreateChangelog_WithStripTitlePrefix_StripsKibanaStyleTeamHyph var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/kibana/pull/238555"], - Products = [new ProductArgument { Product = "kibana", Target = "9.2.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "kibana", Lifecycle = "ga" }], Config = configPath, Output = CreateOutputDirectory(), StripTitlePrefix = true @@ -277,7 +277,7 @@ public async Task CreateChangelog_WithExplicitTitle_OverridesPrTitle() Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], Title = "Custom Title Override", Type = "feature", - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Output = CreateOutputDirectory() }; @@ -314,7 +314,7 @@ public async Task CreateChangelog_WithIssues_CreatesValidYaml() { Title = "Fix multiple issues", Type = "bug-fix", - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Issues = [ "https://github.com/elastic/elasticsearch/issues/123", diff --git a/tests/Elastic.Changelog.Tests/Changelogs/Create/ValidationTests.cs b/tests/Elastic.Changelog.Tests/Changelogs/Create/ValidationTests.cs index 7f7021406b..c114be03a4 100644 --- a/tests/Elastic.Changelog.Tests/Changelogs/Create/ValidationTests.cs +++ b/tests/Elastic.Changelog.Tests/Changelogs/Create/ValidationTests.cs @@ -49,7 +49,7 @@ public async Task CreateChangelog_WithPrOptionButNoLabelMapping_ReturnsError() var input = new CreateChangelogArguments { Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Config = configPath, Output = CreateOutputDirectory() }; @@ -73,7 +73,7 @@ public async Task CreateChangelog_WithInvalidProduct_ReturnsError() { Title = "Test", Type = "feature", - Products = [new ProductArgument { Product = "invalid-product", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "invalid-product" }], Output = CreateOutputDirectory() }; @@ -96,7 +96,7 @@ public async Task CreateChangelog_WithInvalidType_ReturnsError() { Title = "Test", Type = "invalid-type", - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Output = CreateOutputDirectory() }; @@ -139,7 +139,7 @@ public async Task CreateChangelog_WithInvalidProductInAddBlockers_ReturnsError() { Title = "Test", Type = "feature", - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Config = configPath, Output = CreateOutputDirectory() }; @@ -248,7 +248,7 @@ public async Task CreateChangelog_WithValidProductInAddBlockers_Succeeds() { Title = "Test", Type = "feature", - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.2.0" }], + Products = [new ProductArgument { Product = "elasticsearch" }], Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], Config = configPath, Output = CreateOutputDirectory() diff --git a/tests/Elastic.Changelog.Tests/Creation/ChangelogCreationServiceTests.cs b/tests/Elastic.Changelog.Tests/Creation/ChangelogCreationServiceTests.cs index 24ecf84144..2bf755b121 100644 --- a/tests/Elastic.Changelog.Tests/Creation/ChangelogCreationServiceTests.cs +++ b/tests/Elastic.Changelog.Tests/Creation/ChangelogCreationServiceTests.cs @@ -239,7 +239,7 @@ public async Task CreateChangelog_OutputDoesNotContainBom() { Title = "Test BOM handling", Type = "feature", - Products = [new ProductArgument { Product = "elasticsearch", Target = "9.1.0", Lifecycle = "ga" }], + Products = [new ProductArgument { Product = "elasticsearch", Lifecycle = "ga" }], Prs = ["https://github.com/elastic/elasticsearch/pull/12345"], Config = Path.Join(Paths.WorkingDirectoryRoot.FullName, "config", "changelog.yml"), Output = tempOutput, diff --git a/tests/Elastic.Changelog.Tests/Reconciliation/FakeS3.cs b/tests/Elastic.Changelog.Tests/Reconciliation/FakeS3.cs index 5681b2a982..f71c8b61a8 100644 --- a/tests/Elastic.Changelog.Tests/Reconciliation/FakeS3.cs +++ b/tests/Elastic.Changelog.Tests/Reconciliation/FakeS3.cs @@ -23,6 +23,7 @@ namespace Elastic.Changelog.Tests.Reconciliation; /// internal sealed class FakeS3 { + private readonly Lock _lock = new(); private readonly Dictionary> _buckets = [with(StringComparer.Ordinal)]; @@ -110,10 +111,14 @@ public IReadOnlyList GetsFor(string bucket) => private ListObjectsV2Response List(ListObjectsV2Request request) { - ListCalls++; - OnList?.Invoke(ListCalls); - - var store = Store(request.BucketName); + int n; + lock (_lock) + n = ++ListCalls; + OnList?.Invoke(n); + + Dictionary store; + lock (_lock) + store = Store(request.BucketName).ToDictionary(kv => kv.Key, kv => kv.Value, StringComparer.Ordinal); var prefix = request.Prefix ?? string.Empty; var objects = new List(); var commonPrefixes = new SortedSet(StringComparer.Ordinal); @@ -154,11 +159,15 @@ private ListObjectsV2Response List(ListObjectsV2Request request) private GetObjectResponse Get(GetObjectRequest request) { - Gets.Add(request); - _gets++; - - if (!Store(request.BucketName).TryGetValue(request.Key, out var obj)) - throw NotFound(); + (string Content, string ETag) obj; + int n; + lock (_lock) + { + Gets.Add(request); + n = ++_gets; + if (!Store(request.BucketName).TryGetValue(request.Key, out obj)) + throw NotFound(); + } // Capture the response before the hook runs, so a hook that reseeds the key simulates a // write landing right after this read. @@ -167,14 +176,18 @@ private GetObjectResponse Get(GetObjectRequest request) ETag = $"\"{obj.ETag}\"", ResponseStream = new MemoryStream(Encoding.UTF8.GetBytes(obj.Content)) }; - AfterGet?.Invoke(request.Key, _gets); + AfterGet?.Invoke(request.Key, n); return response; } private GetObjectMetadataResponse Head(GetObjectMetadataRequest request) { - if (!Store(request.BucketName).TryGetValue(request.Key, out var obj)) - throw NotFound(); + (string Content, string ETag) obj; + lock (_lock) + { + if (!Store(request.BucketName).TryGetValue(request.Key, out obj)) + throw NotFound(); + } var response = new GetObjectMetadataResponse { @@ -186,40 +199,50 @@ private GetObjectMetadataResponse Head(GetObjectMetadataRequest request) private PutObjectResponse Put(PutObjectRequest request) { - _puts++; - BeforePut?.Invoke(_puts); + int n; + lock (_lock) + n = ++_puts; + BeforePut?.Invoke(n); - Puts.Add(request); + lock (_lock) + { + Puts.Add(request); - var store = Store(request.BucketName); - var exists = store.TryGetValue(request.Key, out var current); - if (request.IfNoneMatch == "*" && exists) - throw PreconditionFailed(); - if (request.IfMatch is { } ifMatch && (!exists || ifMatch.Trim('"') != current.ETag)) - throw PreconditionFailed(); + var store = Store(request.BucketName); + var exists = store.TryGetValue(request.Key, out var current); + if (request.IfNoneMatch == "*" && exists) + throw PreconditionFailed(); + if (request.IfMatch is { } ifMatch && (!exists || ifMatch.Trim('"') != current.ETag)) + throw PreconditionFailed(); - _ = Seed(request.BucketName, request.Key, request.ContentBody); + _ = Seed(request.BucketName, request.Key, request.ContentBody); + } return new PutObjectResponse(); } private DeleteObjectResponse Delete(DeleteObjectRequest request) { - _deletes++; - BeforeDelete?.Invoke(_deletes); - - Deletes.Add(request); + int n; + lock (_lock) + n = ++_deletes; + BeforeDelete?.Invoke(n); - var store = Store(request.BucketName); - var exists = store.TryGetValue(request.Key, out var current); - if (request.IfMatch is { } ifMatch) + lock (_lock) { - if (!exists) - throw NotFound(); - if (ifMatch.Trim('"') != current.ETag) - throw PreconditionFailed(); - } + Deletes.Add(request); - _ = store.Remove(request.Key); + var store = Store(request.BucketName); + var exists = store.TryGetValue(request.Key, out var current); + if (request.IfMatch is { } ifMatch) + { + if (!exists) + throw NotFound(); + if (ifMatch.Trim('"') != current.ETag) + throw PreconditionFailed(); + } + + _ = store.Remove(request.Key); + } return new DeleteObjectResponse(); } diff --git a/tests/Elastic.Changelog.Tests/Reconciliation/NoteAmendReconcilerTests.cs b/tests/Elastic.Changelog.Tests/Reconciliation/NoteAmendReconcilerTests.cs new file mode 100644 index 0000000000..7147428790 --- /dev/null +++ b/tests/Elastic.Changelog.Tests/Reconciliation/NoteAmendReconcilerTests.cs @@ -0,0 +1,286 @@ +// Licensed to Elasticsearch B.V under one or more agreements. +// Elasticsearch B.V licenses this file to you under the Apache 2.0 License. +// See the LICENSE file in the project root for more information + +using System.Text.Json; +using AwesomeAssertions; +using Elastic.Changelog.Reconciliation; +using Elastic.Documentation; +using Elastic.Documentation.Configuration.ReleaseNotes; +using Elastic.Documentation.ReleaseNotes; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Elastic.Changelog.Tests.Reconciliation; + +/// +/// Tests for . Each scenario exercises a discrete path through +/// ProcessVersionBundleAsync: late note, already shipped, already in human amend, missing +/// file annotations, note removed (delete path), idempotent redelivery, no published bundle. +/// +public class NoteAmendReconcilerTests +{ + private const string PublicBucket = "public-bucket"; + private const string Org = "elastic"; + private const string Repo = "elasticsearch"; + private const string Product = "elasticsearch"; + private const string Version = "9.3.0"; + + // Pool key prefix: changelog/elastic/elasticsearch/ + private static string NoteKey(string branch, string file) => + $"changelog/{Org}/{Repo}/{branch}/{file}"; + + private static string AmendNotesKey(string parentFile) => + $"bundle/{Product}/{Path.GetFileNameWithoutExtension(parentFile)}.amend-notes{Path.GetExtension(parentFile)}"; + + private static string RegistryKey() => $"bundle/{Product}/registry.json"; + + private static string BundleKey(string file) => $"bundle/{Product}/{file}"; + + // language=yaml + private const string NoteYaml = + "title: CVE security fix\n" + + "type: security\n" + + "products:\n" + + " - product: elasticsearch\n" + + " versions: [9.3.0]\n" + + " lifecycle: ga\n"; + + private readonly FakeS3 _s3 = new(PublicBucket); + private readonly NotesIndexReconciler _notesReconciler; + private readonly NoteAmendReconciler _reconciler; + + public NoteAmendReconcilerTests() + { + _notesReconciler = new NotesIndexReconciler( + NullLoggerFactory.Instance, _s3.Client, PublicBucket, retryBaseDelay: TimeSpan.Zero); + _reconciler = new NoteAmendReconciler( + NullLoggerFactory.Instance, _s3.Client, PublicBucket, _notesReconciler, + retryBaseDelay: TimeSpan.Zero); + } + + private static ChangelogScope NotesScope() + { + _ = ChangelogScope.TryCreateNotes(Org, Repo, out var scope); + return scope!; + } + + /// Builds a minimal parent bundle YAML with one PR entry that carries a file identity. + private static string ParentBundleYaml(params string[] entryFileNames) + { + var bundle = new Bundle + { + Products = [new BundledProduct(Product, target: Version, lifecycle: Lifecycle.Ga)], + Entries = [.. entryFileNames.Select(n => new BundledEntry + { + File = new BundledFile { Name = n, Checksum = "abc123" }, + Title = $"Entry for {n}", + Type = ChangelogEntryType.BugFix + })] + }; + return ReleaseNotesSerialization.SerializeBundle(bundle); + } + + /// Registry JSON listing the given bundles for the test product, all at . + private static string RegistryJson(params string[] files) + { + var bundles = files.Select(f => new ChangelogRegistryBundle { File = f, Target = Version }).ToList(); + var registry = new ChangelogRegistry { Product = Product, Bundles = bundles }; + return JsonSerializer.Serialize(registry, ChangelogRegistryJsonContext.Default.ChangelogRegistry); + } + + /// Registry JSON listing the given bundles with an explicit target version (use when the bundle should NOT match ). + private static string RegistryJsonWithTarget(string target, params string[] files) + { + var bundles = files.Select(f => new ChangelogRegistryBundle { File = f, Target = target }).ToList(); + var registry = new ChangelogRegistry { Product = Product, Bundles = bundles }; + return JsonSerializer.Serialize(registry, ChangelogRegistryJsonContext.Default.ChangelogRegistry); + } + + private static NotesIndex ReadNotesIndex(string json) => + JsonSerializer.Deserialize(json, NotesIndexJsonContext.Default.NotesIndex)!; + + private static IReadOnlyDictionary> NotesByVersion( + string version, params string[] paths) => + new Dictionary> + { + [version] = [.. paths.Select(p => new NoteIndexEntry { Path = p, BundleSeq = 0 })] + }; + + // ----------------------------------------------------------------------------------------- + + [Fact] + public async Task LateNote_NoBundleAmendYet_WritesAmendNotesSidecar() + { + const string parent = "elasticsearch-9.3.0.yaml"; + _s3.Seed(PublicBucket, RegistryKey(), RegistryJson(parent)); + _s3.Seed(PublicBucket, BundleKey(parent), ParentBundleYaml("main/pr-100.yaml")); + _s3.Seed(PublicBucket, NoteKey("main", "note-cve.yml"), NoteYaml); + + var notesByVersion = NotesByVersion(Version, "main/note-cve.yml"); + await _reconciler.ReconcileAsync(NotesScope(), notesByVersion, TestContext.Current.CancellationToken); + + // Amend sidecar must have been written. + _s3.Exists(PublicBucket, AmendNotesKey(parent)).Should().BeTrue("late note must produce an amend sidecar"); + + // Notes index must be re-written with bundle_seq = 2. + var indexKey = ChangelogKeys.NotesIndexKey(Org, Repo, Version); + _s3.Exists(PublicBucket, indexKey).Should().BeTrue("notes index must be re-written with bundle_seq values"); + var index = ReadNotesIndex(_s3.ContentOf(PublicBucket, indexKey)); + index.Notes.Should().ContainSingle().Which.BundleSeq.Should().Be(2); + } + + [Fact] + public async Task NoteShippedInParent_NoAmendWritten_SeqIsOne() + { + const string parent = "elasticsearch-9.3.0.yaml"; + // Parent bundle already contains the note by leaf name. + _s3.Seed(PublicBucket, RegistryKey(), RegistryJson(parent)); + _s3.Seed(PublicBucket, BundleKey(parent), ParentBundleYaml("main/note-cve.yml", "main/pr-100.yaml")); + _s3.Seed(PublicBucket, NoteKey("main", "note-cve.yml"), NoteYaml); + + var notesByVersion = NotesByVersion(Version, "main/note-cve.yml"); + await _reconciler.ReconcileAsync(NotesScope(), notesByVersion, TestContext.Current.CancellationToken); + + // No amend sidecar should be written. + _s3.Exists(PublicBucket, AmendNotesKey(parent)).Should().BeFalse("note already in parent → no amend needed"); + + // bundle_seq must be 1 (shipped in original bundle). + var indexKey = ChangelogKeys.NotesIndexKey(Org, Repo, Version); + var index = ReadNotesIndex(_s3.ContentOf(PublicBucket, indexKey)); + index.Notes.Should().ContainSingle().Which.BundleSeq.Should().Be(1); + } + + [Fact] + public async Task NoteShippedInHumanAmend_NoAmendNotesWritten_SeqIsOne() + { + const string parent = "elasticsearch-9.3.0.yaml"; + const string humanAmend = "elasticsearch-9.3.0.amend-1.yaml"; + + _s3.Seed(PublicBucket, RegistryKey(), RegistryJson(parent, humanAmend)); + _s3.Seed(PublicBucket, BundleKey(parent), ParentBundleYaml("main/pr-100.yaml")); + + // Human amend adds the note. + var amendBundle = new Bundle + { + Products = [new BundledProduct(Product, target: Version, lifecycle: Lifecycle.Ga)], + Entries = [new BundledEntry { File = new BundledFile { Name = "main/note-cve.yml", Checksum = "def456" }, Title = "CVE", Type = ChangelogEntryType.Security }] + }; + _s3.Seed(PublicBucket, BundleKey(humanAmend), ReleaseNotesSerialization.SerializeBundle(amendBundle)); + _s3.Seed(PublicBucket, NoteKey("main", "note-cve.yml"), NoteYaml); + + var notesByVersion = NotesByVersion(Version, "main/note-cve.yml"); + await _reconciler.ReconcileAsync(NotesScope(), notesByVersion, TestContext.Current.CancellationToken); + + _s3.Exists(PublicBucket, AmendNotesKey(parent)).Should().BeFalse("note already in human amend → reconciler amend-notes must not be created"); + + var indexKey = ChangelogKeys.NotesIndexKey(Org, Repo, Version); + var index = ReadNotesIndex(_s3.ContentOf(PublicBucket, indexKey)); + index.Notes.Should().ContainSingle().Which.BundleSeq.Should().Be(1); + } + + [Fact] + public async Task ParentBundleHasNoFileAnnotations_Skipped_SeqRemainsZero() + { + const string parent = "elasticsearch-9.3.0.yaml"; + _s3.Seed(PublicBucket, RegistryKey(), RegistryJson(parent)); + + // Parent bundle entries have no file blocks (hand-authored legacy format). + var handAuthored = new Bundle + { + Products = [new BundledProduct(Product, target: Version, lifecycle: Lifecycle.Ga)], + Entries = [new BundledEntry { Title = "Hand-authored entry, no file block", Type = ChangelogEntryType.BugFix }] + }; + _s3.Seed(PublicBucket, BundleKey(parent), ReleaseNotesSerialization.SerializeBundle(handAuthored)); + _s3.Seed(PublicBucket, NoteKey("main", "note-cve.yml"), NoteYaml); + + var notesByVersion = NotesByVersion(Version, "main/note-cve.yml"); + await _reconciler.ReconcileAsync(NotesScope(), notesByVersion, TestContext.Current.CancellationToken); + + // No amend written; shipped state is unknown. + _s3.Exists(PublicBucket, AmendNotesKey(parent)).Should().BeFalse("unknown shipped state → skip, no amend"); + + // Notes index is still re-written, but bundle_seq stays 0. + var indexKey = ChangelogKeys.NotesIndexKey(Org, Repo, Version); + var index = ReadNotesIndex(_s3.ContentOf(PublicBucket, indexKey)); + index.Notes.Should().ContainSingle().Which.BundleSeq.Should().Be(0); + } + + [Fact] + public async Task NoteRemovedFromIndex_ExistingAmendSidecarDeleted() + { + const string parent = "elasticsearch-9.3.0.yaml"; + _s3.Seed(PublicBucket, RegistryKey(), RegistryJson(parent)); + _s3.Seed(PublicBucket, BundleKey(parent), ParentBundleYaml("main/pr-100.yaml")); + + // Pre-existing amend-notes sidecar from a previous reconcile. + var staleAmend = new Bundle + { + Products = [new BundledProduct(Product, target: Version, lifecycle: Lifecycle.Ga)], + Entries = [new BundledEntry { File = new BundledFile { Name = "main/note-cve.yml", Checksum = "old" }, Title = "CVE", Type = ChangelogEntryType.Security }] + }; + _s3.Seed(PublicBucket, AmendNotesKey(parent), ReleaseNotesSerialization.SerializeBundle(staleAmend)); + + // No notes for this version (note was deleted from the pool). + var notesByVersion = NotesByVersion(Version); + await _reconciler.ReconcileAsync(NotesScope(), notesByVersion, TestContext.Current.CancellationToken); + + _s3.Exists(PublicBucket, AmendNotesKey(parent)).Should().BeFalse("stale amend sidecar must be deleted when no notes remain"); + _s3.Deletes.Should().ContainSingle().Which.Key.Should().Be(AmendNotesKey(parent)); + } + + [Fact] + public async Task Idempotent_SameStateRedelivered_NoSecondPut() + { + const string parent = "elasticsearch-9.3.0.yaml"; + _s3.Seed(PublicBucket, RegistryKey(), RegistryJson(parent)); + _s3.Seed(PublicBucket, BundleKey(parent), ParentBundleYaml("main/pr-100.yaml")); + _s3.Seed(PublicBucket, NoteKey("main", "note-cve.yml"), NoteYaml); + + var notesByVersion = NotesByVersion(Version, "main/note-cve.yml"); + + // First reconcile → amend sidecar written. + await _reconciler.ReconcileAsync(NotesScope(), notesByVersion, TestContext.Current.CancellationToken); + var putsAfterFirst = _s3.Puts.Count; + putsAfterFirst.Should().BeGreaterThan(0, "first pass must write the amend sidecar and the notes index"); + + // Second reconcile with the same state → content is identical → no additional PUTs. + await _reconciler.ReconcileAsync(NotesScope(), notesByVersion, TestContext.Current.CancellationToken); + var putsAfterSecond = _s3.Puts.Count; + + // The notes index re-write is idempotent too (same content, conditional PUT is a no-op). + // At most 1 extra put for the index (if the reconciler always writes it), and 0 for the amend sidecar. + var amendSidecarPuts = _s3.Puts.Count(p => p.Key == AmendNotesKey(parent)); + amendSidecarPuts.Should().Be(1, "amend sidecar must be written exactly once across both passes"); + } + + [Fact] + public async Task NoBundleForVersion_NoAmend_SeqRemainsZero() + { + // Registry exists for the product but contains no bundle that matches the version. + _s3.Seed(PublicBucket, RegistryKey(), RegistryJsonWithTarget("8.0.0", "elasticsearch-8.0.0.yaml")); + _s3.Seed(PublicBucket, BundleKey("elasticsearch-8.0.0.yaml"), ParentBundleYaml("main/pr-100.yaml")); + _s3.Seed(PublicBucket, NoteKey("main", "note-cve.yml"), NoteYaml); + + var notesByVersion = NotesByVersion(Version, "main/note-cve.yml"); + await _reconciler.ReconcileAsync(NotesScope(), notesByVersion, TestContext.Current.CancellationToken); + + // No amend sidecar: no matching bundle. + _s3.Puts.Should().NotContain(p => p.Key.Contains("amend-notes"), "no matching bundle → no amend possible"); + + // Notes index re-written with bundle_seq = 0. + var indexKey = ChangelogKeys.NotesIndexKey(Org, Repo, Version); + var index = ReadNotesIndex(_s3.ContentOf(PublicBucket, indexKey)); + index.Notes.Should().ContainSingle().Which.BundleSeq.Should().Be(0); + } + + [Fact] + public async Task NoProductsInBundleTree_NoAmend() + { + // Bundle tree is empty (no products listed under bundle/). + // No registry.json objects exist, so ListObjectsV2 returns no common prefixes. + var notesByVersion = NotesByVersion(Version, "main/note-cve.yml"); + await _reconciler.ReconcileAsync(NotesScope(), notesByVersion, TestContext.Current.CancellationToken); + + _s3.Puts.Should().NotContain(p => p.Key.Contains("amend-notes")); + } +} diff --git a/tests/Elastic.Changelog.Tests/Reconciliation/NotesIndexReconcilerTests.cs b/tests/Elastic.Changelog.Tests/Reconciliation/NotesIndexReconcilerTests.cs index b5a34ca9cc..1d85593c14 100644 --- a/tests/Elastic.Changelog.Tests/Reconciliation/NotesIndexReconcilerTests.cs +++ b/tests/Elastic.Changelog.Tests/Reconciliation/NotesIndexReconcilerTests.cs @@ -14,21 +14,28 @@ public class NotesIndexReconcilerTests { private const string PublicBucket = "public-bucket"; + /// New-format note using the versions: field. private const string NoteYaml = "title: Slow rollover known issue\n" + "type: known-issue\n" + "products:\n" + " - product: elasticsearch\n" + + " versions: [9.0.0]\n"; + + /// Legacy-format note using the obsolete target: field for backward-compat tests. + private const string LegacyNoteYaml = + "title: Legacy rollover known issue\n" + + "type: known-issue\n" + + "products:\n" + + " - product: elasticsearch\n" + " target: 9.0.0\n"; - private const string NoteYamlTwoTargets = + private const string NoteYamlTwoVersions = "title: Two-version known issue\n" + "type: known-issue\n" + "products:\n" + " - product: elasticsearch\n" + - " target: 9.0.0\n" + - " - product: elasticsearch\n" + - " target: 9.1.0\n"; + " versions: [9.0.0, 9.1.0]\n"; private readonly FakeS3 _s3 = new(PublicBucket); private readonly NotesIndexReconciler _reconciler; @@ -46,17 +53,31 @@ private static ChangelogScope NotesScope(string org = "elastic", string repo = " private void SeedNote(string branch, string fileName, string yaml) => _s3.Seed(PublicBucket, $"changelog/elastic/elasticsearch/{branch}/{fileName}", yaml); - private NotesIndex ReadIndex(string target) => + private NotesIndex ReadIndex(string version) => JsonSerializer.Deserialize( - _s3.ContentOf(PublicBucket, ChangelogKeys.NotesIndexKey("elastic", "elasticsearch", target)), + _s3.ContentOf(PublicBucket, ChangelogKeys.NotesIndexKey("elastic", "elasticsearch", version)), NotesIndexJsonContext.Default.NotesIndex)!; + // Helper that projects entries to their paths for compact assertions. + private static IEnumerable Paths(NotesIndex index) => index.Notes.Select(e => e.Path); + [Fact] - public void DirectYamlParse_NoteYaml_HasProducts() + public void DirectYamlParse_NoteYaml_HasVersions() { var dto = ReleaseNotesSerialization.GetEntryDeserializer().Deserialize(NoteYaml); dto.Products.Should().NotBeNullOrEmpty("YAML has products"); + dto.Products?[0].Versions.Should().BeEquivalentTo(["9.0.0"]); + } + + [Fact] + public void DirectYamlParse_LegacyTargetField_FallsBackToVersions() + { + // Existing notes in pools still carry `target:` — the reconciler must still read them. + var dto = ReleaseNotesSerialization.GetEntryDeserializer().Deserialize(LegacyNoteYaml); + dto.Products.Should().NotBeNullOrEmpty(); +#pragma warning disable CS0618 // testing backward-compat read of obsolete Target dto.Products?[0].Target.Should().Be("9.0.0"); +#pragma warning restore CS0618 } [Fact] @@ -70,18 +91,40 @@ public async Task ReconcileRepo_SingleNote_WritesIndex() _s3.Gets.Count.Should().BeGreaterThan(0, $"reconciler should have fetched the note; ListCalls={_s3.ListCalls} SeedExists={_s3.Exists(PublicBucket, "changelog/elastic/elasticsearch/main/note-slow-rollover.yml")}"); _s3.Puts.Count.Should().BeGreaterThan(0, $"reconciler should have written the index; ListCalls={_s3.ListCalls} Gets={_s3.Gets.Count}"); var index = ReadIndex("9.0.0"); - index.Notes.Should().BeEquivalentTo(["main/note-slow-rollover.yml"]); + Paths(index).Should().BeEquivalentTo(["main/note-slow-rollover.yml"]); } [Fact] - public async Task ReconcileRepo_NoteWithTwoTargets_AppearsInBothIndexes() + public async Task ReconcileRepo_SingleNote_DefaultsBundleSeqToZero() { - SeedNote("main", "note-two-targets.yml", NoteYamlTwoTargets); + SeedNote("main", "note-slow-rollover.yml", NoteYaml); await _reconciler.ReconcileRepoAsync(NotesScope(), TestContext.Current.CancellationToken); - ReadIndex("9.0.0").Notes.Should().BeEquivalentTo(["main/note-two-targets.yml"]); - ReadIndex("9.1.0").Notes.Should().BeEquivalentTo(["main/note-two-targets.yml"]); + var index = ReadIndex("9.0.0"); + index.Notes.Should().ContainSingle().Which.BundleSeq.Should().Be(0); + } + + [Fact] + public async Task ReconcileRepo_LegacyTargetNote_IndexedViaFallback() + { + // A note that still uses the old `target:` field must still be indexed. + SeedNote("main", "note-legacy.yml", LegacyNoteYaml); + + await _reconciler.ReconcileRepoAsync(NotesScope(), TestContext.Current.CancellationToken); + + Paths(ReadIndex("9.0.0")).Should().BeEquivalentTo(["main/note-legacy.yml"]); + } + + [Fact] + public async Task ReconcileRepo_NoteWithTwoVersions_AppearsInBothIndexes() + { + SeedNote("main", "note-two-versions.yml", NoteYamlTwoVersions); + + await _reconciler.ReconcileRepoAsync(NotesScope(), TestContext.Current.CancellationToken); + + Paths(ReadIndex("9.0.0")).Should().BeEquivalentTo(["main/note-two-versions.yml"]); + Paths(ReadIndex("9.1.0")).Should().BeEquivalentTo(["main/note-two-versions.yml"]); } [Fact] @@ -93,7 +136,7 @@ public async Task ReconcileRepo_SameNoteNameOnTwoBranches_BothInIndex() await _reconciler.ReconcileRepoAsync(NotesScope(), TestContext.Current.CancellationToken); var index = ReadIndex("9.0.0"); - index.Notes.Should().BeEquivalentTo([ + Paths(index).Should().BeEquivalentTo([ "9.0/note-slow-rollover.yml", "main/note-slow-rollover.yml" ]); @@ -129,7 +172,7 @@ public async Task ReconcileRepo_IndexPathsAreSorted() await _reconciler.ReconcileRepoAsync(NotesScope(), TestContext.Current.CancellationToken); var index = ReadIndex("9.0.0"); - index.Notes.Should().Equal(["9.0/note-a.yml", "main/note-b.yml"]); + Paths(index).Should().Equal(["9.0/note-a.yml", "main/note-b.yml"]); } [Fact] @@ -141,7 +184,20 @@ public async Task ReconcileRepo_BranchWithSlashInName_IsIncludedInIndex() await _reconciler.ReconcileRepoAsync(NotesScope(), TestContext.Current.CancellationToken); var index = ReadIndex("9.0.0"); - index.Notes.Should().BeEquivalentTo(["feature/my-fix/note-slow-rollover.yml"]); + Paths(index).Should().BeEquivalentTo(["feature/my-fix/note-slow-rollover.yml"]); + } + + [Fact] + public async Task ReconcileRepo_BranchWithSlashInName_FullPathInIndex() + { + // A feature branch with '/' in its name must index the full pool-relative path. + // The branch is derivable from the path (everything before the last '/'), so it is not stored. + SeedNote("feature/my-fix", "note-slow-rollover.yml", NoteYaml); + + await _reconciler.ReconcileRepoAsync(NotesScope(), TestContext.Current.CancellationToken); + + var index = ReadIndex("9.0.0"); + index.Notes.Should().ContainSingle().Which.Path.Should().Be("feature/my-fix/note-slow-rollover.yml"); } [Fact] @@ -150,7 +206,7 @@ public async Task ReconcileRepo_StaleTargetRemoved_OldIndexDeleted() // Pre-seed a stale notes-8.0.0.json index from a previous reconcile run. _s3.Seed(PublicBucket, ChangelogKeys.NotesIndexKey("elastic", "elasticsearch", "8.0.0"), /*lang=json,strict*/ - """{"notes":["old/note-stale.yml"]}"""); + """{"schema_version":1,"notes":[]}"""); // Only seed a note for 9.0.0. SeedNote("main", "note-slow-rollover.yml", NoteYaml); @@ -158,7 +214,7 @@ public async Task ReconcileRepo_StaleTargetRemoved_OldIndexDeleted() await _reconciler.ReconcileRepoAsync(NotesScope(), TestContext.Current.CancellationToken); // 9.0.0 index should be written. - ReadIndex("9.0.0").Notes.Should().BeEquivalentTo(["main/note-slow-rollover.yml"]); + Paths(ReadIndex("9.0.0")).Should().BeEquivalentTo(["main/note-slow-rollover.yml"]); // 8.0.0 index should have been deleted. _s3.Deletes.Should().ContainSingle() @@ -171,7 +227,7 @@ public async Task ReconcileRepo_NoNotes_DeletesAllExistingIndexes() // Pre-seed a stale notes index. _s3.Seed(PublicBucket, ChangelogKeys.NotesIndexKey("elastic", "elasticsearch", "9.0.0"), /*lang=json,strict*/ - """{"notes":["old/note-stale.yml"]}"""); + """{"schema_version":1,"notes":[]}"""); // No note files — just an unrelated changelog entry. _s3.Seed(PublicBucket, "changelog/elastic/elasticsearch/main/12345.yaml", "title: PR entry"); diff --git a/tests/Elastic.Changelog.Tests/Scrubbing/ScrubberProcessorTests.cs b/tests/Elastic.Changelog.Tests/Scrubbing/ScrubberProcessorTests.cs index 88485b469b..a647117999 100644 --- a/tests/Elastic.Changelog.Tests/Scrubbing/ScrubberProcessorTests.cs +++ b/tests/Elastic.Changelog.Tests/Scrubbing/ScrubberProcessorTests.cs @@ -39,8 +39,10 @@ public ScrubberProcessorTests() NullLoggerFactory.Instance, _s3.Client, PublicBucket, retryBaseDelay: TimeSpan.Zero, metrics: _metrics); var notesReconciler = new NotesIndexReconciler( NullLoggerFactory.Instance, _s3.Client, PublicBucket, sourceBucketName: PrivateBucket, retryBaseDelay: TimeSpan.Zero, metrics: _metrics); + var noteAmendReconciler = new NoteAmendReconciler( + NullLoggerFactory.Instance, _s3.Client, PublicBucket, notesReconciler, retryBaseDelay: TimeSpan.Zero, metrics: _metrics); _processor = new ScrubberProcessor( - NullLoggerFactory.Instance, _s3.Client, PublicBucket, _scrubber, reconciler, shallowReconciler, notesReconciler, _metrics); + NullLoggerFactory.Instance, _s3.Client, PublicBucket, _scrubber, reconciler, shallowReconciler, notesReconciler, noteAmendReconciler, _metrics); } private Cancel Ctx => TestContext.Current.CancellationToken; diff --git a/tests/Elastic.Documentation.Configuration.Tests/ReleaseNotes/CdnChangelogEntryFetcherTests.cs b/tests/Elastic.Documentation.Configuration.Tests/ReleaseNotes/CdnChangelogEntryFetcherTests.cs index d3420acb1c..8bcbd4021a 100644 --- a/tests/Elastic.Documentation.Configuration.Tests/ReleaseNotes/CdnChangelogEntryFetcherTests.cs +++ b/tests/Elastic.Documentation.Configuration.Tests/ReleaseNotes/CdnChangelogEntryFetcherTests.cs @@ -211,7 +211,7 @@ public async Task FetchNotesAsync_HappyPath_FetchesAllListedNotes() { var path = req.RequestUri!.AbsolutePath; if (path.EndsWith("/notes-9.0.0.json", StringComparison.Ordinal)) - return Json(/*lang=json,strict*/ """{"notes":["main/note-slow-rollover.yml","9.0/note-gap.yml"]}"""); + return Json(/*lang=json,strict*/ """{"schema_version":1,"notes":[{"path":"main/note-slow-rollover.yml","bundle_seq":0},{"path":"9.0/note-gap.yml","bundle_seq":0}]}"""); return Yaml(SampleEntry); }); var (errors, _, emitError, _) = Diagnostics(); @@ -234,7 +234,7 @@ public async Task FetchNotesAsync_ListedNoteNotFound_EmitsErrorAndReturnsEmpty() { var path = req.RequestUri!.AbsolutePath; if (path.EndsWith("/notes-9.0.0.json", StringComparison.Ordinal)) - return Json(/*lang=json,strict*/ """{"notes":["main/note-missing.yml"]}"""); + return Json(/*lang=json,strict*/ """{"schema_version":1,"notes":[{"path":"main/note-missing.yml","bundle_seq":0}]}"""); return new HttpResponseMessage(HttpStatusCode.NotFound); }); var (errors, _, emitError, _) = Diagnostics(); @@ -249,7 +249,7 @@ public async Task FetchNotesAsync_ListedNoteNotFound_EmitsErrorAndReturnsEmpty() [Fact] public async Task FetchNotesAsync_EmptyIndex_ReturnsEmpty() { - var handler = new StubHandler(_ => Json(/*lang=json,strict*/ """{"notes":[]}""")); + var handler = new StubHandler(_ => Json(/*lang=json,strict*/ """{"schema_version":1,"notes":[]}""")); var (errors, _, emitError, _) = Diagnostics(); using var fetcher = CreateFetcher(handler);