Cover D/R deletion and in-place overwrite in the PRA test - #2529
Open
delthas wants to merge 3 commits into
Open
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # yq eval 'sortKeys(.)' -i deps.yaml | ||
| backbeat: | ||
| sourceRegistry: ghcr.io/scality | ||
| sourceRegistry: ghcr.io/scality/playground/delthas |
There was a problem hiding this comment.
backbeat points to a personal playground registry (ghcr.io/scality/playground/delthas) and a branch-build tag. drctl (line 24) and zenko-operator (line 134) also carry unreleased commit-hash tags instead of published release tags.
The PR description says these are temporary and will be replaced before merge — flagging so the PR isn't accidentally merged with these values.
delthas
force-pushed
the
wip/ZKOP-562/pra-delete-and-overwrite-tests
branch
from
September 7, 2026 16:12
d333eb2 to
a87698b
Compare
The examples table carried `Non versioned` alone until 137e69d swapped it for `Versioned`, in the same commit that added the cross-site restore steps. Nothing records why: the commit message says only "Complete PRA logic", the pull request body names the tests it was missing, its review discussed the zenkoversion, the locations and the scripts, and the ticket has no description at all. Nor does the code suggest a reason -- the restore helper that commit extracted keeps the version id optional, so it runs against a bucket that has none. Since the swap, nothing has restored or failed over an object whose bucket holds no version to carry its state, which is where a restore writes `x-amz-restore` onto a master with nothing beneath it. Both rows rather than the swap back: the version delete this scenario covers needs a bucket with versions to delete. Issue: ZKOP-562
delthas
force-pushed
the
wip/ZKOP-562/pra-delete-and-overwrite-tests
branch
3 times, most recently
from
September 10, 2026 08:22
30d1a8b to
ffc9e1c
Compare
Three paths the PRA scenario never touched: it deletes nothing, and its only bucket is versioned, so an object is never overwritten in place and never deleted without a version to remove. None of the deletions D/R has never replicated, nor the metadata a non-versioned overwrite has to carry, is visible to CI today. Removing a version and overwriting in place are steps of the nominal scenario, which already holds the state they need. Removing an unversioned object gets a scenario of its own: a failing step abandons the rest of its scenario, and this is the one whose outcome on a released deployment is not yet known. @pra is assigned at most one pickle at a time, so scenarios cannot fight over the two clusters. Delete markers stay uncovered, as ZENKO-4886 records: a marker carries no placement, so the location the source selects on cannot match it. A suspended bucket gets a scenario of its own too. A put there writes a null version, which the delete addresses as a version even though the master may be the only document holding it, so what removes the object is the teardown of the placeholder left on that master -- a write the source makes in no other case. Issue: ZKOP-562
delthas
force-pushed
the
wip/ZKOP-562/pra-delete-and-overwrite-tests
branch
from
September 11, 2026 09:39
ffc9e1c to
711bbd4
Compare
Not for merge. Exercises ZKOP-562, BB-811 and DRCTL-83 end to end: the @pra CTST run replicates cold objects through the mongo-processor rather than the Kafka Connect mongo sink. - backbeat 5e8a9c0b (BB-811, the D/R processor mode, with an object rewritten in place taken whole, and BB-853 on top: the sink omits the metrics block, and the producer is a hard startup dependency without it) - zenko-operator cf7e60c4 (ZKOP-562, the sink deployment and the source pipeline that shapes the entries it reads, rebased on the 1.9 that carries ZKOP-588 -- so drctl no longer needs DRCTL-81's merge patch to get past the analytics section marshalling as a null) - drctl v1.1.0-preview.1 (carries DRCTL-83, the objects topic on both D/R CRs, and the merged DRCTL-79 that sets the backbeat image on the sink; its vendor tree carries an operator new enough that the analytics section marshals as absent rather than as a null the apiserver rejects) The retag prefix is 9.5.3 because that branch really is 9.5.2, and every operator gate on the backbeat tag sits either below 9.5.1 or at 9.6.0-preview.1, so it reads exactly as 9.5.1 did. drctl is pinned to a released tag rather than to the per-commit image its own CI pushes. backlogMetricsIntervalSeconds moves to configurationOverrides, which is what replaced it: ZKOP-569 added the field, and ZKOP-561 reverted it right after v1.8.16 in favour of the generic override, so the operator this pins rejects the field outright. The interval reaches the same config key through the env var backbeat derives from its schema, on the consumers whose lag the lifecycle conductor waits on. Without it the conductor idles 60-120s per batch, and a Days=0 transition on a versioned bucket takes three batches -- 319.6s measured, past the 300s the CTST transition steps allow.
delthas
force-pushed
the
wip/ZKOP-562/pra-delete-and-overwrite-tests
branch
from
September 11, 2026 10:57
711bbd4 to
3e46a8a
Compare
delthas
marked this pull request as ready for review
September 11, 2026 13:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The PRA scenario deleted nothing, and its only bucket was versioned — so an object was never overwritten in place, and never deleted without a version to remove. This adds those paths and points the solution at the builds that make them pass.
Scenarios
PRA (nominal case)—VersionedPRA (nominal case)—Non versionedPRA (suspended null version deletion)PRA (unversioned deletion)The
Non versionedrow was in the table until137e69dd91swapped it forVersioned, in the same commit that added the cross-site restore steps. Nothing records why — not the commit message, not the pull request, not the ticket — and the restore helper it extracted keeps the version id optional, so it runs against a bucket that has none. Since the swap, nothing has restored or failed over an object whose bucket holds no version to carry its state. Both rows run again.The two deletions get scenarios of their own rather than more steps in the nominal one: a failing step abandons the rest of its scenario, and
@PRAis assigned at most one pickle at a time, so scenarios cannot fight over the two clusters.Deleting an unversioned object already passes on
development/2.16, but for the wrong reason — the delete flag lands on the master, the document a version-lessHeadObjectreads, so the object reads as absent while its document stays behind forever. That row is a regression guard, not a demonstration.What the bumps point at
v1.1.0-preview.1— carries DRCTL-83, the objects topic on both D/R CRs, and the merged DRCTL-79 that sets the backbeat image on the sinkThe two backbeat PRs stay pinned as one build, since the sink needs both and #2833 has not merged. The bump commit is replaced by released versions before this merges, so expect it to be amended until then.
A known race, not fixed here
Bucket definitions reach the sink through the Kafka Connect entity sync; object entries reach it through the mongo-processor. Nothing orders the two. An object entry that arrives before its bucket gets
NoSuchBucket, and the offset is committed regardless — so that object's metadata is dropped for good, and the scenario fails on an assertion no amount of waiting can satisfy.We have not tried to fix it here. The entity path this races is expected to be retired.
Issue: ZKOP-562