CASSSIDECAR-377: Implement job coordination for cluster-wide operations by andresbeckruiz · Pull Request #360 · apache/cassandra-sidecar

andresbeckruiz · 2026-06-10T20:50:52Z

CASSSIDECAR-377

Original PR made against CASSSIDECAR-373 branch with review comments: andresbeckruiz#2.

Changes

OperationalJobCoordinator interface
StorageOperationalJobCoordinator: Implementation that delegates to StorageProvider's compare and set based methods to acquire a lock for an active operation
OperationalJob.operationType(): New abstract method returning OperationType enum, implemented by all concrete jobs
OperationalJobManager integration: If a job requires coordination, the manager acquires the active lock via the coordinator before execution; throws OperationalJobConflictException if rejected
Unit tests for StorageOperationalJobCoordinator and coordinator integration in OperationalJobManagerTest

This ticket only covers the job activation path when a job is submitted to be executed immediately. Clearing locks after completion and status update activation will be implemented in later tickets (CASSSIDECAR-378, CASSSIDECAR-379).

Future Work

CASSSIDECAR-378: Add support for creation and coordination of local Sidecar Jobs
CASSSIDECAR-379: Add PATCH operational-jobs/{id} API to update status of existing operational jobs

pauloricardomg · 2026-06-12T20:49:33Z


            // New job is submitted for all cases when we do not have a corresponding downstream job
-            jobTracker.computeIfAbsent(job.jobId(), jobId -> {
+            OperationalJob tracked = jobTracker.computeIfAbsent(job.jobId(), jobId -> job);


do we need to remove from jobTracker if there's a coordination conflict ? Or perhaps should we just add to jobTracker in case the tryCoordination was successful, so tryCoordination should come before jobtracker.computeIfAbsent ?

This leaves stale job entry in CREATED state if the coordination fails.

Would it make sense to mark the job as FAILED if coordination fails? And set the failureReason to indicate that the job was not able to start due to coordination failure?

Coordination now happens before tracking and jobs that fail coordination are now marked as FAILED, addressed in 3661c4a

skoppu22 · 2026-06-18T15:10:57Z

+    {
+        if (job.requiresCoordination())
+        {
+            Preconditions.checkState(coordinator != null,


Preconditions throws illegal state exception, which is not handled in the caller

No more unhandled IllegalStateException, addressed in 3661c4a

skoppu22 · 2026-06-18T15:13:34Z

+        };
+
+        manager.trySubmitJob(job, onComplete, executorPool.service(), SecondBoundConfiguration.parse("5s"));
+        assertThat(latch.await(10, TimeUnit.SECONDS)).isTrue();


Also verify that tracker doesn't have entry for this job after conflict detected.

Jobs are now marked as FAILED if coordination fails, so no conflict will happen as inflightJobsByOperation does not return completed jobs. Added

assertThat(tracker.inflightJobsByOperation(job.name())).doesNotContain(job);

in 3661c4a to confirm

skoppu22 · 2026-06-18T15:22:48Z

+        {
+            Preconditions.checkState(coordinator != null,
+                                     "Job requires coordination but no OperationalJobCoordinator is configured");
+            boolean activated = coordinator.trySetActive(job.operationType(), job.jobId());


I am afraid to merge this PR without calling clearActive. If we delay adding that in a subsequent PR and if someone enables requiresCoordination() for any job meanwhile, then no more further operations will be allowed. I would recommend implementing calling clearActive in the same PR.

Added in 097a9af

andresbeckruiz · 2026-07-07T20:38:55Z

CI: https://app.circleci.com/pipelines/github/andresbeckruiz/cassandra-sidecar/101/workflows/d27734e4-047e-4299-9029-ed0c9624184c

isaacreath · 2026-07-08T13:07:31Z

+    @Nullable
+    private final OperationalJobCoordinator coordinator;


The nullable OperationalJobCoordinator feels like a bit of a smell to me. It seems like the goal here is to support backwards compatibility for the OperationalJobManager for users of the OperationalJobManager to not have to include coordination (e.g., for existing decommission jobs), am I right about that?

If so, I'd suggest that instead of having a null OperationalJobCoordinator, we have a second implementation, DisabledOperationalJobCoordinator or something similar, that fails all coordination tasks with an error saying that coordination is not currently supported by this sidecar instance (and maybe how to configure coordination if the user wants to have coordinated jobs).

This way we make the coordination enabled / disabled logic part of the behavior of the coordinator itself, as opposed to having a null dependency here. It can also clean up all accesses to the OperationalJobCoordinator as we no longer have to worry about it being null.

Good idea. Replaced the @Nullable OperationalJobCoordinator with a DisabledOperationalJobCoordinator in 405a93d

To clarify, backwards compatibility for existing uncoordinated jobs (eg: decommission) is already handled by requiresCoordination() == false, so they short circuit before touching the coordinator. Therefore, the DisabledOperationalJobCoordinator only matters for a job requiring coordination submitted to an instance without coordination support.

Also updated this patch to bind the OperationalJobCoordinator to the DisabledOperationalJobCoordinator. Configuration to enable this class (as well as durable tracking) will be added in CASSSIDECAR-378.

isaacreath

+1 (nb)

arjunashok · 2026-07-09T20:09:15Z

+        {
+            if (ar.succeeded() && Boolean.TRUE.equals(ar.result()))
+            {
+                job.asyncResult().onComplete(result -> releaseActiveOperationLock(job));


requiresCoordination() is a clean job-level flag the manager just reads.

The release side isn't symmetric though, as trySubmitJob always auto-releases on local completion, with no way for a job to opt out. That's right for today's single-node jobs, but doesn't match clearActive's javadoc (orchestration-layer clearing for jobs that finish elsewhere).

Rather than just documenting that gap, could we add a parallel hook for release?
e.g. releasesOnCompletion(), default true; the manager just reads it instead of special-casing job types, so a future distributed job can skip the automatic release without any change to OperationalJobManager.

Good catch-- addressed in 5d5afc6

andresbeckruiz force-pushed the CASSSIDECAR-377 branch 2 times, most recently from 0e36849 to 817bf51 Compare June 12, 2026 19:22

pauloricardomg reviewed Jun 12, 2026

View reviewed changes

skoppu22 reviewed Jun 18, 2026

View reviewed changes

andresbeckruiz force-pushed the CASSSIDECAR-377 branch from a2bd0e1 to 30a4292 Compare July 7, 2026 18:57

andresbeckruiz requested review from isaacreath, pauloricardomg and skoppu22 July 7, 2026 19:29

isaacreath suggested changes Jul 8, 2026

View reviewed changes

pauloricardomg approved these changes Jul 8, 2026

View reviewed changes

skoppu22 approved these changes Jul 9, 2026

View reviewed changes

andresbeckruiz force-pushed the CASSSIDECAR-377 branch 4 times, most recently from e4bf6fa to 509b15b Compare July 9, 2026 16:15

andresbeckruiz requested a review from isaacreath July 9, 2026 16:31

isaacreath approved these changes Jul 9, 2026

View reviewed changes

arjunashok reviewed Jul 9, 2026

View reviewed changes

andresbeckruiz requested a review from arjunashok July 9, 2026 21:21

arjunashok approved these changes Jul 9, 2026

View reviewed changes

andresbeckruiz force-pushed the CASSSIDECAR-377 branch from 1bd9057 to 58bb7d4 Compare July 14, 2026 16:35

andresbeckruiz added 6 commits July 15, 2026 10:39

CASSSIDECAR-377: Implement job coordination for cluster-wide operations

804817e

Address PR review comments

b492549

Mark OperationalJob as FAILED if coordination fails

3661c4a

Implement releaseActiveOperationLock

097a9af

Implement DisabledOperationalJobCoordinator

405a93d

Add releaseOnCompletion hook to OperationalJob

5d5afc6

andresbeckruiz force-pushed the CASSSIDECAR-377 branch from 58bb7d4 to 5d5afc6 Compare July 15, 2026 14:45

		@Nullable
		private final OperationalJobCoordinator coordinator;

Uh oh!

Conversation

andresbeckruiz commented Jun 10, 2026

Changes

Future Work

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andresbeckruiz Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andresbeckruiz Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andresbeckruiz Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andresbeckruiz Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andresbeckruiz commented Jul 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andresbeckruiz Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

isaacreath left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andresbeckruiz Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

andresbeckruiz Jul 7, 2026 •

edited

Loading

andresbeckruiz Jul 7, 2026 •

edited

Loading

andresbeckruiz Jul 7, 2026 •

edited

Loading

andresbeckruiz Jul 7, 2026 •

edited

Loading

andresbeckruiz Jul 9, 2026 •

edited

Loading

andresbeckruiz Jul 9, 2026 •

edited

Loading