fix(cron): preserve schedule pointer when persist_cloud_task fails

[lovitt]

TL;DR

If Schedule#persist_cloud_task raises (e.g., a transient Cloud Tasks API error), the cron schedule entry is left permanently broken until external re-registration: Redis points at a deleted task and the original task's Cloud Tasks retry silent-aborts via expected_instance? == false. We observed this 7 times in 30 days in production.

Fix: Schedule#save snapshots the prior Redis state and atomically rolls back (via WATCH/MULTI/EXEC) if persist_cloud_task raises; persist_cloud_task schedules the replacement task before deleting the old one. No public API changes. All existing tests pass plus 2 new regression tests.

Problem

When Cloudtasker::Cron::Schedule#save invokes persist_cloud_task, it advances the cron schedule's Redis pointer to the new task before the recursive Cron::Job#schedule! runs that actually creates the replacement task in the backend. If that recursive call raises for any reason (transient Cloud Tasks API error, network blip, instance termination), Redis is left pointing at a task that has been deleted by persist_cloud_task and never replaced.

The original task's Cloud Tasks retry then evaluates expected_instance? == false (because cron_schedule.job_id was advanced past the original's job_id) and silent-aborts. The schedule entry has no live task in the queue and no path to create one. It stays in this state until Schedule.load_from_hash! runs again.

Production observation

We deploy a Rails app on Google Cloud Run with horizontal autoscaling. Cloudtasker schedules are installed via Puma's after_booted callback (equivalent to the on_booted example in docs/CRON_JOBS.md), so each Cloud Run instance runs Schedule.load_from_hash! at boot.

Over a 30-day window we identified 7 incidents where one specific * * * * * cron schedule stopped firing for 28–46 minutes. The last invocation before each gap completed in 8–37 ms, in contrast to the worker's normal 500–3000 ms. Our worker has an app-level guard that returns early if its cloudtasker/cron/job/time_at job_meta is more than 5 minutes behind wall clock; the rapid exits indicate this guard fired — meaning the cron's time_at had already drifted significantly past wall-clock time before the chain broke.

Incident (UTC)	Gap	Last invocation
2026-04-06 16:14 → 17:00	46 min	21 ms
2026-04-06 23:08 → 23:45	37 min	37 ms
2026-04-07 17:32 → 18:06	34 min	24 ms
2026-04-09 21:03 → 21:32	29 min	15 ms
2026-04-13 04:16 → 05:02	46 min	17 ms
2026-04-14 07:01 → 07:38	37 min	23 ms
2026-05-01 18:17 → 18:45	28 min	8 ms

Every incident window contained Cloud Run cold-start clusters (multiple Puma boots within seconds), which trigger Schedule.load_from_hash!. Across the same 30-day window we observed ~1,342 cold-start clusters total; only the 7 listed coincided with a chain break, so boot clusters alone are not sufficient.

We have not directly proven that the failure mode described above is the cause of every one of these incidents. We identified it as a code path in the gem that can produce the observed symptom, and the regression tests below demonstrate it deterministically.

Code path

In lib/cloudtasker/cron/schedule.rb:

1. Cron::Job#schedule! creates successor task B and calls cron_schedule.update(task_id: B.id, job_id: B.job_id).
2. Schedule#save runs redis.write(gid, to_h) — Redis now points at B.
3. Schedule#save evaluates the safety check; if CloudTask.find(B.id) returns nil, persist_cloud_task is invoked.
4. persist_cloud_task calls CloudTask.delete(B.id), removing B.
5. persist_cloud_task runs a recursive Job.new(worker_instance).set(schedule_id: id).schedule! to create a replacement.
6. If the recursive call raises before creating the replacement, the exception propagates. B is deleted, no replacement exists, but cron_schedule.job_id is still B from step 2.
7. Cloud Tasks retries the original task. Cron::Job#execute evaluates expected_instance? = retry_instance? || (cron_schedule.job_id == job_id). Both are false (retry_instance? because flag(:processing) was never reached; the pointer comparison because of step 2). The retry returns true silently — no further dispatches, no schedule! call.

Fix

Two changes in lib/cloudtasker/cron/schedule.rb:

Schedule#save snapshots the previous Redis state before writing and rolls it back if persist_cloud_task raises. The rollback runs inside a Redis WATCH/MULTI/EXEC transaction so a concurrent process that updated the pointer in the meantime is not clobbered — Redis aborts the transaction and the concurrent update is preserved. The original task's Cloud Tasks retry then matches expected_instance? and calls schedule! to repair the chain via the standard cron iteration path.

Schedule#persist_cloud_task creates the replacement task before deleting the old one. If the recursive schedule! raises, the existing task in Cloud Tasks is preserved (worst case: one extra silent-aborted dispatch when the orphan task fires).

Safety

• Success path is unchanged. If persist_cloud_task succeeds, the rescue clause is never reached.
• Failure path is now recoverable. Without the fix, the schedule is unrecoverable until external re-registration. With the fix, the original task's standard Cloud Tasks retry repairs the chain.
• Create-then-delete preserves the old task on creation failure. In the rare case where create succeeds but the subsequent delete fails, both tasks reach Cron::Job#execute; one matches expected_instance? and runs, the other silent-aborts. Net cost: one extra silent-aborted dispatch.
• Rollback is race-free against concurrent writers. WATCH/MULTI/EXEC ensures that if another process updates the schedule pointer between our snapshot and our rollback, the rollback transaction is aborted by Redis and the concurrent update is preserved.
• All existing tests pass. The 4 Schedule#save test contexts in spec/cloudtasker/cron/schedule_spec.rb continue to assert their existing behavior. Total: 688 examples, 0 failures.

Regression tests

spec/cloudtasker/cron/job_spec.rb adds two tests under a describe '#execute (chain-break regression)' block:

1. Chain repair via Cloud Tasks retry. Forces the persist_cloud_task path, makes the recursive CloudTask.create raise, then verifies that on Cloud Tasks' retry of the original task, schedule! is called to repair the chain. Without the rollback fix, the retry silent-aborts and this assertion fails.

2. No clobber on concurrent update. Simulates a concurrent process writing a different pointer to the same gid between the outer save's redis.write and the persist_cloud_task failure. Verifies the rollback leaves the concurrent update in place rather than restoring previous_state. Without the WATCH/MULTI/EXEC guard, the rollback unconditionally clobbers the concurrent write and this assertion fails.

Compatibility

No public API changes, no new dependencies, no configuration changes. Both modifications are contained in private methods of Cloudtasker::Cron::Schedule. The WATCH/MULTI/EXEC rollback reaches into redis.client.with for direct connection access — a pattern not currently used elsewhere in the gem, but it is the standard Redis idiom for atomic compare-and-swap.