[AIT-1204] fix(liveobjects): reject tombstoning of the root object (RTLO4e10, RTO10c1b1) - #2270
Conversation
A faulty OBJECT_DELETE operation (or a tombstoned object state) targeting
the root object would tombstone it: data cleared, all subscriptions
deregistered, and - tombstone being a terminal state - no re-sync could
ever resurrect it. After the GC grace period the sweep would then evict
root from the ObjectsPool entirely, violating RTO3b; root cannot be
re-created as a zero-value object ('root' is not a parseable object id),
so every subsequent objects operation on the channel fails unrecoverably.
Per RTLO4e the realtime system only tombstones orphaned or uninitialized
objects - root is by definition neither - so such a message indicates a
fault and is now rejected client-side, consistent with the existing
warn-and-skip handling for other invalid inbound operations.
- LiveObject.tombstone() rejects the root object with a warning and a
noop update (RTLO4e10); single choke point covering both the RTLO5
operation path and the RTLC6f/RTLM6f sync paths
- ObjectsPool._onGCInterval() never evicts root (RTO10c1b1, mirroring
the existing RTO5c2a guard in deleteExtraObjectIds); root's tombstoned
map entries still get RTLM19 entry-level GC
- overrideWithObjectState() in LiveMap/LiveCounter returns the tombstone
branch early since tombstone() can now return a noop update
Derived UTS tests: RTLO4e10 operation/sync path tests in live_map.test.ts,
channel-level RTO10c1b1 GC test in realtime_object.test.ts.
Spec twin: ably/specification#509
Raised in #2219 review discussion r3558197916
WalkthroughRoot tombstone and delete operations now return no-op updates instead of removing the root object. Garbage collection also excludes the root, while map and counter state overrides use early-return tombstone handling. Tests cover root targeting and post-GC root updates. ChangesRoot object protection
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant ObjectState
participant LiveMap
participant LiveObject
ObjectState->>LiveMap: overrideWithObjectState(tombstone)
LiveMap->>LiveObject: tombstone(objectMessage)
LiveObject-->>LiveMap: noop update
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/uts/objects/unit/realtime_object.test.ts`:
- Around line 924-925: Update the GC completion poll in the realtime object test
to check whether root.get('score') is undefined directly, rather than calling
value() on the missing object. Preserve the existing pollUntil flow and
completion condition.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 8e32b3f1-4075-4f4a-9e92-85feb69d7735
📒 Files selected for processing (6)
src/plugins/liveobjects/livecounter.tssrc/plugins/liveobjects/livemap.tssrc/plugins/liveobjects/liveobject.tssrc/plugins/liveobjects/objectspool.tstest/uts/objects/unit/live_map.test.tstest/uts/objects/unit/realtime_object.test.ts
There was a problem hiding this comment.
Pull request overview
This PR hardens the LiveObjects implementation against faulty inbound operations/state that attempt to tombstone (and later GC-evict) the special root object, ensuring root remains present and usable across sync and GC cycles.
Changes:
- Reject attempts to tombstone the root object by returning a noop update and logging a warning.
- Add an additional safeguard to GC so the root object is never removed from the
ObjectsPool. - Add/extend UTS unit coverage for root-targeted deletes/state overrides and for GC behavior preserving root.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/uts/objects/unit/realtime_object.test.ts | Adds a GC regression test proving root survives a GC sweep even when a rogue delete is received. |
| test/uts/objects/unit/live_map.test.ts | Adds tests asserting root-targeted tombstone/delete paths are rejected and yield noop updates. |
| src/plugins/liveobjects/objectspool.ts | Prevents GC from evicting the root object from the pool. |
| src/plugins/liveobjects/liveobject.ts | Rejects tombstoning of the root object (warn + noop) and widens return types accordingly. |
| src/plugins/liveobjects/livemap.ts | Refactors tombstone override path to return early (supporting noop updates). |
| src/plugins/liveobjects/livecounter.ts | Refactors tombstone override path to return early (supporting noop updates). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ejection warning A rejected root tombstone indicates a faulty inbound message, and the site/serial is what identifies its origin; matches the adjacent RTLO4a3 warning style. Addresses review feedback on #2270.
Fixes https://ably.atlassian.net/browse/AIT-1204
Problem
Raised by the review discussion on #2219 (r3558197916):
deleteExtraObjectIdsprotects the root object during sync cleanup (RTO5c2a), but the GC sweep in_onGCIntervaldeletes any tombstoned object past the grace period — including root.Tracing it showed the gap starts earlier: nothing prevented root from being tombstoned at all. If a faulty
OBJECT_DELETE(or a tombstoned object state during sync) ever targetedroot:overrideWithObjectStateskips tombstoned objects), so no re-sync or re-attach ever recovers it.getRoot()returnsundefined,resetToInitialPool()throws on the next sync, and root can never be re-created as a zero-value object (ObjectId.fromString('root')throws, which would also abort the wholeProtocolMessagebatch).Per RTLO4e the realtime system only tombstones orphaned or uninitialized objects — root is by definition neither — so such a message indicates a fault and is now rejected client-side with a warning, consistent with the existing warn-and-skip handling for invalid inbound operations (RTLO4a3, RTLC7d3/RTLM15d4).
Changes
Source
liveobject.tstombstone()rejects the root object: warning + noop update, no state change. Single choke point — covers theOBJECT_DELETEoperation path (RTLO5) and both sync paths (RTLC6f/RTLM6f). Return type widened toTUpdate | LiveObjectUpdateNoop.objectspool.ts_onGCInterval()never evicts root from the pool (mirrors the RTO5c2a guard). Root's tombstoned map entries still get entry-level GC.livemap.ts/livecounter.tsoverrideWithObjectState()returns the tombstone branch early (it can now yield a noop update); the non-tombstone flow is unchanged, just de-nested.Derived UTS tests
live_map.test.ts—RTLO4e10/object-delete-root-noop-0andRTLO4e10/replace-data-tombstone-root-noop-0: root not tombstoned, data untouched, noop update produced.realtime_object.test.ts—RTO10c1b1/gc-root-never-removed-0: a rogue backdatedOBJECT_DELETEon root, then a full GC sweep (observed via a control counter's eviction), then aMAP_SETthat still applies — proving root survived the sweep and stayed live. Header deviation note extended to cover the GC rendering.Spec twin
ably/specification#509 adds the RTLO4e10/RTO10c1b1 clauses and the UTS test specs, and repoints the three existing tombstone-mechanics test specs (RTLO5, RTLM6f, RTLO4e9) from
objectId: "root"to a non-root map — the derived tests in this repo already used non-root ids, so no changes to those tests were needed here.The same guards are implemented for ably-java (PR to follow).
Testing
live_map.test.ts(41 passing),realtime_object.test.ts(incl. new GC test); full UTS objects unit sweep 316 passing.test/realtime/liveobjects.test.jsagainst sandbox: 510 passing, 0 failing (no legacy test tombstones root — verified by sweep; existingtombstone: trueusages are entry-level only).tsc --noEmitand prettier clean.Summary by CodeRabbit