[api][runtime][python] Add opt-in Kafka tombstones for pruned action state by rob-9 · Pull Request #885 · apache/flink-agents

rob-9 · 2026-07-08T00:36:20Z

Addresses #691

Purpose of change

KafkaActionStateStore.pruneState() currently removes matching entries only from its in-memory cache. The underlying Kafka records remain, so the action-state topic grows indefinitely and replay can restore previously pruned state.

This change adds the opt-in kafkaActionStateTombstoneEnabled option, defaulting to false in Java and Python. When enabled, pruning writes null-valued records for matching state keys so compaction can reclaim them, and rebuildState() treats tombstones as deletions.

The option remains disabled by default because tombstones can invalidate checkpoints or savepoints older than the prune and cause completed actions to execute again.

The pruning path also:

matches parsed Flink keys exactly instead of relying only on prefixes
retains and reports state keys that cannot be parsed
reports asynchronous tombstone delivery failures while still evicting cache entries
avoids an unnecessary producer flush

Flink keys containing _ cannot currently be parsed safely during pruning. Kafka and Fluss retain those entries, and Kafka emits no tombstones for them.

Tests

Passed with JDK 17 and Python 3.12:

KafkaActionStateStoreTest: 22 passed
FlussActionStateStoreTest: 13 passed
Python core-options tests: 5 passed
Java/Python ConfigOption parity check
Ruff
Spotless
git diff --check

API

Adds the public Java and Python KAFKA_ACTION_STATE_TOMBSTONE_ENABLED option. Both expose the kafkaActionStateTombstoneEnabled key with a Boolean type and default value of false.

The public ActionStateStore.pruneState() contract now documents the recovery implications of durable backend deletion. No method signatures changed.

Documentation

doc-needed
doc-not-needed
doc-included

The Kafka configuration table documents the option, default behavior, and recovery trade-off. The shared Flink-key parsing limitation is also documented.

wenjin272 · 2026-07-13T09:02:37Z

@joeyutong Can you take a look at this PR?

joeyutong · 2026-07-14T04:00:13Z

Thanks for addressing the growth of Kafka action-state records. However, writing tombstones unconditionally may break recovery from older checkpoints or savepoints.

For example, if C0 is followed by action state S and then C1 tombstones S, restoring C0 will replay both S and the tombstone. rebuildState() removes S, so the action may execute again. This happens even when Kafka compaction is disabled.

Could tombstone emission be opt-in, for example through kafkaActionStateTombstoneEnabled defaulting to false?

rob-9 · 2026-07-14T18:51:41Z

Could tombstone emission be opt-in, for example through kafkaActionStateTombstoneEnabled defaulting to false?

Yeah, agreed.

Can we make cleanup safe by default though? Maybe we could stamp tombstones with the checkpoint id that triggered the prune; rebuildState() applies one only if id <= restored checkpoint, so restoring C0 skips C1's tombstones. But this would only protect the record until compaction deletes it.

This can be a follow-up. I'll add the opt-in flag to this PR for now.

- add kafkaActionStateTombstoneEnabled (default false) so tombstone emission is opt-in; rebuildState still honors tombstones already in the topic regardless of the flag - match the parsed key part exactly in pruneState so pruning key "a_1" can no longer tombstone state of the distinct key "a" - report async tombstone send failures via producer callback (flush() does not surface per-record errors) - narrow the prune catch to IllegalArgumentException and state the retention consequence in the warning - remove dead inner try/catch in rebuildState (deserialization errors throw from poll(), not from the map ops it wrapped) - document the durable-deletion replay constraint on ActionStateStore.pruneState and add the new option to the config docs - tests: default-off pruning, prefix-collision regression, tombstone replay in rebuildState; simplify assertions

joeyutong · 2026-07-15T03:23:44Z

Thanks for raising the question of whether cleanup can be safe by default. A checkpoint-aligned, user-controlled cleanup path may be a safer general solution:

Persist and expose the mapping from each checkpoint to its effective Kafka recovery marker—the per-partition minimum across all subtasks.
Leave the irreversible cleanup decision to users. Based on their recovery requirements, they can choose the oldest checkpoint to retain and delete records before its marker offsets through Kafka's deleteRecords API.
During rebuildState(), validate the requested offsets against Kafka's current beginning offsets. If a recovery offset is older than the available data, fail fast with a clear error instead of silently rebuilding incomplete state.

Since rebuildState() already starts from the recovery marker, the same marker is a natural boundary for physical cleanup. This aligns cleanup with the actual recovery semantics without automatically invalidating older checkpoints. Once users advance that boundary, fail-fast validation makes any incompatible recovery attempt explicit.

This could be explored as a follow-up.

…neState - testPruneStateEvictsCacheEvenWhenTombstoneSendFails: verifies pruneState degrades gracefully and still evicts the in-memory entry when a tombstone send fails asynchronously (the callback-reporting fix from the prior commit) - testPruneStateSkipsUnparseableKeys: verifies a state key that cannot be parsed into 4 parts is retained rather than pruned (the narrowed IllegalArgumentException catch) Both were verified to fail when the corresponding fix is reverted.

joeyutong · 2026-07-16T03:56:08Z

+     * completed actions to re-execute. Enable only if the job never restores from non-latest
+     * checkpoints or savepoints, or if re-executing actions is acceptable.
+     */
+    public static final ConfigOption<Boolean> KAFKA_ACTION_STATE_TOMBSTONE_ENABLED =


Could we also add the corresponding option to python/flink_agents/api/core_options.py? The cross-language option parity check currently fails because this field exists only on the Java side.

added matching Python option.

joeyutong · 2026-07-16T03:56:19Z

+        actionStateStore = tombstoneEnabledStore(actionStates, mockProducer);
+        String stateKey = ActionStateUtil.generateKey(TEST_KEY, 1L, testAction, testEvent);
+        actionStates.put(stateKey, testActionState);
+        mockProducer.errorNext(new RuntimeException("simulated broker failure"));


mockProducer uses autoComplete=true, so errorNext() is called before any pending completion exists and does not make the subsequent send() fail. This test therefore exercises the successful send path. Could we inject an exception through the callback so the async failure path is actually covered?

fixed, the test now completes the producer callback exceptionally and verifies both callback invocation and cache eviction.

weiqingy · 2026-07-17T07:17:47Z

+            }
+            try {
+                List<String> parts = ActionStateUtil.parseKey(stateKey);
+                if (parts.get(0).equals(keyStr) && Long.parseLong(parts.get(1)) <= seqNum) {


The collision you caught here feels like a thread worth pulling one turn further — I think the same root cause has a second face that this check can't reach.

Both this guard and the collision it fixes trace back to generateKey joining on _ while embedding the raw key unescaped (ActionStateUtil.java:44-56). That's what let pruning key a_1 match agent key a's state key a_1_<uuid>_<uuid>, and the equality check handles that case correctly.

The sibling case is when the agent key itself contains _. For an ordinary keyBy("user_123") the state key is user_123_1_<uuid>_<uuid>, so split("_") yields 5 parts (UUIDs use hyphens) and parseKey's checkArgument(parts.length == 4) throws (ActionStateUtil.java:58-63). This line never gets evaluated, and the entry lands in the catch on line 300 — so it's never evicted from the cache, never tombstoned, and never reclaimed by compaction. For any key containing _, the growth this PR targets would stay as it is. FlussActionStateStore.removeStateEntries catches Exception and skips the same way (FlussActionStateStore.java:242-261).

Worth saying that catching IllegalArgumentException on line 300 is an improvement in its own right — the old code caught only NumberFormatException, so a bad part count escaped pruneState and could take down notifyCheckpointComplete. Warn-and-retain is strictly better.

Escaping the separator properly would touch three files, which feels like more than this PR signed up for — would you rather note the limitation in the option's javadoc and file a follow-up? Or do you see a narrower angle I'm missing?

And on the test: testPruneStateSkipsUnparseableKeys pins this branch with TEST_KEY + "_1_onlythreeparts". Any appetite for a realistic fixture like "user_123" — same branch, but it'd document the case that actually reaches it?

fixed, the test now uses a realistic user_123 key, and the shared Kafka/Fluss limitation is documented and tested.

weiqingy · 2026-07-17T07:17:47Z

-                        });
+                            });
+                }
+                producer.flush();


Does this flush earn its place? The callback just above already reports failures, the eviction below runs regardless of the outcome, and ordering looks covered — put() flushes before returning (line 146), so a tombstone is always sent after its record is acked.

Curious whether there's a case it's protecting that I'm not seeing.

removed, Kafka ordering is preserved by the state key, and delivery failures remain reported through the callback.

Confirmed, closed on my side.

weiqingy · 2026-07-17T07:17:47Z

+     * checkpoints or savepoints, or if re-executing actions is acceptable.
+     */
+    public static final ConfigOption<Boolean> KAFKA_ACTION_STATE_TOMBSTONE_ENABLED =
+            new ConfigOption<>("kafkaActionStateTombstoneEnabled", Boolean.class, false);


Defaulting to false reads as the right call to me, and I think for a sharper reason than the javadoc gives itself credit for: with cleanup.policy=compact on the topic (KafkaActionStateStore.java:396), a tombstone is a durable delete instruction to the log cleaner — so the older-checkpoint erasure you documented above isn't something the replay side could have absorbed. Gating emission is the lever that actually controls it. A safe default-on would seem to need emission gated on the oldest still-restorable checkpoint, and I can't see a cheap way to know that here, so I'm not suggesting you flip it.

That does leave #691's original ask — the unbounded growth — unaddressed while the option is off. Where do you see this landing: is opt-in the endpoint, or would a follow-up be worth filing so the issue has somewhere to point?

Saw #1034, that answers this. It's the right home for what #691 still has open, so I'll follow the discussion there.

weiqingy · 2026-07-17T07:17:47Z

 | `kafkaActionStateTopic`             | (none)                   | String  | The config parameter specifies the Kafka topic for action state.            |
 | `kafkaActionStateTopicNumPartitions`| 64                       | Integer | The config parameter specifies the number of partitions for the Kafka action state topic. |
 | `kafkaActionStateTopicReplicationFactor` | 1                     | Integer | The config parameter specifies the replication factor for the Kafka action state topic. |
+| `kafkaActionStateTombstoneEnabled`  | false                    | Boolean | Whether pruning sends tombstone records so log compaction can reclaim pruned keys. Off by default: without tombstones the topic grows unboundedly, but restoring any checkpoint or savepoint replays correctly. When enabled, restoring from the latest completed checkpoint is unaffected, but restoring an older checkpoint or savepoint may replay tombstones written after that restore point and re-execute already completed actions. Enable only if the job never restores from non-latest checkpoints or savepoints, or if re-executing actions is acceptable. |


Small one: this row and the new public AgentConfigOptions.KAFKA_ACTION_STATE_TOMBSTONE_ENABLED are both in the diff, while the description checks doc-not-needed and says "No public API changes". It also still describes the unconditional design rather than the opt-in one that shipped. Mind refreshing it before merge?

Checked the updated body and docs, this one's closed.

weiqingy · 2026-07-17T07:17:47Z

+
+        // Assert - tombstones should have been sent to Kafka
+        var history = mockProducer.history();
+        assertThat(history).hasSize(2);


testPruneStateSendsTombstonesWithCorrectKeys pins this same contract more precisely — containsExactlyInAnyOrder(key1, key2) on the keys, containsOnlyNulls() on the values. A bug in tombstone keys, count, or nullness would fail both together, so I'm not sure this block catches anything the dedicated test would miss.

Would it read cleaner to leave testPruneState as it was — default store, cache eviction only, which also drops the tombstoneEnabledStore swap at the top — and keep the tombstone contract in one place? testPruneStateNoTombstonesByDefault already pins the default-off behavior.

addressed. testPruneState now covers cache eviction, while the dedicated tests own tombstone and default-off assertions.

Closed on my side, thanks.

rob-9 · 2026-07-17T16:27:43Z

thanks for the reviews @weiqingy @joeyutong . i'll try to address these soon.

wenjin272 · 2026-08-21T04:05:56Z

Hi @rob-9, just checking in on this PR. The comments from the last review round are still pending, and the branch now has merge conflicts. Are you still planning to continue working on it? If so, could you rebase and address the remaining comments when you have time? Otherwise, please let us know so someone else can pick it up. Thanks!

rob-9 · 2026-08-21T04:07:17Z

Hi @rob-9, just checking in on this PR. The comments from the last review round are still pending, and the branch now has merge conflicts. Are you still planning to continue working on it? If so, could you rebase and address the remaining comments when you have time? Otherwise, please let us know so someone else can pick it up. Thanks!

Sorry about this, forgot about it. I'll address the comments and fix conflicts soon.

…neState to cache eviction only

rob-9 · 2026-08-21T23:09:45Z

Hey all, comments should be addressed. The follow-up issue for checkpoint-aligned cleanup is #1034 .

weiqingy

Thanks for working through these. The limitation around keys containing _ is documented and pinned by a test on both stores now, and #1034 gives what's left of #691 somewhere to point.

A few questions inline.

weiqingy · 2026-08-23T02:21:58Z

+
+        store.pruneState(agentKey, 1L);
+
+        assertThat(actionStates).containsKey(stateKey);


This pins one half of the Kafka fix on the Fluss side. The other half, the exact first-part check at KafkaActionStateStore.java:301, doesn't look like it came across.

FlussActionStateStore.pruneState:489 calls removeStateEntries(key.toString() + "_", stateSeqNum -> stateSeqNum <= seqNum), and removeStateEntries:243 filters on startsWith(keyPrefix) plus the parsed sequence number, with no comparison of parts.get(0) against the key being pruned.

So Flink key a at seq 1 stores a_1_<eventUuid>_<actionUuid>, which parses cleanly as 4 parts with parts.get(1) == "1". Pruning the distinct key a_1 at any seqNum >= 1 matches the prefix a_1_ and satisfies 1 <= seqNum, so key a's completed state is evicted. FlussActionStateStore.get:227 reads the cache only, so the next lookup returns null and the action re-runs.

Is leaving Fluss on the prefix match deliberate, or would the same parts.get(0) check belong in removeStateEntries?

added the exact parsed-key check to Fluss, with tests covering both pruning and lookup cleanup.

weiqingy · 2026-08-23T02:21:58Z

+        String stateKey = ActionStateUtil.generateKey(flinkKey, 1L, testAction, testEvent);
+        actionStates.put(stateKey, testActionState);
+
+        assertThat(actionStateStore.get(flinkKey, 2L, testAction, testEvent)).isNull();


Does this discriminate the way its name suggests? The entry is stored at seq 1 and looked up at seq 2. The divergence cleanup only evicts entries with stateSeqNum > seqNum (KafkaActionStateStore.java:182), and 1 is not greater than 2, so the entry is retained whether or not parseKey throws. The lookup returns null either way, since no seq-2 key was ever stored, so both assertions hold against an implementation that parses user_123 correctly.

Inverting the two sequence numbers would separate the cases, in case that's useful: store at seq 3, then get(flinkKey, 1L, ...). Correct parsing evicts (3 > 1), only the parse failure retains it, and the seq-1 lookup is still a cache miss so the removeIf fires.

testPruneStateSkipsUnparseableKeys and the Fluss twin both do fail against a fixed implementation, so this looks like the odd one out.

you're right. I inverted the sequence numbers so the test now fails if the key parses successfully and gets evicted.

weiqingy · 2026-08-23T02:21:58Z

+                                            "Cannot parse state key: {}. The entry cannot be "
+                                                    + "considered for divergence cleanup and will "
+                                                    + "be retained.",
+                                            entry.getKey(),


The widening isn't what I'm asking about here, it's how often this line can fire.

The removeIf scans the whole cache whenever !actionStates.containsKey(stateKey) || hasDivergence (:170), get() sits on the durable-execution path with three call sites in DurableExecutionManager (:212, :218, :255), and entries that fail to parse are retained by design. So one user_123 entry produces a WARN on every subsequent cache miss for the life of the job, each with a full stack trace, since e is passed as a trailing arg.

pruneState:305 already warns for the same key on the prune path. Would dropping the throwable here, or demoting this to DEBUG, cost signal you'd want to keep?

Demoted this to DEBUG and removed the throwable, so repeated cache misses no longer produce WARN stack traces.

weiqingy · 2026-08-23T02:21:58Z

 |------------------------------|------------------|---------|------------------------------------------------------------------------------------------|
 | `actionStateStoreBackend`    | (none)           | String  | The backend for action state store. Supported values: `"kafka"`, `"fluss"`.              |

+Durable action state stores currently join raw Flink keys and other key parts with an unescaped `_`. Flink keys containing `_` cannot be parsed safely during pruning, so both Kafka and Fluss retain their state in memory and backend storage. Kafka also emits no tombstones for those keys.


nit: "both Kafka and Fluss retain their state in memory and backend storage" reads as though backend retention follows from the parse failure. On the Fluss side it doesn't. FlussActionStateStore.pruneState:487 never deletes from the backend for any key, parseable or not, and its javadoc says why: "The Fluss log is append-only; physical cleanup relies on Fluss log retention configuration."

Two smaller things alongside it. All three doc sites frame the limitation as growth, but the same parse failure also makes the divergence cleanup in get() inert (KafkaActionStateStore.java:184, FlussActionStateStore.java:257), so stale higher-seq state survives a divergence that was actually detected. That's a correctness cost rather than a storage one.

And #1034 covers the checkpoint boundary, but nothing seems to track the key encoding itself. Worth its own issue?

nit: the Fluss claim and the divergence-cleanup cost both read accurately now across all three sites, so those two are closed. On the key encoding I also had a look at #1010, but that one is about foreign-key retention after a restore, so the encoding itself still has nowhere to point.

weiqingy · 2026-08-23T02:21:58Z

 ### Exactly-Once Action Consistency

-To ensure exactly-once action consistency, you must configure an external action state store. Flink Agents record action state in this store on a per-action basis. After recovering from a checkpoint, Flink Agents consult the external store and will not re-execute actions that were already completed. This guarantees each action is executed exactly once after recovering from a checkpoint.
+To ensure exactly-once action consistency, you must configure an external action state store. Flink Agents record action state in this store on a per-action basis. After recovering from a checkpoint, Flink Agents consult the external store and reuse completed action state when its backing record remains available. This prevents re-execution for checkpoints supported by the store's retained recovery history.


nit: this paragraph covers both backends, including Fluss and the default Kafka setup with tombstones off. The hazard it now hedges for is opt-in and Kafka-only, and the hint warning box a few lines below states it with that scope. "backing record remains available" and "the store's retained recovery history" also appear nowhere else in either doc, so a reader has nothing to resolve them against.

Is the hedge doing work here that the warning box below doesn't already do? One way this could read is with the original sentence restored: "This guarantees each action is executed exactly once after recovering from a checkpoint."

nit: the hedge is gone, thanks. That was my wording though. I quoted only the last sentence and called it the original, so the one before it got dropped along the way. On main the paragraph also has "After recovering from a checkpoint, Flink Agents consult the external store and will not re-execute actions that were already completed." Without it, "This guarantees..." now attaches to recording state per action rather than to the consult-and-skip behavior it described. Worth putting back?

weiqingy · 2026-08-24T04:09:31Z

@@ -175,10 +181,11 @@ public ActionState get(Object key, long seqNum, Action action, Event event) thro
                                        // the requested seqNum
                                        return stateSeqNum > seqNum;


The Fluss get() test you just added looks like it would fail if it were pointed at Kafka. Copying FlussActionStateStoreTest.java:145-152 into KafkaActionStateStoreTest: get("a_1", 0L, ...) misses, this removeIf parses the cached a_1_<eventUuid>_<actionUuid> (Flink key a, seq 1) into 4 parts, 1 > 0 fires, and the containsKey assertion fails. Kafka has the pruneState twin at KafkaActionStateStoreTest.java:261-274, but not this one.

The reason looks like this scan isn't tied to a key at all. There's no prefix filter and no parts.get(0) check, just stateSeqNum > seqNum applied to every entry in the map. One map holds every key the subtask owns (:82, store created per operator at DurableExecutionManager.java:118-125), and sequence numbers count per key (DurableExecutionManager.java:233), so one key's lookup can drop another key's state.

The cache isn't the durable record, so how much that costs depends on when the evicted key is read next. The case I can't rule out is just after a restore, when the replay path reads the rebuilt cache: processActionTaskForKey skips execution only when actionState != null && actionState.isCompleted() (ActionExecutionOperator.java:435), so an entry that got dropped runs again. That needs action tasks for different keys to interleave during replay, which I haven't reproduced, so I'd call it a hazard worth checking rather than a break.

To be clear, this predicate is base code and not something the PR introduced. The only change inside the block was the catch clause and its log line, and the other three scan sites do carry the guard at head (:294-301 here, and FlussActionStateStore.java:249-256 for both of its callers). Would the same parts.get(0) check you just added on Fluss fit here as well, or would you rather it went out separately?

weiqingy · 2026-08-24T04:09:32Z

+                                long stateSeqNum = Long.parseLong(parts.get(1));
+                                return seqNumFilter.test(stateSeqNum);
                            } catch (Exception e) {
                                LOG.warn("Failed to parse state key: {}", entry.getKey(), e);


nit: the trailing e with a single {} means this logs a full stack trace, and it's the Fluss twin of the Kafka site you just demoted. A Flink key containing _ never parses, so :262 keeps the entries and get() runs this scan again on every new sequence number for that key, which repeats the trace for as long as the job lives.

It's narrower than the Kafka one, since this scan is prefix-filtered so only that key's own entries reach it, and the docs now describe the retention as expected. Would the same DEBUG treatment fit here, and is the stack trace earning its place on this path?

rob-9 added 2 commits July 7, 2026 17:22

fix: handle null-valued records in KafkaActionStateStore.rebuildState()

0db12a6

feat: send tombstone records in KafkaActionStateStore.pruneState()

98b1aaa

github-actions Bot added doc-not-needed Your PR changes do not impact docs fixVersion/0.4.0 priority/major Default priority of the PR or issue. labels Jul 8, 2026

rob-9 added 2 commits July 15, 2026 11:16

style: fix spotless formatting violations

a02a28a

joeyutong reviewed Jul 16, 2026

View reviewed changes

weiqingy reviewed Jul 17, 2026

View reviewed changes

rob-9 added 8 commits August 21, 2026 15:05

Merge fork/main into feat/691-kafka-prune-tombstones

55f7874

feat: add kafkaActionStateTombstoneEnabled to python core options

41c9fee

test: exercise the real async tombstone send failure path

07958ff

docs: document that keys containing '_' are never pruned or tombstoned

f46cffc

refactor: drop redundant producer.flush() in pruneState

2cb557c

docs: clarify tombstone recovery scope

2cb89d8

test: keep tombstone assertions in the dedicated test, revert testPru…

14b894f

…neState to cache eviction only

[runtime][docs] Harden tombstone recovery edge cases

16b44b6

github-actions Bot added doc-included Your PR already contains the necessary documentation updates. and removed doc-not-needed Your PR changes do not impact docs labels Aug 21, 2026

rob-9 changed the title ~~[runtime] send Kafka tombstones on state pruning for durable log compaction~~ [api][runtime][python] Add opt-in Kafka tombstones for pruned action state Aug 21, 2026

github-actions Bot removed the doc-included Your PR already contains the necessary documentation updates. label Aug 21, 2026

github-actions Bot added the doc-included Your PR already contains the necessary documentation updates. label Aug 21, 2026

rob-9 mentioned this pull request Aug 21, 2026

[Feature] Add checkpoint-aligned cleanup for Kafka action state #1034

Open

1 task

weiqingy reviewed Aug 23, 2026

View reviewed changes

rob-9 added 5 commits August 23, 2026 01:05

[runtime][java] Protect Fluss cleanup from key collisions

d09d94c

[runtime][java] Strengthen unparseable Kafka key test

a8bf18a

[runtime][java] Quiet repeated Kafka key parse failures

fca59c2

[api][runtime][docs] Clarify action state key limitation

8b8db63

[docs] Restore exactly-once action wording

a6c68e3

weiqingy reviewed Aug 24, 2026

View reviewed changes


		store.pruneState(agentKey, 1L);

		assertThat(actionStates).containsKey(stateKey);

		@@ -175,10 +181,11 @@ public ActionState get(Object key, long seqNum, Action action, Event event) thro
		// the requested seqNum
		return stateSeqNum > seqNum;

Uh oh!

Conversation

rob-9 commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of change

Tests

API

Documentation

Uh oh!

wenjin272 commented Jul 13, 2026

Uh oh!

joeyutong commented Jul 14, 2026

Uh oh!

rob-9 commented Jul 14, 2026

Uh oh!

joeyutong commented Jul 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rob-9 commented Jul 17, 2026

Uh oh!

wenjin272 commented Aug 21, 2026

Uh oh!

rob-9 commented Aug 21, 2026

Uh oh!

rob-9 commented Aug 21, 2026

Uh oh!

weiqingy left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

rob-9 commented Jul 8, 2026 •

edited

Loading

weiqingy left a comment •

edited

Loading