OBSERVE notifications larger than one block are delivered truncated

[QuiteYellow]

Summary

_dispatch_coap hands an OBSERVE notification's payload straight to on_notification without looking at its Block2 option (dtls_session.py:629-643). When a notification's representation spans more than one block, only the first block reaches the callback, and the consumer decodes a partial CBOR buffer.

One-shot get() is not affected: it already runs a token-stable Block2 reassembly loop (dtls_session.py:649+). The gap is specific to the notification path.

Symptom

The consumer sees a CBOR decode failure on a large observed resource, most often /mode/vs/0. Two independent reports, same path, same error shape:

• Samsung Refrigerator (TP2X_REF_20K) will not reconnect on HA startup #37 (TP1X_DA-KS-RANGE-0101X): observe /mode/vs/0: cbor decode failed: premature end of stream (expected to read 5880 bytes, got 555 instead)
• localthings #361: poll cbor decode: premature end of stream (expected to read 1 bytes, got 0 instead) on /mode/vs/0

The error message belongs to the consumer, but the truncation happens here.

Why

RFC 7959 §2.6 covers blockwise notifications. When sending a 2.05 notification the server sends only the first block of the representation, and the client retrieves the rest "as if it had caused this first response by a GET request." This library never does that second step for notifications, so anything past the first block is lost.

Design constraints (learned on these devices)

Anyone fixing this has to account for the device behaviour already documented in the dtls_session.py docstring and oven-findings.md:

• Block2 transfers must reuse one token across every block. A fresh token per block is silently dropped.
• A fresh token has to begin its transfer at block 0. Samsung keys per-transfer state on the token, so a request that opens with Block2 NUM=1 under a token the server has not seen gets no reply at all.
• The continuation cannot borrow the notification's token. RFC 7959 §3.4 is explicit that requests for additional blocks cannot use the token of the observation relationship, and in this codebase it would also make _dispatch_coap resolve real notifications into the transfer's pending waiter.
• OBSERVE registrations use 1-byte tokens; Samsung silently drops TKL>1 registrations. One-shot GETs use 4-byte tokens.
• The resource can change mid-transfer. ETag validation exists for exactly that race.
• Requests are paced at 5/s. A notification storm that triggers a fetch per notification could cross the firmware ceiling.

The fix

The two constraints above leave one shape: when a notification arrives carrying Block2 with the M bit set, discard the partial and re-read the whole resource from block 0 on a fresh 4-byte one-shot token. That is an ordinary GET, which is what §2.6 asks for, and it reuses the token-stable reassembly get() already performs. It costs one redundant block against a transfer that would otherwise be lost entirely.

The re-read has to be ETag-guarded against mid-transfer change and rate-limited so a notification storm cannot exceed the 5/s pacing ceiling. Deliver the reassembled representation to on_notification only once it is complete. get() already does token-stable Block2 reassembly (dtls_session.py:649+); factor that loop into a shared helper and reuse it here. This can land independently, on current main, and gives the codebase the shared Block2 primitive #36 was reaching for instead of waiting on it.

Two things need care:

• The re-read runs on the reader thread's callback path, and get() blocks on an event only the reader thread can set. It has to be handed to a worker, not called inline.
• When the re-read fails or has not finished (device silent, ETag mismatch mid-transfer, pacing ceiling hit), drop the notification and log at debug rather than hand the first block to the callback.

[QuiteYellow]

Correcting my own framing above, before anyone builds on it.

Two things in the original text were wrong. Observe combined with Block2 is RFC 7959 §2.6, not §2.4; §2.4 is plain Block2 usage. And the continuation does not reuse the notification's token. §3.4 says the opposite, in a note attached to the worked example: "tokens are just shown in this example to illustrate that the requests for additional blocks cannot make use of the token of the Observation relationship."

That matters here beyond pedantry, because the spec-literal shape is the one we have evidence against. §2.6 has the client continue at Block2 NUM≥1 under a fresh token. Our own Block2 diagnostics recorded exactly that request timing out on RT-OCF: the server keys per-transfer state on the token, so a fresh token has to begin its transfer at NUM=0 or nothing comes back. The only variant that ever retrieved a later block was a single token held across a transfer that started at block 0.

So the fix is a full re-read of the resource from block 0 on a fresh 4-byte one-shot token, not a continuation. It is still an ordinary GET as far as §2.6 is concerned, and it is already proven against hardware: the MQTT bridge in this repo has been doing precisely this as a consumer-side workaround, triggered off CBOR-decode failure instead of off the Block2 option. The change is moving it into the library and driving it from the Block2 option.

The token question I flagged as needing a hardware test is therefore answered. One constraint I had missed is now in the issue body: the re-read cannot be called inline from _dispatch_coap, because that runs on the reader thread and get() waits on an event only the reader thread can set. The rest is unchanged: if the re-read fails, drop the notification and log at debug rather than hand over the first block.

I have edited the issue body to match.

[QuiteYellow]

Fixed in #40, and the open question in the issue body is now settled against hardware. Two runs, on separate sessions:

refetch /mode/vs/0 tok=bfadc91e blocks=10 bytes=10068 ok
refetch /mode/vs/0 tok=d50961b3 blocks=10 bytes=10068 ok

Four-byte tokens both times, ten blocks reassembled, byte-identical across the two. A re-read from block 0 on a fresh one-shot token is what this server answers. No §2.6 continuation was attempted and none is needed.

Those two arrived as OBSERVE registration responses, because the appliance is normally kept off the internet and its notify dispatch is cloud-gated. It was then given access briefly, with a cook started, which produced eleven re-reads: six delivered, five superseded by a newer notification mid-read, none failed. Every run of superseded results ended in a delivery, so the final state always reached the callback. The one that closed the burst reassembled 10066 bytes across ten blocks and applied mode: NoOperation -> Convection to the cache from the observe path, which is the exact notification that used to arrive as a truncated CBOR buffer.

One detail for anyone hitting this: the truncation fires on every session start for any observed resource larger than one block, not only when push is live. subscribe() leaves the initial 2.05 to the observe branch, so a registration response takes the same path a notification does. That is likely why this reproduced so readily.