Skip to content

feat: add opt-in h264 image storage#2504

Draft
TomCC7 wants to merge 3 commits into
cc/mem2-image-encodefrom
cc/feat/h264-encode-module
Draft

feat: add opt-in h264 image storage#2504
TomCC7 wants to merge 3 commits into
cc/mem2-image-encodefrom
cc/feat/h264-encode-module

Conversation

@TomCC7

@TomCC7 TomCC7 commented Jun 16, 2026

Copy link
Copy Markdown
Member

Summary

This patch updates the existing H.264 image PR so Image is not overloaded with encoded-video semantics.

Image now remains a decoded, pixel-addressable raster type: Image.data is always a NumPy array. H.264 bytes are used only as a transport/storage representation, matching the way the repo already treats JPEG: modules keep using In[Image] / Out[Image], while codecs encode/decode behind the transport or storage boundary.

What changed

  • Restored raw-only Image semantics.

    • Removed public encoded-Image fields and helpers such as encoding, codec_metadata, require_raw(), and Image.encoded(...).
    • Updated call sites back to direct .data access.
  • Kept H.264 transport as an end-to-end wire codec.

    • Module-facing API stays Image -> Image.
    • H264LcmTransport encodes Image to H.264 packet bytes on publish and decodes back to Image before subscribers receive it.
    • Encoded packets are not exposed as Image.
  • Added opt-in H.264 storage for logical Image streams.

    • store.stream(name, Image, codec="h264")
    • RecorderConfig(stream_codecs={"color_image": "h264"})
    • Default Image storage remains JPEG.
  • Reused one shared H.264 implementation.

    • Transport and storage both call the same core H264Encoder, H264Packet, and H264Decoder logic.
    • Storage adds stream-specific behavior around that core: append locking, keyframe metadata, keyframe seeking, lazy decode, and id-ordered sequential decode.
  • Updated demos/docs/tests for the split semantics.

    • Webcam recorder now uses raw transport into the recorder and lets the storage backend do the H.264 encode.
    • Playback reads H.264-backed memory2 observations as decoded Image values.

Why

The earlier version made Image.data sometimes mean decoded pixels and sometimes mean encoded H.264 bytes. That made the type harder to reason about and forced unrelated image consumers to handle encoded-video cases.

This patch follows the repo’s existing JPEG pattern instead:

public API:       Image
physical layer:   encoded bytes
public API:       Image
The encode/decode logic lives behind transport/storage boundaries, not in the Image message type.

@codecov

codecov Bot commented Jun 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.43671% with 111 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/memory2/video/h264.py 75.57% 20 Missing and 12 partials ⚠️
dimos/protocol/video/h264.py 77.95% 19 Missing and 9 partials ⚠️
dimos/msgs/sensor_msgs/Image.py 63.33% 6 Missing and 5 partials ⚠️
dimos/protocol/pubsub/impl/h264_lcm.py 76.74% 8 Missing and 2 partials ⚠️
dimos/memory2/module.py 14.28% 6 Missing ⚠️
dimos/memory2/store/sqlite.py 87.50% 1 Missing and 2 partials ⚠️
dimos/memory2/store/base.py 75.00% 1 Missing and 1 partial ⚠️
dimos/models/vl/florence.py 0.00% 2 Missing ⚠️
dimos/msgs/sensor_msgs/PointCloud2.py 0.00% 2 Missing ⚠️
...rimental/temporal_memory/temporal_utils/helpers.py 0.00% 2 Missing ⚠️
... and 12 more
@@                   Coverage Diff                    @@
##           cc/mem2-image-encode    #2504      +/-   ##
========================================================
+ Coverage                 69.77%   69.89%   +0.12%     
========================================================
  Files                       844      844              
  Lines                     74758    75027     +269     
  Branches                   6730     6739       +9     
========================================================
+ Hits                      52164    52442     +278     
+ Misses                    20875    20858      -17     
- Partials                   1719     1727       +8     
Flag Coverage Δ
OS-ubuntu-24.04-arm 63.78% <57.91%> (-0.05%) ⬇️
OS-ubuntu-latest 64.81% <80.37%> (+0.13%) ⬆️
Py-3.10 64.81% <80.37%> (+0.13%) ⬆️
Py-3.11 64.81% <80.37%> (+0.13%) ⬆️
Py-3.12 64.81% <80.37%> (+0.13%) ⬆️
Py-3.13 64.81% <80.37%> (+0.13%) ⬆️
Py-3.14 64.83% <81.01%> (+0.13%) ⬆️
Py-3.14t 64.81% <80.37%> (+0.13%) ⬆️
SelfHosted-Large 30.30% <23.18%> (-0.01%) ⬇️
SelfHosted-Linux 38.12% <25.76%> (-0.06%) ⬇️
SelfHosted-macOS 36.86% <26.24%> (-0.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
dimos/core/transport.py 56.87% <100.00%> (ø)
...imos/experimental/security_demo/security_module.py 68.22% <100.00%> (ø)
dimos/mapping/occupancy/visualize_path.py 100.00% <100.00%> (ø)
dimos/memory2/video/test_h264_storage.py 98.65% <100.00%> (+1.32%) ⬆️
dimos/protocol/video/test_h264.py 98.33% <100.00%> (+1.00%) ⬆️
...ree/go2/blueprints/smart/unitree_go2_h264_video.py 95.23% <100.00%> (+22.82%) ⬆️
...imos/experimental/security_demo/depth_estimator.py 29.03% <0.00%> (ø)
dimos/mapping/osm/current_location_map.py 33.33% <0.00%> (ø)
dimos/models/vl/moondream.py 31.86% <0.00%> (ø)
dimos/models/vl/moondream_hosted.py 34.28% <0.00%> (ø)
... and 18 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant