Add the scene-entity contract and MJCF entity composition#2479
Conversation
EntityDescriptor (what an entity is) + EntityStateBatch (where every entity is): an authority-agnostic wire contract between whatever owns physics truth (sim today, perception later) and whatever mirrors the scene (planning worlds, visualizers, recorders). Versioned, length-prefixed JSON over LCM — the EntityMarkers pattern — so Rust/browser consumers hand-decode it without generated lcm_msgs types; wire tests pin the schema. msg_name is namespaced simulation_msgs.* (LCM channel keys derive from it) ahead of the type's eventual move out of experimental. entity_scene composes scene entities into an MjSpec — cooked collision hulls when the package provides them, AABB boxes otherwise. The mujoco lint stub gains the spec/kinematics surface this and downstream consumers use.
Codecov Report❌ Patch coverage is
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 17 files with indirect coverage changes 🚀 New features to boost your workflow:
|
|
@Nabla7 - @mustafab0 said this isnt blocking and no rush to merge and proposes we refactor to add clean integration |
|
|
||
|
|
||
| def test_mesh_entity_loads_cooked_hulls(tmp_path: Path) -> None: | ||
| mujoco = pytest.importorskip("mujoco") |
There was a problem hiding this comment.
Please no importorskip. We want tests to fail if imports are missing. Otherwise how can we ensure that things don't break.
mustafab0
left a comment
mustafab0
left a comment
There was a problem hiding this comment.
@Nabla7 I am conflicted approving this PR.
I understand what you are trying to build here with the entities as these are the building blocks for your sidecar dynamic loading of objects. However I do not believe this should be a blocker for the reachability analysis.
Happy to approve this for it being part of the experimental branch, but I want to wait until we have a complete spec for PimSim, to see how this fits in the larger system.
We already have "entities" called Objects as dimos standard messages and could just be reused here. That is not as sophisticated as your EntityDescriptor but should essentially do the same thing.
Have you taken a look at it?
4 participants
Problem
There is no shared contract for scene objects. Three consumers need the same
two facts — what objects exist (shape, mesh, mass) and where they are
right now — and each is on track to invent its own answer: the manipulation
planning world needs live obstacle poses (a moved chair is invisible to the
planner today), the upcoming viser manipulation visualizer needs renderable
objects (
PlanningSceneInfocarries robots only), and MuJoCo-based sims needto compose objects as collision bodies. Without one wire format, these grow
into parallel, incompatible sync paths.
Closes DIM-XXX
Solution
Two types and one composition function, deliberately small so dependent
branches (
pim/feat/g1-reachability,cc/feat/viser-vis-rework) can buildagainst main instead of against each other:
EntityDescriptor— what an entity is: stable id, kind(static/kinematic/dynamic),
mesh_ref(GLB), shape hint + extents forprimitives, mass, optional rgba. Everything a collision backend or a
renderer needs to instantiate it.
EntityStateBatch— where everything is: a timestamped snapshot of(descriptor, pose) entries, streamed by whatever owns physics truth — a
simulator today, perception later. Producers and consumers never see each
other; the batch is their only coupling.
entity_scene.add_entities_to_spec— composes descriptors into anMjSpec as collision bodies: cooked CoACD hulls when package metadata
provides them, AABB boxes otherwise.
Wire format: versioned, length-prefixed JSON over LCM (the existing
visualization_msgs.EntityMarkerspattern) — hand-decodable from Rust andbrowser consumers, no generated lcm_msgs type, unknown keys ignored for
forward compatibility. The schema is pinned by tests.
msg_nameissimulation_msgs.EntityStateBatch(LCM channel keys derive from it, so thenamespace is chosen for the type's eventual home, not its current
experimental/path).Proven, not speculative: this exact stream already drives a browser viewer
and a Rust lidar consumer on the pimsim dev branch, and the planning-world
consumer (port →
sync_entity_poses, with a moved-crate-flips-plan-validitytest) follows in the reachability PR stacked on this one.
Also included: the mujoco lint stub gains the
MjSpec/kinematics surfacethat
entity_sceneand downstream consumers use.How to Test