Repository navigation
Conversation
|
📝 Docs preview is not auto-deployed for fork PRs. A maintainer with write access to |
|
No actionable comments were generated in the recent review. ℹ️ Recent review info
Walkthrough
Merge Risk:
 |
| Check name | Status | Explanation |
|---|---|---|
| Linked Issues check |  |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | |
Check skipped because no linked issues were found for this pull request. |
| Description Check | |
Check skipped - CodeRabbit’s high-level summary is enabled. |
| Title check | |
The title clearly and concisely describes the main changes: adding the AgileMaster full-body plugin and the G1 teleoperation example. |
✨ Finishing Touches
-
🧪 Generate unit tests (beta)
-
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
|
/preview-docs |
|
✅ Preview deployed: https://NVIDIA.github.io/IsaacCapture/preview/pr-1250/ |
|
@aristarkhovNV Can you please help review the core change part. |
ivany-nv
left a comment
ivany-nv
left a comment
There was a problem hiding this comment.
Summary
This adds a MOXI Receiver plugin that publishes AgileMaster poses as FullBodyPose, a body.jmex live tracker that reads them (same shape as the Noitom and Xsens readers), and an Isaac Lab G1 example. CI is green on every job. Built locally without the SDK, the jmex tests pass, and the plugin's own publisher, run against a CloudXR runtime, round-trips through body.jmex into a standard full-body MCAP with every joint value intact. Read independently, the axis change, slot mapping, quaternion order, timestamps and MCAP wiring all hold up. One suggestion, nothing blocking. The src/core/live_trackers reader and factory registration are with @aristarkhovNV; this review covers the plugin, example, tests and docs.
Legend: 🚫 BLOCKER = merge-blocking | 💡 SUGGESTION = meaningful improvement | 🧹 NIT = precise small correction
| Finding | |
|---|---|
| 🚫 | none |
| 💡 | src/plugins/jmex/app/main.cpp:126-129 - An unrecognized argument is only logged as a warning. So --channel 2 (space instead of =) or a typo like --chanel=2 leaves the plugin on channel 1, waiting for Player forever. That is the silent mismatch the strict --channel= parsing at L92-L93 exists to prevent. The Xsens plugin rejects unknown options; doing the same here makes the plugin fail fast with exit code 1. |
| 🧹 | none |
ivany-nv
left a comment
ivany-nv
left a comment
There was a problem hiding this comment.
Summary
A second pass, this time against this PR merged onto current main, which is 55 commits past your base. It merges cleanly and builds. ctest -L jmex, the vendor-validation cases, clang_format, pre-commit on the changed files and make current-docs all pass. The body.jmex reader is safe to add: the default vendor is unchanged, there is no schema or recording change, and it compiles against main's newer SchemaTracker. Before approval there are three things: two example fixes (the first two 💡 below) and a rebase onto main so CI reruns against it. The rest are questions and small items.
Legend: 🚫 BLOCKER = merge-blocking | 💡 SUGGESTION = meaningful improvement | ❓ QUESTION = needs your answer | 🧹 NIT = precise small correction
| Finding | |
|---|---|
| 🚫 | none |
| 💡 | jmex_g1_tasks.py:30-32 - Isaac Lab 3.0.0 EA, the release main's README names as current, moved this config to isaaclab_tasks.contrib.locomanip_pick_place.locomanipulation_g1_env_cfg (isaac-sim/IsaacLab#5891). The manager_based path no longer exists there, so --external_callback ...register_tasks fails with ModuleNotFoundError. Please import the new path with a fallback to the beta2 one, and name the tested Isaac Lab version in README prerequisite 4. |
| 💡 | jmex_g1_tasks.py:711-725 - Returning [] tells teleop_se3_agent.py to reject none of the leftover arguments: list_intersection(remaining_args, []) is empty, and with None it is all of them. So --cloudxr-env none (hyphen) is silently ignored and Isaac Lab starts its own auto-webrtc runtime. That is the failure README L97-99 warns about, now with no error. The callback parses no arguments of its own, so return None is right. |
| 💡 | main.cpp:59-75 - This extends my earlier comment. parse_port checks that the whole string is an integer but not its range. So --tcp-port=70000, --channel=0 or --channel=300 go straight to the SDK, and udp_start_port + 1 + channel (L151) can overflow. The noitom and xsens plugins reject out-of-range ports, and receiver_session.hpp:27-28 already states the valid channels. |
| ❓ | app/CMakeLists.txt:8-14 - Is Moxi::Receiver static or shared? If it is shared, the installed jmex_plugin has no RPATH to find it. The plugins that link a vendor shared library set INSTALL_RPATH; sharpa_avatar points it at an external SDK. readelf -d install/plugins/jmex/jmex_plugin on your machine answers this. |
| ❓ | receiver_session.cpp:136-153 - The joint table is re-read only after the channel drops. Can Player load a different model, as the README's troubleshooting tells operators to do, without the pairing dropping? If it can, slots keep the old joint indices and go out valid. Re-checking MxRGetBoneJointCount on each new sequence id would cover it. |
| ❓ | jmex_g1_tasks.py:769-777 - retargeting_execution is unset, so Isaac Lab runs the pipeline in pipelined mode, and the overlay's debug-draw calls run on the retarget worker thread. Which mode was the live run in? Setting mode="sync", or drawing from process_actions, settles it either way. |
| 🧹 | .gitignore:5 / ClangFormat.cmake:43 - Use MOXIReceiverSDK*/ and MOXIReceiverSDK[^/]*, so an archive unpacked under its versioned name is not picked up by git add -A or the format check. |
| 🧹 | full_body_publisher.hpp:42-48 - body.jmex accepts max_flatbuffer_size, but the plugin always creates a 16 KiB collection, so a smaller value makes the reader throw. Either drop the parameter or add --max-flatbuffer-size= as noitom and xsens do. |
| 🧹 | device_clock.cpp:52-60 - A Player timestamp of 2^63 or more turns negative, push_buffer throws, and publish() treats every such frame as a lost session. Falling back to the sample time there keeps push failures transport-only. |
| 🧹 | Naming, after the rebase: main now calls the library and the repository "Isaac Capture"; Isaac Teleop stays the teleop component. This affects the plugin README L8-9, the example README L11/29/31/49 and record_jmex_full_body.py:4. The devices.yaml row's URL should be NVIDIA/IsaacCapture. The example README's wheel steps (L22-43) can link the build-from-source page and keep only the Isaac-Lab-specific parts. |
| 🧹 | src/plugins/jmex/CMakeLists.txt - The README says Linux x86_64 only. A CMAKE_SYSTEM_NAME check with a "Linux only" skip message, as the xsens and noitom plugins have, reports the right reason on other platforms. |
Optional, not needed for approval:
- Build the SDK-free sources as an always-built core library, with the receiver in the SDK-gated part. The test could then link it, and those files would get the project warning set, as gamepad and spacemouse do.
- Split
JmexG1ActionSourceintoFullBodySourceplus a retargeter, to match theIDeviceIOSource"pure converter" contract.
Please rebase onto main, or we can update the branch, so CI reruns against it before approval.
Adds `jmex`, a plugin for the j-mex AgileMaster motion capture suit, and an example, in two parts, that consumes what it publishes. The plugin reads one MOXI Player channel through the MOXI Receiver SDK (0.8.0 or newer, obtained from j-mex) and runs on Linux only. It fills the 24 `FullBodyPose` slots from the standard human skeleton Player has loaded, one slot per joint, already in OpenXR axes, and pushes them under the `body.jmex` vendor on every frame. A slot with no joint, or with no usable value in a frame, goes out invalid; nothing is inferred from its neighbors. The raw device clock carries the time Player stamped on the frame (`--device-clock=local` uses the pickup time instead), and if the CloudXR runtime restarts underneath the plugin, the plugin reconnects rather than ending the run. An argument it cannot use stops it with exit code 1, naming the argument. Everything but the receiver is a library that builds without the SDK, so on Linux its unit tests run in every build that has tests enabled; without the SDK only the plugin executable is skipped. Isaac Capture learns `body.jmex` as a full-body vendor: a live tracker that reads the plugin's collection, with a `collection_id` parameter, its entry in the DeviceIO factory, the vendor validation cases, and a row in `devices.yaml`. An SDK unpacked into the plugin directory, under any versioned name, is kept out of git and the clang-format check. `isaaccapture_examples.jmex.jmex_g1_tasks` registers a task that inherits Isaac Lab's `LocomanipulationG1EnvCfg`, on Isaac Lab 3.0.0 EA or 3.0.0 beta2, and takes its action from the operator; nothing in Isaac Lab is modified. It is upper-body teleop. Each arm is rebuilt segment by segment onto the robot's own shoulder, so the operator's arm angles survive while the lengths stay the robot's, and the elbows and shoulders get IK targets of their own from that same chain. The wrist's orientation is the operator's hand, carried across through a forearm frame. The operator's torso turns the robot's shoulders about its waist, so leaning, bending sideways and twisting come through; their heading does not, so the operator can turn to face a screen without moving the robot. Only a frame that carries all 24 slots drives the robot, and slots that stay missing are named on the console. `isaaccapture_examples.jmex.record_jmex_full_body` writes the same stream to a standard full-body MCAP. Tests cover the plugin's argument parsing, slot mapping, change of axes, device clock choice and session rebuild after a failed push, the vendor's parameters, and the example's action source, including the geometry of the rebuilt arm and the orientation carried through the forearm frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: j-mex <jmex.dev@jmex.com.tw>
a91f3af to
8dcc95c
Compare
|
Thanks for the second pass. I rebased onto
The PR description has the updated local gate (both toolchains, six entries each) and the device |
|
Heads-up: #1394 just merged. It moves Could you switch [tool.hatch.build]
only-include = ["python/isaaccapture_examples/jmex"]
[tool.hatch.build.targets.wheel]
sources = ["python"]Nothing checks where |
Description
Adds support for the j-mex AgileMaster motion capture suit: a device plugin that publishes the
operator's body pose as
FullBodyPoseunder a newbody.jmexvendor, the core registration thatlets Isaac Capture read it, and an Isaac Lab example that drives a Unitree G1 from it. None of this
is upstream yet; the plugin, the example and their tests are all new.
AgileMaster has three parts: the Mocap Suit the operator wears; MOXI Player, which solves the suit
onto a skeleton and streams it on a numbered channel; and the MOXI Receiver SDK, which the plugin
links against.
The plugin (
src/plugins/jmex/) receives one Player channel and fills the 24BodyJointslotsfrom the standard human skeleton loaded in Player, one slot per joint, in OpenXR axes. It pushes one
FullBodyPoseper frame, on thejmex_full_bodycollection by default. A slot with no joint, orwith no usable value in a frame, goes out with
is_validfalse; nothing is inferred fromneighboring joints. The raw device clock carries the time Player stamped on the frame
(
--device-clock=localuses the pickup time instead). If the CloudXR runtime restarts underneathit, the plugin re-establishes its session and carries on. An argument it cannot use, such as a
misspelt flag or a channel outside 1–255, stops it with exit code 1 and names the argument. It runs
on Linux and needs MOXI Receiver SDK 0.8.0 or newer. Without the SDK, configure prints
Skipping jmex plugin build: MOXIReceiverSDK 0.8.0 or newer not foundand builds only the plugin'sSDK-free core library, which the tests link; the rest of the tree builds as before.
Where the conversion happens. Player streams its own skeleton. The plugin maps it onto the 24
slots and into OpenXR axes (
body_joint_mapping,openxr_convention); Player does not emitFullBodyPose. There are two reasons for this. The change of axes was measured in this tree and ispinned by the plugin's tests, next to the code that consumes the slots. Done in Player, those tests
would live where this repository's CI never runs them. And Player does not have to produce Isaac
Capture's internal schema, so a change to
FullBodyPosestays in this tree instead of waiting on aPlayer release.
Registration in core. Outside the new directories, each change adds an entry to an existing
list. The one existing line changed is the Linux-only note in
tests/cpp/plugins/CMakeLists.txt,which now names jmex too.
src/core/live_trackers/cpp/live_full_body_tracker_jmex_impl.{hpp,cpp}body.jmex. It reads the plugin's collection and takes one vendor parameter,collection_idsrc/core/live_trackers/cpp/CMakeLists.txtlive_deviceio_factory.{hpp,cpp}try_create_*function; one row ink_tracker_dispatch, after the existing full-body vendors, soFullBodyTracker's default vendor is unchanged; one branch invalidate_vendor_selectionstests/cpp/core/live_trackers/test_vendor_validation.cppSECTIONs forbody.jmex, beside the existing vendors'src/plugins/CMakeLists.txtadd_subdirectory(jmex)in the ungated list; the plugin skips itself off Linux, and without the SDK builds only its core librarytests/cpp/plugins/CMakeLists.txtadd_subdirectory(jmex)for the plugin's tests, in the Linux-only blocktests/python/examples/CMakeLists.txtadd_subdirectory(jmex)for the example's testscmake/ClangFormat.cmakedocs/source/_data/devices.yamlThe example (
examples/jmex/, packageisaaccapture_examples.jmex) has two parts:jmex_g1_tasksregistersIsaac-PickPlace-Locomanipulation-G1-Jmex-Abs-v0, which inherits IsaacLab's
LocomanipulationG1EnvCfg(Isaac Lab 3.0.0 EA or 3.0.0 beta2) and takes its action from theoperator; nothing in Isaac Lab is modified. It is upper-body teleop. Each arm is rebuilt segment
by segment onto the G1's own shoulder, so the operator's arm angles carry over at the robot's arm lengths, and the elbows and
shoulders get IK targets from that same chain. The wrist's orientation is the operator's hand,
carried through a forearm frame. The operator's torso turns the robot's shoulders about its waist,
so leaning, bending sideways and twisting come through; their heading does not, so the operator
can turn to face a screen without moving the robot. Only a frame that carries all 24 slots drives
the robot.
record_jmex_full_bodyrecords the same stream to a standard full-body MCAP(
core.FullBodyPoseRecordon thefull_bodychannel), which the sharedreplay_full_bodyscriptplays back unmodified.
Type of change
Testing
The tests in this PR need no SDK, so CI builds and runs them.
ctest -L jmexselects 24: theplugin's 23 Catch2 cases (argument parsing, slot mapping, change of axes, device clock choice, one
frame becoming one
FullBodyPose, session recovery after a failed push), which link the plugin'sSDK-free core library; and the example's Python test, whose 91 pytest cases cover its action source, including
the geometry of the rebuilt arm and the orientation carried through the forearm frame. The Python
test is registered when
BUILD_EXAMPLESandBUILD_VIZare on, as in the ubuntu and Windows jobs.The
body.jmexsections of the vendor validation test run underctest -R 'vendor validation'.Local gate, Ubuntu x86_64. We ran the same Python and build-type matrix as the x64 entries of
build-ubuntu(Python 3.11, 3.12 and 3.13, each in Debug and Release) without the SDK, each from anempty tree, twice: once with CI's toolchain (GCC 11.4, CMake 3.31.6, uv 0.10.6, clang-format 14.0.0)
and once with GCC 13.3. Local builds left
BUILD_OVXR_CAMERA_QUADSandBUILD_PLUGIN_OAK_CAMERAdisabled; CI enables both. In all twelve:
jmextests and the 3 vendor validation cases passed;.cppfiles that build without the SDK compiled without warnings (theother two are covered below);
ctest, 566 passed, 9 skipped and 3 failed. The three are upstreamGPU tests (
VkContextwith an explicit device index, and twocamera_viztests) that fail on thisworkstation's GPU setup and are unrelated to this change; CI's runners have no GPU and skip them.
The two plugin sources that compile only with the SDK were built against SDK 0.8.0 with GCC 13 and
syntax-checked with GCC 11, without warnings. With CI's toolchain we also repeated the Release
install with
auditwheel repair, thepip-editable-installjob (its import check, and itswheel-content tests: 10 passed) and the configure of
test-viz-sanitizers.SKIP=check-copyright-year pre-commit run --all-files,make current-docs(-W),scripts/check_doc_refs.py --all, ruff, and clang-format 14 on the changed files all pass. Windowsand arm64 were not built locally.
Since this PR comes from a fork, its workflows wait for a maintainer's approval. The same commit has
already been through the
Build Ubuntuworkflow on our fork, where all twelvebuild-ubuntuentries(x64 and arm64) passed, along with DCO and pre-commit:
https://github.com/jmex-sw/IsaacTeleop/actions/runs/41028280279
With the device. Driven live by an operator wearing the upper-body part of the suit, with MOXI
Player on the same machine (Isaac Lab 3.0.0 beta2 on Isaac Sim 6.0.1, XR running, retargeting in sync
mode), the G1's waist followed lean, side bend and twist with a correlation of 0.97–0.99, 90–270 ms
behind. The operator went past the waist's limits in lean and twist, where the waist stops following.
An earlier run in the full suit with pipelined retargeting gave 0.99 at the same latencies, and
turning on the spot through ±90° moved the G1's yaw less than 12°. Replayed headless from a
recording, the waist followed all three with a correlation of 1.00, one or two physics steps behind.
On Isaac Lab 3.0.0 EA (Isaac Sim 6.1.0) the G1 followed a replayed recording with Kit's window open
and the reference overlay drawn.
Running it yourself. Isaac Lab needs this repository's
isaaccaptureinstalled in its Python:no released package carries the
body.jmexvendor yet, and Isaac Lab pins the package under itsprevious name. The example README has the steps. The SDK and MOXI Player are not distributed with
this PR. A maintainer validation package can be provided through an agreed private channel on
request.
Checklist
SKIP=check-copyright-year pre-commit run --all-filesgit commit -s) per the DCO