Skip to content

Add kitless teleoperation and replay with SO101 support - #8788

Draft
hougantc-nvda wants to merge 9 commits into
isaac-sim:mainfrom
hougantc-nvda:hougantc/kitless-live-recorded-teleop
Draft

hougantc-nvda wants to merge 9 commits into
isaac-sim:mainfrom
hougantc-nvda:hougantc/kitless-live-recorded-teleop

Conversation

@hougantc-nvda

@hougantc-nvda hougantc-nvda commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Description

Connect live Capture tracking and recorded MCAP input to kitless simulation through the OVXR adapter from #8787. SO101 gains CPU OvPhysX selection and the current Capture home-transform, clutch and gripper integration.

  • Teleop integration: isaaclab_teleop owns control events, target-frame rebasing, input policy and session cleanup; task-specific action preparation remains with the task.
  • Runtime selection: reuse upstream Run the teleop scripts without Kit when nothing needs it #8828's non-Kit keyboard/gamepad/SpaceMouse support and Capture 1.7 APIs. Kit keeps its separate --xr path; kitless uses --ovxr.
  • Replay: MCAP supplies tracker/control input while the connected headset supplies the view. Reset resets the environment without rewinding the recording. HDF5 recording/replay remains on the Kit path.

Shared fixes address empty CPU Warp arrays, Newton graph capture overlapping legacy-stream uploads, and simulation-context ownership after a failed asynchronous drain. Kit fixes defer native USD imports until runtime launch and keep headless XR updating without changing ordinary headless on-demand capture.

See the live/replay runtime guide.

Builds on #8787. This draft targets main and includes its predecessors; review the incremental diff.

Dependencies: compatible Capture OVXR packages and the provider upgrade in #8627. Native-provider defaults are unchanged; use the source-build setup until matching packages and metadata are released.

The CloudXR.js push-device default remains 0. Root pyproject.toml and uv.lock retain upstream policy; NumPy compatibility for the source-built OVXR environment is explicit local setup, not a project-wide pin.

Type of change

  • New feature
  • Bug fix
  • Documentation update

Testing

Results cover the cumulative stack, not an independent full-suite run of this PR.

  • The cumulative stack passed 1,062 cases across 51 selected Lab/Kit test files, including control/replay/bootstrap and failure cleanup, CPU Warp conversion, GPU Newton and 52 Kit teleop startup cases.
  • IWER/headset trials exercised SO101 clutch/gripper/Reset and humanoid control; SO101 cube retention remains unresolved. Kit headless and Scene UI/camera checks passed; Kit Franka XR passed stereo, motion, Reset, reconnect and shutdown.

Formatting, changelog, incremental warning-as-error Sphinx and the kitless dependency check passed. The clean Sphinx check was blocked by external-inventory DNS failures.

Open limits: cold attach over 300 seconds, intermittent GR1T2 first-Play/Reset recovery, SO101 cube retention and eight Kit SDK metadata mismatches remain unresolved. Full-repository, hosted-CI, arm64 and full task-success coverage are not claimed. See #8791 for the exact validation checkpoint, performance samples and limitations.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with uv run isaaclab --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have validated the changed behavior with focused coverage and applied the test-audit criteria to any test changes
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump package versions — CI handles that)
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

The warnings checkbox remains open: dependency warnings and the separate Kit metadata audit are unresolved.

@github-actions github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team infrastructure labels Oct 23, 2026
@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-live-recorded-teleop branch 2 times, most recently from e933899 to 4d7c54f Compare October 26, 2026 17:12
@hougantc-nvda hougantc-nvda changed the title Add live teleoperation and MCAP replay through OVXR Add kitless teleoperation and replay with SO101 support Oct 26, 2026
Consolidate the reviewed PR boundary before rebasing onto current main.
Previous cumulative head: 12208c4

Signed-off-by: Hougant Chen <hougantc@nvidia.com>
Signed-off-by: Hougant Chen <hougantc@nvidia.com>
Consolidate the reviewed PR boundary before rebasing onto current main.
Previous cumulative head: 48f8b49

Signed-off-by: Hougant Chen <hougantc@nvidia.com>
Signed-off-by: Hougant Chen <hougantc@nvidia.com>
Signed-off-by: Hougant Chen <hougantc@nvidia.com>
Signed-off-by: Hougant Chen <hougantc@nvidia.com>
@hougantc-nvda
hougantc-nvda force-pushed the hougantc/kitless-live-recorded-teleop branch from 48f8b49 to e118cbc Compare October 27, 2026 16:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation infrastructure isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant