[chrisvoncsefalvay]

Description

Adds reusable XR retargeters for da Vinci Research Kit (dVRK) Patient Side Manipulators (PSMs).

The retargeters provide a simulator-independent boundary between an XR controller and a PSM integration: they emit a bounded absolute tool-tip pose and paired jaw-joint targets, while robot-specific differential IK, articulation control, collision handling, and emergency-stop behaviour remain with the downstream consumer.

What is included

• DVRKPSMClutchRetargeter emits a float32 ee_pose in [x, y, z, qx, qy, qz, qw] order.
  • Requires an explicit home transform.
  • Supports either controller hand, translation scaling, workspace bounds, orientation calibration, and a configurable clutch threshold.
  • Captures the controller origin on engagement, applies absolute translation and conjugated relative rotation, and re-clutches without a target jump.
• DVRKPSMGripperRetargeter emits float32 jaw_targets for the two native PSM jaw joints.
  • Supports configurable open and closed endpoints, initial closedness, clutch threshold, trigger deadband, and opening-intent dwell.
  • Closing responds immediately; opening requires deliberate trigger release observed for the configured graph-time duration.
• NumPy-only Cartesian-clutch and jaw-intent state machines keep the control semantics reusable without importing a simulator runtime.
• The four public config and retargeter classes are lazy-exported from isaacteleop.retargeters and included in Python package staging.
• The retargeting reference now includes a dVRK overview, controller and frame contracts, configuration guidance, and a runnable right-controller pipeline with explicit source connections.

Control and safety semantics

The squeeze input (deadman) gates target updates. Releasing it holds the last commanded target and re-arms controller-origin capture; it does not disable downstream actuation or latch the measured articulation state. Command disable, measured-state hold, joint and collision limits and emergency-stop behaviour remain the responsibility of the simulator or robot integration.

The output is designed to be fail-safe, i.e. tracking loss, inactive sessions, missing controller data, malformed poses, degenerate quaternions and non-finite samples hold the last safe output and require a fresh engagement. Configuration values that will be narrowed to float32 are validated against the finite float32 range.

The jaw retargeter preserves the last jaw target across clutch release and tracking transitions. Its opening dwell uses observed graph time rather than frame count, including protection against repeated or regressed timestamps.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

Testing

• Release CMake configure and build: passed.
• Full CTest suite: 145/145 passed.
• Focused simulator-independent dVRK suite: 58/58 passed.
• Mypy: 42 source files clean, including the new dVRK modules in the repository CTest target.
• Sphinx documentation build with warnings treated as errors: passed.
• Runnable documentation pipeline construction: passed.
• Ruff lint and format checks: passed.
• SKIP=check-copyright-year rtk uvx pre-commit run --all-files: passed.
• REUSE/SPDX and git diff --check: passed.

Coverage includes clutch state transitions, workspace clipping, no-jump re-engagement, an independent non-commuting orientation-conjugation oracle, paired jaw interpolation, deliberate opening dwell, graph-time regressions, malformed and non-finite samples, float32-range configuration rejection and, hopefully, public wrapper/kernel parity.

Scope

This PR contains the reusable dVRK retargeters, public API and configuration, focused tests, packaging, and retargeter documentation only. For the sake of separation of concerns, the Isaac Sim/Lab example, diff IK and articulation validation will be in the downstream dual PSM integration.

Checklist

• I have read and understood the contribution guidelines
• I have run the linter and formatter with SKIP=check-copyright-year pre-commit run --all-files
• I have made corresponding changes to the documentation
• I have added tests that prove my fix/feature works (or explained why not)
• I have signed off all my commits (git commit -s) per the DCO

Summary by CodeRabbit

• New Features

  • Added dVRK Patient Side Manipulator retargeters for clutch-controlled end-effector pose and trigger-controlled paired-jaw targets.
  • Added configurable workspace limits, clutch thresholds, jaw endpoints, initial jaw positions, and orientation offsets.
  • Added robust handling for tracking loss, invalid input, clutch transitions, and reset behavior.
  • Exposed the new retargeters through the public package interface.

• Documentation

  • Added setup guidance, input/output contracts, frame conventions, usage examples, and test instructions for dVRK retargeting.

• Tests

  • Added comprehensive coverage for pose, jaw, clutch, reset, validation, timing, and invalid-input behavior.

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/IsaacCapture/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 838ec848-426b-489c-bf7b-4b48bb095393

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Adds dVRK PSM clutch and gripper retargeters with NumPy state machines for absolute tool poses and paired jaw targets. The change exposes public configurations and retargeter classes, integrates the package into builds, adds extensive simulator-free tests, and documents behavior, reference frames, Python wiring, and validation commands.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ControllersSource
  participant DVRKPSMClutchRetargeter
  participant DVRKPSMGripperRetargeter
  participant OutputCombiner
  participant SimulatorIK
  ControllersSource->>DVRKPSMClutchRetargeter: grip pose and squeeze
  ControllersSource->>DVRKPSMGripperRetargeter: trigger and squeeze
  DVRKPSMClutchRetargeter-->>OutputCombiner: ee_pose
  DVRKPSMGripperRetargeter-->>OutputCombiner: jaw_targets
  OutputCombiner->>SimulatorIK: combined pose and jaw targets

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: adding dVRK PSM retargeters.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

📝 Docs preview is not auto-deployed for fork PRs.

A maintainer with write access to NVIDIA/IsaacTeleop can deploy a preview by
commenting /preview-docs on this PR. Once deployed, the preview
will live at:

https://nvidia.github.io/IsaacTeleop/preview/pr-769/

[Copilot]

Pull request overview

Adds simulator-independent XR retargeters for the dVRK Patient Side Manipulator (PSM), providing a clean boundary that outputs a bounded absolute tool pose plus paired jaw targets for downstream IK / articulation control.

Changes:

• Introduces DVRKPSMClutchRetargeter (7D ee_pose) and DVRKPSMGripperRetargeter (2D jaw_targets) with NumPy-only control kernels.
• Wires the new dVRK retargeters into the Python package surface and build/test tooling (mypy path, package staging).
• Adds extensive simulator-free unit tests and new retargeter documentation/reference material.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/retargeters/DVRK/psm_retargeter.py	New engine-facing retargeter wrappers for pose clutch and jaw targets.
src/retargeters/DVRK/control.py	New NumPy-only clutch + jaw-intent state machines and math helpers.
src/retargeters/DVRK/init.py	Lazy-export surface for the DVRK retargeter subpackage.
src/retargeters/CMakeLists.txt	Ensures DVRK __pycache__ is removed during Python package staging.
src/retargeters/init.py	Exposes new dVRK retargeters/configs from isaacteleop.retargeters.
src/core/retargeting_engine_tests/python/test_dvrk_psm_retargeters.py	New simulator-free test suite for dVRK PSM retargeters and kernels.
src/core/retargeting_engine_tests/python/CMakeLists.txt	Adds mypy coverage for the new src/retargeters/DVRK modules.
src/core/python/pyproject.toml.in	Includes isaacteleop.retargeters.DVRK in the packaged modules list.
docs/source/references/retargeting/index.rst	Adds dVRK entry to the retargeter reference index.
docs/source/references/retargeting/dvrk.rst	New documentation page describing dVRK PSM retargeter usage and contracts.

[chrisvoncsefalvay]

@jiwenc-nv &al. - this is very closely linked with a PR in Isaac Lab that's contingent on it: #6527, and might make sense to be reviewed together. The latter is still in draft, so when this is merged, I can undraft it.

[jiwenc-nv]

Hey Chris, thanks for contributing. Will take a look soon. Wondering whether you have more context on the overall effort? Looks interesting, but wanna to know how we can better support your work.

[chrisvoncsefalvay]

Hi @jiwenc-nv, very fair point, and my apologies for not getting this in sooner.

The dVRK (da Vinci Research Kit) is the closest that surgical robotics has to a SO-ARM 101. It is a research tool made available by Intuitive Surgical (ISRG), who make the da Vinci surgical robots. These consist of four arms, of which typically one holds an endoscopic camera (the ECM) and up to three hold what are known as patient-side modules (PSMs), which are the tool arms. They come loaded with instruments such as needle drivers, forceps &c. Unfortunately, you can't just go out and buy a dVRK - there are around 35 or so of them around in the whole world, all of them at universities, and the requirements to get one are fairly stringent. This makes developing policies and testing them on surgical tasks rather difficult.

At the same time, exactly because of the academic abundance (relative to all other surgical robots) of the dVRK, as well as the fact that it's effectively open-source and is arguably the closest thing we have to a broadly adopted, purpose-built research embodiment, and because it has an explicit non-clinical research boundary, it is very widely used. Open-H-Embodiment, an NVIDIA-sponsored initiative to collect surgical robotic training data, uses it extensively (around 1/7th of the data comes from dVRK or da Vinci units, which are kinematically identical - the CMR Versius is more abundant but less open), and most of the basic robotic surgical routine primitives (needle passing being one example) are implemented.

It is thus an optimal embodiment to offer an entry into surgical robotic teleoperation. Not only is this a way for users without access to a dVRK to collect high quality teleop data, it is also representative of a class of retargeters that is not present here (there are some kinematic quirks to how laparoscopic surgical robots work that are not replicated by the conventional structures) and since it's almost always (at least) bimanual, it's a good veer into that realm as well.

I have previously explored dVRK + Isaac in LeOnardo⁠, an Isaac Sim extension for bimanual PSM teleoperation using two SO-ARM 101 leader arms. That was useful proof that the combination works, but it also reinforced that there is little value in maintaining a separate bespoke stack when IsaacTeleop, Isaac Lab, OpenXR/CloudXR and Isaac Sim already form such a strong ecosystem. I would much rather put the reusable control seam upstream here and keep task-specific IK, articulation, contacts, limits and emergency-stop behaviour downstream.

An example of an immediate downstream use case is now IsaacLab #6570⁠, which supersedes the earlier #6527. It adds a contact- and load-qualified bimanual needle pass—one of the foundational subtasks within suturing—with paired-controller XR teleoperation. The same primitive already supports SurgiSabre⁠, essentially Beat Saber for surgical robots: a deliberately playful, low-stakes familiarisation environment for clutching, remote-centre motion and workspace limits. Longer term, I would love to see this shared dVRK teleoperation layer become part of the backbone of CERTIS⁠, an Isaac-native collection of composable surgical tasks for teleoperation, demonstration collection, training and reproducible evaluation, and also be a starting point for others developing in surgical robotics.

I have deliberately kept #769 small and simulator-independent: controller input in, bounded PSM pose and jaw targets out. I would be grateful for any guidance and support on making sure that my way of making this work aligns with what your longer term architectural ideas are for IsaacTeleop - after all, I'm the guest here, I have tried to align as well as I could with the preexisting codebase but happy to accommodate any changes.

[jiwenc-nv]

Hi @jiwenc-nv, very fair point, and my apologies for not getting this in sooner.

The dVRK (da Vinci Research Kit) is the closest that surgical robotics has to a SO-ARM 101. It is a research tool made available by Intuitive Surgical (ISRG), who make the da Vinci surgical robots. These consist of four arms, of which typically one holds an endoscopic camera (the ECM) and up to three hold what are known as patient-side modules (PSMs), which are the tool arms. They come loaded with instruments such as needle drivers, forceps &c. Unfortunately, you can't just go out and buy a dVRK - there are around 35 or so of them around in the whole world, all of them at universities, and the requirements to get one are fairly stringent. This makes developing policies and testing them on surgical tasks rather difficult.

Very thoughtful and nice writeup! Thanks for posting. I will encourage you to include a version of that explanation into: docs/source/references/retargeting/dvrk.rst as well.

Also on dvrk.rst, I'd recommend you include a gif (no larger than 5MB please to show case the teleoperation you have).

We are also brining in SO-101 leader arm within Isaac Teleop as well. (also more leader arm to come), would love you to also think about how those hardware can be useful for you as well.

@rwiltz can you take a pass on the PR as well? as you are the expert for our retargeting pipeline.

[chrisvoncsefalvay]

@jiwenc-nv added, thanks!

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

[chrisvoncsefalvay]

Thanks @jiwenc-nv and @rwiltz - ought to hopefully be good to go!

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/python/isaacteleop/retargeters/DVRK/control.py:566

• In the jaw-intent state machine, the initial “reference-capture” thresholds use >= / <= against trigger_deadband (lines 558 and 564). If trigger_deadband is configured to 0.0 (allowed by validation), a zero trigger delta satisfies trigger_delta <= -trigger_deadband and can start (and eventually mature) an opening intent without any actual trigger release, which then bypasses the deliberate-opening guard on subsequent samples. Using strict comparisons avoids this edge case and matches the docstring wording (“must exceed”).

        if not self._interaction_started:
            # Reference capture makes re-clutch a no-op until the operator
            # moves the index trigger far enough to be unambiguous.
            if trigger_delta >= self._config.trigger_deadband:
                self._interaction_started = True
                self._cancel_opening()
                self._set_closedness(self._closedness + trigger_delta)
                self._last_trigger = trigger_value
                return self.targets
            if trigger_delta <= -self._config.trigger_deadband:
                if self._opening_anchor_trigger is None:

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (4)

src/python/isaacteleop/retargeters/DVRK/control.py:40

• np.ndarray implements __dlpack__ in recent NumPy versions, so this will route NumPy arrays through np.from_dlpack(...) unnecessarily. Consider prioritizing np.asarray(...) and only using np.from_dlpack(...) for non-NumPy DLPack producers (e.g., hasattr(values, '__dlpack__') and not isinstance(values, np.ndarray)) to avoid extra overhead and potential DLPack ownership semantics surprises.

def _as_vector(values: object, *, length: int, name: str) -> np.ndarray:
    """Return a finite vector, accepting NumPy and DLPack producers."""
    if hasattr(values, "__dlpack__"):
        values = np.from_dlpack(values)
    vector = np.asarray(values, dtype=np.float64)
    if vector.shape != (length,) or not np.all(np.isfinite(vector)):
        raise ValueError(f"{name} must be a finite vector with shape ({length},)")
    return vector.copy()

src/core/retargeting_engine_tests/python/test_dvrk_psm_retargeters.py:1047

• This test asserts against a private implementation detail (_clutch_state). To reduce brittleness, prefer asserting purely via public behavior/outputs (e.g., verify that a subsequent squeeze frame only latches without changing the emitted pose) rather than peeking into private state.

        assert not retargeter._clutch_state.engaged

src/core/retargeting_engine_tests/python/test_dvrk_psm_retargeters.py:140

• The docstring says the transform is built with identity orientation, but the helper accepts (and applies) an optional rotation. Update the docstring to reflect the true behavior (identity unless rotation is provided).

def _make_home_transform(position, rotation: np.ndarray | None = None) -> np.ndarray:
    """Build a PSM base-to-tool home transform with identity orientation."""
    transform = np.eye(4, dtype=np.float64)
    if rotation is not None:
        transform[:3, :3] = rotation

src/python/isaacteleop/retargeters/DVRK/psm_retargeter.py:59

• The local alias _TRIGGER_DEADZONE is later used as the default for the config field named trigger_deadband. Renaming the alias to something like _DEFAULT_TRIGGER_DEADBAND (or aligning the config field name) would avoid mixing two similar terms for the same concept.

    DEFAULT_OPENING_INTENT_DURATION_S as _DEFAULT_OPENING_INTENT_DURATION_S,
    DEFAULT_TRIGGER_DEADZONE as _TRIGGER_DEADZONE,

[chrisvoncsefalvay]

@jiwenc-nv Thanks for looking after this. As far as I can discern, this seems to be a CI config side issue with puling the cxr SDK that was recently introduced. Let me know if this needs anything from me/if there's any way I can assist!

[chrisvoncsefalvay]

Updated to pass the cloudxr setup hitch - this should, given a CI run, pass fine. @ivany-nv / @rwiltz / @jiwenc-nv - let me know if this needs anything else from my end! Thanks!

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

[chrisvoncsefalvay]

@rwiltz / @jiwenc-nv — PR #769 is conflict-free and the existing review approval still counts, but the four Actions suites on 734f are awaiting maintainer approval. Would one of you approve and run the workflows? I’ll leave the branch untouched unless the fresh run exposes a real failure. Thanks!

[ivany-nv]

🚫 BLOCKER — control.py#L352: rotation and translation use different frames

L352 right-composes the rotation delta onto the engagement orientation (q_engage ⊗ q_delta), so it acts in the tool frame. Ten lines up, control.py#L333-L342 applies translation as home_position + scale * (p_now - p_origin) — pure reference-frame arithmetic. With any non-identity home_orientation the two channels disagree.

Measured at head 7144734f by importing control.py (numpy only) and driving step(): engage upright, then rotate the hand +30° about reference Z. Reference-frame rotation vectors, degrees.

home_orientation	orientation_offset	Commanded tool rotation
identity — shipped default	identity	[0, 0, +30]	✅ 1:1
180° about X — PSM, tool down	identity	[0, 0, −30]	❌ inverted
60° about Y	identity	[25.98, 0, 15]	❌ axis rotated 60°
180° about X	conj(home_orientation)	[0, 0, +30]	✅ 1:1

The last row is the left-composed form, so both conventions are already reachable — the module never says which one it implements. control.py#L182 requires home, controller samples and workspace bounds to "share one reference frame": translation obeys that, rotation does not. SO101ClutchRetargeter has the identical translation formula and left-composes, documented as load-bearing — "a hand rotation about base Z maps to an EE rotation about base Z" (clutch_retargeter.py#L15).

No test combines a non-identity home_orientation with a rotation delta, so the 58 passing tests do not cover it.

---

💡 control.py#L248-L251: clutch_threshold=0.0 permanently defeats the deadman

Validated on the closed interval [0, 1] but compared with < at control.py#L306, so engagement is squeeze >= threshold. At 0.0 the tool tracks the controller and the jaws drive fully closed with squeeze at exactly 0.0. Same at L438-L441 / L537. SO101ClutchRetargeter pairs a strict > with [0, 1).

💡 control.py#L96: orthonormality atol=1e-5 rejects valid configs

The neighbouring _ROTATION_ATOL is 1e-4, measured in clutch_retargeter.py#L110-L124. Over 20 000 random SO(3), a home rotation written with 4 decimal literals is rejected 20000/20000 at 1e-5 against 2664/20000 at 1e-4; 5 decimals still fails 504/20000 against 0.

---

Note. Copilot's suppressed trigger_deadband=0.0 comment does not hold: at zero trigger delta the closing branch matches first and returns, so the opening branch is unreachable there.

[chrisvoncsefalvay]

Hi @ivany-nv, thanks for taking the time to work through this so carefully, esp the concrete rotation examples. All fair points.

I agree on the frame convention. Translation follows the shared reference axes, and rotation should do the same by default. I've fixed this to calculate the controller delta as q_now * inverse(q_origin) and apply the calibrated delta on the left of the held tool orientation. Both multiplication orders need to change for this to work when the controller is already rotated at engagement. I’ll also clarify the meaning of orientation_offset and extend the tests to cover non-commuting home and controller orientations, including after re-clutching. The existing test with a rotated home uses rotations about the same axis, which is why it missed this.
The zero clutch threshold is a good catch too. This will align both state machines with SO101’s strict squeeze > threshold rule and [0, 1) configuration range, too, with boundary tests for both pose and jaw targets. Zero squeeze should always hold the last target.

On matrix validation, I’m happy to relax the absolute tol to 1e-4 and add coverage for rounded configuration matrices, while retaining the checks that reject improper rotations. Agreed also on the suppressed zero-deadband point - the closing branch returns first without changing the target when the delta is zero. I’ll retain that behaviour and add an explicit regression test. Thanks for checking that one too. Cheers again for a very thorough and insightful review!

[rwiltz]

Hi @chrisvoncsefalvay, thanks for the dVRK retargeters. I've been running them end to end with the needle-pass task from isaac-sim/IsaacLab#6570, and they work well.

main has moved on since this branch was cut. The main change is that the Python package was renamed from isaacteleop to isaaccapture in cb12c0c (isaacteleop imports stay as an alias until 1.9). The
new retargeters/DVRK/ files therefore need to move into src/python/isaaccapture/.

I put together a version rebased onto current main: rwiltz/dvrk-psm-retargeters-isaaccapture (compare). It contains:

• Your commits, rebased with you as author. The only conflict was the mypy path in tests/python/core/retargeting_engine/CMakeLists.txt.
• One extra commit from me that moves DVRK/ into isaaccapture and updates the imports in the module, tests, mypy target, and the docs example. It's kept separate so it's easy to review.
• Your docs(agents) commit left out. The pre-commit advice is good, but it's repo-wide rather than dVRK-specific, so it would probably land more easily as its own small PR.

On that branch, pre-commit passes and all 27 retargeting_* tests pass, including the dVRK suite and mypy. from isaacteleop.retargeters import DVRKPSM... still resolves through the alias.

How would you like to proceed?

1. I force-push it to this PR's branch.
2. You rebase yourself, and my branch is just a reference.
3. I open it as a separate PR that credits you, and we close this one.

[chrisvoncsefalvay]

Hi @rwiltz - thank you! I'm fine with 1 if force pushes are generally ok here. Thanks again for your work on this.

[chrisvoncsefalvay]

@rwiltz this ought to be mergeable - let me know if this is in line with what was discussed. Thanks again!