[hougantc-nvda]

Description

Review stack · 4/5: #8787 → #8788 → #8789 → #8790 → #8791

Review only this layer. These PRs target main; the comparison excludes predecessors.

Add robot-camera panels (PiP) and a desktop mirror to OVXR teleoperation. Live and MCAP replay entrypoints enable the mirror by default; --headless disables the window while retaining headset streaming and selected PiP. The programmatic adapter keeps its opt-in default.

• Lab selects cameras and supplies Isaac Capture's CameraPanelConsumer and OwnedDesktopMirror through OvxrSessionBinding.frame_consumers.
• The renderer and lease own camera allocation, scene publication and native-resource lifetime; Isaac Capture owns presentation.
• Original camera-feed values are validated before layout/conversion, including update-rate and placement inputs. Zero-rate sentinel behavior is preserved.

GR1T2 and G1 locomanipulation expose a robot-view camera. Fixed-base G1 adds two wrist feeds. Breaking observation change: its default policy includes head/wrist images, including non-XR use. State-only callers must disable all three scene cameras and policy terms, and clear image_obs_list before construction; the task guide provides the exact recipe. SO101 has no declared PiP camera.

Panels use head-locked placement and the existing renderer. The Vulkan mirror displays completed eye or camera images; closing it requests orderly shutdown. PiP/mirror use the CUDA 13 package selection. The mirror requires Python 3.12+, the desktop-mirror extra and a usable GPU-backed display.

See camera and mirror setup.

A failed camera drain retains the environment instead of releasing resources that a camera may still use. Cleanup preserves the first error.

Installation: use published isaaccapture==1.7.93a1 from NVIDIA’s Python index with the scoped OVXR installation. The guide creates the optional ovxr selection; project-wide Kit/provider defaults are unchanged.

Type of change

• New feature
• Bug fix
• Documentation update

Testing

The combined stack passed 877 tests across 33 affected test files, with no failures or skips. This includes the native Newton graph-capture/replay/reset tests, renderer/lease contracts, recorded-control gating, camera cleanup failures, registered task selection and CPU collision checks, plus the inherited upstream noise-reset cases. Formatting, all five changelog boundaries and the warning-as-error documentation build pass. These are cumulative integration results; individual PR boundaries received static/API and changelog checks. Runtime and performance scope is detailed in #8791.

Remaining qualification: native Kit execution and MCAP-file playback were not rerun; control/replay owner tests substitute external I/O. Native consumer-destruction failure is not established by normal shutdown. Cold renderer attachment, intermittent GR1T2 first-Play recovery and SO101 cube retention remain open. Existing USD, render-product GPU-assignment and native runtime warnings are retained; this is not a warning-free qualification.

Checklist

• Contribution guidelines read
• Changed-file formatting/lint and Python 3.12 changelog checks pass
• Public documentation and release fragments updated
• Affected behavior and cumulative IWER regression checks complete within the scope above
• Native Kit and physical-headset qualification at this revision
• No new warnings

[hougantc-nvda]

@greptile Please review the latest changes in this PR.

[greptile-apps]

Confidence Score: 4/5

[High impact] The PR adds a new XR rendering path that shares the renderer across teleoperation and simulation workflows.

Fix the publication test fixture before merging.

Findings

1.  Publication tests fail during setup ▶

Summary

This PR adds optional OVXR camera panels and a desktop mirror. The latest changes publish visual materials through the render lease and prepare registered humanoid tasks before scene construction.

• Live teleoperation and replay can use OVXR, with optional camera panels and a desktop mirror.
• OVRTX and OVStage can be borrowed safely by an external renderer.
• Robot cameras now supply view feeds for contributed humanoid teleoperation tasks.
• Contributed robot tasks can run with kitless physics backends and task-specific teleoperation helpers.
• Pink hand joints keep their expected order, and Warp proxy arrays expose their updated operations.
• Headless XR keeps updating its scene, and applications can control simulation-context cleanup.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Task[Registered task] --> Prepare[Prepare selected physics]
    Prepare --> Env[Construct environment]
    Env --> Materials[Register visual materials]
    Materials --> Publish[Publish scene through render lease]
    Publish --> Capture[Capture renders completed images]
    Capture --> Panels[Optional camera panels]
    Capture --> Mirror[Optional desktop mirror]

Loading

_{Reviews (4) · Last reviewed commit: "Bind material addresses after deferred c..." · Reviewed by Greptile}

[hougantc-nvda]

@greptile Please review the current head (5421767) again. Recheck the fixes and remaining issues against the published code, including the actual interfaces and callers at this PR boundary.

[hougantc-nvda]

@greptile please review this PR again at the latest head (4d4579389784e7d22681b2d32cf6a97275ab489d), including the changes since your previous review.

[isaaclab-review-bot]

Isaac Lab Review Bot

Needs attention: Design and architecture (2), Scope and over-engineering (2), API (1), Compatibility and deprecation (7), Implementation (1), Style consistency (5), Test quality (5).

• PR description: The title understates renderer leasing, task backend preparation, and lifecycle changes; replace stale comparison and validation revisions with head-specific results, plus reproducible baseline/candidate XR frame-rate, step-rate, and peak-memory measurements.
• Design and architecture: Backend preparation has duplicate owners in HumanoidTeleopEnv and action_policy, while hand-presence policy bypasses IsaacTeleopDevice through private lifecycle state; centralize dispatch and expose a device property.
• Scope and over-engineering: Minimal implementation: extend OVRTXRenderer, task configs, and teleop entrypoints; keep render_lease, g1_kitless_newton, packing_table, gr1t2_kitless_action/_cfg, steering_wheel, teleop_physics_cfg, action_policy, ov_libraries_xr, ovxr_session, and xr_scene_partition; merge publication_buffers into OVRTXRenderer, delete teleop_env, and remove packing-table forwarding wrappers. Remaining flow is task selection → leased renderer → Capture consumers. Suggested PR breakdown: core fixes; task preparation; renderer lease; OVXR lifecycle; panels and mirror.
• API: OvLibrariesXrConfig exposes OvLibrariesXrCameraConfig in its public contract, but the lazy package export omits that required construction type; add it to all and the stub import.
• Compatibility and deprecation: Breaking changes: existing G1/GR1T2 registry classes, fixed-base G1 observations, SO101 reset defaults, optional-backend importability, and TeleopSessionLifecycle.stop failure behavior changed without deprecation bridges.
• Implementation: G1NewtonBoundedPinkInverseKinematicsAction allocates and stacks per environment on every IK step; cache stacked Pink limits and apply intersections and clamps as batched tensor operations.
• Style consistency: Cross-module helpers are exposed through underscore-private names and inconsistent aliases, obscuring their supported ownership; give shared helpers public names and use consistent relative imports within packages.
• Test quality: Added tests grep or AST-recompile production source, assert private module locations, and include assertion-free or unrelated negative controls; replace them with executed public-boundary behavior tests or delete them.

Significant concerns. Posted 23 inline findings.

The full PR diff was reviewed; some supplemental surrounding file context was omitted.

Automated review; human maintainers own approval decisions.

[hougantc-nvda]

@isaaclab-review-bot review