Skip to content

feat(teleop): show live camera feeds beside the 3D viewer#42

Open
cn0303 wants to merge 1 commit into
huggingface:mainfrom
cn0303:feat/teleop-camera-feed
Open

feat(teleop): show live camera feeds beside the 3D viewer#42
cn0303 wants to merge 1 commit into
huggingface:mainfrom
cn0303:feat/teleop-camera-feed

Conversation

@cn0303

@cn0303 cn0303 commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

What

Adds an optional camera panel to the teleoperation page so you can watch your cameras live while
you drive the arm; handy for verifying framing and catching a camera that came loose.

  • Off by default. Landing on the page never calls getUserMedia; the user opts in with a
    toggle; the same consent pattern as the calibration camera toggle (fix(calibration): don't open cameras until the user turns them on #21).
  • Mirrors the robot's configured cameras. The panel shows one live feed per camera on the
    selected robot record (e.g. wrist_cam, webcam), stacked vertically with their names.
  • Strict mirror. If the robot has no cameras configured, the panel shows nothing but a hint to
    add them during calibration — it never surfaces a device that wasn't deliberately configured.
  • A configured-but-currently-absent camera still appears (name + "preview failed" placeholder) so
    it's obvious it's expected but not detected.

Why this works during teleop

Teleoperation opens no cv2 cameras (SO101FollowerConfig has no cameras), so the browser is
free to stream them directly via getUserMedia while the arm runs. (Recording is different; there
cv2 owns the devices, so this browser-source approach is teleop-only by design.)

Implementation notes

  • CameraFeed is a small presentational component bound to a browser deviceId.
  • TeleopCameraPanel reads the selected robot's cameras via useRobots() and renders a CameraFeed
    per configured camera. A retry button remounts the feeds for a fresh getUserMedia attempt that is
    handy if a camera was unplugged and reconnected. (No backend camera enumeration: the configured
    cameras already carry the browser device_id we stream directly.)
  • Mounted in the teleop VisualizerPanel via a new rightSlot.

Testing

  • Manual: configured cameras show stacked with correct names; zero-config shows the hint only;
    unplugged-but-configured shows the placeholder.
  • npx tsc --noEmit clean; ESLint clean on the changed files (the repo's full npm run lint has
    pre-existing errors/warnings unrelated to this change); npm run build succeeds.
  • Source-only PR — frontend/dist is left to CI's build_frontend.yml to rebuild on merge.

Add an optional camera panel to the teleoperation page, off by default so
landing on the page never calls getUserMedia (same consent pattern as the
calibration camera toggle). Teleoperation opens no cv2 cameras, so the
browser can stream them directly while the arm runs.

The panel mirrors the selected robot's configured cameras: it shows a live
feed for each camera on the robot record (e.g. wrist_cam, webcam),
stacked vertically, and shows nothing when the robot has none -- teleop
never surfaces a device that was not deliberately configured.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant