ros2-voice-command-robot

Talk to a ROS 2 robot and it drives there. Speech is cut into utterances with voice activity detection, transcribed, parsed into an intent ("go to the kitchen", "move forward 2 meters", "turn left", "patrol the lab and the hallway", "stop"), checked against safety limits and named places, and sent to Nav2 as NavigateToPose goals. The simulation is a TurtleBot3 in Gazebo with Nav2 from nav2_bringup on ROS 2 Humble.

The speech side follows the design of offline-voice-assistant: everything can run on the robot with no internet, and every model has a mock so CI needs no weights, microphone or GPU.

This is a portfolio demo by Arpan Bom. CI builds the package in ROS 2 Humble and tests the logic with mocks. It does not run Gazebo in CI, and no navigation performance numbers are claimed. The place coordinates in config/places.yaml were picked from the turtlebot3_world map for the demo.

Architecture

flowchart LR
  MIC[Microphone / WAV] --> VAD[Energy VAD<br/>utterance segments]
  VAD --> STT{Speech to text}
  STT -- default/CI --> MS[Scripted mock]
  STT -- stt:=whisper --> FW[faster-whisper<br/>on device]
  KB[Keyboard node] --> T
  MS --> T[/voice/transcript/]
  FW --> T
  T --> VC[voice_command_node]
  subgraph VC_core [core, no ROS imports]
    R[Rule grammar] -->|unknown| L[Local LLM<br/>OpenAI-compatible server<br/>JSON + validation]
    L -->|bad JSON / unknown place| R
    R --> P[GoalPlanner<br/>named places, limits]
    P --> S[CommandSession<br/>queue, preempt, stop]
  end
  VC --- VC_core
  VC -- /voice/goal --> NC[nav_commander_node]
  VC -- /voice/cancel --> NC
  NC -- NavigateToPose action --> NAV[Nav2]
  NAV --> GZ[Gazebo TurtleBot3]
  NC -- /voice/goal_result --> VC
  VC -- /voice/reply --> TTS[TTS / log]
  AMCL[/amcl_pose/] --> VC

Loading

Piece	File	Notes
VAD	core/vad.py	RMS energy with hangover, minimum speech length and padding
STT	core/stt.py	ScriptedSTT mock (default) or WhisperSTT (optional faster-whisper)
Intent	core/intent.py	Rule grammar first; for anything it cannot parse, a local LLM (llama.cpp server, Ollama or vLLM) returns JSON. Output is validated: unknown kinds, invented places and missing numbers fall back to the rules
Planner	core/planner.py	Named places from YAML, relative moves along the current heading (max 5 m), rotations, patrols (max 6 stops), "did you mean" hints
Session	core/session.py	One goal at a time; patrol queue; new commands preempt; "stop" cancels
ROS nodes	voice_command_node.py, nav_commander_node.py, mic_listener_node.py, keyboard_node.py	Thin wrappers around the core

Run the simulation

ROS 2 Humble with nav2_bringup and turtlebot3_gazebo installed:

mkdir -p ~/voice_ws/src && cd ~/voice_ws/src
git clone https://github.com/applycoding/ros2-voice-command-robot
cd .. && colcon build --symlink-install && source install/setup.bash
export TURTLEBOT3_MODEL=waffle
export GAZEBO_MODEL_PATH=$GAZEBO_MODEL_PATH:/opt/ros/humble/share/turtlebot3_gazebo/models

ros2 launch voice_command_robot voice_nav_sim.launch.py headless:=False
# set the initial pose in RViz, then in another terminal:
ros2 run voice_command_robot keyboard
you> go to the kitchen
you> patrol the lab and the hallway
you> stop

Speak instead of typing (needs pip install faster-whisper):

ros2 run voice_command_robot mic_listener --ros-args -p stt:=whisper -p wav_file:=command.wav

Use a local LLM for free-form commands (any OpenAI-compatible server, for example llama-server -m qwen2.5-1.5b-instruct-q4_k_m.gguf --port 8080):

ros2 launch voice_command_robot voice_nav_sim.launch.py intent_backend:=llm

Tests

pip install -r requirements-dev.txt
pytest

The tests cover the grammar, LLM output validation and fallback (with a fake HTTP client), the planner's limits and geometry, the session state machine, VAD on synthetic audio, an audio-to-goals pipeline run with the scripted STT, and the launch and package files. CI also runs colcon build and imports the nodes inside the ros:humble-ros-base image.

License

MIT