Add voice-controlled home agent with skills and guardrails

[nfaggian]

Summary

Adds a home voice agent you can talk to from your iPhone to control smart home devices. Built on Google ADK with Gemini Live API for real-time bidirectional voice.

What's included

Native iOS app (ios/) — recommended

• SwiftUI iPhone app with real microphone capture and audio playback
• Connects to the Python voice server over WebSocket
• Settings screen for server host + TLS
• Open ios/HomeVoiceAgent.xcodeproj in Xcode, set your team, run on device

Home agent (src/agents/home_agent/)

• Skills system — enable/disable tool modules via config/skills.yaml
  • home_assistant — list/control entities through Home Assistant REST API
  • routines — predefined multi-device routines (goodnight, away, movie mode)
• Guardrails — safety policies via config/guardrails.yaml
  • Block dangerous topics and protected entities
  • Restrict controllable HA domains
  • Require confirmation for sensitive tool calls
  • Temperature bounds for climate control

Voice server (src/server/)

• FastAPI WebSocket server with ADK streaming (text + audio)
• Browser PWA client (alternative to native app)

Tests

• Skills loading
• Guardrail enforcement (topics, domains, confirmations)

How to use (native iOS)

1. Copy .env.example → .env and set Gemini + Home Assistant credentials
2. Run make voice on your home server
3. Open ios/HomeVoiceAgent.xcodeproj in Xcode
4. Build to your iPhone (same Wi‑Fi as server)
5. Settings → enter 192.168.x.x:8000 → tap Start Voice

Architecture

flowchart LR
  iPhone[Native iOS App] -->|WebSocket audio/text| VoiceServer[FastAPI Voice Server]
  VoiceServer --> ADK[ADK home_voice_agent]
  ADK --> Skills[Skills Loader]
  ADK --> Guardrails[Guardrail Callbacks]
  Skills --> HA[Home Assistant API]

Loading

Notes

• Voice requires a Gemini model that supports the Live API (HOME_AGENT_MODEL)
• For remote iPhone access outside LAN, use HTTPS/WSS via Tailscale or a reverse proxy
• Update routine entity IDs (light.living_room, etc.) to match your Home Assistant setup
• Add an app icon in Xcode (Assets.xcassets → AppIcon) before App Store distribution