voice-agent

Demo A real-time voice agent that watches nuScenes driving footage with you and talks about it. Speak into the browser; a Qwen3 orchestrator answers directly or calls tools — look at the current scene (Cosmos-Reason2), compare two scenes side by side, search a FalkorDB knowledge graph of the scene library, or control playback. Replies stream back as speech, sentence by sentence, and you can interrupt the agent by talking over it.

All models are self-hosted (vLLM on GPU machines, reached over SSH tunnels).

browser mic ── 16 kHz PCM over websocket ──► FastAPI server
                                              │  VAD: calibration, endpointing, barge-in
                                              ▼
                                            faster-whisper STT
                                              ▼
                                            Qwen3-8B orchestrator (LangGraph tool loop)
                                              ├─ look_at_scene / compare_scenes ──► Cosmos-Reason2-8B
                                              ├─ search_scenes / open_comparison ─► FalkorDB graph
                                              ├─ control_playback ───────────────► UI events
                                              │
                                              │  session context injected per turn:
                                              │   · recent frame captions (precomputed perception)
                                              │   · ego + lead-vehicle turn telemetry (ground truth)
                                              ▼
                                            edge-tts ── mp3 per sentence ──► browser playback

Quickstart

Prereqs: Python 3.12, WSL2 with Docker Engine, the nuScenes v1.0-mini dataset extracted to ./v1.0-mini, and the model endpoints in .env reachable (see docs/OPERATIONS.md for the server-side commands and tunnels).

python -m venv .venv
.\.venv\Scripts\pip install -r requirements.txt
wsl docker compose up -d               # FalkorDB (6379) + Postgres/pgvector (5432)

# one-time data pipeline (already done if data/ and the graph are populated):
.\.venv\Scripts\python scripts\nuscenes_export.py      # scene mp4s -> data/scenes
.\.venv\Scripts\python scripts\build_scene_graph.py    # trajectory attrs -> data/index
.\.venv\Scripts\python scripts\load_falkordb.py        # bundle + attrs -> FalkorDB

.\.venv\Scripts\uvicorn server.main:app --port 3000

Open http://localhost:3000, press Start, allow the mic, stay quiet for one second (noise calibration), then talk.

Things to say: "what do you see right now?" · "is it safe to overtake?" · "find a scene with pedestrians at night" · "switch" · "compare this with another scene" · "is the van in front turning?" · "jump to the moment".

Configuration (.env)

Variable	Purpose
ORCH_BASE_URL / ORCH_MODEL	Qwen3 orchestrator (vLLM with tool calling). Unset = single-model fallback mode.
LLM_BASE_URL / LLM_MODEL	Cosmos-Reason2 vision endpoint. Unset = echo agent (pipeline testable with no models).
LLM_TEMPERATURE, LLM_*_PENALTY	Sampling for the vision model (repetition_penalty matters — see docs).
STT_URL	Remote GPU whisper service; unset = local faster-whisper (WHISPER_MODEL, WHISPER_DEVICE).
EMBED_URL	Cosmos-Embed1 service for semantic retrieval ranking; unset = caption-keyword ranking.
VIDEO_FILE	Startup scene (defaults to first file in media/).
TTS_VOICE	edge-tts voice.
DATABASE_URL, VECTOR_BACKEND	Postgres/pgvector (provisioned alternative vector backend).

Repository layout

server/
  pipeline.py     VAD, barge-in, turns, playback actions, session controls
  orchestrator.py Qwen3 tool loop + session context + tracing + durable memory
  search.py       FalkorDB graph search, contrast pick, recent captions
  trajectory.py   ego + lead-vehicle turn segments from telemetry
  nuscenes_tables.py · scene_graph.py · stt.py · tts.py · video.py · config.py · main.py
scripts/          data pipeline (export / graph build / FalkorDB load)
                  + deployable GPU services (stt_server.py, embed_server.py)
web/              single-page UI: mic worklet, dual players, result cards, chat
nuscenes_graph_bundle/  Cosmos-Embed1 vectors + Cosmos-Reason2 annotations (prebuilt)
data/             exported scene mp4s + built indexes
docs/             ARCHITECTURE.md (how it works) · OPERATIONS.md (runbook)
run.py            launcher (Windows event-loop setup) — use this, not bare uvicorn

Status

Working and end-to-end tested: voice loop with barge-in, orchestrated tool calling, vision Q&A and scene comparison, graph search with result cards, playback control by voice and UI, ego + lead-vehicle telemetry answers, precomputed temporal context. Pending: GPU STT deployment (biggest latency + accuracy lever), Cosmos-Embed1 service (semantic ranking), observability stack, packaging/CI. See docs/OPERATIONS.md for the full runbook and docs/ARCHITECTURE.md for design rationale and known limitations.