Ask questions about any video and get real-time, evidence-backed answers with clickable timestamp citations that jump to the exact moment in the video.
- YouTube & URL import — paste any YouTube link or direct video URL;
yt-dlpdownloads and indexes it automatically - Local video upload — drag & drop MP4, WebM, MKV, MOV, AVI, MP3, and more
- Speech-to-text — OpenAI Whisper (
smallmodel) transcribes audio with word-level timestamps - On-screen text (OCR) — Tesseract extracts text from sampled video frames
- Semantic search —
sentence-transformers/all-MiniLM-L6-v2+ FAISS vector index for fast, accurate retrieval - LLM-powered answers — Mistral (via Ollama) generates fluent, cited answers with real-time token streaming (SSE)
- Timestamp citations — every answer comes with clickable evidence cards; click to seek the video to that exact moment
- Conversation memory — the assistant remembers the last 2 turns for natural follow-up questions
┌─────────────────────────────────────────────────────────┐
│ Browser (React) │
│ UploadPanel │ VideoPlayer │ VideoList │ QuestionPanel │
└────────────────────────┬────────────────────────────────┘
│ HTTP / SSE
┌────────────────────────▼────────────────────────────────┐
│ FastAPI Backend │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────┐ │
│ │ yt-dlp │ │ Whisper │ │Tesseract │ │ FAISS │ │
│ │(download)│ │ (ASR) │ │ (OCR) │ │(index) │ │
│ └──────────┘ └──────────┘ └──────────┘ └────────┘ │
│ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Sentence Transformers → FAISS Vector Search │ │
│ └──────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Ollama / Mistral (SSE streaming) │ │
│ └──────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
- Download — yt-dlp fetches video from URL (skipped for local uploads)
- Transcribe — Whisper generates timestamped transcript segments
- Extract frames — sample one frame every 15 seconds
- OCR — Tesseract reads on-screen text from each frame
- Chunk — transcript + OCR segments are chunked (≤320 chars, ≤45 s span)
- Embed — sentence-transformers encodes every chunk into a 384-dim vector
- Index — FAISS flat inner-product index built over all chunk vectors
- Ready — video is searchable; Q&A available instantly
| Layer | Technology |
|---|---|
| Frontend | React 18, TypeScript, Vite, SSE streaming |
| Backend | FastAPI, Uvicorn, Python 3.11 |
| Speech-to-text | OpenAI Whisper (faster-whisper, small model) |
| OCR | Tesseract 5 + pytesseract + OpenCV |
| Embeddings | sentence-transformers/all-MiniLM-L6-v2 |
| Vector search | FAISS (CPU) with custom reranker |
| LLM | Mistral 7B via Ollama (streaming SSE) |
| URL import | yt-dlp |
| Deployment | Docker + Docker Compose + nginx |
| Tests | pytest (52 tests, unit + integration) |
- GPU Passthrough in Docker: Seamlessly leverages NVIDIA GPU hardware (
deploy.resources) within isolated Linux containers for blazingly fast Mistral and Whisper inference. - Handling Cross-OS File Locks: Implements robust retry loops for real-time JSON file reading to circumvent 0-byte OS-level race conditions between Windows hosts and Docker volumes during active SSE streaming.
- Reverse Proxy Optimization: Configured custom NGINX
client_max_body_sizethresholds to securely route and handle massive 500MB+ user video uploads bypassing default API limits. - Asynchronous Fault Tolerance: Built robust Python exception guards around
yt-dlpsubprocesses to intercept streaming crashes, preventing background worker thread-death and unlocking stuck UI states. - Hallucination Filtering: Intelligently drops phantom audio transcriptions (< 2 character word-noise) generated by Whisper during long instrumental video silence, keeping the semantic RAG context purely factual.
Prerequisites: Docker Desktop
# 1. Clone & start
git clone https://github.com/your-username/multimodal-rag.git
cd multimodal-rag
docker-compose up --build
# 2. Pull the Mistral model (first run only — ~4 GB)
docker exec -it $(docker ps -qf "name=ollama") ollama pull mistral
# 3. Open the app
open http://localhostThe stack runs on:
- Frontend → http://localhost
- Backend API → http://localhost:8000
- Ollama → http://localhost:11434
| Tool | Notes |
|---|---|
| Python 3.11+ | python --version |
| Node.js 20+ | node --version |
| Ollama | Install and run ollama pull mistral |
| Tesseract 5 | Windows: installer from link |
| ffmpeg | Windows: winget install ffmpeg |
# Create and activate virtual environment
python -m venv .venv
.venv\Scripts\activate # Windows
# source .venv/bin/activate # macOS/Linux
# Install dependencies
pip install -r backend/requirements.txt
pip install -r backend/requirements-optional.txt # Whisper, FAISS, etc.
# Start
cd backend
uvicorn app.main:app --reload --port 8000cd frontend
npm install
npm run dev # http://localhost:5173Copy backend/.env and adjust as needed:
# Ollama
MMRAG_OLLAMA_BASE_URL=http://localhost:11434
MMRAG_OLLAMA_MODEL=mistral
# Whisper (tiny | base | small | medium | large)
MMRAG_WHISPER_MODEL_SIZE=small
MMRAG_WHISPER_DEVICE=cpu
# Retrieval
MMRAG_RETRIEVAL_TOP_K=5
MMRAG_CHUNK_MAX_CHARS=320
MMRAG_FRAME_SAMPLING_SECONDS=15
MMRAG_MAX_OCR_FRAMES=12
# Upload limit
MMRAG_MAX_UPLOAD_MB=500Tests run without Whisper, Ollama, or any ML model — fast and CI-friendly.
cd backend
pip install -r requirements-test.txt
pytest tests/ -v52 passed in 0.93s
Coverage:
tests/test_chunking.py— 11 unit tests (chunking logic, OCR prefix, ordering)tests/test_retrieval.py— 27 unit tests (summarization detection, deduplication, scoring)tests/test_api.py— 14 integration tests (upload, delete, URL import, Q&A validation)
multimodal-rag/
├── backend/
│ ├── app/
│ │ ├── main.py # FastAPI app + lifespan
│ │ ├── config.py # Settings (pydantic-settings)
│ │ ├── models.py # Pydantic models
│ │ ├── routes/api.py # All HTTP endpoints
│ │ └── services/
│ │ ├── pipeline.py # Orchestrates full processing
│ │ ├── transcription.py # Whisper wrapper
│ │ ├── video.py # Frame extraction + OCR
│ │ ├── chunking.py # Text chunking logic
│ │ ├── embeddings.py # Sentence-transformer wrapper
│ │ ├── retrieval.py # FAISS search + reranker
│ │ ├── qa.py # LLM answer generation (SSE)
│ │ ├── storage.py # File-based repository
│ │ └── downloader.py # yt-dlp integration
│ ├── tests/
│ │ ├── conftest.py # Fixtures (mocked ML, temp data)
│ │ ├── test_api.py
│ │ ├── test_chunking.py
│ │ └── test_retrieval.py
│ ├── Dockerfile
│ ├── requirements.txt
│ └── requirements-optional.txt
├── frontend/
│ ├── src/
│ │ ├── App.tsx
│ │ ├── styles.css
│ │ ├── components/
│ │ │ ├── UploadPanel.tsx # File + URL tab upload
│ │ │ ├── VideoList.tsx # Video library sidebar
│ │ │ ├── VideoPlayer.tsx # Range-aware video player
│ │ │ └── QuestionPanel.tsx # Streaming Q&A + citations
│ │ └── lib/api.ts # API client (SSE + REST)
│ ├── Dockerfile
│ └── nginx.conf
├── docker-compose.yml
├── .dockerignore
└── .gitignore
- Progress bar for yt-dlp download stage
- Multi-video search — ask across all indexed videos at once
- Qdrant migration — production-grade vector database for scale
- GPU support — Whisper on CUDA for ~10× faster transcription
- Auth — user sessions and private video libraries
MIT