Voice-first RAG

Voice AI Assistant (RAG)

Speak the question, hear the cited answer. Full voice loop in production.

Demo limits

·5 questions and 3 documents per session, evicted on expiry
·300 answers and 600 transcriptions per UTC day, across everyone
·Kill switch that stops paid calls without a redeploy

A public AI demo is a spending endpoint. Cost control is part of what I ship.

Problem

Most "AI assistants" force the user to read walls of text. Voice-first interfaces win on accessibility and time-to-answer, but stitching mic capture, STT, retrieval, and streaming TTS without 5-second lag is non-trivial.

Approach

→Browser captures mic via MediaRecorder (webm-opus / mp4-aac) and posts a single multipart blob, with no WebRTC complexity.
→gpt-4o-transcribe handles speech-to-text server-side with a language hint; the transcript pre-fills a textarea so the user can edit before submitting.
→Hybrid retrieval: pgvector HNSW + tsvector GIN fused by RRF in a single SQL round-trip, then optional cross-encoder reranking. Two relevance thresholds, one per score scale, because a fusion rank and a calibrated reranker score are not comparable numbers.
→Embeddings run locally: FastEmbed on ONNX with paraphrase-multilingual-MiniLM-L12-v2, so ingest makes no external embedding call and answers work in the visitor's language.
→Answer streams as PCM audio chunks over SSE directly into the Web Audio API. Sentence-pipelined TTS starts speaking sentence one while the model is still writing sentence two, which takes first-audible-word from ~1500ms down to ~600ms.

Stack

Next.js 16React 19FastAPIPostgreSQL 17 + pgvectorgpt-4o-transcribegpt-4o-mini-ttsFastEmbed (ONNX, local)DeepSeek via OpenRouter