BrainHub: Multimodal RAG

Problem

Real teams have meeting recordings, scanned contracts, screenshot threads in Slack, and PDFs in Drive. Most RAG demos handle PDFs only. Operations leaders end up grepping their own Notion when an answer already exists in a video from three weeks ago.

Approach

• →libmagic MIME sniffing on upload (not file extension). Allowlist enforced server-side.
• →Images are embedded as images, not as captions. Text, image and video share one vector space (voyage-multimodal-3.5), so a written question can retrieve a figure because both live in the same space, not because a model described the figure first. Captioning throws away everything the caption did not mention.
• →Scanned PDFs do not vanish. A page that yields no text layer is rendered at 150 DPI and takes the visual path, while native pages keep the cheap text path. Before that, a scanned contract was silently absent from the index.
• →URL crawl with SSRF defense: scheme allowlist, IPv4 obfuscation rejection, DNS re-resolution on every redirect hop, cloud-metadata IP block.
• →Contextual retrieval with prompt caching on the document block, so chunks after the first pay about 10% on the shared document tokens instead of the full price. Roughly 85% cheaper per ingested document.
• →Grading is decoupled from the vendor score scale. An absolute threshold is only applied when the score is a calibrated reranker score, never against fusion ranks. Coupling the two silently discarded every retrieved chunk on every question.
• →Tenant isolation enforced at retrieval: every chunk row carries user_id, both pgvector and tsvector queries WHERE-filter on it before RRF or rerank.
• →JWT auth, per-IP rate limiting backed by Redis, a global daily ceiling consumed before any paid call, and Server-Sent Events for streaming answers.

Stack