Python Library

Vision RAG v0.1.2

A modular Python library for Retrieval-Augmented Generation over video. Ask questions about any video and get answers powered by transcript analysis and visual frame understanding.

MIT License Python ≥ 3.9 1 core dependency

Installation

$ pip install vision-rag

Vision RAG ships with only one hard dependency — pymediainfo. All other dependencies are installed based on which providers you choose to use.

How It Works

Vision RAG follows a clean 6-stage pipeline that takes a raw video file and turns it into an answerable knowledge base:

Stage 1Ingestvideo_ingestion.py

→

Stage 2Chunkvideo_chunker.py

→

Stage 3Embedembedding.py

→

Stage 4Indexvectorstores.py

→

Stage 5Retrieveretriever.py

→

Stage 6Generategenerator.py

Video Ingestion

Reads video metadata (duration, resolution, FPS, codec) via pymediainfo.

Smart Chunking

Time-based overlapping chunks with keyframe extraction and ASR transcription.

Dual Embedding

Embeds both text transcripts and visual frames into vector space.

Hybrid Retrieval

Searches text and image indexes simultaneously with deduplication.

Multi-Model Generation

Works with GPT-4o, Claude, Gemini, Ollama, or any custom VLM.

Fully Pluggable

Every component has a base class — bring your own ASR, embedder, store, or LLM.

Overview

Vision RAG is designed around a simple principle: every stage is pluggable. The library provides base classes at each layer, along with built-in implementations for popular providers. You can mix and match, or bring your own.

Bring Your Own Everything. Vision RAG doesn't lock you into any model or API. Subclass any base class (BaseASR, BaseTextEmbedder, BaseImageEmbedder, BaseVectorStore, BaseGenerator) and plug in whatever you want.

Quick Start

Here's a complete end-to-end example that ingests a video, chunks it, embeds it, indexes it, retrieves relevant segments, and generates an answer:

python

from vision_rag.video_ingestion import VideoLoader
from vision_rag.video_chunker import Chunker, WhisperLocalASR
from vision_rag.embedding import EmbeddingBuilder, OpenAITextEmbedder, CLIPImageEmbedder
from vision_rag.vectorstores import FAISS
from vision_rag.retriever import Retriever
from vision_rag.generator import Generator, OllamaGenerator

# Stage 1 — Ingest
video_doc = VideoLoader().load("video.mp4")

# Stage 2 — Chunk
chunks = Chunker(
    asr=WhisperLocalASR(model_size="base"),
    use_asr=True, use_frames=True,
    chunk_size=5.0, chunk_overlap=1.0,
).chunk("video.mp4")

# Stage 3 — Embed (using built-in providers)
text_embedder  = OpenAITextEmbedder(api_key="your_openai_key")
image_embedder = CLIPImageEmbedder()
embedded_chunks = EmbeddingBuilder(
    text_embedding=text_embedder, image_embedding=image_embedder,
).embed(chunks)

# Stage 4 — Index
store = FAISS()
store.index(embedded_chunks)

# Stage 5 + 6 — Retrieve and Generate
query   = input("Ask a question: ")
results = Retriever(store=store, text_embedder=text_embedder).retrieve(query)
answer  = Generator(llm=OllamaGenerator(model="llava:7b")).generate(query=query, results=results)

print(answer.text)

Stage 1 — Video Ingestion

VideoLoader reads a video file and returns a VideoDocument containing raw metadata. No frames, no audio — just file information passed to the next stage.

python

from vision_rag.video_ingestion import VideoLoader

loader = VideoLoader()
video_doc = loader.load("sample.mp4")

print(video_doc)
# VideoDocument(file='sample.mp4', duration=120.50s, fps=30.0,
#   resolution=1920x1080, frames=3615, codec='avc1')

VideoDocument Fields

Field	Type	Description
`source_path`	str	Absolute path to the video file
`filename`	str	Base filename (e.g. "sample.mp4")
`format`	str	File extension (e.g. "mp4", "avi")
`file_size_bytes`	int	File size in bytes
`duration_seconds`	float	Video duration in seconds
`fps`	float	Frames per second
`total_frames`	int	Total number of frames
`width` / `height`	int	Video resolution
`codec`	str	Video codec (e.g. "avc1", "HEVC")

Stage 2 — Video Chunking

The Chunker splits a video into overlapping time-based chunks. Each chunk can include a keyframe image and a transcript segment from ASR.

python

from vision_rag.video_chunker import Chunker, WhisperLocalASR

chunker = Chunker(
    asr=WhisperLocalASR(model_size="medium"),
    use_asr=True,
    use_frames=True,
    chunk_size=5.0,       # seconds per chunk
    chunk_overlap=1.0,   # overlap between chunks
)
chunks = chunker.chunk("video.mp4")

Chunker Parameters

Parameter	Default	Description
`asr`	None	Any `BaseASR` provider for transcription
`use_asr`	True	Whether to transcribe audio
`use_frames`	True	Whether to extract keyframe per chunk
`chunk_size`	5.0	Duration of each chunk in seconds
`chunk_overlap`	1.0	Overlap between consecutive chunks in seconds
`keyframe_strategy`	"middle"	"middle" or "first" — where to sample keyframe
`frames_dir`	None	Custom directory for frames (defaults to .vision_rag_cache/frames/)

Chunk Fields

Field	Description
`chunk_id`	Chunk index (0, 1, 2...)
`start` / `end`	Start/end time in seconds
`duration`	Duration in seconds
`text`	ASR transcript for this chunk
`frame_path`	Path to keyframe .jpg image
`metadata`	Dict with video_filename, chunk_index, total_chunks, keyframe_strategy, asr_provider

ASR — Bring Your Own

Vision RAG ships with built-in ASR providers but you can plug in anything by subclassing BaseASR:

python

from vision_rag.video_chunker import BaseASR

# Built-in providers
from vision_rag.video_chunker import WhisperLocalASR, OpenAIASR, DeepgramASR

# Your own — any model, any API
class MyASR(BaseASR):
    def transcribe(self, audio_path: str) -> list[dict]:
        return [{"start": 0.0, "end": 5.0, "text": "..."}]

chunker = Chunker(asr=MyASR(), use_asr=True)

WhisperLocalASRBuilt-in

Local Whisper via faster-whisper or openai-whisper. No API key needed. Supports model sizes: tiny, base, small, medium, large.

OpenAIASRBuilt-in

OpenAI Whisper API. Requires OPENAI_API_KEY env var or pass api_key=.

DeepgramASRBuilt-in

Deepgram nova-2 API. Requires DEEPGRAM_API_KEY env var or pass api_key=.

Stage 3 — Embedding

The EmbeddingBuilder converts each chunk's text and keyframe image into vectors. You need at least one embedder (text or image).

python

from vision_rag.embedding import EmbeddingBuilder, BaseTextEmbedder, BaseImageEmbedder

# Your own text embedder
class MyTextEmbedder(BaseTextEmbedder):
    def embed(self, text: str) -> list[float]:
        return [...]  # your model or API

# Your own image embedder
class MyImageEmbedder(BaseImageEmbedder):
    def embed(self, image_path: str) -> list[float]:
        return [...]  # your model or API

embedder = EmbeddingBuilder(
    text_embedding=MyTextEmbedder(),
    image_embedding=MyImageEmbedder(),
)
embedded_chunks = embedder.embed(chunks)

Built-in Embedding Providers

OpenAITextEmbedderText

OpenAI text-embedding-3-small by default. Requires API key. Pass model= to override.

SentenceTransformerTextEmbedderText

Local embeddings via all-MiniLM-L6-v2. No API key needed. Supports device="auto".

CLIPTextEmbedderText

CLIP ViT-B/32 text encoder. Local, no API key. Pair with CLIPImageEmbedder (same model=) for cross-modal retrieval — vectors must share the same joint space.

CLIPImageEmbedderImage

CLIP ViT-B/32 image encoder. Local, no API key. Supports device="auto" | "cpu" | "cuda" | "mps".

OpenAIImageEmbedderImage

OpenAI vision embeddings via base64 encoding. Requires API key.

CLIP cross-modal pairing. When using CLIP, always pair CLIPTextEmbedder with CLIPImageEmbedder using the same model= string. Mixing CLIP image vectors with an unrelated text embedder (e.g. SentenceTransformer) produces vectors from incompatible spaces — Retriever will reject mismatched dimensions outright.

Stage 4 — Vector Stores

Built-in Stores

python

from vision_rag.vectorstores import FAISS, Chroma

# FAISS — fast local search
store = FAISS()
store.index(embedded_chunks)
store.save("my_index")
store.load("my_index")

# Chroma — persistent local DB
store = Chroma(path="my_chroma_db")
store.index(embedded_chunks)

Custom Vector Store. Subclass BaseVectorStore and implement index(), search_text(), search_image(), search_by_time(), save(), and load().

Stage 5 — Retrieval

The Retriever takes a text query, embeds it, and searches both text and image indexes simultaneously. Results are deduplicated by chunk_id and fused via Reciprocal Rank Fusion (RRF) — a rank-based fusion method that works correctly even when text and image similarity scores are on different scales.

python

from vision_rag.retriever import Retriever

retriever = Retriever(
    store=store,
    text_embedder=text_embedder,  # same embedder used at indexing time
    top_k_text=5,
    top_k_image=5,
)

# Semantic search
results = retriever.retrieve("What did they say about frozen yogurt?")
results.text_results    # top text matches
results.image_results   # top image matches
results.all             # both, fused via RRF (deduplicated, ranked)
results.by_time         # same results, sorted chronologically

# Time-based search (no embedding needed)
chunks = retriever.retrieve_by_time(start=10.0, end=20.0)

Reciprocal Rank Fusion (RRF). results.all fuses the text and image ranked lists using RRF: score(chunk) = Σ 1 / (60 + rank) across each list the chunk appears in. This sidesteps score-scale incompatibility between different embedding models — only the internal ordering of each list needs to be meaningful.

Stage 6 — Generation

Built-in VLM Providers

The Generator wraps any BaseGenerator provider and controls how top-k chunks are selected and ordered before being sent to the model.

Generator Parameters

Parameter	Default	Description
`llm`	—	Any `BaseGenerator` provider (required)
`top_k`	5	How many RRF-ranked chunks to pass to the LLM
`chronological`	True	If True, selected chunks are reordered by timestamp before generation — the model sees a coherent timeline rather than a relevance-ranked jumble

python

from vision_rag.generator import Generator, OpenAIGenerator

generator = Generator(llm=OpenAIGenerator(api_key="sk-..."))
answer = generator.generate(query="What happened?", results=results)
print(answer.text)

python

from vision_rag.generator import Generator, AnthropicGenerator

# Default model: claude-sonnet-4-6
generator = Generator(llm=AnthropicGenerator(api_key="sk-ant-..."))
answer = generator.generate(query="What happened?", results=results)
print(answer.text)

python

from vision_rag.generator import Generator, GeminiGenerator

# Default model: gemini-2.0-flash
generator = Generator(llm=GeminiGenerator(api_key="..."))
answer = generator.generate(query="What happened?", results=results)
print(answer.text)

python

from vision_rag.generator import Generator, OllamaGenerator

# VLM — text + images
generator = Generator(llm=OllamaGenerator(model="llava:7b"))
# Text only
generator = Generator(llm=OllamaGenerator(model="llama3.2"))

answer = generator.generate(query="What happened?", results=results)
print(answer.text)

python

from vision_rag.generator import Generator, BaseGenerator

class MyGenerator(BaseGenerator):
    def generate(self, query: str, chunks) -> str:
        return "answer..."

generator = Generator(llm=MyGenerator())

The Generator takes the top-k results (default 5) ranked by score, interleaves text transcripts and keyframe images, and sends them to the VLM for answer generation.

The returned GeneratorAnswer object contains:

answer.text — the generated answer string
answer.query — the original question
answer.sources — list of EmbeddedChunk objects used to generate the answer

Custom Providers — Bring Your Own

Every component in Vision RAG is designed to be extended. Here are the base classes you can subclass:

Base Class	Method(s) to Implement	Returns
`BaseASR`	`transcribe(audio_path)`	`list[dict]` with start, end, text
`BaseTextEmbedder`	`embed(text)`	`list[float]`
`BaseImageEmbedder`	`embed(image_path)`	`list[float]`
`BaseVectorStore`	`index()`, `search_text()`, `search_image()`, `search_by_time()`, `save()`, `load()`	Various
`BaseGenerator`	`generate(query, chunks)`	`str`

Device Selection (`_device.py`)

All local torch-backed components (CLIPTextEmbedder, CLIPImageEmbedder, SentenceTransformerTextEmbedder, WhisperLocalASR) share a common device-resolution utility. Device resolution is lazy — it happens on first use, not at construction, so simply importing these classes never requires torch to be installed.

Value	Behaviour
`"auto"` (default)	Picks the best available: CUDA → MPS → CPU
`"cpu"`	Always available, never requires torch
`"cuda"` / `"cuda:N"`	Explicit GPU; raises `RuntimeError` if CUDA is unavailable
`"mps"`	Apple Silicon; raises `RuntimeError` if MPS is unavailable

WhisperLocalASR device note. WhisperLocalASR restricts device selection to cpu and cuda only — faster-whisper's ctranslate2 backend has no MPS support, and this restriction is enforced uniformly across both backends (faster-whisper and openai-whisper fallback) so behaviour is consistent regardless of which backend is installed.

API Reference

All public classes are exported from the top-level vision_rag package:

python

from vision_rag import (
    # Stage 1
    VideoLoader, VideoDocument,
    # Stage 2
    Chunker, Chunk, BaseASR, WhisperLocalASR, OpenAIASR, DeepgramASR,
    # Stage 3
    EmbeddingBuilder, EmbeddedChunk,
    BaseTextEmbedder, BaseImageEmbedder,
    OpenAITextEmbedder, SentenceTransformerTextEmbedder,
    CLIPImageEmbedder, OpenAIImageEmbedder,
    # Stage 4
    BaseVectorStore, SearchResult, FAISS, Chroma,
    # Stage 5
    Retriever, RetrievalResult,
    # Stage 6
    Generator, GeneratorAnswer, BaseGenerator,
    OpenAIGenerator, AnthropicGenerator, GeminiGenerator, OllamaGenerator,
)

Dependencies

Vision RAG has only one hard dependency — pymediainfo. Everything else is optional and installed based on what you use:

Feature	Install Command
ASR (local Whisper)	`pip install faster-whisper`
ASR (OpenAI API)	`pip install openai`
ASR (Deepgram)	`pip install deepgram-sdk`
Frames + Audio extraction	`brew install ffmpeg` (or apt)
FAISS vector store	`pip install faiss-cpu`
Chroma vector store	`pip install chromadb`
OpenAI embeddings	`pip install openai`
Sentence Transformers	`pip install sentence-transformers`
CLIP image embeddings	`pip install git+https://github.com/openai/CLIP.git torch Pillow`
Ollama generation	`pip install ollama`
Anthropic generation	`pip install anthropic`
Gemini generation	`pip install google-genai`

License: Vision RAG is released under the MIT License.

Vision RAG v0.1.2

Installation

How It Works

Overview

Quick Start

Stage 1 — Video Ingestion

VideoDocument Fields

Stage 2 — Video Chunking

Chunker Parameters

Chunk Fields

ASR — Bring Your Own

Stage 3 — Embedding

Built-in Embedding Providers

Stage 4 — Vector Stores

Built-in Stores

Stage 5 — Retrieval

Stage 6 — Generation

Built-in VLM Providers

Generator Parameters

Custom Providers — Bring Your Own

Device Selection (_device.py)

API Reference

Dependencies

Device Selection (`_device.py`)