Key Takeaways: turbovec shrinks large embedding stores dramatically while staying fast, making local, privacy‑first RAG and semantic search practical on commodity hardware.
What is turbovec?
turbovec is an open‑source vector index written in Rust with first‑class Python bindings, built on Google Research’s TurboQuant algorithm for high‑dimensional vector quantization. It is designed to store and search large embedding corpora with aggressive 2–4‑bit compression, while matching or beating FAISS on recall and query latency in common RAG workloads. The headline example from the README and community posts is a 10‑million‑document corpus that would occupy roughly 31 GB as float32 embeddings, but fits into about 4 GB with turbovec’s compressed index — and still searches faster than FAISS on tested hardware.
For developers building retrieval‑augmented generation (RAG), semantic search, or agent memory systems that must run locally or inside strict VPCs, turbovec offers a way to keep everything on CPU and dramatically lower memory pressure.
- GitHub repository: https://github.com/RyanCodrai/turbovec
- PyPI package: https://pypi.org/project/turbovec/
Why turbovec matters for modern RAG
Most RAG tutorials quietly assume that you can afford a big in‑memory vector store, but at tens of millions of documents the naïve float32 footprint quickly becomes unmanageable. If you want to serve LLMs from laptops, edge devices, or on‑prem clusters without spinning up a separate vector database service (or paying for a managed one), compression and CPU‑friendly search become critical. turbovec is designed specifically for this environment: it keeps all data local, uses Rust and SIMD kernels (NEON on ARM, AVX2/AVX‑512BW on x86), and exposes a simple Python and Rust API so you can integrate it into your existing stack.
Because TurboQuant is data‑oblivious — it does not require training data or codebook fitting — turbovec avoids the usual “train a PQ codebook, rebuild the index when the distribution shifts” operational burden found in many vector stores.
Core features at a glance
Extreme compression and competitive speed
At typical OpenAI‑style embedding dimensions, turbovec compresses float32 vectors by up to 16× using 2‑bit or 4‑bit quantization while preserving high recall. For example, a 1,536‑dimensional float32 vector shrinks from 6,144 bytes to around 384 bytes at 2‑bit, cutting memory usage dramatically for large corpora. Benchmark suites in the repo show that turbovec’s TurboQuant‑based kernels beat FAISS’s IndexPQFastScan by roughly 12–20% on ARM (Apple M‑series) and match or slightly exceed FAISS on x86 for many 4‑bit configurations.
Online ingest and training‑free indexing
turbovec’s index can accept new vectors at any time without a separate training step: you just call add() and they become searchable immediately.
Because TurboQuant’s codebook and quantization scheme come from math rather than sampled data, there is no k‑means training phase, no retraining when the corpus grows, and no need to rebuild the entire index after distribution shifts. That makes turbovec very friendly for streaming or continuously updated document collections where embeddings are added incrementally throughout the day.
Filtered search and hybrid retrieval
The Python search() API accepts an allowlist of IDs (or a slot bitmask) so you can restrict scoring to a candidate set produced by another system, like SQL, BM25, or time‑window filters. Filtering happens inside the SIMD kernel at 32‑vector block granularity, short‑circuiting blocks with no allowed entries and dropping disallowed entries before heap insertion, which avoids wasting cycles on vectors you’ll discard anyway. This hybrid retrieval pattern (candidate generation + dense re‑rank with turbovec) is ideal for multi‑signal search pipelines and structured‑plus‑semantic query flows.
Local‑only, privacy‑first behavior
turbovec is a pure local library: there is no hosted service and no data leaves your machine or VPC unless you choose to export it. You can pair turbovec with any embedding model — open‑source or your own in‑house — to build an air‑gapped RAG stack where both inference and indexing run entirely on hardware you control. The project is MIT‑licensed, making it straightforward to use in proprietary products as well as open‑source tooling.
Installing turbovec
Python installation
For most users, the Python bindings are the quickest way to get started.
Install from PyPI:
pip install turbovecThis pulls the turbovec wheel built via maturin and gives you access to the TurboQuantIndex and IdMapIndex classes from Python.
If you are working on ARM or AVX‑512‑capable x86 machines, the wheel includes architecture‑specific kernels that activate at runtime based on CPU feature detection.
Building the Python wheel from source (optional)
If you want to build from source — for example, to tweak SIMD flags or run local benchmarks — clone the repo and use maturin:
pip install maturin
cd turbovec-python
maturin build --release
pip install target/wheels/*.whlThis compiles the Rust core and packages it as a Python wheel you can install locally.
Rust crate installation
Rust users can depend on turbovec directly from Cargo.toml.
Add the crate:
cargo add turbovecThen build your project as usual; turbovec’s .cargo/config.toml targets x86-64-v3 (AVX2 baseline) on x86 while dynamically enabling AVX‑512BW where available.
To build from source for testing:
cargo build --releaseThis produces a release binary with optimized SIMD kernels ready to use in your own Rust code.
Getting started in Python
Basic indexing and search
The core Python class for most use cases is TurboQuantIndex.
from turbovec import TurboQuantIndex
index = TurboQuantIndex(dim=1536, bit_width=4)
index.add(vectors)
index.add(more_vectors)
scores, indices = index.search(query, k=10)
index.write("my_index.tq")
loaded = TurboQuantIndex.load("my_index.tq")Here:
dimis your embedding dimension (for example, OpenAI text embeddings at 1,536).bit_widthcontrols compression: typically 2‑bit or 4‑bit; 4‑bit trades a bit more memory for slightly higher recall.add()can be called multiple times as new data arrives — there’s no training phase.write()andload()give you straightforward persistence: save the compressed index to disk and reload it later.
Stable external IDs with IdMapIndex
For production systems, you often want index‑internal positions separate from external IDs that survive deletions, migrations, or merges.
turbovec ships an IdMapIndex type that stores a 64‑bit external ID alongside each vector:
import numpy as np
from turbovec import IdMapIndex
index = IdMapIndex(dim=1536, bit_width=4)
index.add_with_ids(
vectors,
np.array([1001, 1002, 1003], dtype=np.uint64),
)
scores, ids = index.search(query, k=10) # ids are your external uint64 IDs
index.remove(1002) # O(1) deletion by ID
index.write("my_index.tvim")
loaded = IdMapIndex.load("my_index.tvim")This lets you treat turbovec as a fast ANN layer while still using semantic IDs from your own database or document store.
Hybrid retrieval and filtered search
A common pattern is to first narrow candidates via SQL or BM25, then re‑rank that subset with turbovec:
import numpy as np
from turbovec import IdMapIndex
idx = IdMapIndex(dim=1536, bit_width=4)
idx.add_with_ids(vectors, ids)
# Stage 1: external system narrows candidate IDs
allowed = np.array(
db.execute(
"SELECT id FROM docs WHERE tenant = ?",
(tenant,),
).fetchall(),
dtype=np.uint64,
)
# Stage 2: dense re-rank within the candidate set
scores, ids = idx.search(query, k=10, allowlist=allowed)The allowlist ensures the SIMD kernel only scores vectors you’ve pre‑selected, and the output length never exceeds the size of your allowed set.
Using turbovec from Rust
Rust users get analogous APIs with TurboQuantIndex and IdMapIndex types.
Basic Rust indexing
use turbovec::TurboQuantIndex;
let mut index = TurboQuantIndex::new(1536, 4);
index.add(&vectors);
let results = index.search(&queries, 10);
index.write("index.tv").unwrap();
let loaded = TurboQuantIndex::load("index.tv").unwrap();This code mirrors the Python example: create an index, add vectors, search with queries, and persist to disk.
Stable IDs in Rust
use turbovec::IdMapIndex;
let mut index = IdMapIndex::new(1536, 4);
index.add_with_ids(&vectors, &[1001, 1002, 1003]);
let (scores, ids) = index.search(&queries, 10);
index.remove(1002);
index.write("index.tvim").unwrap();
let loaded = IdMapIndex::load("index.tvim").unwrap();Because the crate is pure Rust, you can integrate turbovec directly into your own retrieval engine, build custom metadata layers, or wrap it in higher‑level services.
Framework integrations and turbovecdb
To ease adoption, turbovec provides optional extras that plug into popular Python RAG frameworks.
With extras on PyPI, you can use turbovec as a drop‑in replacement for in‑memory stores in:
- LangChain (
pip install turbovec[langchain]). - LlamaIndex (
pip install turbovec[llama-index]). - Haystack (
pip install turbovec[haystack]). - Agno (
pip install turbovec[agno]).
For an embedded vector database experience, the companion project turbovecdb combines turbovec’s ANN index with a durable SQLite backend, storing exact float32 vectors and metadata while using turbovec as a rebuildable cache for fast approximate search. This gives you filters, persistence, multi‑process safety, and exact cosine re‑ranking on top of turbovec’s compressed index — useful when you want “vector DB without a server,” just local files and Python.
