Google just found a way to take 31 gigabytes of AI memory and shrink it down to just 4. And a developer already turned it into a free, open-source tool you can run on your laptop today.

Links & Resources

Why This Matters

Every time an AI model thinks, it stores massive lists of numbers, millions of them, in something called the Key-Value (KV) Cache. Think of it as the model's working memory. The problem? This memory eats up insane amounts of RAM, and it's one of the biggest reasons running AI is so expensive.

A single 10-million-document corpus takes 31 GB of RAM stored as standard 32-bit floats. That's a beefy GPU just to hold vectors in memory, before the model even starts doing anything useful.

Google's TurboQuant compresses those same vectors down to 4 GB, and actually searches them faster than before.

What Is TurboQuant?

TurboQuant is a compression algorithm from Google Research, published in March 2026 and presented at ICLR 2026. It compresses the way AI models store and retrieve information, specifically, the high-dimensional vectors that represent everything an AI "knows."

Think of it like a photo. A giant high-res image and a compressed copy look almost identical to your eye, but one takes a fraction of the space. Google pulled off that exact trick on AI's memory: keep what actually matters, throw out the bulk.

The Numbers

  • 31 GB → 4 GB, 8x memory reduction on a 10M document corpus
  • 8x faster attention computation vs uncompressed on H100 GPUs
  • 3-bit quantization with zero accuracy loss, no retraining, no fine-tuning needed
  • 12-20% faster search than Meta's FAISS on ARM chips (Apple Silicon)

How It Works, The Simple Version

TurboQuant uses a two-step compression pipeline backed by two supporting algorithms:

Step 1: PolarQuant, Rewrite the Coordinates

Instead of storing AI memory as standard X-Y-Z coordinates, PolarQuant converts everything into polar coordinates, a radius (how strong the signal is) and angles (what direction/meaning the data points in).

It's like replacing "Go 3 blocks East, 4 blocks North" with "Go 5 blocks at a 37-degree angle." Same destination, way less data to store.

This eliminates the expensive normalization step that traditional methods need, removing the memory overhead entirely.

Step 2: QJL, The 1-Bit Error Checker

After compression, there's always tiny errors left over. TurboQuant uses the Quantized Johnson-Lindenstrauss (QJL) algorithm to clean those up using just 1 extra bit per number. It acts like a mathematical spell-checker, catches bias, keeps accuracy intact, adds almost nothing to the file size.

The Result

The combined system hits near the theoretical lower bound on compression distortion (the Shannon limit). In plain English: it's almost mathematically impossible to compress better than this without losing information.

What Is Turbovec? (The Open-Source Tool)

A developer named Ryan Codrai took Google's TurboQuant paper and built turbovec, a free, open-source vector index written in Rust with Python bindings. It already has 10K+ GitHub stars and 851 forks.

What turbovec gives you:

  • No training step, add vectors and they're indexed instantly. No parameter tuning, no rebuilds.
  • Faster than FAISS, hand-written SIMD kernels (NEON for ARM, AVX-512 for x86) beat Meta's FAISS IndexPQ by 12-20% on ARM.
  • Filtered search, pass an allowlist to search() and it respects it inside the SIMD kernel. No wasted compute.
  • Fully local, no cloud service, no data leaving your machine. Pair with any open-source embedding model for a completely air-gapped RAG stack.
  • Framework integrations, drop-in replacements for LangChain, LlamaIndex, Haystack, and Agno vector stores.

How to Set Up Turbovec

Requirements

  • Python 3.8+ or Rust toolchain
  • Any modern CPU (Apple Silicon, Intel Haswell 2013+, or newer)

Install via pip

pip install turbovec

Basic Usage, Python

from turbovec import TurboQuantIndex

# Create an index (1536 = OpenAI embedding dimension, 4 = bit width)
index = TurboQuantIndex(dim=1536, bit_width=4)

# Add your vectors
index.add(vectors)

# Search
scores, indices = index.search(query, k=10)

# Save and load
index.write("my_index.tq")
loaded = TurboQuantIndex.load("my_index.tq")

With Stable IDs (For Real Apps)

import numpy as np
from turbovec import IdMapIndex

index = IdMapIndex(dim=1536, bit_width=4)
index.add_with_ids(vectors, np.array([1001, 1002, 1003], dtype=np.uint64))

scores, ids = index.search(query, k=10)  # returns your custom IDs
index.remove(1002)                        # O(1) delete by ID

index.write("my_index.tvim")

Hybrid Search (Filter + Vector)

import numpy as np
from turbovec import IdMapIndex

idx = IdMapIndex(dim=1536, bit_width=4)
idx.add_with_ids(vectors, ids)

# Step 1: SQL/BM25 narrows to candidate IDs
allowed = np.array(
    db.execute("SELECT id FROM docs WHERE tenant=?", (t,)).fetchall(),
    dtype=np.uint64
)

# Step 2: Dense rerank within candidates only
scores, ids = idx.search(query, k=10, allowlist=allowed)

Filtering happens inside the SIMD kernel, blocks with no allowed slots are skipped entirely before any scoring work. This means selective filters are fast, not a brute-force scan that throws away results.

Install via Rust

cargo add turbovec

use turbovec::TurboQuantIndex;

let mut index = TurboQuantIndex::new(1536, 4);
index.add(&vectors);
let results = index.search(&queries, 10);
index.write("index.tv").unwrap();

Framework Integrations

Turbovec ships drop-in replacements for popular AI frameworks, same API surface, just swap the import:

FrameworkInstallReplaces
LangChainpip install turbovec[langchain]InMemoryVectorStore
LlamaIndexpip install turbovec[llama-index]SimpleVectorStore
Haystackpip install turbovec[haystack]InMemoryDocumentStore
Agnopip install turbovec[agno]LanceDb

Compression & Speed Benchmarks

Memory Compression

A 1536-dimensional vector (standard OpenAI embedding size):

FormatSize per vector10M vectors
Float32 (standard)6,144 bytes~57 GB
4-bit TurboQuant768 bytes~7.2 GB
2-bit TurboQuant384 bytes~3.6 GB

That's 8x to 16x compression depending on bit width.

Search Speed (vs FAISS)

Tested on 100K vectors, 1K queries, k=64:

  • ARM (Apple M3 Max): turbovec beats FAISS FastScan by 12-20% across every config
  • x86 (Intel Xeon Sapphire Rapids): turbovec wins every 4-bit config by 1-6%, within ~1% on 2-bit

Recall (Accuracy)

On OpenAI d=1536 and d=3072 embeddings, TurboQuant beats FAISS by 0.4-3.4 points at Recall@1 across 2-bit and 4-bit. Both converge to perfect recall by k=4.

Why You Should Care

If You're Running AI Locally

Your laptop or desktop can now handle vector databases that previously needed a GPU server. A RAG pipeline over millions of documents, running on a MacBook, fully offline. Your data never leaves your machine.

If You're Building AI Products

Memory costs drop dramatically. The same GPU that handled one workload can now handle 8x more. That means cheaper inference, more users per server, lower cloud bills.

If You're Into Privacy

turbovec is fully local. Pair it with an open-source embedding model (like nomic-embed-text via Ollama) and you have a completely air-gapped retrieval system. No API calls, no cloud, no data leaks.

Links

Follow AI Adventure YT for more AI breakdowns and guides.