Search Everything: Full-Text Indexing with Tantivy Under the Hood

What’s new since this post (v0.3.8). Text fields can name a language analyzer, with Snowball stemmers for English, German, French, Spanish and Italian and a dictionary-based Croatian analyzer built from 1.7 million word forms. Every schema now carries a version and a thumbprint, and a search fetches only the documents on the page it returns, so deep pages and large limits stay fast.

The Inverted Index Advantage

Traditional databases struggle with text search. A LIKE query with wildcards scans every row, O(n) complexity that destroys performance as data grows. Full-text search engines use inverted indexes, mapping terms to document IDs for O(log n) lookups regardless of dataset size.

CameoDB integrates Tantivy, a Rust-based search library inspired by Lucene. Tantivy provides inverted indexes, term dictionaries, positional data for phrase queries, and BM25 relevance scoring. Combined with redb's ACID KV storage, CameoDB delivers hybrid search, fast point lookups via redb, rich full-text queries via Tantivy.

Index-Only Storage Strategy

Most search engines duplicate data, storing complete documents in the index for retrieval. This bloats index size and slows writes. CameoDB takes a different approach, index-only storage.

// Tantivy stores only indexed fields (no STORED flag)
schema = SchemaBuilder::new()
 .add_text_field("title", TEXT | STORED)
 .add_text_field("content", TEXT) // No STORED
 .build()

// redb stores complete JSON documents
redb.insert("doc:123", full_json_document)

Tantivy stores only the id field and indexed fields without the STORED flag. Complete JSON documents live exclusively in redb. When you search, Tantivy returns matching document IDs and scores, then we batch-fetch the full documents from redb. This split-storage strategy means smaller indices, faster search performance, and zero data duplication.

Field-Level Control and Schema Evolution

CameoDB provides fine-grained control over field indexing through schema configuration. Each field can be configured with:

INDEXED

Enables range queries, filtering, and term lookups on the field.

FAST

Enables sorting and aggregations on the field value.

STORED

Stores the field value in Tantivy for direct retrieval (rarely used).

TEXT

Tokenizes and indexes for full-text search. Name a language analyzer on the field, such as en_stem or hr_stem_fold, and a search for one form of a word finds the others.

Schema evolution is built in. Declare the fields you want to search with PUT /api/{index}/_config, or let the first write define them: every field on the first document is indexed. A field that arrives later is learned as well, recorded with the type its values have shown, and every value is kept. Each schema carries a version and a thumbprint, so every node, and every search, knows exactly which schema it holds.

Query Capabilities: Beyond Simple Search

Tantivy supports sophisticated query patterns that go beyond keyword matching:

// Phrase queries with proximity
query = "database systems"~2 // Within 2 words

// Range queries on numeric fields
query = price:[100 TO 500]

// Boolean combinations
query = (title:rust OR title:go) AND year:[2020 TO 2024]

// Boosting for relevance tuning
query = title:rust^3 OR body:rust

Phrase queries use positional indexes to find terms in specific order with configurable proximity slop. Range queries work on numeric, date, text, string and ip fields. Boolean operators enable complex logic with AND, OR, NOT. Field boosting allows you to weight certain fields higher in relevance scoring.

BM25 Relevance Scoring

Tantivy uses BM25, the industry-standard relevance scoring algorithm. BM25 considers term frequency (how often terms appear in a document), inverse document frequency (how rare terms are across the corpus), and document length normalization. This means documents matching rare terms score higher, and longer documents aren't unfairly penalized.

CameoDB returns scores alongside search results, enabling your application to implement custom ranking logic, result re-sorting, or confidence thresholds for filtering low-quality matches.

Segment Merging and Performance

Tantivy stores indexes in immutable segments. As documents are added, new segments are created. Periodic merging combines smaller segments into larger ones, reducing the number of segments that must be searched during queries. This improves read performance while maintaining write throughput.

CameoDB commits segments on a steady clock while writes arrive, and once more when they settle, so a new document is searchable after the next commit and no write ever waits for one. When deletes pile up, POST /_admin/index/{index}/merge folds each shard's segments into one while writes and searches carry on.

The Takeaway

Tantivy provides the full-text search engine that powers CameoDB's hybrid architecture. The index-only storage strategy keeps indices lean while redb provides fast document retrieval. Schema evolution, phrase queries, BM25 scoring, and segment merging deliver production-grade search capabilities without the complexity of managing a separate search service. The result is rich search integrated with ACID-compliant storage.