ACID Guarantees in a Leaderless World: CameoDB's Transaction Model

What's new since this post (v0.3.8). The write path described below is the current one: Redb commits the document and its WAL entry first, and Tantivy indexes it after. Each Tantivy commit records the WAL position it covers, so a restart replays exactly the writes the index has not seen yet. A shard now reopens in milliseconds after any stop, whatever its size.

The ACID Challenge in Leaderless Systems

Traditional databases achieve ACID guarantees through consensus algorithms like Raft or Paxos. These protocols elect a leader to serialize writes, ensuring consistency across replicas. But consensus comes with costs: leader election latency, quorum requirements, and limited scalability.

CameoDB takes a different approach. Instead of global consensus, it provides ACID guarantees at the storage engine level through redb's transactional model. Every document has exactly one owner in the cluster, computed from a consistent-hash ring, and that shard commits the write with full ACID semantics. No leader, no quorum, and no second copy to reconcile.

Atomicity: All-or-Nothing Writes

Atomicity ensures that a transaction is either fully applied or not applied at all. CameoDB guarantees atomicity through redb's transaction system. The document and its write-ahead log entry are written in the same redb transaction, so either both are committed or neither is.

// One redb transaction per write
txn = db.begin_write()
data.insert(id, json_blob)   // the document
wal.insert(seq_id, id)       // which document changed
txn.commit()                 // durable: the write is safe

// Tantivy follows the store
writer.add_document(...)     // indexed
writer.commit(wal_seq)       // the commit records the WAL position it covers

Redb is the source of truth and Tantivy is a view derived from it. The WAL holds only the document id, because the document itself is already in the same transaction. That keeps every write small and leaves one authoritative copy of each document.

Consistency: Invariants Across Storage

Consistency ensures that the database transitions from one valid state to another. CameoDB maintains consistency through schema validation: a declared field keeps its declared type and refuses values that do not fit it, and every schema carries a version and a thumbprint.

Between the two engines, the WAL is the bridge. A lookup by id is answered from redb the moment the write returns, and a search sees the write after the next Tantivy commit. On restart, CameoDB reads the WAL position stamped into the last Tantivy commit and replays only the ids above it, each from its committed row. Replay converges on the committed state of each document, so a document updated twenty times in that tail is indexed once, as it stands.

Isolation: Concurrent Operation Safety

Isolation ensures that concurrent transactions don't interfere with each other. CameoDB inherits it from redb's MVCC design. Readers work on snapshots, seeing a consistent view of the store at a point in time, unaffected by concurrent writes.

Writes go through a single writer per shard, so commits are serializable by construction. Concurrent writes to the same document are applied one after another, with no lost updates and no race conditions, and readers never wait for writers.

Durability: Surviving Failures

Durability ensures that committed transactions survive failures. With wal_sync on, the default, every write is fsynced when redb commits it, so an acknowledged write survives crashes, power failures and unclean shutdowns.

# cameodb.toml
[storage]
wal_sync = true      # fsync every committed write (default)

For bulk loads of data you can replay from elsewhere, wal_sync = false lets redb commit in memory and flush with the next durable commit. On write-only bulk ingest the two measure the same at saturation, around 370,000 documents a second on one node, so durability comes at no cost where throughput matters most.

Distributed Consistency Model

Writes are routed to the node that owns the shard, based on consistent hashing over 256 virtual nodes. Because each document lives on exactly one shard, there is one version of it to read: a lookup by id goes straight to its owner and sees the latest committed write, with no routing key needed on a normal index.

Searches fan out to every owner in parallel and merge on the way back. Each shard answers from its latest Tantivy commit, so a search sees every write up to that commit.

Schema Agreement

Schemas are the one thing every node shares, and CameoDB agrees on them across the cluster. When a write introduces a new field, the node that saw it asks every node to agree, and the whole cluster moves to the same schema version together. Each search reports the schema version and thumbprint it ran under, so an agent always knows which schema answered it.

The Shape of It

Strong Local ACID

• Full ACID on every shard
• Fast local transactions
• No leader election overhead
• Predictable latency

A Simple Global Model

• One owner per document
• Id lookups read the latest write
• Searches read the latest index commit
• Schemas agreed by every node

The Takeaway

CameoDB delivers ACID guarantees through redb's transactional storage engine, with Tantivy kept in step through a WAL that lives inside the same transaction. The cluster layer adds placement by consistent hashing and schema agreement, and no consensus on the data path. The result is a database that combines transactional integrity on every shard with a mesh that grows by adding nodes.