Leaderless Mesh: How CameoDB Scales Without Consensus

What’s new since this post (v0.3.8). Peer traffic can be authenticated with a cluster pre-shared key, schema changes are agreed by every node so the whole mesh moves to one schema version together, and a node notices a peer that has stopped answering and replies for it at once instead of waiting on it. A search that spans the mesh reports how many shards answered in stats.shards.

The Consensus Problem

Most distributed databases rely on consensus algorithms like Raft or Paxos. These protocols ensure consistency by electing a leader, replicating writes, and requiring majority agreement before commits. The tradeoff, complexity and latency. Every write must travel through the leader, wait for quorum acknowledgment, and handle leader elections during failures. As clusters grow, consensus overhead increases, and scaling becomes painful.

CameoDB takes a different approach, a leaderless mesh. No coordinator, no leader elections, no quorum waits on the data path. Instead, data distribution and routing are handled through consistent hashing and a Kademlia DHT. Each node knows where data belongs, and operations route directly to the owner. The result, a mesh that grows by adding nodes, with minimal coordination overhead.

Consistent Hashing: Data Distribution Without Coordination

At the heart of CameoDB's distribution is a consistent hash ring. Each node is assigned virtual node tokens, and each shard maps to a specific position on the ring. When you write a document with a routing key, the system hashes the key using XXH3, finds the position on the ring, and determines which node owns that shard.

// Routing decision algorithm
hash = xxh3_64(routing_key)
shard_id = ConsistentRing.get_owner(hash)
node_id = shard_assignments[shard_id].owner
decision = if node_id == local_node
 RoutingDecision::Local
 else
 RoutingDecision::Remote { node_id, peer_addr }

This means deterministic routing without coordination. The same routing key always maps to the same shard and node, regardless of which node receives the request. Every node computes the owner from the same ring of 256 virtual nodes per member, so placement needs no lookup service and no central coordinator.

Kademlia DHT: Peer Discovery From a Single Seed

How do nodes find each other? CameoDB uses a Kademlia DHT built on libp2p. Each node maintains a routing table of known peers, organized by XOR distance from its own node ID. When a node joins, it contacts bootstrap nodes and discovers the network through iterative lookups.

The DHT provides three critical capabilities:

Peer Discovery

Point a node at a seed and it finds and connects to the rest of the cluster on its own.

Routing Metadata

Distribute shard ownership information across the cluster for decentralized routing.

Failure Detection

Notice when a peer stops answering, mark it lost, and answer requests for it at once rather than leaving them to wait.

Actor-Based Remote Execution

CameoDB uses the Kameo actor framework for distributed execution. Each node runs a NodeOrchestrator actor that manages local shards, and these actors can communicate remotely over libp2p. When the router decides an operation belongs on a remote node, it uses Kameo's remote messaging to forward the request.

// Remote call path
orchestrator_name = format!("orchestrator-{}", target_node_id)
remote_ref = RemoteActorRef::lookup(orchestrator_name).await
result = remote_ref.ask(&ClientOp::Search { ... }).await

Remote actors are registered with stable names like orchestrator-{node_id} and shard-{shard_id}, making them discoverable across the cluster. The same ClientOp message type is used locally and remotely, so operation semantics are consistent regardless of where they execute.

Scatter-Gather: Fan-Out Without Bottlenecks

For queries without a routing key, CameoDB uses scatter-gather broadcast. The router asks the cluster coordinator for known peers, selects up to a fanout limit, and executes the query in parallel across local and remote nodes.

// Broadcast search algorithm
peers = coordinator.get_known_peers()
selected = peers.take(broadcast_fanout_limit)

local_result = handle_client_op(op) // Local search
remote_results = parallel(selected, |peer| {
 remote.ask(peer, op, timeout)
}) // Fan-out to remotes

merged = top_k_merge(local_result, remote_results, limit)

Results are merged using score-aware top-K aggregation, allowing higher-scoring remote hits to displace weaker local results. Bounded concurrency ensures the system doesn't overwhelm itself with parallel requests, and per-call timeouts prevent stragglers from blocking the response.

Event-Driven State Management

Cluster metadata is persisted without polling or background tasks. All state transitions occur on membership events, PeerDiscovered when a node joins, PeerLost when a node leaves. The cluster coordinator maintains a simple state machine with three states, Active (all nodes present), Degraded (one node missing), Failed (more than one node missing).

Metadata is written to metadata.redb inline with state changes, providing crash-safe persistence without separate background processes. On boot, nodes load the persisted snapshot and reconcile with actual cluster state, logging discrepancies for operational visibility.

Why Leaderless

A leaderless mesh keeps the data path short and the operational model simple:

Advantages

• No leader election latency
• No coordinator to run
• Grows by adding nodes
• Simpler operational model
• Better write throughput

Design Notes

• One owner per document
• Id lookups go straight to the owner
• Searches fan out and merge in parallel
• A routing key groups related documents

The Takeaway

CameoDB's leaderless mesh trades consensus complexity for deterministic routing. Consistent hashing ensures data lands in predictable places, Kademlia DHT takes discovery from a single seed to the whole mesh, and actor-based remote execution provides clean distributed semantics. The result is a database that grows by adding nodes, spots a lost peer quickly, and keeps operational complexity low. No Raft, no Paxos, just math and networking doing what they do best.