Skip to main content

Raft in Kahuna: Consensus and High Availability System

Overview​

Raft serves as the foundation for ensuring consistency and high availability in Kahuna's distributed data storage system. Operating at the core of Kahuna's architecture. Raft manages data replication across multiple nodes through a partition-based approach where each partition is independently governed by a dedicated Raft group. This document describes how Raft functions within the Kahuna ecosystem and the mechanisms that enable reliable distributed data management.

Consensus and Replication Mechanisms​

Raft implements a leader-based consensus protocol within each partition group. When client applications submit write requests, the designated leader node for that partition coordinates the operation across all replicas in the group.

The leader election process automatically selects a coordinator for each Raft group. This leader becomes the sole node authorized to handle client write operations and assumes responsibility for propagating log entries (representing data changes) to follower nodes. This centralized decision-making approach streamlines coordination and maintains operation ordering.

Log replication follows a structured process where the leader appends new operations to its local log before transmitting these entries to followers. Once a majority of nodes within the group have acknowledged receipt and storage of the entry, the leader marks it as committed. This majority-based commitment strategy ensures that data changes persist even when some nodes experience failures.

Fault Tolerance and Recovery Systems​

Raft's resilience to node failures comes from its replicated log architecture. If a leader node becomes unavailable, the remaining nodes in the group initiate a new election process based on their current log state. This automated failover mechanism minimizes system downtime while preserving fault tolerance capabilities.

Data consistency across the system is achieved through strict ordering guarantees. Raft ensures that all committed log entries are applied to each node's state machine in identical sequence. This strong consistency model is essential for maintaining transactional integrity throughout the Kahuna ecosystem, especially when handling complex operations.

Operational Architecture​

Kahuna implements a partitioned data model where information is segmented into discrete partitions. Each partition functions as an independent Raft group with its own consensus process. This partitioned design enables horizontal scalability since consensus operations occur independently across different data segments rather than requiring system-wide agreement.

When network partitions or communication disruptions occur, Raft's intrinsic recovery processes help affected nodes synchronize upon reconnection. The leader's log serves as the authoritative record, allowing followers to reconcile any discrepancies and reestablish consistency within the group.

Raft Group Configuration​

Each Kahuna partition is a distinct Raft group. In the default full-replication mode, every roster voter hosts every partition. With replication factor and replica placement, each partition has an explicit voter set, such as three or five replicas, and quorum is computed over that partition's voter replicas only.

One replica is elected leader and processes requests for that partition. Followers replicate committed data from the leader. Nodes that do not host a partition can still accept client requests and forward them to a hosting replica.

Leadership Management​

Leadership transitions occur through a voting mechanism triggered when the current leader becomes unresponsive or experiences failures. Each leadership election establishes a new "term" represented by a monotonically increasing counter value. The protocol strictly enforces that only one leader can exist per term within each Raft group, preventing split-brain scenarios.

In multi-partition clusters, independent elections can leave one node leading more partitions or more high-traffic partitions than its peers. Kahuna's optional leader balancer monitors leader count and partition load, then suggests normal Raft leadership handoffs. It changes leadership placement without moving partition data. Replica placement is a separate feature that changes which nodes store and vote for a partition.

Stale Leader Fencing​

A node can briefly believe it is still leader after it has been cut off from the voters that elect a replacement. Kahuna fences that window at multiple layers:

  • Replicated writes and durable transaction phases still require a Raft quorum, so a cut-off leader returns a retryable result instead of committing data
  • Authoritative reads confirm leadership through a quorum before serving from the leader
  • Actor-only state changes, including interactive transaction staging and point, prefix, or range locks, also confirm leadership before changing local memory
  • Proposals carry the term observed when they were admitted. If the node lost leadership and regained it in a newer term before appending the proposal, the proposal is rejected as stale and the caller retries
  • When a node loses leadership for a partition, Kahuna drops belief-only state for that partition: staged transactional writes, write intents, and exclusive prefix or range locks

Check-quorum step-down is enabled by default. A leader that stops hearing same-term acknowledgement from a majority steps down, bounding the stale-leader window. See Server Configuration for --raft-enable-check-quorum and --raft-check-quorum-interval-multiplier.

Transaction Support​

Raft plays a crucial role in Kahuna's transaction processing by ensuring changes to individual partitions are durably replicated before confirming transaction commitment. This integration helps the transaction layer maintain atomicity guarantees even when operations span multiple nodes or partitions.

Multi-Raft Management​

Kahuna's architecture efficiently manages thousands of concurrent Raft groups (partitions) on individual nodes. The system employs sophisticated batch processing techniques and intelligent scheduling algorithms to coordinate multiple Raft groups simultaneously without compromising performance or reliability.

Kommander Implementation​

Kahuna implements the Raft protocol through the Kommander library. Kommander handles Raft message processing, elections, log persistence, and state-machine callbacks. Raft log storage is configured separately from Kahuna's materialized key/value backend; each layer can use RocksDB or SQLite depending on server configuration.