1 Preface

Kahuna started as a question: what would it take to build a coordination system that a working developer could pick up in an afternoon and run in production the next week? Not a research prototype. Not a system that requires a PhD to operate. A system with locks, key-value storage, transactions, and sequences, all backed by consensus, all accessible through a simple API.

That question turned into code, and the code turned into something that needed a book.

1.1 Who This Book Is For

This book is for developers and operators who build and run distributed systems. You do not need prior experience with consensus protocols, MVCC, or actor models. The book starts with practical usage (how to store a value, acquire a lock, run a transaction) and progresses to internal mechanisms (how Raft replicates entries, how the two-phase commit protocol works, how the WAL is structured).

If you are evaluating Kahuna for a project, Part I (Chapters 1 through 7) gives you everything you need to make that decision. If you are already running Kahuna and need to understand its behavior under failure, Part III (Chapters 14 through 23) explains the internals. If you are operating a production cluster, Part IV (Chapters 24 through 29) covers deployment, scaling, backup, observability, and testing.

1.2 How This Book Is Organized

The book is divided into four parts:

Part I: Using Kahuna (Chapters 1 through 13). These chapters teach you how to use Kahuna’s three primitives (key-value store, distributed locks, distributed sequences), how to write transactions and scripts, and how to apply these tools to common patterns such as leader election, idempotent operations, and transactional workflows.

Part II: Architecture (Chapters 14 and 15). These two chapters provide the map. Chapter 14 introduces the component hierarchy. Chapter 15 traces a single request through every layer from client to disk and back. Every subsequent chapter references these two.

Part III: Mechanisms (Chapters 16 through 23). Each chapter explains one internal subsystem: partitioning, consensus, the actor model, MVCC, transactions, locks and sequences, persistence, and recovery. These chapters are independent of each other. Read them in any order after Part II.

Part IV: Operations (Chapters 24 through 29). These chapters cover deployment, cluster membership, range management, backup and point-in-time recovery, observability and tuning, and testing. They combine the external knowledge from Part I with the internal knowledge from Part III.

The appendices provide reference material: the script language syntax, the configuration reference, the CLI command reference, error codes, a comparison with other systems, and a glossary.

1.3 Conventions

Code examples use C# for client code and Kahuna Script for server-side scripts. Command-line examples show the kahuna-cli interactive shell or the kahuna-server flags. Configuration flags appear with their default values in tables throughout the text.

Each internal chapter ends with a “Failure Scenarios” section that describes what goes wrong when specific conditions occur. These sections are based on real behavior observed through the Jepsen test suite.

1.4 Acknowledgments

Building a distributed system is not a solo effort, even when it feels like one at 3 AM. The Jepsen test suite found real bugs that made Kahuna better. The Raft paper by Diego Ongaro and John Ousterhout provided the foundation. The open-source community around .NET provided the runtime and tooling that made the implementation possible.

Thank you to everyone who filed an issue, asked a question, or tried Kahuna on a workload it was not designed for. Those conversations shaped the system more than any design document.

And thank you for picking up this book. I hope it helps you build something reliable.

Andres Gutierrez

2 What Is Kahuna?

Imagine you are running three copies of a payment service behind a load balancer. A customer submits an order. The request lands on node A, which checks the customer’s balance, deducts the amount, and records the charge. But what happens if the same request, retried after a timeout, lands on node B? Node B checks the balance, sees the old value (the deduction hasn’t propagated yet), and charges the customer a second time.

This is not a hypothetical scenario. It is one of the most common bugs in distributed systems: two processes operating on shared state without coordination. The fix is not more careful coding. The fix is a coordination primitive (a lock, a conditional write, or an atomic transaction) that both processes agree to respect.

Building those coordination primitives yourself is hard. Getting them right under network partitions, process crashes, and clock drift is harder still. This is the problem that Kahuna solves.

2.1 The Coordination Problem

When an application runs on a single machine, coordination is straightforward. A thread acquires a mutex, does its work, and releases the mutex. The operating system guarantees that only one thread holds the mutex at a time. The application doesn’t need to think about what happens if the CPU loses power mid-operation, because the OS handles that, too.

Distributed systems don’t have a single machine. They have multiple processes running on multiple nodes, connected by a network that can delay, reorder, or drop messages. There is no shared memory. There is no global clock. There is no single entity that can decide “this process goes first.”

And yet, distributed applications need all of the things that a single-machine application takes for granted:

• Mutual exclusion. Only one worker should process a given job at a time.
• Consistent state. If two clients update the same key, every subsequent read should see the same value, regardless of which node handles the read.
• Atomic updates. When a transaction writes to three keys, either all three writes take effect or none of them do.
• Ordered identifiers. Order numbers, event IDs, and log sequence numbers need to be unique and monotonically increasing across the entire cluster.

Each of these requirements is a coordination problem. Solving any one of them requires agreement among multiple nodes. Solving them correctly, under failures, requires a consensus protocol: a formal mechanism by which a group of nodes agrees on a single value or a single order of operations, even when some nodes are unreachable.

Consensus protocols are well-studied. Raft, Paxos, and Zab have been implemented in production systems for over a decade. But implementing a consensus protocol is not the same as having a coordination system. A coordination system wraps the protocol in a usable API: locks you can acquire, keys you can read and write, transactions you can commit, and sequences you can allocate from.

That is what Kahuna provides.

2.2 What Kahuna Is

Kahuna is an open-source distributed coordination system built in C# on .NET. It provides three core primitives:

1. A distributed key-value store. Store and retrieve data across a cluster of nodes, with strong consistency, multi-version concurrency control (MVCC), and support for transactions.

2. Distributed locks. Acquire time-bounded locks (leases) on named resources, with monotonically increasing fencing tokens that prevent stale lock holders from corrupting data.

3. A distributed sequencer. Generate globally unique, monotonically increasing numbers without a single-node bottleneck.

These three primitives share a single cluster, a single consensus layer (Raft), and a single operational surface. You deploy one system instead of three.

Kahuna organizes data into partitions. Each partition is an independent unit with its own Raft group: its own leader, its own replicas, and its own log. This design allows Kahuna to scale horizontally: more partitions means more throughput, because different partitions can be led by different nodes.

Every write in Kahuna goes through Raft consensus. This means that once a write is acknowledged, it is durable on a majority of replicas. If a node fails, another node can take over as leader for that partition without losing committed data.

Kahuna is a Hawaiian word that refers to an expert in any field. Historically, it has been used to refer to doctors, surgeons, priests, and sorcerers.

2.3 The Three Primitives

2.3.1 Key-Value Storage

At its core, Kahuna is a distributed key-value store. You write a value under a string key, and you can read it back from any node in the cluster. Kahuna routes the key to the correct partition, forwards the request to the partition’s leader, and replicates the write through Raft before acknowledging it.

But Kahuna’s key-value store goes well beyond simple get and set. It provides:

• Revisions. Every write to a key increments a revision number. You can read the current revision, and you can use it for compare-and-swap operations: “set this key to X, but only if its current revision is 7.” If another writer changed the key in the meantime, the operation fails instead of silently overwriting.

• Conditional writes. Beyond compare-and-swap on revisions, you can set a key only if it doesn’t exist yet, only if it already exists, or only if its current value matches a specific value.

• Time-to-live (TTL). Keys can expire after a specified duration. This is useful for session tokens, ephemeral state, and presence detection.

• Durability levels. Keys can be persistent (survive restarts, replicated via Raft) or ephemeral (fast, in-memory only, lost on restart). You choose per key.

• Range queries. You can scan keys by prefix or by range, with pagination support for large result sets.

• Transactions. You can read and write multiple keys atomically, with either optimistic or pessimistic concurrency control.

Here is what a simple key-value operation looks like in C#:

var client = new KahunaClient("http://localhost:2070");

// Set a key
await client.SetKeyValue("user:1001:name", "Alice");

// Get a key
KahunaKeyValue result = await client.GetKeyValue("user:1001:name");
Console.WriteLine(result.ValueAsString()); // "Alice"
Console.WriteLine(result.Revision);         // 1

The key user:1001:name is routed to a partition based on its prefix. The write goes through Raft consensus. The read goes to the partition’s leader, which confirms it is still the leader before responding.

2.3.2 Distributed Locks

A distributed lock is a lease: a time-bounded claim on a named resource. While a process holds the lock, no other process can acquire it. When the lease expires (because the holder crashed, or because it simply didn’t renew in time), the lock becomes available to other processes.

Leases alone are not enough, though. Consider this scenario:

1. Process A acquires a lock with a 10-second lease.
2. Process A starts a long operation.
3. The lease expires because the operation takes longer than expected.
4. Process B acquires the lock.
5. Process A finishes the operation and writes its result, but it no longer holds the lock.

Process A just wrote to a shared resource without holding the lock. This is the stale lock holder problem, and it can corrupt data even when the lock implementation itself is correct.

Kahuna prevents this with fencing tokens. Every time a lock is acquired, Kahuna assigns it a monotonically increasing integer. Process A might get fencing token 5. When the lease expires and process B acquires the lock, process B gets fencing token 6. If the downstream system checks the fencing token before accepting a write, it can reject process A’s stale write (token 5) because it has already seen a higher token (6).

// Acquire a lock with a 10-second lease
await using KahunaLock lockHandle =
    await client.GetOrCreateLock(
        "resource:payment:1001",
        expiresMs: 10000);

if (lockHandle.IsAcquired)
{
    Console.WriteLine($"Lock acquired, fencing token: {lockHandle.FencingToken}");
    // Do the protected work
}
// Lock is released automatically when lockHandle is disposed

The await using pattern ensures the lock is released even if the code throws an exception. If the process crashes instead of throwing, the lease expires and the lock becomes available.

2.3.3 Distributed Sequencer

Many applications need unique identifiers: order numbers, event IDs, invoice numbers, log sequence numbers. These identifiers must be:

• Unique across the entire cluster.
• Monotonically increasing. A higher number always means a later event.
• Available without a single-node bottleneck. The system should not funnel all ID generation through one machine.

Database auto-increment columns satisfy the first two requirements, but not the third. UUIDs satisfy the first and third, but not the second (they are not sortable by generation time in their standard form). Snowflake IDs satisfy all three, but require careful clock management and machine ID assignment.

Kahuna’s sequencer takes a different approach. It uses block-based allocation: a node reserves a block of sequence values (for example, 1000 at a time) with a single Raft commit. It then serves individual values from that block without any further consensus operations. When the block is exhausted, it reserves another one.

This design means that one Raft commit can produce hundreds or thousands of unique values. The trade-off is gaps: if a node crashes with 300 values remaining in its block, those 300 values are never used. The sequence jumps from wherever it was to the next block boundary. Values are guaranteed to be unique, but they are not guaranteed to be gap-free.

// Create a sequence
await client.CreateSequence("order-ids");

// Allocate the next value
KahunaSequence seq = await client.NextSequenceValue("order-ids");
Console.WriteLine($"Order ID: {seq.CurrentValue}"); // 1

// Allocate a batch of values
KahunaSequenceRange range = await client.ReserveSequenceRange("order-ids", 100);
Console.WriteLine($"Range: {range.Start} to {range.End}"); // e.g. 2 to 101

For most applications, gaps are acceptable. If you need gap-free sequences (for example, invoice numbers in jurisdictions that require them), you can set the block size to 1, which forces a Raft commit per value. This eliminates gaps but reduces throughput.

2.4 When to Use Kahuna

Kahuna is designed for coordination workloads: operations where multiple processes need to agree on shared state. Here are the scenarios where it fits well:

Distributed locking. Protecting shared resources across services: job processing queues, rate limiters, payment flows, leader election, and any workflow where “only one at a time” is a correctness requirement.

Coordination metadata. Storing configuration that multiple services read and update: feature flags, service registrations, routing tables, cluster membership records.

Transactional updates to small key sets. Atomically updating a handful of related keys: account balances, inventory counts, state machines. Kahuna’s transactions are designed for small working sets (tens of keys), not bulk data operations.

Unique ID generation. Producing globally unique, monotonically increasing identifiers without a centralized bottleneck.

Session and presence management. Tracking which services or users are currently active, using ephemeral keys with TTL.

2.5 When Not to Use Kahuna

Kahuna is not a general-purpose database, and using it as one will lead to poor results.

Bulk data storage. If you need to store millions of rows with complex queries, use a database (PostgreSQL, CockroachDB, CamusDB). Kahuna stores coordination metadata and small working sets, not application data.

Message queuing. If you need publish-subscribe, message routing, or stream processing, use a message broker (Kafka, RabbitMQ, NATS). Kahuna can coordinate producers and consumers, but it is not a message transport.

High-throughput caching. For large-scale cache workloads with millions of reads per second, use Redis or Memcached. Kahuna can serve as a cache for low-to-medium workloads (it supports TTL, ephemeral durability, and no-revision writes), but its strong consistency and Raft consensus path add overhead that dedicated cache systems avoid.

Analytics and reporting. If you need aggregations, joins, or full-text search, use a database or search engine. Kahuna’s query model is key-based, not relational.

The simplest rule: if the data is coordination state (who holds the lock, what is the current sequence number, what is the latest configuration version), Kahuna is a good fit. If the data is application state (user profiles, product catalogs, order histories), use a database.

2.6 How Kahuna Compares

Several established systems provide overlapping capabilities. Understanding the differences helps you choose the right tool.

2.6.1 etcd

etcd is a distributed key-value store built on Raft. It is the coordination backbone of Kubernetes. Like Kahuna, it provides linearizable reads and writes, leases, and watches.

Differences: etcd does not provide a built-in sequencer, a transaction scripting language, or MVCC-based snapshot isolation for multi-key transactions. etcd’s lock API (in the concurrency package) does not return a fencing token directly. A developer can read the key’s CreateRevision and use it as a fencing value, but the application must implement that pattern manually. Kahuna returns a fencing token as an explicit field on every lock acquisition. etcd uses a single Raft group, so every node holds every key. This limits etcd to small clusters (typically 3 or 5 nodes) and a total data size of a few gigabytes. Kahuna uses multiple Raft groups (one per partition) and a configurable replication factor. Each partition is replicated to a subset of nodes, not all of them. This means a Kahuna cluster can grow well beyond 5 or 9 nodes: keys and locks are distributed across many partitions, and each partition only consumes resources on its replicas. etcd is written in Go; Kahuna is written in C#.

2.6.2 Apache ZooKeeper

ZooKeeper is the original distributed coordination service. It provides a hierarchical namespace (similar to a filesystem), ephemeral nodes, watches, and sequential nodes for ordering.

Differences: ZooKeeper’s data model is tree-structured (znodes), while Kahuna’s is flat (key-value). ZooKeeper does not provide multi-key transactions, a scripting language, or a built-in sequencer with block allocation. ZooKeeper’s lock recipe uses sequential ephemeral znodes. The sequence number on the znode (or the znode’s czxid) is monotonically increasing and can serve as a fencing token, but the application must read and propagate it manually. Like etcd, ZooKeeper uses a single consensus group (Zab), so every node holds every znode. This limits cluster size and total data capacity. Kahuna distributes data across partitions with per-partition replication, which allows it to scale to larger clusters. ZooKeeper is written in Java; Kahuna is written in C#.

2.6.3 HashiCorp Consul

Consul provides service discovery, health checking, and a key-value store. Its KV store supports CAS operations and distributed locks via sessions.

Differences: Consul is primarily a service mesh and discovery tool; its KV store is a secondary feature. Consul does not provide multi-key transactions, MVCC, snapshot isolation, or a sequencer. Consul’s KV entries carry a ModifyIndex that increases monotonically, so a lock holder can use it as a fencing value, but the application must implement that pattern. Consul is written in Go; Kahuna is written in C#.

2.6.4 Redis

Redis is an in-memory data structure store. With Redlock (a distributed lock algorithm), it provides distributed locking. Redis Cluster provides horizontal scaling with hash-slot partitioning.

Differences: Redis prioritizes throughput and low latency over strong consistency. Redlock has been the subject of academic debate regarding its safety guarantees under certain failure modes. Kahuna’s locks are built on Raft consensus with linearizable operations and fencing tokens. Redis does not provide multi-key transactions with snapshot isolation across partitions. Redis is written in C; Kahuna is written in C#.

2.6.5 Summary

This comparison is factual, not promotional. Each system makes different trade-offs, and the right choice depends on your workload, your team’s expertise, and your operational requirements.

2.7 Architecture at a Glance

Before diving into the details in later chapters, here is a brief sketch of how Kahuna is organized internally. You don’t need to understand all of this now. Part III of the book covers each component in depth.

A Kahuna cluster consists of multiple nodes. Each node runs the same server binary and can serve any request by routing it to the appropriate partition leader.

Data is organized into partitions. Each partition is a range of keys managed by its own independent Raft group. A Raft group has a leader (which handles writes and linearizable reads) and followers (which replicate the leader’s log). If the leader fails, the remaining replicas elect a new leader.

When a client sends a request, the receiving node determines which partition owns the key (based on hashing or range mapping), finds the leader for that partition, and forwards the request. If the receiving node is the leader, it handles the request locally. The leader proposes the write to Raft, waits for a majority of replicas to acknowledge, and then responds to the client.

Client
  → Node (any node can receive)
  → Partition resolution (which partition owns this key?)
  → Leader (forward to leader if this node isn't it)
  → Raft consensus (replicate to majority)
  → Acknowledge to client

Kahuna uses Hybrid Logical Clocks (HLC) for ordering events across nodes. HLC combines a physical timestamp with a logical counter to produce globally unique, causally ordered timestamps without requiring synchronized clocks. Every committed write gets an HLC timestamp that establishes its position in the total order of operations.

The key-value store uses Multi-Version Concurrency Control (MVCC): each key retains multiple versions, indexed by their HLC timestamp. This allows transactions to read a consistent snapshot of the data at a specific point in time, even while other transactions are writing to the same keys.

2.8 A First Look at the Client SDK

To close this chapter, here is a complete C# program that connects to a Kahuna cluster, uses all three primitives, and handles basic errors. Don’t worry about understanding every detail. The following chapters will explain each operation thoroughly.

using Kahuna.Client;

// Connect to a Kahuna cluster
var client = new KahunaClient("http://localhost:2070");

// --- Key-Value ---
// Store a value
await client.SetKeyValue("config:feature:dark-mode", "enabled");

// Read it back
var kv = await client.GetKeyValue("config:feature:dark-mode");
Console.WriteLine($"Key: {kv.Key}, Value: {kv.ValueAsString()}, Revision: {kv.Revision}");

// Conditional write: update only if the revision hasn't changed
var updated = await client.TryCompareRevisionAndSetKeyValue(
    "config:feature:dark-mode",
    "disabled",
    kv.Revision);
Console.WriteLine($"Update succeeded: {updated.Success}");

// --- Distributed Lock ---
// Acquire a lock with a 5-second lease
await using var lockHandle = await client.GetOrCreateLock(
    "job:nightly-report",
    expiresMs: 5000);

if (lockHandle.IsAcquired)
{
    Console.WriteLine($"Acquired lock with fencing token: {lockHandle.FencingToken}");
    // Protected work goes here
}

// --- Sequencer ---
// Create a sequence and allocate values
await client.CreateSequence("invoice-numbers");
var next = await client.NextSequenceValue("invoice-numbers");
Console.WriteLine($"Next invoice number: {next.CurrentValue}");

This program doesn’t handle retries, leader changes, or transaction conflicts yet. Chapter 13 covers all of those. For now, the point is to see the shape of the API: connect, operate, and let the cluster handle replication and consistency.

2.9 What’s Ahead

The rest of this book is organized into four parts:

Part I, Using Kahuna (Chapters 2 to 7), walks through each primitive in detail. You’ll start a cluster, learn the full key-value API, master distributed locks and fencing tokens, use the sequencer, write transactions, and learn the Kahuna scripting language.

Part II, Building Distributed Applications (Chapters 8 to 13), applies these primitives to real engineering problems: leader election, idempotent workflows, service coordination, transactional pipelines, and failure handling.

Part III, Understanding Kahuna Internals (Chapters 14 to 23), opens the hood. You’ll trace a request from the client through routing, consensus, the actor model, MVCC, the two-phase commit protocol, persistence, and recovery. Every chapter connects to actual source code.

Part IV, Operating and Extending Kahuna (Chapters 24 to 29), covers production operations: deployment, cluster membership, range management, backup and point-in-time recovery, observability, performance tuning, and Jepsen testing.

You don’t need to read the book linearly. Parts I and II are designed for developers who want to use Kahuna. Parts III and IV are for those who want to understand or operate it. But the concepts build on each other, and the book is designed to be read in order if you have the time.

Let’s begin. The next chapter gets a cluster running on your machine.

3 Getting Started

This chapter takes you from zero to a working Kahuna environment. By the end, you will have a running cluster, a command-line client, and a C# application that reads, writes, locks, and generates sequence numbers. You will also crash a node and see the cluster recover on its own.

3.1 Prerequisites

Kahuna runs on .NET. You need:

• .NET 10 SDK (or later). Download it from dotnet.microsoft.com.
• Docker (optional, for multi-node clusters). Any recent version of Docker Desktop or Docker Engine works.

Verify your .NET installation:

dotnet --version

If this prints a version number of 10.0 or higher, you are ready.

3.2 Installing the Server

Kahuna is distributed as a .NET global tool. Install it with a single command:

dotnet tool install -g Kahuna.Server

This places the kahuna-server command on your PATH. Run it:

kahuna-server

With no arguments, the server starts a standalone node. It listens for HTTP connections on port 2070. The output shows the storage paths and the ports the node is using.

By default, the node stores its key-value data and Raft write-ahead log under your user data directory:

• Linux/macOS: ~/.local/share/kahuna
• Windows: %LOCALAPPDATA%\kahuna

Both paths are printed at startup. To store data in a different location, set the KAHUNA_HOME environment variable, or pass --storage-path and --wal-path explicitly.

The standalone node elects itself as the leader for every partition. It does not need any peer configuration. This makes it the fastest way to start experimenting.

3.2.1 Storage Backends

Kahuna supports three storage backends:

The default is RocksDB. For quick experiments where you do not need data to survive a restart, use the memory backend:

kahuna-server --storage memory --wal-storage memory

3.2.2 HTTPS

The standalone node serves HTTP only. To enable HTTPS, supply a certificate:

kahuna-server \
  --https-ports 2071 \
  --https-certificate /path/to/certificate.pfx

For local development, a self-signed certificate is sufficient. The client SDK has an option to skip certificate validation (covered later in this chapter).

3.3 Installing the Command-Line Client

The CLI is a separate .NET global tool:

dotnet tool install -g Kahuna.Control

This installs the kahuna-cli command. By default, it connects to http://localhost:2070, which matches the standalone server’s default port.

3.4 First Operations with the CLI

Start a standalone server in one terminal window. Open a second terminal and try the following commands.

3.4.1 Setting and Getting a Key

Write a value:

kahuna-cli --set "users/alice" --value '{"name":"Alice","role":"admin"}'

Read it back:

kahuna-cli --get "users/alice"

The output shows the value and metadata, including the revision number. Every write increments the revision.

3.4.2 Setting a Key with an Expiry

Keys can have a time-to-live (TTL). The --expires flag takes a value in milliseconds:

kahuna-cli --set "session/abc123" --value "active" --expires 30000

This key expires after 30 seconds. After that, a --get on this key returns nothing.

3.4.3 Acquiring and Releasing a Lock

Acquire a lock named jobs/send-email. The --expires flag specifies the lease duration in milliseconds:

kahuna-cli --lock "jobs/send-email" --owner "worker-1" --expires 10000

The output confirms the lock is acquired. It also shows the fencing token, a monotonically increasing number. Each time the lock changes hands, the fencing token increases.

Release the lock:

kahuna-cli --unlock "jobs/send-email" --owner "worker-1"

Only the owner that acquired the lock can release it.

3.4.4 Creating and Using a Sequence

Create a sequence named order-ids:

kahuna-cli --create-sequence "order-ids"

Retrieve the next value:

kahuna-cli --next-sequence "order-ids"

Each call returns a new, unique, monotonically increasing number. You can also reserve a range of values at once:

kahuna-cli --reserve-sequence "order-ids" --count 10

This reserves 10 values and returns the start and end of the range.

3.4.5 Output Formats

By default, the CLI prints results in a human-readable console format. For scripting, use JSON output:

kahuna-cli --get "users/alice" --format json

3.5 Running a Three-Node Cluster

A single node is fine for development. For testing replication and fault tolerance, you need multiple nodes. Kahuna provides two ways to run a local cluster: Docker Compose and shell scripts.

3.5.1 Option 1: Docker Compose

The Kahuna repository includes a Docker Compose file at docker/local.yml. This file defines a three-node cluster on a Docker bridge network.

Clone the repository and start the cluster:

git clone https://github.com/kahunakv/kahuna.git
cd kahuna
docker compose -f docker/local.yml up -d

The three nodes expose these ports on your host:

Each node has its own storage volume. Each node’s --initial-cluster flag lists the other two nodes’ Raft endpoints. This is how nodes discover each other at startup.

Verify the cluster is running:

docker compose -f docker/local.yml ps

All three containers should show a healthy status.

Connect the CLI to any node in the cluster. For the Docker cluster, the HTTPS ports use a self-signed certificate, so add --insecure to skip validation:

kahuna-cli -c "https://localhost:8082" --insecure --set "test/key" --value "hello"
kahuna-cli -c "https://localhost:8084" --insecure --get "test/key"

Notice that you write to node 1 and read from node 2. Kahuna routes the request to the correct partition leader internally, and the value is replicated through Raft. The read returns the same value regardless of which node you connect to.

3.5.2 Option 2: Local Shell Script

If you prefer to run the cluster outside Docker, the repository includes a script that starts three nodes as local processes:

cd kahuna
./scripts/run-cluster.sh

This script builds the server from source, then starts three nodes on localhost. The port layout is the same as the Docker cluster:

Press Ctrl+C to stop all three nodes.

For a standalone node from source, use:

./scripts/run-standalone.sh

Both scripts accept environment variables for customization:

For example, to run a fully ephemeral cluster:

KAHUNA_STORAGE=memory ./scripts/run-cluster.sh

3.5.3 How Cluster Discovery Works

When a node starts, it needs to know about its peers. Kahuna uses static discovery: each node’s --initial-cluster flag lists the Raft endpoints of the other nodes. The node contacts those peers and forms a Raft group for each partition.

The key configuration flags for clustering are:

For example, to manually start node 1 of a three-node cluster:

kahuna-server \
  --raft-nodename kahuna1 \
  --raft-nodeid 1 \
  --raft-host 127.0.0.1 \
  --raft-port 8082 \
  --http-ports 8081 \
  --https-ports 8082 \
  --https-certificate certificate.pfx \
  --initial-cluster 127.0.0.1:8084 127.0.0.1:8086 \
  --initial-cluster-partitions 3

Nodes 2 and 3 use the same pattern, with their own node name, ID, ports, and peer list. Each node lists only the other nodes in --initial-cluster, not itself.

3.6 Connecting from C#

The Kahuna .NET client library communicates with the server over gRPC. Install it from NuGet:

dotnet add package Kahuna.Client

3.6.1 Creating a Client

For a single endpoint:

using Kahuna.Client;

var client = new KahunaClient("http://localhost:2070");

For a cluster with multiple endpoints, pass an array of URLs:

var client = new KahunaClient(new[]
{
    "https://localhost:8082",
    "https://localhost:8084",
    "https://localhost:8086"
});

The client distributes requests across the endpoints using round-robin. Kahuna routes each request to the correct partition leader internally, so any endpoint can handle any request.

3.6.2 Skipping Certificate Validation

When connecting to a cluster that uses self-signed certificates (such as the local Docker cluster), configure the client to skip TLS validation:

var client = new KahunaClient(
    new[] { "https://localhost:8082", "https://localhost:8084", "https://localhost:8086" },
    options: new KahunaOptions { AllowInsecureCertificateValidation = true }
);

Do not use this setting in production.

3.6.3 Key-Value Operations

Write and read a value:

KahunaKeyValue setResult = await client.SetKeyValue("users/alice", "admin");
Console.WriteLine($"Revision: {setResult.Revision}");

KahunaKeyValue getResult = await client.GetKeyValue("users/alice");
Console.WriteLine($"Value: {getResult.ValueAsString()}");
Console.WriteLine($"Revision: {getResult.Revision}");

Every successful write returns a KahunaKeyValue object. The Success property tells you whether the write went through. The Revision property is a monotonically increasing number that changes with every update to that key.

Write with a TTL (in milliseconds):

await client.SetKeyValue("session/token-xyz", "active", expiryTime: 60000);

This key expires after 60 seconds.

3.6.4 Distributed Locks

Acquire a lock, do some work, and release it:

await using KahunaLock lockHandle = await client.GetOrCreateLock(
    resource: "jobs/send-email",
    expiry: TimeSpan.FromSeconds(10),
    durability: LockDurability.Persistent
);

if (lockHandle.IsAcquired)
{
    Console.WriteLine($"Lock acquired. Fencing token: {lockHandle.FencingToken}");
    // Do the protected work here.
}
else
{
    Console.WriteLine("Could not acquire the lock. Another process holds it.");
}
// The lock is released automatically when the using block ends.

The GetOrCreateLock method returns a KahunaLock object. If the lock is already held by another owner, IsAcquired is false. The lock implements IAsyncDisposable, so it is released automatically at the end of the await using block.

The FencingToken is a monotonically increasing number. Each time a lock changes hands, the fencing token increases. You can pass this token to downstream services to ensure that a stale lock holder cannot overwrite newer data. Chapter 4 covers fencing in detail.

3.6.5 Sequences

Create a sequence and generate values:

KahunaSequence seq = await client.CreateSequence("order-ids");
Console.WriteLine($"Sequence created. Current value: {seq.CurrentValue}");

long nextId = await client.NextSequenceValue("order-ids");
Console.WriteLine($"Next value: {nextId}");

KahunaSequenceRange range = await client.ReserveSequenceRange("order-ids", count: 10);
Console.WriteLine($"Reserved range: {range.Start} to {range.End}");

NextSequenceValue returns a single value. ReserveSequenceRange reserves a batch of values and returns the start and end of the range. The values in a reserved range are guaranteed to be unique and monotonically increasing across all clients.

3.7 A Complete Example

The following console application connects to a standalone Kahuna node and exercises all three primitives:

using Kahuna.Client;
using Kahuna.Shared.Locks;

var client = new KahunaClient("http://localhost:2070");

// --- Key-Value ---
KahunaKeyValue kv = await client.SetKeyValue("demo/greeting", "Hello, Kahuna!");
Console.WriteLine($"Set key 'demo/greeting' at revision {kv.Revision}");

KahunaKeyValue read = await client.GetKeyValue("demo/greeting");
Console.WriteLine($"Read: {read.ValueAsString()} (revision {read.Revision})");

// Update the key.
KahunaKeyValue updated = await client.SetKeyValue("demo/greeting", "Hello again!");
Console.WriteLine($"Updated to revision {updated.Revision}");

// --- Distributed Lock ---
await using KahunaLock lockHandle = await client.GetOrCreateLock(
    resource: "demo/my-lock",
    expiry: TimeSpan.FromSeconds(30),
    durability: LockDurability.Persistent
);

if (lockHandle.IsAcquired)
{
    Console.WriteLine($"Lock acquired. Fencing token: {lockHandle.FencingToken}");
}

// --- Sequence ---
await client.CreateSequence("demo/counter");
for (int i = 0; i < 5; i++)
{
    long id = await client.NextSequenceValue("demo/counter");
    Console.WriteLine($"Sequence value: {id}");
}

Console.WriteLine("Done.");

To run this example, create a new console project:

mkdir kahuna-demo && cd kahuna-demo
dotnet new console
dotnet add package Kahuna.Client

Replace the contents of Program.cs with the code above. Start a standalone Kahuna server in another terminal (kahuna-server), then run the application:

dotnet run

You should see output similar to:

Set key 'demo/greeting' at revision 1
Read: Hello, Kahuna! (revision 1)
Updated to revision 2
Lock acquired. Fencing token: 1
Sequence value: 0
Sequence value: 1
Sequence value: 2
Sequence value: 3
Sequence value: 4
Done.

3.8 Your First Failure Experiment

Distributed systems earn their value when something goes wrong. This experiment demonstrates that Kahuna keeps serving requests after a node failure.

3.8.1 Setup

Start the three-node Docker cluster:

docker compose -f docker/local.yml up -d

Write a key through node 1:

kahuna-cli -c "https://localhost:8082" --insecure \
  --set "experiment/counter" --value "42"

Verify the key is readable from all three nodes:

kahuna-cli -c "https://localhost:8082" --insecure --get "experiment/counter"
kahuna-cli -c "https://localhost:8084" --insecure --get "experiment/counter"
kahuna-cli -c "https://localhost:8086" --insecure --get "experiment/counter"

All three should return 42.

3.8.2 Kill a Node

Stop the second container:

docker stop kahuna2

Now try reading and writing through the remaining nodes:

kahuna-cli -c "https://localhost:8082" --insecure --get "experiment/counter"
kahuna-cli -c "https://localhost:8086" --insecure \
  --set "experiment/counter" --value "43"
kahuna-cli -c "https://localhost:8082" --insecure --get "experiment/counter"

The reads and writes still work. Two out of three nodes form a majority (a quorum), so Raft can still reach consensus on new writes.

3.8.3 Bring the Node Back

docker start kahuna2

After a few seconds, node 2 catches up with the other two. Read the key through node 2:

kahuna-cli -c "https://localhost:8084" --insecure --get "experiment/counter"

It returns 43, the value that was written while it was down. Raft replicated the missed writes when node 2 rejoined the cluster.

3.8.4 What Happens If Two Nodes Go Down?

Stop two nodes:

docker stop kahuna2 kahuna3

Now try to write:

kahuna-cli -c "https://localhost:8082" --insecure \
  --set "experiment/counter" --value "44"

This request fails or times out. With only one node remaining out of three, Raft cannot form a quorum. The cluster refuses to accept writes because it cannot guarantee that the write will be durable on a majority of replicas.

This is intentional. Kahuna chooses consistency over availability: it will not accept a write that it cannot safely replicate.

Bring the nodes back:

docker start kahuna2 kahuna3

After a few seconds, the cluster resumes normal operation.

3.9 Summary

In this chapter you installed Kahuna’s server and CLI tools, ran a standalone node, operated a three-node cluster, and connected a C# application. You used all three primitives (key-value storage, distributed locks, and sequences) from both the CLI and the client SDK. You also ran a failure experiment that demonstrated Raft’s ability to maintain availability with a majority of nodes and to replicate missed writes when a failed node returns.

The next chapter explores key-value operations in depth: conditional writes, revisions, TTL, range queries, and durability levels.

4 Key-Value Operations

Chapter 2 showed how to set and get a key. This chapter covers the full key-value API: conditional writes, revisions, time-to-live, prefix queries, range scans, batch operations, durability levels, and snapshot reads. By the end, you will know every tool Kahuna provides for working with key-value data.

4.1 Keys and Values

A Kahuna key is a string. A value is a byte array (or a UTF-8 string, which the client converts to bytes). There is no schema, no type enforcement, and no size hierarchy. Every key is flat.

Keys often follow a convention of prefixed namespaces separated by /:

users/alice
users/bob
config/feature-flags/dark-mode
orders/2024/00042

This convention is not just cosmetic. Kahuna uses the key’s prefix (everything up to and including the last /) to determine which partition stores the key. Keys that share a prefix route to the same partition. This matters for prefix queries (covered later in this chapter) and for transactions (covered in Chapter 6).

4.2 Revisions

Every key in Kahuna has a revision number. The revision starts at 0 on the first write and increases by one with every subsequent write to the same key.

var client = new KahunaClient("http://localhost:2070");

KahunaKeyValue r1 = await client.SetKeyValue("counter/a", "first");
Console.WriteLine(r1.Revision);  // 0

KahunaKeyValue r2 = await client.SetKeyValue("counter/a", "second");
Console.WriteLine(r2.Revision);  // 1

KahunaKeyValue r3 = await client.SetKeyValue("counter/a", "third");
Console.WriteLine(r3.Revision);  // 2

Revisions are per-key. Writing to counter/a does not affect the revision of counter/b.

The revision number serves two purposes:

1. Conflict detection. Conditional writes can check the revision before updating. If another client changed the key since you last read it, the revision will not match and the write will fail. This is optimistic concurrency control.

2. History. Kahuna stores previous revisions. You can read the value of a key at any past revision.

4.2.1 Reading a Specific Revision

KahunaKeyValue atRev1 = await client.GetKeyValueRevision("counter/a", revision: 1);
Console.WriteLine(atRev1.ValueAsString());  // "second"

If the requested revision does not exist, Success is false and Value is null.

4.3 Conditional Writes

An unconditional SetKeyValue always overwrites the current value. Conditional writes add a check: the write only succeeds if a condition is met. If the condition fails, Success is false and the key is unchanged.

Kahuna provides four conditional write modes, controlled by the KeyValueFlags parameter.

4.3.1 Set If Not Exists

Write only if the key does not already exist:

KahunaKeyValue result = await client.SetKeyValue(
    "config/db-connection",
    "Server=primary;Database=app",
    flags: KeyValueFlags.SetIfNotExists
);

if (result.Success)
    Console.WriteLine("Configuration key created.");
else
    Console.WriteLine("Key already exists. Not overwritten.");

This is useful for one-time initialization. Multiple processes can race to set the key. Exactly one will succeed.

4.3.2 Set If Exists

Write only if the key already exists:

KahunaKeyValue result = await client.SetKeyValue(
    "config/db-connection",
    "Server=secondary;Database=app",
    flags: KeyValueFlags.SetIfExists
);

if (result.Success)
    Console.WriteLine("Configuration updated.");
else
    Console.WriteLine("Key does not exist. Nothing to update.");

4.3.3 Compare-Revision-And-Set (CRAS)

Write only if the key’s current revision matches a specified value. This is the primary mechanism for optimistic concurrency control.

// Read the current value and its revision.
KahunaKeyValue current = await client.GetKeyValue("inventory/widget-stock");
Console.WriteLine($"Current stock: {current.ValueAsString()}, revision: {current.Revision}");

// Update the value, but only if nobody else changed it.
KahunaKeyValue updated = await client.TryCompareRevisionAndSetKeyValue(
    "inventory/widget-stock",
    "95",
    compareRevision: current.Revision
);

if (updated.Success)
    Console.WriteLine($"Updated to revision {updated.Revision}.");
else
    Console.WriteLine("Conflict: another client modified the key. Read and retry.");

The pattern is read, modify, write-if-unchanged. If another client wrote to the key between your read and your write, the revision will differ and the write will fail. Your code can then re-read and retry.

4.3.4 Compare-Value-And-Set (CVAS)

Write only if the key’s current value matches a specified byte sequence:

KahunaKeyValue result = await client.TryCompareValueAndSetKeyValue(
    "state-machine/order-123",
    value: "shipped",
    compareValue: "paid"
);

if (result.Success)
    Console.WriteLine("State transitioned from 'paid' to 'shipped'.");
else
    Console.WriteLine("Current value is not 'paid'. Transition rejected.");

This is useful for state machines. The transition from “paid” to “shipped” only succeeds if the current state is “paid.” If another process already moved the state to “cancelled,” the write fails.

4.4 Time-to-Live (TTL)

Keys can have an expiration time. After the TTL elapses, the key is no longer readable. The TTL is specified in milliseconds.

// This key expires after 60 seconds.
await client.SetKeyValue("session/token-abc", "active", expiryTime: 60000);

You can also use a TimeSpan:

await client.SetKeyValue("session/token-abc", "active", TimeSpan.FromMinutes(5));

An expiry of 0 (the default) means the key does not expire.

4.4.1 Extending a TTL

If a key is about to expire but you need it to live longer, extend its TTL without changing its value:

await client.ExtendKeyValue("session/token-abc", expiresMs: 60000);

Or using a TimeSpan:

await client.ExtendKeyValue("session/token-abc", TimeSpan.FromMinutes(5));

The extension resets the expiration clock from the current time. If the key does not exist, Success is false.

4.4.2 TTL and Conditional Writes

You can combine TTL with conditional write flags:

// Create a session key only if it doesn't already exist, with a 30-minute TTL.
KahunaKeyValue session = await client.SetKeyValue(
    "session/user-42",
    "session-data-here",
    expiryTime: 1800000,
    flags: KeyValueFlags.SetIfNotExists
);

4.5 Durability Levels

Every key-value operation accepts a durability parameter with two options:

// Persistent (default): safe, replicated.
await client.SetKeyValue("config/important", "value", durability: KeyValueDurability.Persistent);

// Ephemeral: fast, local, not replicated.
await client.SetKeyValue("cache/temp-result", "value", durability: KeyValueDurability.Ephemeral);

Use ephemeral durability for data you can afford to lose: caches, temporary counters, rate-limit windows. Use persistent durability for anything that must survive failures.

A read with a specific durability level only sees keys written at that same level. A key written as Ephemeral is not visible to a Persistent read, and vice versa.

4.6 Deleting Keys

Delete a key:

KahunaKeyValue deleted = await client.DeleteKeyValue("users/alice");
if (deleted.Success)
    Console.WriteLine("Key deleted.");
else
    Console.WriteLine("Key did not exist.");

A delete on a non-existent key sets Success to false.

4.7 Checking Existence

If you only need to know whether a key exists (without fetching its value), use ExistsKeyValue:

KahunaKeyValue exists = await client.ExistsKeyValue("users/alice");
if (exists.Success)
    Console.WriteLine($"Key exists at revision {exists.Revision}.");
else
    Console.WriteLine("Key does not exist.");

This is lighter than GetKeyValue because the server does not transfer the value bytes.

4.8 No-Revision Writes

By default, every write archives a revision entry so you can read the key’s history. For high-throughput keys where you never need history (caches, counters, ephemeral state), you can skip revision archiving:

await client.SetKeyValueNoRevision("metrics/request-count", "4217");

The revision counter still increments and conditional writes still work, but the previous value is not stored for historical reads. This reduces storage overhead for keys that change frequently.

4.9 Batch Operations

When you need to read, write, or delete many keys at once, batch operations reduce the number of round trips.

4.9.1 Batch Set

var items = new List<KahunaSetKeyValueRequestItem>
{
    new() { Key = "users/alice", Value = "admin"u8.ToArray() },
    new() { Key = "users/bob", Value = "editor"u8.ToArray() },
    new() { Key = "users/carol", Value = "viewer"u8.ToArray() }
};

List<KahunaKeyValue> results = await client.SetManyKeyValues(items);

foreach (KahunaKeyValue kv in results)
    Console.WriteLine($"{kv.Key}: revision {kv.Revision}, success: {kv.Success}");

Each item in the batch can have its own flags, expiry, and durability level.

4.9.2 Batch Get

var requests = new List<KahunaGetManyKeyValuesRequestItem>
{
    new() { Key = "users/alice" },
    new() { Key = "users/bob" },
    new() { Key = "users/carol" }
};

List<KahunaKeyValue> results = await client.GetManyKeyValues(requests);

foreach (KahunaKeyValue kv in results)
{
    if (kv.Success)
        Console.WriteLine($"{kv.Key} = {kv.ValueAsString()}");
    else
        Console.WriteLine($"{kv.Key}: not found");
}

4.9.3 Batch Delete

// Simple form: pass a list of keys.
List<KahunaKeyValue> deleted = await client.DeleteManyKeyValues(
    new[] { "users/alice", "users/bob", "users/carol" }
);

foreach (KahunaKeyValue kv in deleted)
    Console.WriteLine($"{kv.Key}: deleted = {kv.Success}");

4.9.4 Batch Exists

var requests = new List<KahunaGetManyKeyValuesRequestItem>
{
    new() { Key = "users/alice" },
    new() { Key = "users/bob" }
};

List<KahunaKeyValue> results = await client.ExistsManyKeyValues(requests);

All batch operations complete in a single round trip to the server.

4.10 Prefix Queries

Kahuna provides two ways to query keys by prefix. The difference between them is scope.

4.10.1 Single-Partition Prefix Query

GetByBucket returns all keys that share a prefix, within the partition that owns that prefix:

List<KahunaKeyValue> users = await client.GetByBucket(
    "users",
    KeyValueDurability.Persistent
);

foreach (KahunaKeyValue kv in users)
    Console.WriteLine($"{kv.Key} = {kv.ValueAsString()}");

Because keys with the same prefix hash to the same partition, this query contacts only one partition. It is efficient, but it only finds keys that share the exact prefix.

The result set is capped at 4,096 entries. For larger sets, use range scans (next section).

4.10.2 All-Node Prefix Scan

ScanAllByPrefix searches all nodes for keys that match a prefix:

List<KahunaKeyValue> allConfigs = await client.ScanAllByPrefix(
    "config",
    KeyValueDurability.Persistent
);

This is a broader operation. It contacts every node in the cluster. Use it when you need to find keys across partitions. Like GetByBucket, the result is capped at 4,096 entries.

4.11 Range Scans

For large data sets or fine-grained control over which keys to return, use range scans.

4.11.1 Paginated Range Query

GetByRange returns keys within a prefix, optionally bounded by start and end keys:

List<KahunaKeyValue> page = await client.GetByRange(
    prefix: "orders/2024",
    startKey: "orders/2024/00100",
    startInclusive: true,
    endKey: "orders/2024/00200",
    endInclusive: false,
    limit: 50
);

This returns up to 50 keys in the range [orders/2024/00100, orders/2024/00200) within the orders/2024 prefix. The startInclusive and endInclusive parameters control whether the boundary keys are included.

If you omit startKey and endKey, the query returns the first limit keys in the prefix.

4.11.2 Streaming Range Scan

ScanByRange returns an IAsyncEnumerable<KahunaKeyValue> that pages through results automatically:

await foreach (KahunaKeyValue kv in client.ScanByRange(
    prefix: "logs/2024-03",
    pageSize: 100))
{
    Console.WriteLine($"{kv.Key}: {kv.ValueAsString()}");
}

The client fetches pages of 100 entries from the server. Your code iterates over them as a continuous stream. This is the right choice when you do not know how many keys exist and want to process them one at a time without loading all results into memory.

You can also bound the scan with start and end keys:

await foreach (KahunaKeyValue kv in client.ScanByRange(
    prefix: "events",
    startKey: "events/2024-03-01",
    endKey: "events/2024-03-31",
    endInclusive: true,
    pageSize: 200))
{
    ProcessEvent(kv);
}

4.12 Snapshot Reads

By default, every read returns the latest committed value. Snapshot reads let you read the state of the key-value store at a specific point in time.

The snapshotMs parameter accepts a Unix epoch timestamp in milliseconds. When set, the read returns the value that was current at that moment.

// Write a value.
KahunaKeyValue v1 = await client.SetKeyValue("price/widget", "10.00");
long snapshotTime = v1.LastModified;

// Write a new value.
await client.SetKeyValue("price/widget", "12.00");

// Read the current value.
KahunaKeyValue current = await client.GetKeyValue("price/widget");
Console.WriteLine(current.ValueAsString());  // "12.00"

// Read the value at the snapshot time.
KahunaKeyValue snapshot = await client.GetKeyValue("price/widget", snapshotMs: snapshotTime);
Console.WriteLine(snapshot.ValueAsString());  // "10.00"

The LastModified property on a KahunaKeyValue is a Unix epoch timestamp in milliseconds. You can use it as a snapshot anchor: read a key, save its LastModified, and later re-read at that exact point.

Snapshot reads also work with GetByBucket, ScanAllByPrefix, GetByRange, and ScanByRange. This gives you a consistent view across multiple keys at the same point in time.

// Read all user keys as they were at a specific moment.
List<KahunaKeyValue> usersAtSnapshot = await client.GetByBucket(
    "users",
    KeyValueDurability.Persistent,
    snapshotMs: snapshotTime
);

Snapshot reads depend on Kahuna’s MVCC (multi-version concurrency control) layer. Chapter 19 covers the internals of how snapshots work.

4.13 The KahunaKeyValue Object

Every key-value operation returns a KahunaKeyValue object. Here is a summary of its properties and methods:

The Extend and Delete methods operate on the same key and durability level that produced the result. They are convenience methods so you can chain operations without repeating the key name.

KahunaKeyValue session = await client.GetKeyValue("session/user-42");
if (session.Success)
{
    // Extend the session by 30 more minutes.
    await session.Extend(TimeSpan.FromMinutes(30));
}

4.14 CLI Reference

The command-line client supports all key-value operations. Here is a quick reference:

To connect to a specific node, use -c:

kahuna-cli -c "https://localhost:8082" --insecure --get "users/alice"

4.15 Putting It Together

The following example demonstrates several key-value patterns in a single program:

using Kahuna.Client;
using Kahuna.Shared.KeyValue;

var client = new KahunaClient("http://localhost:2070");

// 1. Create a key only if it doesn't exist (one-time initialization).
KahunaKeyValue init = await client.SetKeyValue(
    "app/version", "1.0.0", flags: KeyValueFlags.SetIfNotExists);
Console.WriteLine($"Init: success={init.Success}, revision={init.Revision}");

// 2. Read, modify, write with optimistic concurrency.
KahunaKeyValue config = await client.GetKeyValue("app/version");
KahunaKeyValue updated = await client.TryCompareRevisionAndSetKeyValue(
    "app/version", "1.1.0", compareRevision: config.Revision);

if (updated.Success)
    Console.WriteLine($"Updated to {updated.Revision}");
else
    Console.WriteLine("Conflict detected. Retry needed.");

// 3. State machine transition via compare-value-and-set.
await client.SetKeyValue("order/99/status", "pending");

KahunaKeyValue transition = await client.TryCompareValueAndSetKeyValue(
    "order/99/status", value: "confirmed", compareValue: "pending");
Console.WriteLine($"Transition: {transition.Success}");

// 4. Batch write with TTL.
var items = new List<KahunaSetKeyValueRequestItem>
{
    new() { Key = "cache/result-a", Value = "42"u8.ToArray(), ExpiresMs = 30000 },
    new() { Key = "cache/result-b", Value = "99"u8.ToArray(), ExpiresMs = 30000 }
};
await client.SetManyKeyValues(items);

// 5. Prefix query.
List<KahunaKeyValue> cacheEntries = await client.GetByBucket(
    "cache", KeyValueDurability.Persistent);
Console.WriteLine($"Cache entries: {cacheEntries.Count}");

// 6. Snapshot read.
KahunaKeyValue v1 = await client.SetKeyValue("price/gold", "2000");
long anchor = v1.LastModified;

await client.SetKeyValue("price/gold", "2050");

KahunaKeyValue latest = await client.GetKeyValue("price/gold");
KahunaKeyValue past = await client.GetKeyValue("price/gold", snapshotMs: anchor);

Console.WriteLine($"Latest: {latest.ValueAsString()}");   // 2050
Console.WriteLine($"At snapshot: {past.ValueAsString()}"); // 2000

4.16 Summary

Kahuna’s key-value API goes beyond simple get and set. Conditional writes (SetIfNotExists, SetIfExists, CRAS, CVAS) give you optimistic concurrency control and state machine transitions without external locks. Revisions provide a per-key version history. TTL lets keys expire automatically. Prefix queries and range scans let you retrieve groups of related keys efficiently. Snapshot reads give you a consistent view of the store at a past point in time. Batch operations reduce round trips when you work with many keys at once.

The next chapter covers distributed locks: lease semantics, fencing tokens, lock contention, and what happens when a lock holder crashes.

5 Distributed Locking

On a single machine, a mutex is enough. A thread acquires the mutex, does its work, and releases it. The operating system guarantees that only one thread holds the mutex at a time.

In a distributed system, there is no shared memory and no single operating system. Two processes on different machines can both believe they hold “the lock” if the mechanism that coordinates them fails. This chapter explains how Kahuna’s distributed locks work, why leases and fencing tokens are necessary, and how to use them correctly.

5.1 Why Locks Need Leases

A naive distributed lock works like this: write your name into a key. If the key is empty, you hold the lock. Delete the key when you are done.

The problem is “when you are done.” What if the lock holder crashes before it deletes the key? The lock is held forever. No other process can acquire it.

The solution is a lease: a lock with an expiration time. If the holder does not release the lock before the lease expires, the lock becomes available to other clients. This guarantees progress. A crashed process cannot block the system indefinitely.

Kahuna’s locks are lease-based. Every lock acquisition requires an expiry time. When the expiry elapses, the lock is automatically released.

5.2 Acquiring a Lock

The simplest lock acquisition takes a resource name and an expiry:

await using KahunaLock lockHandle = await client.GetOrCreateLock(
    resource: "jobs/send-invoice",
    expiry: TimeSpan.FromSeconds(30),
    durability: LockDurability.Persistent
);

if (lockHandle.IsAcquired)
{
    Console.WriteLine($"Lock acquired. Fencing token: {lockHandle.FencingToken}");
    // Do the protected work here.
}
else
{
    Console.WriteLine("Lock is held by another process.");
}

GetOrCreateLock returns a KahunaLock object. If the lock is already held by another owner, IsAcquired is false. The call does not block or wait. It tries once and returns immediately.

The lock implements IAsyncDisposable. When the await using block ends, the lock is released automatically. If the process crashes before the block ends, the lease expires and the lock becomes available.

5.2.1 Lock Parameters

5.2.2 Lock Owner

Every lock acquisition generates a unique owner token (a GUID). This token identifies the holder. Only the owner that acquired the lock can release or extend it. You do not need to manage this token yourself; the KahunaLock object holds it internally.

You can read the owner for debugging:

if (lockHandle.IsAcquired)
    Console.WriteLine($"Owner: {lockHandle.OwnerAsString}");

5.3 Fencing Tokens

Leases solve the problem of a crashed lock holder. But they introduce a new problem: the stale lock holder.

5.3.1 The Stale Lock Holder Problem

Consider this scenario:

1. Process A acquires a lock with a 10-second lease.
2. Process A starts writing to a shared resource (a database, a file, an external API).
3. Process A experiences a long garbage collection pause, a network delay, or a slow disk. It does not crash, but it stops making progress for 15 seconds.
4. The lease expires after 10 seconds. Process A does not know this yet.
5. Process B acquires the lock. It starts writing to the same shared resource.
6. Process A resumes. It still believes it holds the lock. It writes to the shared resource, overwriting what process B just wrote.

Both processes believe they are the rightful lock holder. The data is now corrupted.

Extending the lease duration does not fix this. No matter how long the lease is, a sufficiently long pause can outlast it. The fundamental problem is that process A has no way to know its lease expired.

5.3.2 How Fencing Tokens Solve This

Kahuna assigns a fencing token to every lock acquisition. The fencing token is a monotonically increasing integer that is scoped to a specific resource. Each time the lock changes hands, the fencing token increases.

Here is the sequence with fencing tokens:

1. Process A acquires the lock. Fencing token: 5.
2. Process A pauses for 15 seconds. The lease expires.
3. Process B acquires the lock. Fencing token: 6.
4. Process A resumes and tries to write to the shared resource, passing fencing token 5.
5. The shared resource checks the fencing token. It has already seen token 6 from process B. It rejects token 5 because it is stale.

The downstream system (a database, an API, a message queue) must participate in fencing. It stores the highest fencing token it has seen and rejects any write with a lower token.

5.3.3 Fencing Token Behavior

• The first acquisition of a resource gets fencing token 0.
• Each new acquisition (by a different owner, or after expiry) increments the token by 1.
• Extending a lock does not change the fencing token.
• The same owner re-acquiring a lock before it expires gets the same fencing token back (no increment).
• Fencing tokens are per-resource. Locking jobs/a and jobs/b produce independent token sequences.

// First acquisition: token = 0
await using (KahunaLock first = await client.GetOrCreateLock("demo/fence", TimeSpan.FromSeconds(5)))
{
    Console.WriteLine(first.FencingToken);  // 0
}

// Second acquisition (after release): token = 1
await using (KahunaLock second = await client.GetOrCreateLock("demo/fence", TimeSpan.FromSeconds(5)))
{
    Console.WriteLine(second.FencingToken);  // 1
}

5.3.4 Using Fencing Tokens in Practice

Pass the fencing token to every downstream operation that the lock protects:

await using KahunaLock lockHandle = await client.GetOrCreateLock(
    "orders/process-batch",
    TimeSpan.FromSeconds(30)
);

if (lockHandle.IsAcquired)
{
    long token = lockHandle.FencingToken;

    // Pass the token to the database.
    await db.ExecuteAsync(
        "UPDATE orders SET status = 'processed' WHERE batch_id = @batch AND fence_token < @token",
        new { batch = batchId, token }
    );
}

The database query rejects the update if a higher fencing token has already written to this batch. This prevents a stale lock holder from overwriting newer data.

Not every system supports fencing natively. For systems that do not, you can store the fencing token as a column or field and check it in your application logic before writing.

5.4 Waiting for a Lock

By default, GetOrCreateLock tries once. If the lock is held, it returns immediately with IsAcquired = false. For workloads where you want to wait until the lock becomes available, pass wait and retry parameters:

await using KahunaLock lockHandle = await client.GetOrCreateLock(
    resource: "jobs/send-invoice",
    expiry: TimeSpan.FromSeconds(30),
    wait: TimeSpan.FromSeconds(10),
    retry: TimeSpan.FromMilliseconds(200)
);

This tries to acquire the lock. If it fails, it retries every 200 milliseconds (with a small random jitter of plus or minus 50 ms) until either the lock is acquired or 10 seconds elapse.

If the wait period expires without acquiring the lock, IsAcquired is false. The call does not throw an exception.

The retry parameter must be greater than zero when wait is greater than zero. Otherwise the client throws a KahunaException.

5.4.1 Contention with Multiple Clients

When multiple clients compete for the same lock using wait and retry, each client retries independently. Kahuna does not queue waiters or guarantee fairness. The first client to attempt acquisition after the lock is released wins. The jitter on the retry interval reduces the chance that all clients retry at exactly the same time.

// 10 workers competing for the same lock.
var tasks = Enumerable.Range(0, 10).Select(async i =>
{
    await using KahunaLock lk = await client.GetOrCreateLock(
        "shared/critical-section",
        expiry: TimeSpan.FromSeconds(5),
        wait: TimeSpan.FromSeconds(60),
        retry: TimeSpan.FromMilliseconds(100)
    );

    if (lk.IsAcquired)
    {
        Console.WriteLine($"Worker {i} acquired lock. Token: {lk.FencingToken}");
        await Task.Delay(500);  // Simulate work.
    }
});

await Task.WhenAll(tasks);

Each worker eventually acquires the lock because the lease is short (5 seconds) and the wait budget is long (60 seconds).

5.5 Extending a Lease

If your work takes longer than the original lease, extend it before it expires:

await using KahunaLock lockHandle = await client.GetOrCreateLock(
    "jobs/long-running",
    TimeSpan.FromSeconds(10)
);

if (lockHandle.IsAcquired)
{
    // Start the work.
    await ProcessFirstBatch();

    // Extend the lease by another 10 seconds.
    (bool extended, long token) = await lockHandle.TryExtend(TimeSpan.FromSeconds(10));

    if (extended)
    {
        Console.WriteLine($"Lease extended. Fencing token still: {token}");
        await ProcessSecondBatch();
    }
    else
    {
        Console.WriteLine("Extension failed. The lease may have expired.");
        // Stop work. Another process may now hold the lock.
    }
}

Key details about extension:

• TryExtend resets the expiration clock from the current time.
• Only the current owner can extend the lock. If the lease already expired and another process acquired it, TryExtend returns false.
• Extending does not change the fencing token. The token remains the same for the duration of a single ownership period.
• There is no automatic renewal. Your application must call TryExtend explicitly.

You can also extend a lock through the client directly, if you have the owner token:

(bool success, long token) = await client.TryExtendLock(
    "jobs/long-running",
    lockHandle.Owner,
    TimeSpan.FromSeconds(10),
    LockDurability.Persistent
);

5.6 What Happens When a Lock Holder Crashes

This is the scenario leases are designed for:

1. Process A acquires a lock with a 10-second lease.
2. Process A crashes (power failure, OOM kill, unhandled exception).
3. The DisposeAsync call never runs because the process is gone.
4. After 10 seconds, the lease expires.
5. Process B acquires the lock with a new, higher fencing token.

No manual intervention is required. The cluster does not need to detect the crash. The lease simply expires, and the lock becomes available.

This is why lease duration matters:

• Too short: The lock expires before the holder finishes its work. The holder must extend frequently, adding network round trips.
• Too long: After a crash, other processes wait for the lease to expire before they can make progress.

A good starting point is 10 to 30 seconds for most workloads. Adjust based on how long the protected work takes and how quickly you need recovery after a crash.

5.7 Unlocking

The await using pattern releases the lock automatically. You can also unlock manually:

KahunaLock lockHandle = await client.GetOrCreateLock(
    "jobs/send-email",
    TimeSpan.FromSeconds(30)
);

if (lockHandle.IsAcquired)
{
    await DoWork();
    bool released = await client.Unlock(
        "jobs/send-email",
        lockHandle.Owner,
        LockDurability.Persistent
    );
    Console.WriteLine($"Released: {released}");
}

Only the owner can unlock. If a different process tries to unlock with a different owner token, the call returns false.

If you do not unlock and do not dispose, the lock remains held until the lease expires. The KahunaLock finalizer logs a warning if the object is garbage-collected without being disposed.

5.8 Querying Lock State

You can inspect a lock’s current state without acquiring it:

KahunaLockInfo? info = await client.GetLockInfo(
    "jobs/send-email",
    LockDurability.Persistent
);

if (info != null)
{
    Console.WriteLine($"Owner: {info.Owner}");
    Console.WriteLine($"Fencing token: {info.FencingToken}");
    Console.WriteLine($"Expires: {info.Expires}");
}

If no lock exists for the resource, GetLockInfo returns null.

You can also query through a KahunaLock object, even if you did not acquire the lock:

KahunaLock lockHandle = await client.GetOrCreateLock("jobs/send-email", TimeSpan.FromSeconds(5));
KahunaLockInfo? info = await lockHandle.GetInfo();

Lock info is a diagnostic tool. Do not use it to make decisions about whether to proceed with work. Between the time you read the lock info and the time you act on it, the lock state can change. Use GetOrCreateLock for coordination, not GetLockInfo.

5.9 Ephemeral Locks

By default, locks are persistent: the acquisition goes through Raft consensus and is replicated to a majority of nodes. This provides strong guarantees but adds latency.

For workloads where speed matters more than durability (coordinating in-memory caches, local-only rate limiting), use ephemeral locks:

await using KahunaLock lockHandle = await client.GetOrCreateLock(
    "cache/rebuild",
    expiry: TimeSpan.FromSeconds(5),
    durability: LockDurability.Ephemeral
);

Ephemeral locks are stored in memory on the receiving node only. They are faster but do not survive a node restart. If the node crashes, the lock is gone immediately (no need to wait for lease expiry, but also no replication safety).

Use ephemeral locks only when losing the lock on a node failure is acceptable.

5.10 CLI Lock Operations

The command-line client supports lock operations:

Acquire a lock:

kahuna-cli --lock "jobs/send-email" --owner "worker-1" --expires 10000

Extend a lock:

kahuna-cli --extend-lock "jobs/send-email" --owner "worker-1" --expires 10000

Release a lock:

kahuna-cli --unlock "jobs/send-email" --owner "worker-1"

When using the CLI, you supply the owner name yourself. In the C# SDK, the owner is generated automatically as a GUID.

5.11 Common Patterns

5.11.1 Try-Once Pattern

Attempt the lock once. If it fails, skip the work or report the conflict:

await using KahunaLock lk = await client.GetOrCreateLock(
    "jobs/daily-report", TimeSpan.FromMinutes(5));

if (!lk.IsAcquired)
{
    Console.WriteLine("Another instance is already generating the report.");
    return;
}

await GenerateDailyReport();

This is appropriate when duplicate work is harmless (another instance is already doing the job) or when you have an external retry mechanism (a job scheduler that retries later).

5.11.2 Wait-and-Retry Pattern

Wait for the lock to become available:

await using KahunaLock lk = await client.GetOrCreateLock(
    "orders/checkout",
    expiry: TimeSpan.FromSeconds(15),
    wait: TimeSpan.FromSeconds(30),
    retry: TimeSpan.FromMilliseconds(100)
);

if (!lk.IsAcquired)
    throw new TimeoutException("Could not acquire checkout lock within 30 seconds.");

await ProcessCheckout();

5.11.3 Extend-While-Working Pattern

For work that may take longer than the initial lease:

await using KahunaLock lk = await client.GetOrCreateLock(
    "etl/load-customers", TimeSpan.FromSeconds(30));

if (!lk.IsAcquired) return;

foreach (var batch in customerBatches)
{
    await LoadBatch(batch);

    // Extend the lease before it expires.
    (bool ok, _) = await lk.TryExtend(TimeSpan.FromSeconds(30));
    if (!ok)
    {
        Console.WriteLine("Lost the lock. Stopping work.");
        return;
    }
}

Each iteration resets the lease clock. If the extension fails (because the lease expired between iterations), the code stops immediately to avoid conflicting with a new lock holder.

5.12 Summary

Kahuna’s distributed locks combine leases and fencing tokens to solve two problems: a crashed lock holder (the lease expires and frees the lock) and a stale lock holder (the fencing token lets downstream systems reject outdated writes). Locks can wait and retry when contended, extend their lease for long-running work, and be queried for diagnostic purposes. Ephemeral locks trade durability for speed when replication is not needed.

The next chapter covers the distributed sequencer: generating globally unique, monotonically increasing numbers across the cluster.

6 The Distributed Sequencer

Many applications need unique, monotonically increasing numbers: order IDs, invoice numbers, event sequence numbers, log offsets. On a single machine, an auto-incrementing counter solves the problem. In a distributed system, there is no single machine to own the counter. Two nodes incrementing their own local counters will produce duplicates.

Kahuna’s distributed sequencer generates globally unique, monotonically increasing numbers across the entire cluster. It does this without making every allocation wait for Raft consensus, thanks to a block-based allocation scheme that amortizes the cost of consensus over many values.

6.1 Creating a Sequence

A sequence is a named, persistent counter. Create one with a name:

KahunaSequence seq = await client.CreateSequence("order-ids");

The sequence starts at 0 with an increment of 1. You can customize the starting value, the increment, and an optional maximum:

KahunaSequence invoiceSeq = await client.CreateSequence(
    name: "invoice-numbers",
    initialValue: 1000,
    increment: 1,
    maxValue: 999999
);

This sequence starts allocating from 1001 (initialValue + increment) and stops at 999,999. If you try to allocate past the maximum, the client throws a KahunaException with a MaxValueExceeded response.

Creating a sequence that already exists throws a KahunaException with an AlreadyExists response. Use GetSequence to check whether a sequence exists before creating it.

6.2 Getting the Next Value

The simplest operation is requesting a single value:

long id = await client.NextSequenceValue("order-ids");
Console.WriteLine(id);  // 1

Each call returns the next value in the sequence. Values are unique across all clients and all nodes. Two clients calling NextSequenceValue concurrently on the same sequence will never receive the same number.

6.3 Reserving a Range

If you need many values at once, reserve a range:

KahunaSequenceRange range = await client.ReserveSequenceRange("order-ids", count: 100);
Console.WriteLine($"Start: {range.Start}, End: {range.End}, Count: {range.Count}");

The returned range contains count consecutive values. Your application can use them locally without making further calls to Kahuna:

KahunaSequenceRange range = await client.ReserveSequenceRange("order-ids", count: 50);

for (long id = range.Start; id <= range.End; id++)
{
    await InsertOrder(id);
}

Reserving a range is more efficient than calling NextSequenceValue in a loop because it requires only one round trip to the server.

6.4 How Block Allocation Works

Understanding block allocation explains why the sequencer is fast and what trade-offs it makes.

6.4.1 The Problem with Naive Allocation

A naive sequencer would store a counter in Raft and increment it on every request. Each request would require a Raft proposal, replication to a majority of nodes, and a durable write. At high throughput, this becomes a bottleneck: hundreds of Raft round trips per second for a single counter.

6.4.2 Block-Based Allocation

Kahuna uses a different approach. Instead of incrementing the counter by 1 on every request, the sequencer reserves a block of values in a single Raft operation.

Here is how it works:

1. The server stores a high-water mark for each sequence. The high-water mark is the highest value that has been reserved (not necessarily issued to clients).

2. When a client requests a value, the sequencer actor checks if it has any values left in its current block. If it does, it returns the next value from memory. No storage operation is needed.

3. When the current block is exhausted, the actor reserves a new block by bumping the high-water mark in the durable record. This is a single Raft compare-and-swap operation. The new block provides another batch of values to serve from memory.

The default block size is 1,000. This means one Raft commit is amortized over 1,000 values. The throughput of the sequencer is roughly 1,000 times higher than a naive per-value approach.

6.4.3 The Trade-Off: Gaps

Block allocation introduces the possibility of gaps in the sequence. If a node reserves a block of 1,000 values and only issues 50 before a restart or a leadership change, the remaining 950 values in that block are never issued. The next block starts from the new high-water mark.

For example:

1. The sequencer reserves block [1, 1000].
2. It issues values 1 through 50 to clients.
3. The node restarts.
4. The sequencer reserves a new block [1001, 2000].
5. Values 51 through 1000 are never issued.

This is the same trade-off that PostgreSQL makes with CACHE on sequences, or that SQL Server makes with sequence caching. The values are unique and monotonically increasing, but they are not contiguous.

6.4.4 When Gaps Are Not Acceptable

If your application requires gap-free numbering (some financial regulations require contiguous invoice numbers), set the block size to 1:

kahuna-server --sequencer-block-size 1

With a block size of 1, every allocation requires a Raft round trip. This is slower but produces no gaps. The right choice depends on your requirements.

6.4.5 Block Lease Revalidation

To bound stale-leader exposure, each block has a lease duration (default: 5 seconds). If a block sits in memory longer than the lease, the sequencer revalidates it against the durable record before serving more values. This prevents a stale leader from issuing values that conflict with values issued by a new leader.

6.5 Increment

The increment parameter controls the step size between values. With the default increment of 1, values go 1, 2, 3, 4. With an increment of 5:

KahunaSequence seq = await client.CreateSequence(
    "batch-ids", initialValue: 0, increment: 5);

long a = await client.NextSequenceValue("batch-ids");  // 5
long b = await client.NextSequenceValue("batch-ids");  // 10
long c = await client.NextSequenceValue("batch-ids");  // 15

The first value issued is initialValue + increment. Subsequent values increase by increment.

6.6 Maximum Value

Sequences can have an optional maximum. When the sequence reaches its maximum, further allocations fail:

KahunaSequence seq = await client.CreateSequence(
    "limited-ids", initialValue: 0, increment: 1, maxValue: 100);

// Allocate values until the sequence is exhausted.
for (int i = 0; i < 200; i++)
{
    try
    {
        long val = await client.NextSequenceValue("limited-ids");
        Console.WriteLine(val);
    }
    catch (KahunaException ex)
    {
        Console.WriteLine($"Sequence exhausted: {ex.Message}");
        break;
    }
}

When the maximum is reached, the client throws a KahunaException. The sequence does not wrap around.

If no maximum is set (the default), the sequence can grow until it reaches long.MaxValue.

6.7 Idempotent Allocation

In a distributed system, a client might send a request, experience a timeout, and retry. Without protection, the retry allocates new values. The client does not know whether the original request succeeded, so it may end up with two sets of values for the same logical operation.

Kahuna solves this with idempotency keys. Pass a key with your allocation request:

long id = await client.NextSequenceValue(
    "order-ids",
    idempotencyKey: "checkout-request-abc123"
);

If you retry with the same idempotency key, Kahuna returns the same value that was allocated on the first call. No new value is consumed. This is true even if a leadership change occurred between the first call and the retry, because the idempotent result is stored in the durable record.

Idempotent reserves work the same way:

KahunaSequenceRange range = await client.ReserveSequenceRange(
    "order-ids",
    count: 10,
    idempotencyKey: "batch-import-42"
);

Retrying with the same key and the same count returns the identical range. If you retry with the same key but a different count, Kahuna returns an InvalidInput error.

6.7.1 Retention Limits

Idempotency entries are retained for a limited time and number:

After an entry is evicted (by age or count), a retry with that key allocates fresh values instead of replaying the original allocation. Set these values based on how long your retry windows last.

6.8 Querying a Sequence

Read the metadata of an existing sequence:

KahunaSequence? seq = await client.GetSequence("order-ids");

if (seq != null)
{
    Console.WriteLine($"Name: {seq.Name}");
    Console.WriteLine($"Current value: {seq.CurrentValue}");
    Console.WriteLine($"Initial value: {seq.InitialValue}");
    Console.WriteLine($"Increment: {seq.Increment}");
    Console.WriteLine($"Max value: {seq.MaxValue}");
    Console.WriteLine($"Revision: {seq.Revision}");
}

The CurrentValue property is the high-water mark: the highest value that has been reserved (by block allocation), not the last value issued to a client. It may be higher than the last value any client received.

If the sequence does not exist, GetSequence returns null.

6.9 Deleting a Sequence

bool deleted = await client.DeleteSequence("order-ids");

if (deleted)
    Console.WriteLine("Sequence deleted.");
else
    Console.WriteLine("Sequence not found.");

Deleting a sequence removes its durable record. If you create a sequence with the same name afterward, it starts fresh from its initial value.

6.10 CLI Commands

6.11 Server Configuration

These server-side settings control sequencer behavior:

6.12 Use Cases

6.12.1 Order Numbers

await client.CreateSequence("orders", initialValue: 10000);

// In the order placement handler:
long orderId = await client.NextSequenceValue(
    "orders",
    idempotencyKey: $"place-order-{requestId}"
);
await SaveOrder(orderId, orderDetails);

The idempotency key ensures that a retried placement does not consume a second order number.

6.12.2 Event Offsets

await client.CreateSequence("event-log-offset");

// When appending an event:
KahunaSequenceRange batch = await client.ReserveSequenceRange("event-log-offset", count: events.Count);

long offset = batch.Start;
foreach (var evt in events)
{
    evt.Offset = offset++;
    await AppendEvent(evt);
}

Reserving a range gives each event in the batch a unique, contiguous offset within that batch.

6.12.3 Sharded ID Generation

Multiple sequences can generate IDs for different shards:

// One sequence per shard, with non-overlapping ranges via increment and initial value.
await client.CreateSequence("ids-shard-0", initialValue: 0, increment: 4);
await client.CreateSequence("ids-shard-1", initialValue: 1, increment: 4);
await client.CreateSequence("ids-shard-2", initialValue: 2, increment: 4);
await client.CreateSequence("ids-shard-3", initialValue: 3, increment: 4);

// Shard 0 produces: 4, 8, 12, 16, ...
// Shard 1 produces: 5, 9, 13, 17, ...
// Shard 2 produces: 6, 10, 14, 18, ...
// Shard 3 produces: 7, 11, 15, 19, ...

Each shard generates IDs independently. The increment of 4 and staggered initial values ensure the ranges never overlap.

6.13 Guarantees and Non-Guarantees

The sequencer provides these guarantees:

• Uniqueness. A value is issued at most once for the lifetime of the sequence, across all clients, nodes, and leadership changes.
• Monotonic increase. Values always climb. A later allocation returns a higher number than an earlier one (within the same ownership period).
• Contiguous within a reserve. A ReserveSequenceRange(count: N) call returns exactly N consecutive values.

The sequencer does not guarantee:

• No gaps. Block allocation can leave gaps on restart, eviction, or leadership change. Set --sequencer-block-size 1 for gap-free numbering at the cost of throughput.
• Strict ordering across leadership changes. During a leadership transition, the new leader starts from the next block boundary. This preserves uniqueness and monotonicity but may skip values.

6.14 Summary

Kahuna’s distributed sequencer generates unique, monotonically increasing numbers across the cluster. Block-based allocation amortizes Raft consensus over many values, trading contiguity for throughput. Idempotency keys protect against duplicate allocation on retries. Sequences support custom increments, maximum values, and range reservations for batch workloads.

The next chapter covers transactions: how to read and write multiple keys atomically, with snapshot isolation and conflict detection.

7 Transactions

The previous chapters covered operations on individual keys, locks, and sequences. Each operation targets a single resource and completes in one round trip. Many real workloads need more than that. Transferring a balance between two accounts requires reading both keys, checking constraints, and writing both keys as a single atomic unit. If the process crashes after writing one key but before writing the other, the data is inconsistent.

Kahuna supports multi-key transactions with snapshot isolation. A transaction groups multiple reads and writes into a unit that either commits entirely or rolls back entirely. No partial results are visible to other clients.

Kahuna offers two transaction models:

• Script transactions send a self-contained script to the server. The server parses, plans, and executes the entire script as one atomic unit. This is the simpler model and the better choice when the logic can be expressed as a short sequence of key-value operations.

• Interactive transactions open a session between the client and the server. The client sends operations one at a time within the session, reads results, makes decisions in application code, and then commits or rolls back. This model is necessary when the transaction logic depends on intermediate results that only the application can evaluate.

Both models use two-phase commit (2PC) internally when the transaction touches keys on multiple partitions.

In both models, the server is the transaction coordinator. The client sends operations and a commit request, but the server drives the 2PC protocol and applies the writes. If the client crashes after it sends the commit request, the server still completes the transaction. The commit decision does not depend on the client remaining connected. This makes Kahuna transactions resilient to client failures at the critical moment.

7.1 Script Transactions

A script transaction is a string of key-value commands that the server executes atomically. If any command fails, the entire transaction is aborted and no changes are applied.

7.1.1 A Simple Script

string script = @"
    SET accounts/alice '950'
    SET accounts/bob '1050'
";

KahunaKeyValueTransactionResult result =
    await client.ExecuteKeyValueTransactionScript(script);

Console.WriteLine(result.Type);  // Set

This script writes two keys in a single atomic operation. Both writes succeed together or fail together. The Type property on the result indicates the outcome of the last command in the script.

7.1.2 Parameters

Hard-coding values into a script string is inconvenient and error-prone. Use parameters to pass values at runtime:

string script = "SET @key @value";

var parameters = new List<KeyValueParameter>
{
    new() { Key = "@key", Value = "accounts/alice" },
    new() { Key = "@value", Value = "950" }
};

KahunaKeyValueTransactionResult result =
    await client.ExecuteKeyValueTransactionScript(script, parameters: parameters);

A parameter is a placeholder that starts with @. At execution time, Kahuna replaces each placeholder with the corresponding value from the parameter list.

7.1.3 The Result Object

ExecuteKeyValueTransactionScript returns a KahunaKeyValueTransactionResult. This object contains the outcome of the script:

When a script contains one or more GET commands, the returned values appear in the Values list in order. Each entry carries the key, value, revision, expiration time, and last modification timestamp.

string script = @"
    SET users/1 'Alice'
    SET users/2 'Bob'
    GET users/1
";

KahunaKeyValueTransactionResult result =
    await client.ExecuteKeyValueTransactionScript(script);

Console.WriteLine(result.FirstValueAsString);  // Alice

7.1.4 Pre-Hashing for Reuse

Every time you call ExecuteKeyValueTransactionScript with a string script, the server must parse it. If you execute the same script many times (with different parameters), you can avoid repeated parsing by loading the script once:

KahunaTransactionScript transferScript = client.LoadTransactionScript(
    "SET @from @fromBalance SET @to @toBalance"
);

LoadTransactionScript computes a Blake3 hash of the script text. On the first execution, the server parses the script and caches the result keyed by this hash. On subsequent executions, the server finds the cached plan and skips parsing.

Execute the pre-hashed script with Run:

KahunaKeyValueTransactionResult result = await transferScript.Run(
    parameters: new()
    {
        new() { Key = "@from", Value = "accounts/alice" },
        new() { Key = "@fromBalance", Value = "900" },
        new() { Key = "@to", Value = "accounts/bob" },
        new() { Key = "@toBalance", Value = "1100" }
    }
);

7.1.5 Script Priority

By default, script transactions run at Normal priority. When the server is at its concurrency ceiling, you can influence which transactions start first:

KahunaKeyValueTransactionResult result = await transferScript.Run(
    priority: TransactionPriority.High,
    parameters: new()
    {
        new() { Key = "@key", Value = "config/critical" },
        new() { Key = "@value", Value = "updated" }
    }
);

Priority levels, from lowest to highest: Background, Low, Normal, High, Critical. Priority only affects admission order when the server is saturated. Below the concurrency ceiling, all transactions start immediately regardless of priority.

The next chapter covers the full script language in detail: variables, conditionals, loops, built-in functions, and multi-statement control flow.

7.2 Interactive Transactions

When the transaction logic requires reading a value from the server, making a decision in your application, and then writing based on that decision, use an interactive transaction session.

7.2.1 Starting a Session

await using KahunaTransactionSession session = await client.StartTransactionSession(
    new KahunaTransactionOptions
    {
        Timeout = 5000,
        Locking = KeyValueTransactionLocking.Pessimistic
    }
);

StartTransactionSession opens a session with the server. The session receives a unique transaction ID and is pinned to a coordinator node for its lifetime. All operations within the session are part of the same transaction.

The session implements IAsyncDisposable. When the await using block ends:

• If the session is still in Pending status (no explicit commit or rollback), it rolls back automatically.
• If the session was committed or rolled back, disposal is a no-op.

7.2.2 Session Operations

Inside a session, you can perform the same key-value operations as the main KahunaClient, but they execute within the transaction scope:

await using KahunaTransactionSession session = await client.StartTransactionSession(
    new KahunaTransactionOptions { Timeout = 5000 }
);

KahunaKeyValue alice = await session.GetKeyValue("accounts/alice");
KahunaKeyValue bob = await session.GetKeyValue("accounts/bob");

long aliceBalance = long.Parse(alice.ValueAsString ?? "0");
long bobBalance = long.Parse(bob.ValueAsString ?? "0");

if (aliceBalance >= 100)
{
    await session.SetKeyValue("accounts/alice", (aliceBalance - 100).ToString());
    await session.SetKeyValue("accounts/bob", (bobBalance + 100).ToString());
    await session.Commit();
}
else
{
    await session.Rollback();
}

This transaction reads two balances, checks a constraint in application code, and writes updated balances. The reads and writes are part of a single atomic unit. If another client modifies either account between the reads and the commit, the transaction may be aborted (depending on the locking mode and read validation settings).

The session supports these operations:

All operations check that the session is still in Pending status. If you try to operate on a committed, rolled back, or aborted session, the client throws a KahunaException.

7.2.3 Committing

Call Commit to finalize the transaction:

bool committed = await session.Commit();

The commit drives a two-phase commit protocol across all partitions that the transaction touched. If the commit succeeds, committed is true and the session status moves to Committed. All writes become visible to other clients.

If a conflict is detected (another transaction modified a key that this transaction read or wrote), the commit throws a KahunaException with an Aborted error code. The session moves to the Aborted status. This is terminal: you cannot retry a commit on an aborted session. Start a new session instead.

If a transient error occurs (network timeout, temporary leader unavailability), the session returns to Pending status. You can retry the commit.

7.2.4 Rolling Back

Call Rollback to discard all changes:

await session.Rollback();

After a rollback, the session status is Rolledback. All acquired locks are released and no writes from this session are applied.

If you do not call either Commit or Rollback, the await using block calls Rollback automatically when the session is disposed.

7.2.5 Transaction Status

A session moves through these states:

7.3 Locking Modes

Kahuna interactive sessions support two locking strategies: pessimistic (the default) and optimistic.

7.3.1 Pessimistic Locking

new KahunaTransactionOptions
{
    Locking = KeyValueTransactionLocking.Pessimistic
}

In pessimistic mode, the session acquires an exclusive lock on every key it reads or writes. This means:

• GetKeyValue("k") acquires an exclusive lock on k before reading.
• SetKeyValue("k", "v") acquires an exclusive lock on k before writing.
• GetByBucket("prefix/") acquires an exclusive prefix lock before scanning.
• GetByRange(...) acquires an exclusive range lock before scanning.

No other transaction can read or write a locked key until this transaction commits or rolls back. This prevents conflicts at the cost of concurrency: if two transactions touch the same key, one waits (or aborts) while the other holds the lock.

Pessimistic locking is the safer default. Use it when conflicts are common or when you cannot tolerate aborted transactions.

7.3.2 Optimistic Locking

new KahunaTransactionOptions
{
    Locking = KeyValueTransactionLocking.Optimistic
}

In optimistic mode, the session acquires exclusive locks only on writes, not on reads:

• GetKeyValue("k") reads without locking.
• SetKeyValue("k", "v") acquires an exclusive lock on k before writing.

This allows higher concurrency for read-heavy workloads. Multiple transactions can read the same keys simultaneously. Conflicts are detected at commit time: if a key that this transaction read was modified by another transaction after the read, the commit is rejected.

Optimistic locking works best when conflicts are rare. If conflicts are frequent, transactions abort and retry repeatedly, which wastes work.

7.4 Read Validation and Write-Skew Detection

By default (ReadValidation.None), the commit does not check whether keys that the transaction read were modified by other transactions. This is efficient but allows a class of anomaly called write-skew.

7.4.1 The Write-Skew Problem

Consider two doctors on call. A business rule says at least one doctor must remain on call. Both doctors check the on-call roster, see the other is on call, and each removes themselves. Both transactions commit because neither wrote to the same key. The result: zero doctors on call.

7.4.2 Enabling Read Validation

new KahunaTransactionOptions
{
    Locking = KeyValueTransactionLocking.Optimistic,
    ReadValidation = ReadValidation.TrackAndValidate
}

With TrackAndValidate, the session records every key it reads (the read set). At commit time, the server checks whether any key in the read set was modified after this transaction read it. If so, the commit is aborted. This prevents write-skew at the cost of additional validation work during commit.

TrackAndValidate is most useful with optimistic locking. Pessimistic locking already prevents concurrent modifications through exclusive locks, so read validation adds little value in that mode.

7.5 RetryableTransaction

Many applications follow the same pattern: start a session, do work, commit, and retry on conflict. Kahuna provides a helper that encapsulates this pattern:

await client.RetryableTransaction(
    new KahunaTransactionOptions
    {
        Timeout = 5000,
        Locking = KeyValueTransactionLocking.Pessimistic
    },
    async (session, ct) =>
    {
        KahunaKeyValue counter = await session.GetKeyValue("stats/visits");
        long count = long.Parse(counter.ValueAsString ?? "0");
        await session.SetKeyValue("stats/visits", (count + 1).ToString());
        await session.Commit(ct);
    }
);

RetryableTransaction starts a session, runs your callback, and handles retry logic automatically. If the callback throws a KahunaException with an Aborted, MustRetry, or AlreadyLocked error code, the method waits and retries with a new session.

The retry strategy uses decorrelated jitter backoff:

• Up to 10 retries.
• The first retry delay has a median of 50 milliseconds.
• Each subsequent delay grows with jitter but is capped at 500 milliseconds.
• If all 10 retries fail, the method throws a KahunaException with an Aborted error code.

Any exception that is not Aborted, MustRetry, or AlreadyLocked propagates immediately without retry. This includes application logic errors, cancellation, and permanent failures.

7.6 Transaction Options Reference

7.6.1 Decision Durability

The DecisionDurability option controls how the server handles the 2PC decision record:

• BestEffort (default): The server tells the client the outcome as soon as the two-phase commit protocol completes in memory. The decision is persisted asynchronously. This is faster, but in a narrow crash window the decision could be lost and the transaction outcome becomes unknown.

• Durable: The server persists the decision record through Raft consensus before returning the outcome to the client. This adds latency but guarantees that the outcome survives a crash.

For most workloads, BestEffort is sufficient. Use Durable when you need an external guarantee that a committed transaction will remain committed even if the coordinator node crashes immediately after the commit response.

7.6.2 Snapshot Reads

The ReadTimestamp option pins all reads in the session to a specific point in time. This is useful for reporting queries that must see a consistent snapshot while other transactions continue to write:

await using KahunaTransactionSession session = await client.StartTransactionSession(
    new KahunaTransactionOptions
    {
        Timeout = 10000,
        Locking = KeyValueTransactionLocking.Optimistic,
        ReadTimestamp = snapshotTimestamp
    }
);

KahunaKeyValue balance = await session.GetKeyValue("accounts/alice");

When ReadTimestamp is non-zero, the server serves the value as of that timestamp without recording a read dependency. This means the read does not participate in conflict detection, because it reads historical data that cannot be changed.

7.7 Common Patterns

7.7.1 Read-Modify-Write

The most common transaction pattern: read a value, compute a new value, and write it back.

await client.RetryableTransaction(
    new KahunaTransactionOptions { Timeout = 5000 },
    async (session, ct) =>
    {
        KahunaKeyValue item = await session.GetKeyValue("inventory/widget-a");
        int stock = int.Parse(item.ValueAsString ?? "0");

        if (stock < 1)
            throw new InvalidOperationException("Out of stock.");

        await session.SetKeyValue("inventory/widget-a", (stock - 1).ToString());
        await session.Commit(ct);
    }
);

Wrapping this in RetryableTransaction ensures that if another client modifies the same key concurrently, the transaction retries with fresh data.

7.7.2 Multi-Key Transfer

Move a value from one key to another atomically:

await client.RetryableTransaction(
    new KahunaTransactionOptions { Timeout = 5000 },
    async (session, ct) =>
    {
        KahunaKeyValue from = await session.GetKeyValue("accounts/alice");
        KahunaKeyValue to = await session.GetKeyValue("accounts/bob");

        long fromBalance = long.Parse(from.ValueAsString ?? "0");
        long toBalance = long.Parse(to.ValueAsString ?? "0");
        long amount = 100;

        if (fromBalance < amount)
            throw new InvalidOperationException("Insufficient funds.");

        await session.SetKeyValue("accounts/alice", (fromBalance - amount).ToString());
        await session.SetKeyValue("accounts/bob", (toBalance + amount).ToString());
        await session.Commit(ct);
    }
);

If accounts/alice and accounts/bob live on different partitions, Kahuna uses 2PC to commit both writes atomically.

7.7.3 Conditional Insert

Insert a key only if it does not already exist, as part of a larger transaction:

await using KahunaTransactionSession session = await client.StartTransactionSession(
    new KahunaTransactionOptions { Timeout = 5000 }
);

KahunaKeyValue existing = await session.GetKeyValue("users/alice@example.com");

if (existing.Success)
{
    Console.WriteLine("User already exists.");
    await session.Rollback();
}
else
{
    await session.SetKeyValue("users/alice@example.com", "Alice");
    await session.SetKeyValue("user-count", "1");
    await session.Commit();
}

7.8 Summary

Kahuna provides two transaction models. Script transactions send a self-contained script to the server for atomic execution. Interactive transactions open a session and let the client make decisions based on intermediate results. Both models use 2PC for multi-partition atomicity.

Pessimistic locking (the default) acquires exclusive locks on every read and write, preventing conflicts at the cost of concurrency. Optimistic locking acquires locks only on writes, allowing higher concurrency but detecting conflicts at commit time. Read validation with TrackAndValidate prevents write-skew anomalies by checking the read set at commit time.

RetryableTransaction wraps the common start, work, commit, retry loop with decorrelated jitter backoff.

The next chapter covers the transaction script language in full: variables, conditionals, loops, built-in functions, and error handling.

8 The Script Language

Chapter 6 introduced script transactions: self-contained programs that the server executes atomically. This chapter is the complete reference for the script language. It covers the syntax, data types, commands, control flow, expressions, built-in functions, and transaction options.

8.1 Language Overview

The Kahuna script language is a small, domain-specific language designed for one purpose: reading and writing key-value data inside a transaction. It is not a general-purpose programming language. It has no file I/O, no networking, no user-defined functions, and no classes. What it does have is direct access to Kahuna’s key-value operations, conditional logic, loops, and built-in functions for type checking and data manipulation.

Scripts are case-insensitive. SET, set, and Set are the same command. String literals use single quotes ('hello') or double quotes ("hello"). Identifiers that collide with reserved words can be escaped with backticks (`delete`).

A script is a sequence of statements. The server parses the script into an abstract syntax tree, executes the statements in order, and returns the result of the last statement (or the value passed to RETURN).

8.2 Data Types

The language supports five data types:

Arrays are created with the range operator (1..10) or returned by commands like GET BY BUCKET and SCAN BY PREFIX. You cannot construct an array literal directly.

8.3 Variables

Use LET to assign a value to a variable:

LET x = 42
LET name = 'Alice'
LET total = x + 10
LET found = true

Variables are dynamically typed. You can reassign a variable to a different type:

LET x = 42
LET x = 'now a string'

Variable names follow identifier rules: letters, digits, and underscores, starting with a letter or underscore.

8.4 Key-Value Commands

8.4.1 SET

SET writes a value to a key:

SET mykey 'hello world'

The key can be an identifier, a string literal, or a placeholder:

SET 'users/alice' 'active'
SET @key @value

8.4.1.1 SET Flags

Flags modify the behavior of SET. You can combine multiple flags on one command.

NX (Not Exists): Write only if the key does not exist yet.

SET mykey 'first' NX

If mykey already exists, the SET does not modify it.

XX (Exists): Write only if the key already exists.

SET mykey 'updated' XX

If mykey does not exist, the SET does nothing.

CMP (Compare Value): Write only if the current value equals the given expression.

SET counter 'closed' CMP 'open'

This sets counter to 'closed' only if its current value is 'open'.

CMPREV (Compare Revision): Write only if the current revision equals the given number.

SET config 'new-value' CMPREV 5

This sets config only if its revision is exactly 5. This is useful for optimistic concurrency: read the revision, do your work, then write only if nobody else changed the key.

EX (Expires): Set a TTL in milliseconds.

SET session 'token-abc' EX 30000

The key expires and is deleted after 30 seconds.

NOREV (No Revision): Do not track the revision for this write.

SET cache/item 'data' NOREV

This is useful for cache-like workloads where you do not need version tracking.

Combining flags:

SET mykey 'value' NX EX 60000

This creates the key only if it does not exist, with a 60-second TTL.

8.4.2 ESET

ESET is the ephemeral variant of SET. It works the same way but stores the value in memory only, without Raft replication. Ephemeral data does not survive a node restart.

ESET cache/user 'data' EX 5000

ESET supports all the same flags as SET: NX, XX, CMP, CMPREV, EX, NOREV.

8.4.3 GET

GET reads a value by key:

GET mykey

To capture the result in a variable, use LET:

LET value = GET mykey

8.4.3.1 Reading a Specific Revision

Read the value at a specific revision number:

LET old = GET mykey AT 3

This returns the value that mykey had at revision 3.

8.4.3.2 Snapshot Reads

Read the value as it was at a specific point in time (Unix timestamp in milliseconds):

LET snapshot = GET mykey AS OF 1700000000000

This returns the value that mykey had at the given timestamp.

8.4.4 EGET

EGET reads from ephemeral storage:

LET cached = EGET cache/user

EGET supports the same AT and AS OF variants as GET.

8.4.5 EXISTS

Check whether a key exists:

LET found = EXISTS mykey

EXISTS also supports AT (revision) and AS OF (timestamp) variants.

8.4.6 EEXISTS

Check whether an ephemeral key exists:

LET found = EEXISTS cache/user

8.4.7 DELETE

Delete a key:

DELETE mykey

8.4.8 EDELETE

Delete an ephemeral key:

EDELETE cache/user

8.4.9 EXTEND

Extend the TTL of a key by a given number of milliseconds:

EXTEND mykey 30000

This resets the expiry clock. The key will live for another 30 seconds from now.

8.4.10 EEXTEND

Extend the TTL of an ephemeral key:

EEXTEND cache/user 5000

8.4.11 GET BY BUCKET

Retrieve all keys that share a common prefix (bucket). The prefix is everything up to and including the last / in the key name:

LET items = GET BY BUCKET 'users/'

This returns an array of key-value results for all keys whose names start with users/. You can iterate over the results with a FOR loop.

GET BY BUCKET supports AS OF for snapshot reads:

LET items = GET BY BUCKET 'users/' AS OF 1700000000000

8.4.12 EGET BY BUCKET

The ephemeral variant:

LET items = EGET BY BUCKET 'cache/'

8.4.13 SCAN BY PREFIX

Scan all keys that match a given prefix:

LET results = SCAN BY PREFIX 'config/'

Like GET BY BUCKET, this returns an array. It supports AS OF for snapshot reads.

8.4.14 ESCAN BY PREFIX

The ephemeral variant:

LET results = ESCAN BY PREFIX 'cache/config/'

8.5 Placeholders

Placeholders let you pass values into a script from the calling application. A placeholder starts with @:

SET @key @value EX @ttl

In C#, pass placeholders as a list of KeyValueParameter objects:

string script = "SET @key @value EX @ttl";

var result = await client.ExecuteKeyValueTransactionScript(
    script,
    parameters: [
        new() { Key = "@key", Value = "users/alice" },
        new() { Key = "@value", Value = "active" },
        new() { Key = "@ttl", Value = "30000" }
    ]
);

Placeholders prevent injection and let you reuse the same compiled script with different values. When you use LoadTransactionScript (covered in Chapter 6), the server caches the parsed AST and reuses it across calls. Only the placeholder values change.

8.6 Expressions

The language supports arithmetic, comparison, and logical expressions.

8.6.1 Arithmetic

LET a = 10 + 5      // 15
LET b = 10 - 3      // 7
LET c = 4 * 3       // 12
LET d = 10 / 2      // 5

Arithmetic works on integers and floats. Mixing types promotes the result to float.

8.6.2 Comparison

LET eq  = (x = 5)        // equality (single = or ==)
LET neq = (x != 5)       // not equal (or <>)
LET lt  = (x < 10)       // less than
LET gt  = (x > 0)        // greater than
LET lte = (x <= 100)     // less than or equal
LET gte = (x >= 1)       // greater than or equal

Both = and == test equality. Both != and <> test inequality.

8.6.3 Logical Operators

LET both = (a > 0) && (b > 0)   // AND
LET either = (a > 0) || (b > 0) // OR
LET no = !(a > 0)               // NOT

You can also use the word forms: AND, OR, NOT.

LET both = (a > 0) AND (b > 0)

8.6.4 Ranges

The .. operator creates an array of integers:

LET nums = 1..10

This creates an array [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].

8.6.5 Array Indexing

Access array elements by index (zero-based):

LET nums = 1..5
LET first = nums[0]    // 1
LET third = nums[2]    // 3

8.6.6 Special Expressions

NOT SET: Evaluates to a sentinel that indicates a key was never written. Use it to check whether a GET returned no data:

LET value = GET mykey
IF value = NOT SET THEN
    SET mykey 'default'
END

NOT FOUND: Similar to NOT SET, indicates the key was not found:

LET value = GET mykey
IF value = NOT FOUND THEN
    SET mykey 'initialized'
END

8.7 Control Flow

8.7.1 IF / THEN / ELSE / END

Conditional execution:

LET status = GET account/status

IF status = 'active' THEN
    SET account/last-login current_time()
END

With an else branch:

LET balance = GET account/balance

IF to_long(balance) >= 100 THEN
    SET account/balance to_string(to_long(balance) - 100)
ELSE
    THROW 'Insufficient balance'
END

Conditions can use any expression. Nested IF statements are allowed.

8.7.2 FOR / IN / DO / END

Loop over a range or an array:

FOR i IN 1..5 DO
    SET concat('key/', to_string(i)) to_string(i * 10)
END

This creates five keys: key/1 through key/5 with values 10 through 50.

You can loop over the results of a bucket query:

LET items = GET BY BUCKET 'users/'

FOR item IN items DO
    SET concat('backup/', item) item
END

8.8 Transactions in Scripts

8.8.1 Implicit Transactions

When a script contains multiple statements without a BEGIN block, the server wraps the entire script in an auto-commit transaction. If any statement fails, the entire script rolls back.

SET account/a '900'
SET account/b '1100'

Both writes succeed together or fail together.

8.8.2 Explicit Transactions

Use BEGIN and END to define a transaction block explicitly. Inside the block, you can use COMMIT or ROLLBACK:

BEGIN
    LET balance = GET account/a
    LET amount = 100

    IF to_long(balance) >= amount THEN
        SET account/a to_string(to_long(balance) - amount)

        LET target = GET account/b
        SET account/b to_string(to_long(target) + amount)
        COMMIT
    ELSE
        ROLLBACK
    END
END

If you call ROLLBACK, all writes inside the BEGIN block are discarded. If you call COMMIT, the writes become durable. If you reach END without calling either, the transaction commits automatically.

8.8.3 Transaction Options

BEGIN accepts options in parentheses:

BEGIN (locking = optimistic, timeout = 5000)
    LET a = GET account/a
    LET b = GET account/b
    SET account/a to_string(to_long(a) - 50)
    SET account/b to_string(to_long(b) + 50)
END

Available options:

8.9 RETURN

RETURN ends the script and returns a value to the caller:

LET x = GET mykey

IF x = NOT SET THEN
    RETURN 'not found'
END

RETURN concat('found: ', x)

Without a RETURN, the script returns the result of the last executed statement.

RETURN without a value ends the script immediately:

LET x = GET mykey
IF x = NOT SET THEN
    RETURN
END
SET mykey to_string(to_long(x) + 1)

8.10 SLEEP

SLEEP pauses execution for a given number of milliseconds:

SLEEP 1000

Use SLEEP with caution. It holds the transaction open during the pause. The primary use case is testing and debugging, not production workloads.

8.11 THROW

THROW raises an error and aborts the script:

LET balance = GET account/balance

IF to_long(balance) < 0 THEN
    THROW 'Balance cannot be negative'
END

The error message is returned to the caller in the transaction result’s Reason field.

8.12 Built-in Functions

The language provides built-in functions for type checking, type conversion, string manipulation, math, and metadata access.

8.12.1 Type Checking Functions

These functions test the type of a value and return a boolean:

LET x = 42
RETURN is_long(x)    // true

LET y = 'hello'
RETURN is_string(y)  // true

LET z = 1..10
RETURN is_array(z)   // true

8.12.2 Type Conversion Functions

These functions convert a value from one type to another:

Values stored in Kahuna are byte arrays. When you read a value with GET, the result is a string. To do arithmetic on it, convert it to an integer or float first:

LET raw = GET counter
LET n = to_long(raw)
SET counter to_string(n + 1)

Conversion rules:

• to_long(1.7) truncates to 1.
• to_long(true) returns 1. to_long(false) returns 0.
• to_float(true) returns 1. to_float(false) returns 0.
• to_bool(0) returns false. to_bool(1) returns true.
• to_string(null) returns an empty string.
• to_json works on integers, floats, strings, and arrays. It produces a JSON representation of the value.

8.12.3 String Functions

LET name = 'Alice'
RETURN upper(name)         // 'ALICE'
RETURN lower(name)         // 'alice'
RETURN concat('Hi, ', name) // 'Hi, Alice'
RETURN len(name)           // 5

concat takes exactly two string arguments. To concatenate more than two values, nest the calls:

LET full = concat(concat(first, ' '), last)

8.12.4 Math Functions

LET a = abs(-5)       // 5
LET b = pow(2, 10)    // 1024
LET c = round(3.7)    // 4
LET d = ceil(3.1)     // 4
LET e = floor(3.9)    // 3
LET f = min(10, 20)   // 10
LET g = max(10, 20)   // 20

8.12.5 Metadata Functions

LET value = GET mykey
LET r = revision(value)
LET e = expires(value)
LET now = current_time()

revision and expires are useful for conditional logic based on a key’s metadata:

LET value = GET mykey
IF revision(value) > 10 THEN
    SET mykey 'stable'
END

count works only on arrays:

LET items = GET BY BUCKET 'users/'
LET total = count(items)
RETURN to_string(total)

8.13 Complete Example: Balance Transfer

This example combines many features: variables, GET, SET with conditions, type conversion, explicit transactions, and error handling.

BEGIN (locking = pessimistic, timeout = 5000)
    LET source_balance = GET @source
    LET target_balance = GET @target

    IF source_balance = NOT SET THEN
        THROW concat('Source account not found: ', @source)
    END

    IF target_balance = NOT SET THEN
        THROW concat('Target account not found: ', @target)
    END

    LET amount = to_long(@amount)
    LET src = to_long(source_balance)
    LET tgt = to_long(target_balance)

    IF src < amount THEN
        THROW 'Insufficient balance'
    END

    SET @source to_string(src - amount)
    SET @target to_string(tgt + amount)
    COMMIT
END

Call this script from C#:

var result = await client.ExecuteKeyValueTransactionScript(
    script,
    parameters: [
        new() { Key = "@source", Value = "account/alice" },
        new() { Key = "@target", Value = "account/bob" },
        new() { Key = "@amount", Value = "500" }
    ]
);

The transaction reads both balances, checks the source has enough funds, updates both accounts, and commits. If any step fails or the source balance is too low, the script throws and no writes take effect.

8.14 Complete Example: Batch Initialization

This example uses a FOR loop to initialize a set of keys:

FOR i IN 0..9 DO
    LET key = concat('sensor/', to_string(i))
    LET exists_result = EXISTS key

    IF exists_result = NOT SET THEN
        SET key '0' EX 60000
    END
END

This creates keys sensor/0 through sensor/9, each with a 60-second TTL, but only if the key does not exist yet.

8.15 Complete Example: Snapshot Audit

Read the state of several keys as they were at a specific moment:

BEGIN (snapshot = @timestamp)
    LET a = GET account/a
    LET b = GET account/b
    LET c = GET account/c
    LET total = to_long(a) + to_long(b) + to_long(c)
    RETURN to_string(total)
END

The snapshot option ensures all three reads see a consistent view of the data at the given timestamp, even if the keys changed after that point.

8.16 Script Caching

When you send a script to the server, the server parses it into an AST (abstract syntax tree). Parsing is fast, but for scripts that run frequently, you can avoid the overhead by using LoadTransactionScript (covered in Chapter 6). The server hashes the script with Blake3, caches the AST, and reuses it on subsequent calls with the same hash. Placeholder values change between calls, but the structure stays cached.

The server evicts cached scripts when they have not been used within the cache TTL. The TTL is configured with --script-cache-expiration (default: 600 seconds).

8.17 Reserved Words

These words are reserved by the language. You cannot use them as bare key names or variable names without backtick escaping:

SET, GET, LET, IF, THEN, ELSE, END, FOR, DO, IN, BEGIN, COMMIT, ROLLBACK, RETURN, SLEEP, DELETE, DEL, EXTEND, EXISTS, ESET, EGET, EDELETE, EDEL, EEXTEND, EEXISTS, TRUE, FALSE, NULL, NX, XX, EX, CMP, CMPREV, NOREV, THROW, FOUND, NOT, AT, AS, OF, SCAN, ESCAN, BY, BUCKET, PREFIX, AND, OR

If your key name collides with a reserved word, use a string literal or backtick escaping:

SET 'delete' 'some value'
SET `delete` 'some value'

8.18 Operator Precedence

Operators follow this precedence, from lowest to highest:

Use parentheses to override the default order:

LET result = (a + b) * c

8.19 Summary

The Kahuna script language provides the building blocks for server-side transaction logic: key-value commands with conditional flags, variables, arithmetic and logical expressions, control flow with IF and FOR, explicit transaction blocks with configurable options, and built-in functions for type checking, conversion, string manipulation, and math. Placeholders separate data from logic, and script caching amortizes parsing cost.

Each persistent command (SET, GET, EXISTS, DELETE, EXTEND) has an ephemeral counterpart (ESET, EGET, EEXISTS, EDELETE, EEXTEND) for in-memory workloads. Bulk reads use GET BY BUCKET and SCAN BY PREFIX.

The next chapter covers real-world patterns: how to combine these primitives to build distributed coordination workflows, leader election, and idempotent processing.

9 Leader Election and Job Coordination

Many distributed applications need exactly one active process to do a specific job. A scheduler that sends reminder emails, a worker that processes a payment queue, a service that aggregates metrics: each of these must run on one node at a time. If two instances run simultaneously, they send duplicate emails, process payments twice, or produce incorrect aggregates.

On a single machine, this is simple. You run one process. If it crashes, a supervisor restarts it. In a distributed system, you run multiple instances for availability, and you need a way to decide which one is the active leader. The others stand by and take over if the leader fails.

This chapter shows how to build leader election and job coordination patterns with Kahuna’s locks, fencing tokens, and key-value operations.

9.1 The Simplest Leader Election

A distributed lock is a natural fit for leader election. The process that holds the lock is the leader. All other processes are followers. When the leader fails, the lock’s lease expires and a follower acquires it.

Here is the simplest version:

while (!stoppingToken.IsCancellationRequested)
{
    await using KahunaLock lockHandle = await client.GetOrCreateLock(
        resource: "leader/email-scheduler",
        expiry: TimeSpan.FromSeconds(15),
        wait: TimeSpan.FromSeconds(10),
        retry: TimeSpan.FromMilliseconds(500),
        durability: LockDurability.Persistent,
        cancellationToken: stoppingToken
    );

    if (lockHandle.IsAcquired)
    {
        Console.WriteLine($"I am the leader. Fencing token: {lockHandle.FencingToken}");
        await RunSchedulerLoop(lockHandle, stoppingToken);
    }
    else
    {
        Console.WriteLine("Another instance is the leader. Retrying...");
        await Task.Delay(2000, stoppingToken);
    }
}