5 posts tagged with "distributed-systems"

Every write to Kahuna passes through Kommander. Kommander is the open-source Raft library that elects leaders, replicates the log, and decides when a write is durable. If Kommander loses a committed entry, Kahuna loses your data. If Kommander elects two leaders for one term, Kahuna can serve two different histories.

This post explains how we test a component with that much responsibility. You will learn which test suites exist, what the coverage number means, what runs on every push, what runs every night, and what deterministic simulation testing adds that normal tests cannot. If you plan to run Kahuna in production, this is the evidence behind its consistency promise.

Last Sunday, I was having lunch with a friend and talking about Kahuna, CamusDB, and all the time I've been putting into these projects lately. He didn't really understand why I would build Kahuna when most of what it does could probably be achieved with some combination of Redis and Postgres.

My explanation may not have been very convincing, and I understand why. Every business has a different tolerance for durability, availability, and consistency. Not every business operates 24/7, and not every business suffers significantly if its systems are down for a few minutes or even a few hours. In many operations, problems and mistakes can be fixed manually by people, or sometimes with an apology to the customer.

My story is very different. For some reason, I've spent almost my entire career working in industries that operate 24/7: airline reservations, hospitality, and video games. There is always a customer or user somewhere interacting with the system, whether it's just a few people or hundreds of thousands at the same time. That changed the way my brain works and the kinds of failure modes I instinctively try to avoid.

Building systems where performance is high and consistency is non-negotiable gradually became part of my DNA.

Kahuna started as a simple idea: build a lightweight tool that centralizes the distributed concerns every team eventually runs into. Locks, leader election, configuration, sequences. Things that are easy to get wrong and painful to debug when they break at 3 a.m.

Somewhere along the way, it became something else entirely.

Today I am releasing Kahuna 1.0, and the system that ships is far more than the coordination library I set out to build. Kahuna is a distributed key/value database with transactions, a scripting engine, horizontal scalability, and enough correctness testing to bet real workloads on it.

Distributed systems usually look reliable when everything is healthy. The network is fast, every server is online, and requests arrive in a convenient order.

But what happens when a server loses contact with the rest of the cluster? What if a process stops while handling a request? What if Kahuna saves a value, but the reply never reaches the client?

We recently added a public Jepsen test suite for Kahuna to explore these difficult situations. It creates real Kahuna clusters, sends many requests at the same time, and introduces failures on purpose. It then checks whether Kahuna still keeps its promises.

Our goal is not only to see green test results. We want to find rare problems, understand them, and make sure they do not return.

Distributed transactions rarely fail in dramatic ways. They fail in boring ways.

A client writes key account:alice, the response gets lost, the client retries, another request starts commit, and now the system has to answer a simple question: did that write count?

That is the kind of question Kahuna should answer on the server, not push onto every client library and application.

The next step for Kahuna transactions is a reusable transaction coordinator: one server-owned place that tracks what a transaction has read, written, locked, retried, committed, or rolled back.