2 posts tagged with "testing"

Every write to Kahuna passes through Kommander. Kommander is the open-source Raft library that elects leaders, replicates the log, and decides when a write is durable. If Kommander loses a committed entry, Kahuna loses your data. If Kommander elects two leaders for one term, Kahuna can serve two different histories.

This post explains how we test a component with that much responsibility. You will learn which test suites exist, what the coverage number means, what runs on every push, what runs every night, and what deterministic simulation testing adds that normal tests cannot. If you plan to run Kahuna in production, this is the evidence behind its consistency promise.

Distributed systems usually look reliable when everything is healthy. The network is fast, every server is online, and requests arrive in a convenient order.

But what happens when a server loses contact with the rest of the cluster? What if a process stops while handling a request? What if Kahuna saves a value, but the reply never reaches the client?

We recently added a public Jepsen test suite for Kahuna to explore these difficult situations. It creates real Kahuna clusters, sends many requests at the same time, and introduces failures on purpose. It then checks whether Kahuna still keeps its promises.

Our goal is not only to see green test results. We want to find rare problems, understand them, and make sure they do not return.