One post tagged with "testing"

Distributed systems usually look reliable when everything is healthy. The network is fast, every server is online, and requests arrive in a convenient order.

But what happens when a server loses contact with the rest of the cluster? What if a process stops while handling a request? What if Kahuna saves a value, but the reply never reaches the client?

We recently added a public Jepsen test suite for Kahuna to explore these difficult situations. It creates real Kahuna clusters, sends many requests at the same time, and introduces failures on purpose. It then checks whether Kahuna still keeps its promises.

Our goal is not only to see green test results. We want to find rare problems, understand them, and make sure they do not return.