Keys Eviction

Kahuna serves hot keys from memory, but memory is limited. Eviction is how each node keeps its memory use bounded: it first removes expired and deleted entries, and if that is not enough, it drops the least recently used entries.

What this means for you:

• Persistent keys are safe. Eviction drops only the in-memory copy. The next read loads the value from disk, which is slower but returns the same data.
• Ephemeral keys can disappear. Their only copy is in memory, so an evicted ephemeral key is gone, even before its expiration time. Use ephemeral durability only for data you can rebuild, such as caches and counters.
• The budget is configurable. If a node uses too much memory, or evicts hot data too often, see Configuration Surface.

Most deployments do not need to change anything. The rest of this page describes the algorithm.

Kahuna keeps key/value state in memory inside each KeyValueActor, an actor that owns a group of keys. Persistent keys also have durable storage behind them, but the in-memory copy avoids disk reads for hot data. Ephemeral keys live only in memory, so eviction of an ephemeral key removes the value from the node.

The current key/value eviction algorithm is a bounded collector. It reclaims definite garbage first, then uses true least-recently-used ordering only when the actor is over its configured entry or byte budget. Revision and MVCC metadata are trimmed where they grow so hot keys with large histories do not grow without bound.

The diagram shows one collection pass. Both phases share the CollectBatchMax limit.

When Collection Runs

Collection can run in three ways:

• A KeyValueCollectorActor periodically fans out Collect messages to all ephemeral and persistent key/value actors.
• The fan-out is staggered across the configured collection interval so every actor does not sweep at the same time.
• Each KeyValueActor checks its memory budget every few hundred non-collection operations. If the actor is over budget, it runs collection immediately.

If one collection pass reaches its batch limit and the actor is still over budget, the actor schedules a follow-up collection message for itself. This prevents one request from doing unbounded cleanup work while still draining memory pressure over multiple passes.

Actor Budgets

Each key/value actor tracks:

• number of entries in its in-memory BTree
• approximate bytes held by keys, values, revision history, and MVCC metadata

Collection pressure is based on:

Setting	Meaning	Default
MaxEntriesPerActor	Maximum cached key/value entries per actor before LRU pressure applies.	50000
MaxBytesPerActor	Approximate maximum bytes per actor before LRU pressure applies.	268435456
CollectBatchMax	Maximum entries removed in one collection pass.	1000

The byte estimate is intentionally approximate. It includes key text, value bytes, fixed entry overhead, revision metadata, revision value bytes, and MVCC metadata.

Memory limits are per key/value actor in each durability ring (the separate set of actors for persistent keys and for ephemeral keys), not per Raft partition. KeyValueWorkers determines actor count; one actor can own keys from multiple partitions and a partition can span actors. These approximate budgets are collection thresholds rather than hard heap caps. Tune actor count and cache budgets against observed residency; adding partitions alone does not add an actor budget.

Phase 1: Garbage Reclamation

Every collection pass first removes entries that are safe to reclaim without considering the actor budget:

• entries in Deleted state
• entries in Undefined state
• entries whose expiration timestamp is set and has passed

Entries with an active write intent or replication intent are skipped. Kahuna does not evict objects currently participating in local mutation or replication work.

Kahuna keeps auxiliary structures for this work:

• Deleted and undefined entries are tracked through a tombstone queue. The collector drains the queue and re-checks the current entry before removing it, so a key that was deleted and later recreated is not removed accidentally.
• Expiring entries are tracked in an expiry heap ordered by expiration time. The collector pops expired entries until it reaches the first future expiration, so cleanup work is proportional to the number of expired keys instead of the number of cached keys.

This phase is bounded by CollectBatchMax. If the actor is still over budget after the pass reaches its batch limit, the actor schedules a follow-up collection.

Phase 2: LRU Eviction

After tombstones and expired entries are reclaimed, Kahuna projects the actor's remaining entry count and byte usage. If the actor is still over either budget, it evicts cold live entries using least-recently-used ordering.

Each cached entry participates in an intrusive LRU list:

• reads and writes move the entry to the hot end of the list
• the cold end identifies the least recently used entries without sorting or scanning the whole BTree
• eviction walks from the cold end until the actor is under budget or the batch limit is reached

Entries already selected for garbage eviction, entries with write intents, entries with replication intents, and dirty persistent entries are excluded from LRU eviction.

Dirty Persistent Entries

Persistent writes become visible in memory before the background writer flushes the materialized state to RocksDB or SQLite. During that short window, the in-memory entry can be newer than disk. Evicting it would make the next read reload stale data, so Kahuna pins dirty entries until they are safe to drop.

The collector uses the entry's flushed revision and a safety window based on DirtyObjectsWriterDelay to avoid evicting a persistent entry whose latest change might not be on disk yet. The same rule protects deletes, so an unflushed delete cannot be evicted in a way that resurrects an older value from disk.

This safety rule affects memory behavior: under heavy write load, recently modified persistent keys may remain in memory longer than cold clean keys.

Metadata Trimming

Eviction is not the only way Kahuna reduces memory. Revision and transaction metadata are bounded inline, where they grow:

• Each new archived revision trims settled history toward RevisionRetention; committed revisions still awaiting backend flush are retained beyond that target.
• If an MVCC snapshot hold is active, trimming also keeps the boundary revision at or before the protective registered-hold floor.
• When a transaction commits, rolls back, or releases a lock, its MVCC snapshot is removed and expired sibling snapshots are cleaned up.

This matters for hot keys. A hot key may never be selected by LRU, but its revision or transaction metadata can still grow. Inline trimming lets Kahuna reduce memory without scanning the entire store or evicting the current value. Snapshot holds protect a boundary revision in memory; unflushed revisions can add more. Persistent history keeps the boundary and all newer revisions until the protective floor advances.

Setting	Meaning	Default
RevisionRetention (--revisions-to-cache)	Recent in-memory revisions per key; persistent keys can use disk history, ephemeral keys cannot.	4

Persistent revision cleanup is also clamped by the protective registered-hold floor. While a hold remains registered, including after lease expiry until replicated purge or release, the persistent backend must keep the boundary revision and all newer revisions, even if PersistentRevisionRetentionCount or PersistentRevisionRetentionAge would otherwise prune them.

Optional Memory Tuning

--trim-flushed-revisions removes persistent archived revisions once the background writer confirms them on disk, while retaining unflushed revisions and the boundary pinned by a snapshot hold. This reduces resident history for hot keys, but historical reads may perform more disk I/O. It does not delete persisted history or change retention settings. Ephemeral keys keep their usual --revisions-to-cache archive. The option is off by default.

--pinned-value-payloads copies committed payloads of at least --pinned-value-payload-min-bytes (default 256) onto .NET's pinned object heap, where GC does not relocate them. It adds an allocation and copy when retaining a value and shifts collection to generation 2. It is off by default; compare allocation, GC pauses, and memory use under your workload before enabling it. It changes neither durability nor the public byte-payload format.

The actor cache now maintains a point-lookup dictionary alongside its ordered BTree. Reads by key use the dictionary; ordered scans use the BTree. This is automatic and adds index memory; it does not change ordering or visibility semantics.

Persistent Revision Pruning

These rules govern the destructive cleanup of persistent revision history.

Pruning Confirmation

Destructive revision cleanup confirms local system-partition application before sampling the protective hold floor. Failure skips the pass (kahuna.snapshot_floor.prune_skipped_unconfirmed_total). Acquire overlapping a local prune generation returns MustRetry; this is not a cluster-wide atomic barrier and cannot restore already-deleted history. See Snapshot Holds.

Idle RocksDB Revision Sweeps

After a complete clean sweep, RocksDB can skip later backend-wide revision-pruning cycles while no relevant state has changed. New writes, retention settings, snapshot-floor changes, or reaching the next age-expiry frontier reopen sweep work. kahuna.persistence.revision_prune.sweep_idle_total counts skipped idle cycles. Quiescence avoids repeated scans; it does not disable configured retention or remove snapshot-hold protection.

Persistent vs Ephemeral Keys

Persistent and ephemeral keys use the same in-memory key/value collector, but eviction only drops the memory copy of a persistent key, while it removes an ephemeral key for good; see Durability Levels.

Reads, Scans, and Cache Reloads

Latest point gets first try a quiet resident published head after leadership confirmation, then fall back to the actor cache. See Point Reads and Published Heads for guards, transactional pins, and metrics. If a persistent key is missing from memory, Kahuna loads it from the configured backend and usually caches it again. Ephemeral keys cannot be reloaded after eviction because memory was the only copy.

Large ordered scans use paginated range reads. They merge in-memory and on-disk state while carrying a consistent snapshot across pages. Keys that are read from disk during a large scan are not forced back into the cache, so full-table or large-prefix iteration does not flood the cache or disturb hot-key recency.

Non-paginated prefix and bucket scans return the matching set in one call and are capped. They are intended for small, known-bounded prefixes. Use paginated range scans for large or unbounded iteration.

For per-page read amplification, see Range-Scan Observations. A missing page observation means unknown rather than zero.

Locks and Sequences

The algorithm above applies to key/value actors. Lock actors currently use a simpler cleanup path:

• cleanup is checked every 500 lock operations
• cleanup starts only when the actor has at least 200 lock entries
• entries whose LastUsed age exceeds CacheEntryTtl are removed
• each cleanup pass is capped by CacheEntriesToRemove

Sequences are not managed by the key/value collector.

Configuration Surface

The server command line currently exposes:

Command Line Option	Meaning	Default
--cache-entry-ttl	Age threshold used by cache cleanup paths, in seconds.	1800
--cache-entries-to-remove	Maximum number of entries removed by cleanup passes that use this cap.	100
--disable-published-head-reads	Disable the resident published-head get path for diagnosis; all gets use actors.	disabled
--dirty-objects-writer-delay	Delay between dirty object writer flush passes, in milliseconds. Longer delays can increase batching but keep dirty entries pinned longer.	200

Embedded nodes expose the full collector configuration as .NET options. See Embedded Kahuna Node for the full option table.

Observability

When a key/value collector pass evicts entries, Kahuna logs a summary with the number of reclaimed entries, the source breakdown (tombstone, expired, and lru), the resulting store size, byte estimate, elapsed time, and whether a follow-up pass is queued. Frequent backlog or high lru counts indicate sustained memory pressure on that actor.

Log Compaction

Key eviction is separate from Raft write-ahead log (WAL) compaction. Each Raft group also compacts its WAL through Kommander, removing old log entries that are no longer needed after checkpointing. WAL compaction reduces disk usage; key/value eviction reduces in-memory cache usage.