Skip to main content

Embedded Kahuna Node

EmbeddedKahunaNode starts a single-node Kahuna engine inside the current .NET process. It is useful for integration tests, local tools, and applications that need Kahuna's transaction engine without running the ASP.NET server, Kestrel, REST, or external gRPC endpoints.

using System.Text;
using Kahuna;
using Kahuna.Shared.KeyValue;
using Kommander.Time;

await using var node = new EmbeddedKahunaNode(new()
{
Storage = "memory",
WalStorage = "memory",
InitialPartitions = 1
});

await node.StartAsync();
await node.WaitForLeaderForKeyAsync("tenant/table/key-a");

byte[] value = Encoding.UTF8.GetBytes("value-a");

await node.Kahuna.LocateAndTrySetKeyValue(
transactionId: HLCTimestamp.Zero,
key: "tenant/table/key-a",
value: value,
compareValue: null,
compareRevision: -1,
flags: KeyValueFlags.Set,
expiresMs: 0,
durability: KeyValueDurability.Persistent,
cancellationToken: CancellationToken.None
);

API

public sealed class EmbeddedKahunaNode : IAsyncDisposable
{
public IKahuna Kahuna { get; }
public IRaft Raft { get; }

public EmbeddedKahunaNode(
EmbeddedKahunaOptions options,
ILoggerFactory? loggerFactory = null
);

public Task StartAsync(CancellationToken cancellationToken = default);
public Task<string> WaitForLeaderForKeyAsync(string key, CancellationToken cancellationToken = default);
public ValueTask DisposeAsync();
}

Options

OptionDefaultDescription
NodeNameembedded-1Logical node name.
NodeId1Raft node identifier.
HostlocalhostMust be a concrete host, not *.
Port0Unused by in-memory communication, but still part of the Raft configuration.
InitialPartitions1Number of Raft partitions.
StoragememoryKey/value storage backend: memory, sqlite, or rocksdb.
StoragePathemptyStorage directory for persistent backends.
StorageRevisiongeneratedStorage revision name.
WalStoragememoryRaft WAL backend: memory, sqlite, or rocksdb.
WalPathemptyWAL directory for persistent backends.
WalRevisiongeneratedWAL revision name.
WalSyncWritestrueRequire synchronous durable writes for RocksDB or SQLite WAL storage.
RaftWalShardWriteBufferSizeMbnullRocksDB Raft WAL shard memtable size in MiB. Null keeps Kommander's default. Applies only when WalStorage is rocksdb.
RaftWalShardMinWriteBufferNumberToMergenullImmutable memtables merged into one RocksDB WAL shard flush. Null keeps Kommander's default.
RaftWalShardMaxWriteBufferNumbernullMaximum mutable plus immutable memtables per RocksDB WAL shard. Null keeps Kommander's default.
RaftWalShardLevel0FileNumCompactionTriggernullLevel-0 file count that triggers compaction for RocksDB WAL shard column families. Null keeps Kommander's default.
RaftWalShardLevel0SlowdownWritesTriggernullLevel-0 file count at which RocksDB starts slowing WAL writers. Null keeps Kommander's default.
RaftWalShardLevel0StopWritesTriggernullLevel-0 file count at which RocksDB stops WAL writers. Null keeps Kommander's default.
RaftWalShardMaxBytesForLevelBaseMbnullRocksDB max_bytes_for_level_base for WAL shard column families, in MiB. Null keeps Kommander or RocksDB defaults.
RaftWalShardUniversalCompactionnullSelects universal compaction for RocksDB WAL shard column families when set to true; null keeps Kommander's default.
RocksDbSharedMemoryEnabledfalseShare one RocksDB block cache and write-buffer manager between the key/value backend and Raft WAL when both use RocksDB.
RocksDbSharedMemoryBudgetMb320Total shared RocksDB block-cache budget in MiB. The memtable sub-budget is charged inside this total.
RocksDbSharedMemtableBudgetMb128Shared RocksDB memtable sub-budget in MiB. Must be less than or equal to RocksDbSharedMemoryBudgetMb.
RocksDbDirectReadstrueRead RocksDB SST files with direct I/O so the RocksDB block cache is the primary read cache. Applies only when Storage is rocksdb.
RocksDbStatisticsfalseEnable RocksDB internal statistics collection and LOG dumps every 60 seconds. Useful for tuning or diagnosis, but it adds per-operation overhead. Applies only when Storage is rocksdb.
LocksWorkers0Lock actors per durability ring. 0 auto-sizes to max(32, CPU cores * 4).
KeyValueWorkers0Key/value actors per durability ring. 0 auto-sizes to max(32, CPU cores * 4).
BackgroundWriterWorkers1Background persistence worker count.
SequencerWorkers0Sequence actors. 0 auto-sizes to max(8, CPU cores).
SequencerBlockSize1000Values reserved from one durable sequence record per compare-and-swap for sequences without their own block size. Larger blocks reduce storage traffic but can leave larger gaps if an abandoned block is not fully issued.
SequencerIdempotencyRetentionMax256Maximum keyed sequence-reservation idempotency entries retained per sequence record. A non-positive value disables the count cap.
SequencerIdempotencyRetentionTtl10 minutesWindow in which retrying a keyed sequence reservation replays the same allocation. TimeSpan.Zero disables age pruning.
SequencerMaxSequencesPerActor10000Maximum sequence records one sequence actor keeps resident before evicting least-recently-used records and abandoning their reserved blocks. A non-positive value leaves residency unbounded.
SequencerBlockLease5 secondsMaximum time a reserved sequence block can be served from memory before revalidation against the durable record. Safe sequence updates wait this long and refuse allocations during the wait. A non-positive value disables revalidation and sequence updates.
BackendReadIOThreads4Dedicated Kahuna backend read pool size for point gets, existence checks, read-before-write work, and scans. Separate from the Raft WAL read pool.
BackendWriteIOThreads1Dedicated Kahuna backend writer pool size for background batch writes and pruning.
BackendReadQueueDepth4096Per-partition pending queue depth for the backend read scheduler.
ScanPageRetryBudgetMs5000Maximum retry window for one range-scan page that keeps returning retryable state before the scan fails loudly with a retryable server error.
SessionOwnedIntentCeilingMs0Maximum age for session-owned write intents and no-expiry range locks after their owner disappears. 0 derives the ceiling from the transaction timeout, reaper grace, and participant-effect TTL. Prepared durable intents are exempt.
StagedBaseFenceRetentionMs600000How long the prepared-intent store remembers recently committed heads for validated-base fencing. Keep this above the longest transaction lifetime allowed by the deployment. Use the same value on every node when OnePhaseApplyTimeValidation is enabled.
OnePhaseApplyTimeValidationfalseAllows eligible read-modify-write and read-carrying durable transactions to use the one-phase fast path in multi-process Raft groups by validating their reads and bases at apply time against the replicated committed-head ledger. Enable only after every node in the group supports the ledger and gate.
DefaultTransactionTimeout5000Default transaction timeout in milliseconds.
MaxTransactionTimeout300000Maximum admitted interactive transaction timeout in milliseconds. Caller-provided timeouts are clamped to this bound.
MaxConcurrentTransactions0Script transactions that may execute concurrently before further ones queue and start in priority order. 0 disables the script admission gate.
MaxConcurrentSessions0Interactive transaction sessions that may be open concurrently before further ones queue and start in priority order. 0 disables the session admission gate.
TransactionPriorityReservedSlots0Slots out of each transaction concurrency ceiling that only High and Critical transactions may occupy.
TransactionPriorityAgingThreshold1000Milliseconds a queued transaction waits to gain one effective priority level. 0 disables aging.
TransactionPriorityMaxQueued4096Callers that may wait for an admission slot per gate before further ones receive AdmissionRefused. 0 makes the queue unbounded.
DefaultAdmissionWaitMs5000Admission wait used when the caller does not specify one.
MaxAdmissionWaitMs30000Maximum admission wait allowed by the embedded node. Caller-supplied waits are clamped to this value.
StagedWriteIntentLeaseMs15000Lease for transactional staged writes before other transactions may treat them as abandoned and write past them.
ScriptCacheExpiration1 minuteHow long parsed scripts stay cached.
MaxScriptLength65536Largest transaction script accepted, in bytes. Oversized scripts are refused before parsing.
MaxScriptDepth256Deepest transaction script syntax tree accepted. This bounds parser and evaluator stack use.
RevisionsToKeepCached100Number of key revisions to keep cached in memory.
CacheEntryTtl5 minutesAge threshold used by lock cleanup and legacy cleanup paths. Key/value LRU eviction is budget-based.
CacheEntriesToRemove1000Maximum entries removed by cleanup paths that use this cap. Key/value collection uses CollectBatchMax.
CollectionInterval60 secondsInterval for cache collection and eviction checks.
TransactionOutcomeRetentionMax10000Strict maximum retained terminal transaction outcomes. A non-positive value disables best-effort outcome retention.
TransactionOutcomeRetentionTtl5 minutesAge window for retained terminal transaction outcomes. A non-positive value disables age-based removal.
CompletionReceiptRetentionTtl10 minutesAge after which orphaned durable completion receipts can be dropped once their transaction record has already been reclaimed.
DurableDecisionOutstandingMax100000Maximum outstanding undecided canonical durable transaction records admitted by this node. Completed records do not count against this budget.
DurableDecisionDeadlineFloorMs5000Lower clamp for the durable transaction decision-deadline margin.
DurableDecisionDeadlineCeilingMs60000Upper clamp for the durable transaction decision-deadline margin.
DurableRecordRetentionMax200000Maximum retained terminal durable transaction records per node before older records are reclaimed early. A non-positive value disables the count budget.
DurableRecordRetentionMaxBytes268435456Estimated heap-byte budget for terminal durable transaction records plus completion receipts. A non-positive value disables the byte budget.
DurableRecordRetentionHeapPressure0.85Managed-heap load ratio above which terminal records older than the retention floor are reclaimed aggressively. A non-positive value disables this pressure valve.
DurableRecordRetentionFloor90 secondsMinimum age below which terminal durable records are not reclaimed early by count, byte, or heap-pressure budgets.
DurableMaintenanceInterval5 secondsTick interval for prepared-intent recovery and durable record-retention sweeps. TimeSpan.Zero uses CollectionInterval.
DurableRecordGcMaxPerPass4096Maximum terminal transaction records reclaimed in one retention sweep batch.
Functionsempty registryUser-defined script functions callable by this embedded node. Register functions before constructing the node.
FunctionSlowWarnMs50Logs a warning when a user-defined function takes longer than this many milliseconds. 0 disables the warning.
FailFastOnOutOfMemorytrueFail fast on OutOfMemoryException so an orchestrator can restart the embedded process. When disabled, Kahuna records the fatal fault and reports unhealthy through health surfaces that expose the core state.
DurableDeferredSettlementtrueReturn from durable commit once the canonical decision record is durable, then materialize values and settle intents in the background. Set false to await settlement inline.
DurableMaterializeByReferencetrueMaterialize committed durable transactions with a value-free record that references the prepared intent already held by each replica. Set false only during mixed-version rollouts from builds that cannot apply MaterializeIntent.
DurablePreparedIntentMaxCount500000Resident prepared-intent count bound for durable transactions. A non-positive value disables the count bound.
DurablePreparedIntentMaxBytes1073741824Resident prepared-intent value-byte bound for durable transactions. A non-positive value disables the byte bound.
MaxEntriesPerActor50000Maximum cached entries per actor before collection pressure applies.
MaxBytesPerActor268435456Approximate maximum cached bytes per actor before collection pressure applies.
CollectBatchMax1000Maximum number of entries evicted in one collection pass.
RevisionRetention16Number of revisions retained for in-memory revision history.
DirtyObjectsWriterDelay1000Delay between dirty object writer flush passes, in milliseconds. Longer values can increase batching but keep dirty persistent entries pinned in memory longer.
CheckpointInterval30 secondsMinimum checkpoint cadence for dirty partitions after flushes. Shorter intervals advance WAL compaction sooner; longer intervals reduce checkpoint churn.
KeyValueWriteLingerMs1Delay from the oldest queued persistent partition write before a partition batch is proposed. 0 dispatches an idle partition immediately.
KeyValueWritePostCompletionHoldMs0Optional hold after a partition batch completes before the next sub-threshold batch is dispatched. This can improve batch density under saturated same-partition write load.
KeyValueWriteMaxBatchItems512Maximum log entries selected for one partition write coalescing Raft call.
KeyValueWriteMaxInFlightBatchesPerPartition1Maximum coalesced batches a partition may have awaiting Raft results at once. Higher values pipeline quorum waits while preserving FIFO dispatch order.
KeyValueWriteMaxBatchBytes4194304Target serialized bytes selected for one partition write coalescing Raft call.
KeyValueWriteMaxQueuedItemsPerPartition8192Maximum admitted persistent submissions per partition, including writes already in flight.
KeyValueWriteMaxQueuedBytesPerPartition33554432Maximum admitted serialized bytes per partition, including writes already in flight.
KeyValueWriteMaxQueueDelayMs1000Maximum pre-dispatch wait before a queued write is released as MustRetry.
MaxKeyValueWriteAggregatorInboxSize16384Ordinary-submission inbox bound per aggregator lane. Control messages are exempt.
PersistentRevisionRetentionCount0Maximum persisted revisions retained per key. 0 keeps every revision.
PersistentRevisionRetentionAge0Maximum persisted revision age. TimeSpan.Zero disables age-based retention.
PersistentRevisionCleanupInterval5 minutesMinimum interval between full persistent-revision cleanup sweeps.
PersistentRevisionCleanupBatchSize10000Maximum revision records deleted per cleanup pass.
PersistentRevisionCleanupOnWritetrueQueue keys touched by writes for targeted revision cleanup.
PersistentRevisionCleanupTimeBudget250 msWall-clock budget for one targeted persistent revision cleanup pass during a flush cycle.
PersistenceMaxUnflushedItems1000000Maximum committed key/value writes held in memory awaiting background flush before ordinary writes receive retryable backpressure. 0 disables the bound.
PersistenceMaxUnflushedBytes536870912Maximum value bytes held in memory awaiting background flush before ordinary writes receive retryable backpressure. 0 disables the bound.
PitrWindow1 hourRecoverable WAL history. Values are normalized to more than zero and at most 6 hours.
BaseSnapshotInterval30 minutesIntended interval between base checkpoints. It must be positive and no greater than PitrWindow. It also contributes to the protected WAL floor.
BackupDiremptyRoot directory for backup manifests and artifacts. Backup methods on node.Kahuna are disabled when empty.
BackupTargetlocalBackup storage target. local uses BackupDir; other target names require BackupStorageProvider.
BackupScratchDiremptyLocal staging directory for backup targets that cannot receive checkpoints directly. Size it for one full backup.
BackupStorageProvidernullHost-supplied factory for object storage or another non-local backup target. Null uses the local directory implementation.
BackupClusterIdemptyOperator-assigned cluster identity stamped into backup manifests. Use the same value on every node.
BackupMacKeyFileemptyPath to the HMAC-SHA-256 key file used to authenticate backup manifests. Keep it outside BackupDir.
RestoreRootemptyServer-owned root directory that restore targets must be contained within. Setting it enables confined remote restore.
AllowUnconfinedRemoteRestorefalseAllows remote restore without RestoreRoot. Use only in trusted administrative environments.
BackupRetentionMaxChains0Keep at most this many most-recent backup chains. 0 is unbounded. Retention is off unless at least one retention bound is set.
BackupRetentionMaxAge0Delete backup chains whose newest backup is older than this age. TimeSpan.Zero is unbounded.
BackupRetentionMaxBytes0Keep the most-recent backup chains within this artifact byte budget. The newest chain is always kept. 0 is unbounded.
BackupGcInterval1 hourPeriodic backup garbage-collection cadence. TimeSpan.Zero disables the periodic pass, but GC still runs after backups.
BackupRestoreThrottleBytesPerSec0Throughput budget for a restore checkpoint copy. 0 is unlimited.
RangeSplitThreshold1000Sampled key count that triggers count-based range splitting. 0 disables this trigger.
RangeSplitMinRangeSize10Minimum sampled keys required in each child range.
RangeMergeMinSize10Adjacent key ranges smaller than this value can be considered for automatic merging. 0 disables automatic merge.
RangeSplitLoadThreshold0Replicated writes per second required for load-based splitting. 0 disables this trigger.
RangeSplitLoadMinQueueDepth8WAL backlog required alongside the load threshold.
RangeSplitLoadMinCommitWaitMs0Optional commit-wait gate in milliseconds. 0 disables it.
RangeSplitLoadWindow15 secondsTime the complete load predicate must remain satisfied.
RangeSplitLoadPollInterval5 secondsFrequency of load-based split checks.
RangeSplitLoadImbalanceMax0.8Maximum acceptable write fraction assigned to either child.
RangeSplitIndivisibleCooldown5 minutesDelay before reconsidering an indivisible range.
RangeSplitSettleWindow10 secondsPost-split delay before evaluating either child again.
RangeMoveSettleTimeout10 secondsMaximum quiesce wait for in-flight transactions in a moving range before split or merge cutover.
EnableLeaderBalancerfalseEnable cross-node load reports and leader redistribution. Required with load splitting.
LeaderBalancerReportInterval5 secondsInterval between node load reports.
LeaderBalancerInterval30 secondsInterval between balancing passes.
LeaderBalancerReportTtl20 secondsMaximum accepted load-report age.
MinLeaderStability5 secondsMinimum leadership age before transfer.
LeaderBalancerOpsWeight1.0Operations-per-second weight in the balancer load score.
LeaderBalancerQueueWeight0.5Queue-depth weight in the balancer load score.
ReplicationFactor0Desired voter replicas per partition. 0 keeps full replication. Prefer odd values such as 3 or 5 in multi-node deployments.
EnablePlacementRebalancerfalseEnable ongoing replica-placement repair and balancing. Initial placement still applies when ReplicationFactor is positive.
PlacementPassInterval5 secondsCadence for placement-controller passes. Commit-triggered passes can still run immediately.
MaxReplicaMovesPerPass4Maximum new replica add/remove sequences started in one placement pass across repair and balance priorities.
MaxConcurrentReplicaTransfers1Maximum partitions with an in-flight learner catch-up or replica removal caused by balance moves.
MaxConcurrentReplicaRepairs3Maximum in-flight repair moves for under-replicated ranges or replicas stranded on departed nodes.
DecommissionDrainTimeout2 minutesGraceful leave wait for evacuating this node's hosted replicas before removal.
ReplicaCountDeadband1Replica-count imbalance tolerated before balance moves start.
ZonenullOptional zone or rack hint used to spread replicas across failure domains.
EnableLoadReportsfalseGossip per-partition load reports even when leader balancing, placement rebalancing, or replication factor did not already enable them.
JoinExistingSeedsnullSeed endpoints for joining an existing embedded cluster through the cluster constructor. Null or empty boots from the configured discovery roster.
ReadIOThreads8Number of Raft read I/O threads.
WriteIOThreads8Number of Raft write I/O threads.
EnableSharedExecutorPooltrueShare a bounded worker pool across Raft partitions instead of using one OS thread per partition.
PartitionExecutorPoolSize0Shared Raft executor worker count. 0 auto-sizes to the processor count.
HttpSchemehttps://HTTP scheme used by Raft REST communication.
HttpAuthBearerTokenemptyBearer token sent with Raft REST communication.
TransportSecuritynullOptional node-to-node transport security options passed to Kommander. Embedded in-memory peers do not need it, but custom hosts that expose Kahuna gRPC services on a real transport use it for node-only trust checks.
HttpTimeout5Raft REST request timeout in seconds.
HttpVersion2.0HTTP protocol version used by Raft REST communication.
HeartbeatInterval100 msLeader heartbeat interval.
RecentHeartbeatHeartbeatInterval / 4Recent-heartbeat de-duplication window. Leave unset to track the heartbeat cadence safely; set explicitly only when it remains below HeartbeatInterval.
VotingTimeout1500 msVote wait timeout.
CheckLeaderInterval250 msLeader check interval.
TimerInitialDelay2500 msInitial delay before Raft timers start.
UpdateNodesInterval5000 msNode registry update interval.
StartElectionTimeout500Minimum election timeout in milliseconds.
EndElectionTimeout1500Maximum election timeout in milliseconds.
StartElectionTimeoutIncrement100Minimum election timeout increment in milliseconds.
EndElectionTimeoutIncrement200Maximum election timeout increment in milliseconds.
SlowRaftStateMachineLog50Slow state-machine operation log threshold in milliseconds.
SlowRaftWALMachineLog25Slow WAL state-machine operation log threshold in milliseconds.
RaftMaxWalGroupBatchPartitions64Maximum partitions coalesced into one cross-partition WAL group-commit batch.
RaftWalGroupCommitLingerMs0Optional group-commit linger window in milliseconds. 0 disables linger.
RaftWalSingleFsyncCommitfalseEnables Kommander's single-fsync commit fast path for embedded nodes. Server defaults differ; embedded hosts opt in explicitly.
CompactEveryOperations1000Number of committed operations between automatic Raft WAL compaction checks.
CompactNumberEntries50Number of Raft WAL entries removed per compaction batch.
MaxEntriesPerCompaction5000Maximum Raft WAL entries processed per compaction run.

Live snapshot holds clamp persistent revision cleanup. While a hold is active, the boundary revision needed by the held timestamp and every newer revision are kept even if PersistentRevisionRetentionCount or PersistentRevisionRetentionAge would otherwise remove them.

To bound RocksDB memory across both the embedded key/value backend and Raft WAL, enable shared RocksDB memory with both storage layers set to RocksDB:

EmbeddedKahunaOptions options = new()
{
Storage = "rocksdb",
WalStorage = "rocksdb",
RocksDbSharedMemoryEnabled = true,
RocksDbSharedMemoryBudgetMb = 512,
RocksDbSharedMemtableBudgetMb = 128,
RocksDbDirectReads = true,
RocksDbStatistics = false
};

If either Storage or WalStorage is not rocksdb, these shared-memory options are ignored.

Code-Level Configuration

Some KahunaConfiguration options are not currently exposed by either Kahuna.Server command-line flags or EmbeddedKahunaOptions:

OptionDefaultDescription
ScriptCacheMaxEntries1000Maximum parsed scripts retained in the server-side script cache. New entries are dropped when the limit is reached.
MaxKeyValueActorInboxSize16384Maximum ordinary user messages queued in one key/value actor before new ordinary messages receive retryable backpressure. Control messages such as completions, cache-coherence updates, and maintenance are exempt. 0 disables the bound.
KeyValueWriteTerminalReserveItemsPerPartition256Extra per-partition item headroom reserved for terminal durable transaction work such as decision, materialization, settlement, recovery, and metadata handoff.
KeyValueWriteTerminalReserveBytesPerPartition4194304Extra per-partition byte headroom reserved for terminal durable transaction work.
KeyValueWriteMaxQueuedItemsGlobal131072Node-wide ordinary submission item cap across all partitions.
KeyValueWriteMaxQueuedBytesGlobal536870912Node-wide ordinary submission byte cap across all partitions.
KeyValueWriteTerminalReserveItemsGlobal8192Node-wide item headroom reserved for terminal durable transaction work.
KeyValueWriteTerminalReserveBytesGlobal67108864Node-wide byte headroom reserved for terminal durable transaction work.
KeyValueWriteMaxOperationBytes67108864Hard ceiling for one admitted serialized partition write. Values above the ceiling are rejected retryably.
KeyValueWriteBatchExecutionTimeoutMs30000Maximum Raft round-trip time for one aggregator batch before the batch is released retryably.
DurableRecoveryMaxPartitionsPerPass64Maximum partitions driven by prepared-intent recovery in one sweep.
DurableDecisionDeadlineMultiplier4Multiplier applied to observed finalize p99 before clamping the decision-deadline margin.
SnapshotHoldStartupGraceWindow5 minutesGrace window for durable snapshot holds loaded after restart. Expired holds restored from disk are protected briefly so holders can renew before cleanup advances the snapshot floor.

HttpsTrustedThumbprint also exists on KahunaConfiguration, but it is derived from HttpsCertificate by configuration validation rather than being an independent operator setting.

Notes

  • The embedded node uses in-memory Raft and inter-node communication.
  • StartAsync joins the single-node cluster and waits for leaders for the configured partitions.
  • WaitForLeaderForKeyAsync waits for the partition that owns a specific key.
  • Use distinct StoragePath and WalPath values when using sqlite or rocksdb.
  • Set BackupDir to enable backup, catalog, and offline restore methods through node.Kahuna. See Backups and Point-in-Time Recovery.
  • Load-based splitting requires a multi-node embedded deployment, key-range-routed spaces, and EnableLeaderBalancer = true. See Load-Based Range Splitting.
  • Positive ReplicationFactor values are intended for multi-node embedded deployments. 0 keeps the single-node/full-replication default. See Replication Factor and Replica Placement.
  • Always dispose the node with await using or DisposeAsync so Raft leaves the cluster and file-backed resources are released.