Inspection and observability

Two kinds of observability

Mutation inspection separates journal-backed facts from best-effort signals:

The default in-memory journal does not survive process death. Install durable storage before depending on inspection after restart. See Journal storage.

Durable truth

pending(key)

pending(key) returns the current nonterminal intents for the key's terminal identity in durable client-sequence FIFO order. Alias traversal uses durable identity pairs only. Inspection never reconstructs a K, never calls the resolver, and therefore cannot fail because a canonical key is unresolvable.

pendingWrites()

pendingWrites() returns every nonterminal active intent across all durable identities in durable client-sequence order. It is a snapshot, not a subscription. Retired history never appears.

deadLetters()

deadLetters() returns only durably PARKED intents. Parking is legal only before a successful acknowledgement is durably recorded. Once parked, that client sequence never re-enters the executable FIFO. Retired history and post-acknowledgement work do not appear here.

The five public pending states

Five public states cover the six nonterminal durable execution phases:

A parked execution appears only in deadLetters(). A retired execution appears in neither pending inspection nor dead letters.

PendingIntent fields

The failure taxonomy

MutationFailureKind is an append-only classification with nine broad kinds:

MutationFailure is normalized and restart-safe. A raw StoreError or Throwable is never persisted or carried by it. Store removes stack-trace lines and remaining ISO control characters, then truncates at a Unicode code-point boundary. detail is at most 128 UTF-8 bytes and message is at most 1,024 UTF-8 bytes. occurredAtEpochMillis records Unix epoch milliseconds.

Advisory events

events is a read-only SharedFlow<MutationEvent> with replay 0, extra buffer capacity 64, and DROP_OLDEST overflow. Store emits with non-blocking tryEmit only. A new collector receives no history, restart replays nothing, and no event carries a raw Throwable or StoreError.

Intent-scoped events carry a mutationId and durable identity pair:

Checkpoint events are client-scoped and do not include a mutation identity or generation:

Collect for a lifetime your application owns. This example stops after 30 seconds and treats the output only as diagnostics:

withTimeoutOrNull(30_000L) {
    mutations.events.collect { event ->
        println(event::class.simpleName ?: "MutationEvent")
    }
}
// Dropped events are expected under pressure. Reconcile with durable inspection.

The poisoned flow

poisoned reports a projector throw as a PoisonedIntent containing the exact local Throwable. For the durable execution path, a projector throw also parks the row with a normalized PROJECTION failure. The exact throwable is never persisted and does not cross restart.

The flow replays up to 16 in-process reports and drops the oldest under pressure. For the documented projector case, fix the projector rather than treating the report as retry policy; see Authoring mutators.

KeyEvents

MutationStore.keyEvents republishes the delegated core Store's writer-event flow. The mutations facade returns null from runtime(), while keeping this advisory flow available as a public member.

KeyEvents is deliberately open rather than sealed. Its current variants are Written, Invalidated, and Deleted; retain an else branch when matching so a future minor-version variant does not break the consumer. Written covers fetch commits and write-handle adoption, Invalidated covers successful stale marks, and Deleted covers successful destructive removals. There is no Rekeyed variant. Canonical-key changes are observed through the stream swap and inspection described in Aliases and canonical rekeying.

Delivery is a best-effort hot flow with replay 0 and a buffer of 64 that drops the oldest event. It never completes, including after Store.close, so scope collection to an application lifecycle. Purge sweeps, nonresident watermark coverage, external source-of-truth changes, and superseded fetches emit no key event. The extension guide covers the open hierarchy in more detail.

Relationship to store-level telemetry

telemetry(...) in the mutation-store configure lambda installs StoreTelemetry on the delegated core Store. Its callbacks observe fetches, public serves, invalidations, and clears without participating in correctness. The mutations-owned events flow is a separate vocabulary for the journalled write lifecycle, while keyEvents remains the core writer-event stream.

Use Devtools and the inspector for shipped core telemetry sinks and live inspection. None of these advisory surfaces replaces mutation inspection.

Source recorded 2026-08-12 ·main@539614c0· pre-6.0.0-alpha01