Event Replay

Status

This document defines the revised target design for event replay during the early development stage of light-portal. The implementation still contains legacy-backfill, staged-rollout, feature-gate, mandatory-encryption, and allowlist machinery that does not yet match this design. That code will be reviewed separately after this design is accepted.

The design deliberately optimizes for a new development deployment:

• replay works as soon as the code and replay database tables are deployed;
• only failures observed after deployment are replay candidates;
• historical DLQ rows are not migrated or replayed;
• one small configuration provides an execution circuit breaker and identifies exceptional event types that must not be replayed;
• event validity and replay safety are enforced in code;
• Light Gateway controls which roles may call each replay endpoint;
• application-level payload encryption is not required for replay correctness.

Motivation

The portal has two projection processors:

• DbEventConsumerStartupHook reads outbox_message_t, tracks progress in consumer_offsets, and writes failed events to the PostgreSQL dead_letter_queue table.
• PortalEventConsumerStartupHook in user-query reads Debezium-published Kafka records, commits Kafka consumer offsets, and publishes failed records to the configured Kafka DLQ topic.

Both processors ultimately call PortalDbProvider.handleEvent(conn, event). They isolate a failed transaction and advance to later independent work, but a DLQ row alone is not a safe replay instruction:

• one row may be only one member of a multi-event business transaction;
• aggregate versions or graph revisions may require earlier failed transactions first;
• replay can race with newer live projection work;
• rewinding a shared source offset reprocesses unrelated transactions.

Event replay therefore operates on complete canonical failure transactions, not individual notification or DLQ rows. It invokes the same projection executor as live processing without rewinding PostgreSQL or Kafka offsets.

Goals

• Capture every new failed projection transaction in a replayable canonical form.
• Preserve the original transaction boundary, event order, payload, identity, aggregate version, graph revision, and source coordinates.
• Distinguish exact replay after a processor fix from controlled repair of an event whose business data is permanently invalid.
• Validate replay-required metadata when commands append events to event_store_t and outbox_message_t.
• Support PostgreSQL and Kafka through the same planning and execution model.
• Derive ordinary replay policy from validated event metadata instead of maintaining a large event allowlist.
• Permit an explicit exclusion list for exceptional, non-idempotent, or externally side-effecting projection events.
• Use the same projection handler path for live processing and replay.
• Keep unrelated hosts, aggregates, and graph roots processing while a failed scope is repaired.
• Require an immutable plan, a reason, authorization, and a distinct approver before execution.
• Start replay capture and APIs by default after the schema is present; execute only from an explicit operator request.
• Keep configuration small enough that a developer can understand the entire replay setup from one screen.

Non-Goals

• Historical or legacy dead_letter_queue rows are not imported. They remain diagnostic history only.
• The replay planner is not an event editor. It can use the original payload or reference a separately validated and approved repair; it cannot create or modify that repair.
• A repair never overwrites the original event or changes event IDs, transaction membership, aggregate identity/version, graph revision, or source coordinates.
• Replay does not replay successful history to rebuild an empty projection.
• Replay does not make external side effects or non-idempotent handlers safe.
• Replay does not replace event import, promotion, snapshot restore, or a full projection rebuild.
• The initial design does not require an object store, application-managed encryption keys, production rollout stages, canary allowlists, or legacy migration jobs.

Operating Model

command validation
       |
       v
event_store_t + outbox_message_t
       |
       v
PostgreSQL or Kafka live projection processor
       |
       +---------------- success ----------------> projection tables
       |
       +---------------- failure
                              |
                              v
                  canonical failure transaction
                              |
                              v
                     classify failure
                        /       \
                       v         v
             exact replay    propose repair
                       \         /
                        v       v
                  plan -> approve -> execute
                              |
                              v
                 direct command-to-query call
                              |
                              v
                       projection tables

The normal Pub/Sub configuration selects the live processor. Event replay must not introduce another source-selection property. PostgreSQL deployments use the database source metadata; Kafka deployments use the Kafka source metadata.

All of these components are standard, always-active platform behavior after the replay schema and services are deployed:

1. command-side event validation;
2. canonical failure capture;
3. replay query and command APIs;
4. the built-in approval state machine;
5. the request-thread replay processor invoked by the execute command.

Replay and repair are different operations. Exact replay is appropriate when the event was valid and the projection handler was defective or temporarily unable to process it. Repair is appropriate only when the persisted business data itself is invalid and exact replay would deterministically fail again.

There are no independent capture, planning, rollout, source, projection, consumer-group, or host switches in the development configuration. One enabled property is retained solely as the replay-execution circuit breaker.

Event Append Contract

Replayability begins when a command appends events, not after an event fails. The command path must reject invalid transactions before committing event_store_t or outbox_message_t.

Transaction invariants

Every appended transaction must satisfy:

• transaction_id is present and uses the canonical UUID representation;
• transaction_count is positive and identical on every member;
• transaction_ordinal is contiguous from zero through transaction_count - 1;
• every member has the same host and transaction identity;
• event IDs are present and unique;
• the transaction does not contain duplicate ordinals;
• event_store_t and outbox_message_t are committed atomically;
• the persisted source order is deterministic.

Database constraints enforce uniqueness and basic ranges. Java validation enforces cross-row completeness before commit. Tests must prove that a partial transaction cannot become visible to either live processor.

Event invariants

Every event must contain a structurally valid CloudEvent envelope, including:

• event ID and event type;
• host identity;
• schema/specification version;
• event timestamp;
• parseable data;
• aggregate ID, aggregate type, and positive aggregate version for aggregate events;
• root instance ID and positive graph revision for graph-ordered events;
• any handler-specific identity required to make the projection idempotent.

The command path also validates the event data against the registered schema for that exact event type and schema version. It enforces required properties, property types, bounded values, and domain invariants that can be evaluated without projection state. A failure returns a command validation error and appends nothing; it is not a DLQ or replay candidate.

The validation registry is shared by command append, live projection, failure capture, planning, and replay. A handler must not interpret an event as one policy during live processing and another policy during replay.

Every appended event records the exact registry and repair-schema version used to validate it. Later live projection, capture, planning, repair, and replay use that pinned version; a deployment may add a new version but must not remove a version while an event, open failure, repair, or plan references it. Unknown registry entries fail closed on portal/internal append. An external Kafka event with an unknown entry is captured as diagnostic, non-executable evidence rather than interpreted under the current default. This prevents a registry edit from retroactively changing the meaning of already-committed events.

Projection handler contract

Replayable projection handlers must:

• perform database projection work through the caller’s transaction;
• be idempotent for the original event ID and ordering metadata;
• use monotonic aggregate-version or graph-revision checks when ordered;
• avoid network calls, email, message publication, payment submission, or any other non-transactional external side effect;
• produce the same projection outcome in LIVE and REPLAY modes.

An unordered event whose handler cannot satisfy this contract belongs in excludedEventTypes. An aggregate- or graph-ordered handler cannot be excluded: doing so would create a projection gap that cannot be replayed or safely waived. Such a handler must be made transactionally idempotent, normally by moving its external effect behind an outbox, before it can carry ordering metadata.

Replay Eligibility Policy

Shared append-validation registry

Every portal or internal append is validated against the exact event type and event schema version in event-replay-policy-v2. The registry declares the transaction/order policy, dependency metadata, data validator, replay policy, repair disposition and optional repair-schema version. It is code-owned and versioned; it is not another operator switch in event-replay.yml.

The common append boundary validates the complete transaction before reserving an outbox offset or inserting event, outbox, or notification rows. Interactive commands, graph/clone operations, global snapshot imports, and scheduler commands all use this boundary. A successful append pins eventschema and replaypolicy CloudEvent extensions, plus repairschema when applicable, and persists those versions in event_store_t and outbox_message_t. Canonical failure capture copies the pins into the failure transaction and member rows so planning and replay load the referenced version instead of reinterpreting an old event with the latest registry.

Unknown portal/internal event types, malformed transactions, mixed host/user identity, non-contiguous member order, invalid aggregate or graph metadata, and schema-invalid data fail closed and commit no append-side writes. An unknown or structurally invalid external Kafka event can be retained as diagnostic failure evidence, but it is marked non-executable and cannot become a replay candidate.

PORTAL_OBJECT_V1 is the shared baseline validator for current event schema version 1: it proves the CloudEvent data is a parseable JSON object and combines that with the registered envelope, identity, ordering, dependency, and size checks. It does not duplicate every command handler’s required-field and domain rules. Those rules still run before ordinary command event construction. Any event declared SCHEMA_REPAIR must instead name a concrete typed repair/data schema; R9 qualification rejects a repairable event that still relies only on the structural baseline. This distinction prevents the baseline name from being misread as complete per-event domain validation.

The Kafka consumer invokes the same external validation before live projection. A registered, valid portal event remains executable. A structurally valid but unknown or excluded event is captured as canonical diagnostic evidence whose policy rejects planning. A malformed member makes the complete transaction non-executable; it is recorded through the bounded failure notification and configured legacy diagnostic path without creating a canonical replay candidate, and its offset may then advance rather than poison-loop forever.

The default policy is derived from validated metadata:

This removes the need to list hundreds of ordinary replayable events. A new event is replayable when its append contract and projection handler satisfy the derived policy. Exceptional events are excluded explicitly.

For example, UserDeletedEvent contains aggregate identity and version metadata. Its projection handler must use the monotonic aggregate-version guard; it then derives AGGREGATE_VERSION without a dedicated allowlist entry.

An exclusion is an exact event-type match. Unknown patterns, substrings, and wildcards are not accepted because they make policy review ambiguous. Configuration reload also rejects an exclusion whose registered policy is GRAPH_ROOT or AGGREGATE_VERSION. Registry validation rejects an ordered NOT_REPLAYABLE policy for the same reason. Exclusion is therefore available only for transaction-only or explicitly unordered events and cannot brick an ordered scope.

Minimal Configuration

The complete development-facing event-replay.yml is:

# Execution circuit breaker. This does not disable capture, queries, planning,
# approval, or durable replay state.
enabled: ${event-replay.enabled:true}

# Exact event type names whose projection handlers are not safe to replay.
excludedEventTypes: ${event-replay.excludedEventTypes:}

enabled defaults to true and controls execution only. Event validation, canonical failure capture, candidate and status queries, planning, approval, and durable replay state remain active when it is false. An empty exclusion list means every event satisfying the append and projection-handler contracts derives its replay policy from its metadata.

Limits, lease durations, plan expiry, retry counts, approval requirements, and safe error sizes use reviewed application defaults. They do not need operator properties during early development. They can become advanced production configuration later without changing the public replay contract.

The processor identity is derived from the deployed service identity plus a per-process instance identifier. The end-user bearer token is forwarded from command to query; there is no replay worker client identity.

Gateway endpoint roles remain in Light Gateway access-control configuration, not event-replay.yml.

Execution circuit breaker

An emergency may require replay execution to pause while diagnosis continues. Examples include a projection-handler regression, database overload, an incompatible rolling deployment, repeated processor failures, or an authorization incident. These conditions do not justify disabling event validation or failure capture.

Execution pause is controlled by changing event-replay.enabled to false in the config server and pushing the change to every hybrid-command and hybrid-query instance. Config-server change history provides the actor, timestamp, and change evidence. Pausing:

• makes the execute endpoint reject new execution requests with REPLAY_EXECUTION_PAUSED;
• prevents query from claiming a replay request for direct execution;
• does not interrupt an already-running database transaction unsafely;
• does not stop command-side event validation or canonical failure capture;
• does not hide candidates, plans, attempts, or status APIs;
• does not prevent creating or approving a plan;
• does not delete or alter durable replay state;
• does not require a service restart.

Requests with committed intent remain durable while paused; approved plans remain available only until their immutable expiry. An in-flight transaction finishes or rolls back under its existing database fence. Changing enabled back to true does not auto-resume anything: an operator explicitly retries the request. Removing gateway permission prevents new operator requests but does not alter durable state.

EventReplayConfig must be a reloadable light-4j module. Config reload clears the cached event-replay document, and command handlers read the current value before every execute transition, and query checks it again before claiming the named request. Like AuditConfig, it compares the cached configuration-map identity on each load()/current() access and rebuilds its immutable snapshot after ConfigReloadHandler clears the cache. No replay-specific callback, listener, or drain is required. A transition from false to true permits the next explicit operator retry. The effective execution state is exposed through health/status together with the config-server version or generation, reload timestamp, and process instance ID. The deployment health aggregator reports the expected replicas, their effective values and generations, and whether they agree. A multi-instance pause is complete only when that single fleet view reports every expected command/query replica at the same pushed generation with enabled=false; missing or stale replicas keep the pause state unconfirmed.

Shared Projection Transaction Executor

PostgreSQL live processing, Kafka live processing, and replay call one shared executor:

ProjectionResult execute(
    Connection connection,
    ProjectionTransaction transaction,
    ProjectionExecutionMode mode
) throws Exception;

ProjectionExecutionMode is LIVE or REPLAY. It may change telemetry and audit context but must not select a different projection handler. Exact replay passes the original CloudEvent. Repair replay passes a transaction materialized by the approved repair resolver before the executor is called; the executor itself never edits an event.

The executor:

1. validates transaction membership and event order;
2. calls PortalDbProvider.handleEvent(conn, event) for every member;
3. completes graph-revision and aggregate-version bookkeeping;
4. completes clone or other transactional projection outcomes;
5. records notification outcomes through the caller’s connection;
6. leaves commit or rollback to the caller.

The same transaction must contain projection writes, ordering metadata, replay attempt outcome, and failure resolution.

Canonical Failure Capture

Canonical capture applies only to failures observed after the feature is deployed. The legacy DLQ remains visible for diagnostics but is not queried by the replay candidate API and is never backfilled.

PostgreSQL processor

When a complete projection transaction fails:

1. roll back projection writes to the transaction savepoint;
2. construct a canonical envelope from the complete ordered transaction;
3. persist the failure transaction and all event members;
4. update the notification status to DLQ;
5. commit canonical failure capture and claimed source progress atomically;
6. continue with the next independent transaction.

If canonical capture fails, source progress does not advance. A failed event must not be committed past unless the payload needed for replay is durable.

The existing PostgreSQL dead_letter_queue write may remain temporarily for diagnostic compatibility, but replay correctness depends only on canonical failure capture for new failures.

Kafka processor

When a complete Kafka projection transaction fails:

1. roll back projection writes;
2. persist the canonical transaction with original keys, headers, topic, partition, offsets, transaction identity, count, and order;
3. commit the source Kafka offsets only after canonical persistence commits;
4. publish to the external Kafka DLQ independently when that integration is configured.

If canonical persistence fails, the source offset is not committed and Kafka redelivers the records. Capture is idempotent by content fingerprint.

This is intentionally an at-least-once boundary across PostgreSQL persistence and Kafka offset commit, not a distributed transaction. PostgreSQL capture must commit first; Kafka may redeliver after a crash, and the deterministic fingerprint must collapse that delivery into the existing failure. Reversing or parallelizing this order is forbidden because it can lose the only durable replay payload.

External Kafka producers that omit transaction count/order or required event metadata are rejected as executable replay candidates. The system does not guess transaction boundaries.

Canonical Failure Model

One canonical failure represents one complete logical transaction. It records:

• host, projection, and consumer group;
• original transaction ID and ordered member count;
• original source processor and coordinates;
• content fingerprint;
• dependency scopes derived from event metadata;
• bounded error code and message;
• first and latest failure timestamps;
• lifecycle status: OPEN, RESOLVED, or WAIVED.

Each ordered event member records:

• ordinal, event ID, and event type;
• aggregate identity and version when present;
• graph root and revision when present;
• original source coordinates, key, and headers when applicable;
• original payload or a durable payload reference;
• payload format and SHA-256 digest.

The content fingerprint is deterministic over projection identity, transaction identity, ordered event IDs, and ordered payload digests. A redelivery at a different source offset observes the same logical failure rather than creating another candidate.

Once an ordered failure is canonically captured, new commands for the affected aggregate or graph scope are blocked until exact replay or repair restores projection continuity. This bounds further accumulation behind a known poison event, but it cannot guarantee that N+1 is never appended: projection and capture are asynchronous, so commands may append during the interval between the original append and committed failure capture. The block is therefore a prompt, eventually-visible guard after capture, not a synchronous projection cursor.

Before classification, blocked commands return AGGREGATE_PROJECTION_BLOCKED with a safe failure reference. Once a validated repair proposal classifies the failure as invalid data, they return AGGREGATE_REPAIR_REQUIRED; the operator uses the repair flow instead of resubmitting through the stale projection UI. Waiver may close the operator action for diagnostic or unordered failures. A failure with an open AGGREGATE or GRAPH_ROOT scope is rejected with INVALID_REPLAY_STATE; it must be exact-replayed or repaired so the ordered projection gap is closed.

This is a deliberate consistency-over-availability decision. One failed ordered transaction can deny commands for that scope until a fix and exact replay or an approved repair succeeds. There is no generic break-glass that advances ordering metadata without applying the missing projection. Health and alerts report blocked-scope count, age, host, projection, and safe failure ID; crossing the reviewed duration threshold is an operator incident. The existing barrier release can recover worker isolation but cannot pretend an ordered gap is resolved or make later versions safe.

The command guard reads normalized OPEN rows in event_failure_scope_t using the partial command-path index. It matches host plus AGGREGATE (aggregateType:aggregateId) or GRAPH_ROOT scope, so another host, aggregate, or root continues normally. Before a repair proposal it returns AGGREGATE_PROJECTION_BLOCKED; once a validated proposal is awaiting approval or approved it returns AGGREGATE_REPAIR_REQUIRED. The administrative replay status reports the blocked-scope count, oldest blocked timestamp and age. The reviewed code default is 900 seconds; an oldest blocked scope at or above that age sets blockedScopeIncident=true. This threshold is intentionally an internal runtime default, not additional development configuration.

notification_t is the latest user-facing status, not the replay ledger. The candidate APIs read canonical failure tables only.

Payload Storage and Encryption

Application-level encryption is not required for replay correctness and is not mandatory in the early-development design.

Canonical and repaired payloads are stored as immutable bytes (BYTEA for the plain database representation). The SHA-256 content digest is computed over exactly those stored canonical bytes. Replay verifies and parses those same bytes; it never recomputes a digest from a JSONB value or re-serialized object.

event_store_t and outbox_message_t currently store JSONB, which normalizes representation and is not a stable raw-byte archive. A canonical failure may therefore reference those rows for identity and audit, but not as the sole digest-bound payload unless a future schema also stores the versioned canonical bytes. For current PostgreSQL capture, serialize through the versioned canonical JSON encoder once and copy the resulting bytes into the canonical member before source progress commits. Kafka capture stores the received value bytes. Repair creation likewise materializes and stores corrected canonical bytes once.

In either case:

• the payload is immutable after capture;
• a SHA-256 digest over the stored bytes is stored and verified before execution;
• the UI, list APIs, logs, metrics, and audit records never expose the payload, Kafka key, headers, or event JSON;
• the baseline schema revokes payload-column access from PUBLIC; production deployments use dedicated non-owner projection/replay roles with explicit column-scoped grants because owners and explicit table grants bypass that baseline;
• normal database and volume encryption at rest protect the development deployment.

Optional envelope encryption or object storage may be added for production when retention, PII, regulatory, or storage requirements justify it. Enabling that option must not change planning, fingerprints, ordering, or projection behavior, and its key configuration must not be required for ordinary development startup. A secure representation must retain the stable digest of the canonical plaintext bytes separately from any digest of randomized ciphertext or object-storage bytes; ciphertext digests are storage-integrity evidence and must never define the corrected transaction fingerprint.

Repair Model

Repair is an append-only amendment to a canonical failed transaction. It is not an update to event_store_t, outbox_message_t, a Kafka record, or the canonical failure payload. The original event remains available with its original digest for audit and diagnosis.

The minimum repair persistence model is:

• event_repair_t: repair ID, host, target failure, lifecycle status, reason, requester, approver, timestamps, original transaction fingerprint, and corrected transaction fingerprint;
• event_repair_event_t: repair ID, original event ID and ordinal, original digest, corrected data or durable reference, corrected digest, schema version, and the names of changed fields.

Repair lifecycle states are AWAITING_APPROVAL, APPROVED, APPLIED, CANCELLED, and REJECTED. Rows and corrected payloads become immutable when the proposal enters AWAITING_APPROVAL; a change requires a new repair ID and new approval.

Repair proposal contract

A repair proposal always targets one complete canonical failure transaction. It may correct the data of one or more members, but it preserves:

• event IDs, event types, transaction ID, count, order, host, and source coordinates;
• aggregate ID, aggregate type, and aggregate version;
• graph root and graph revision;
• unchanged transaction members byte-for-byte.

Editable fields come from an event-type-specific repair schema. The UI does not provide a generic CloudEvent or JSON editor. The server exposes only authorized, schema-approved business fields, applies field-level redaction, and revalidates the complete corrected transaction with the same schema and domain validators used by command append. Envelope, identity, authorization, and ordering fields are server controlled.

The repair command requires an explicit changeShape discriminator. With SINGLE_EVENT_FIELDS, changes is {field: value} and the complete transaction must have exactly one member for the requested repair schema. With PER_EVENT_FIELDS, changes is keyed by immutable event ID and each value is that member’s typed field object. The server never infers event scoping from whether a business-field value happens to be an object. This is not a raw JSON editor: every event ID must belong to the target transaction, every field must be declared by the pinned repair schema, and the server reconstructs the complete CloudEvent from its immutable canonical envelope. The metadata query returns event IDs, ordinals, digests, changed field names, actors, and lifecycle timestamps, but no original or corrected payload values.

R5 delivers the complete persistence, fingerprint, approval, and API framework, but its only executable repair-schema implementation is the isolated contract fixture (event-replay-contract-fixture-repair-v1). No production portal event is repairable merely because this framework exists. Concrete per-event typed schemas and validators are added to the versioned registry and proven through the UI/API flow by the R9 qualification gate; until then, non-fixture schema requests fail closed with REPAIR_SCHEMA_VALIDATION_FAILED.

Every event policy explicitly declares one repair disposition: SCHEMA_REPAIR, FIX_AND_EXACT_REPLAY_ONLY, or NOT_REPAIRABLE_UNORDERED. SCHEMA_REPAIR names a versioned repair schema and the registry coverage gate requires that schema to exist. FIX_AND_EXACT_REPLAY_ONLY is allowed for an ordered event only as an explicit decision: invalid external data keeps the scope blocked until a deployment makes the original event processable or adds a new repair-schema version. NOT_REPAIRABLE_UNORDERED cannot be used for an ordered policy. There is no implicit empty repair schema.

The proposal stores both payload digests and a bounded audit summary of changed field names. Payload values do not appear in audit messages, logs, metrics, or ordinary replay APIs. The requester cannot approve the repair.

Approval and rejection use one command endpoint with an explicit APPROVE or REJECT decision and an expected corrected-transaction fingerprint. The provider locks the immutable proposal and performs a compare-and-set from AWAITING_APPROVAL; approval records the independent reviewer, while rejection is terminal. There is no caller cancellation endpoint. A waiver or other valid terminal resolution cancels any AWAITING_APPROVAL or APPROVED repair in the same database transaction, and later repair reads reconcile a proposal if they observe that its target failure is already terminal.

Repair planning and execution

An approved repair is input to the planner, not output from it. A repair plan binds the repair ID, approval, original fingerprint, corrected fingerprint, schema version, dependency closure, and projection preconditions into the plan hash. Any change makes the plan stale.

The R5 loadApproved result is a verified snapshot for planning integration, not execution authority. R6 execution must lock the repair and target failure in the canonical failure-then-repair order, recheck both lifecycle states, and reverify original/corrected fingerprints and stored digests in the execution transaction before applying corrected bytes.

The replay worker installs the normal scope barrier and materializes the corrected transaction using the original immutable envelope plus the approved corrected data. The transaction executes at its original logical aggregate version or graph revision through the shared projection handler. This avoids trying to insert another (aggregate_id, aggregate_version) into event_store_t and avoids generating version N+1 while the projection is still at N-1.

Projection writes, ordering metadata, repair status APPLIED, replay attempt completion, and failure status RESOLVED with resolution code RESOLVED_BY_REPAIR commit atomically. Failure leaves the repair approved and retryable and keeps the scope quarantined. Waiver does not apply a repair and never advances projection metadata.

Approved repairs are permanent canonical history. Exact replay of that failure and any future projection rebuild must resolve the original event through the approved repair record and verify both fingerprints. A deployment that loses the repair tables cannot deterministically rebuild repaired projections and must fail closed rather than fall back to the poison payload.

The shared canonical repair resolver accepts both the initial APPROVED materialization and an APPLIED materialization bound by resolved_by_repair_id. Exact replay of an already repaired failure binds the applied repair ID and fingerprints into a new immutable plan, revalidates the original and corrected digests under the normal execution locks, and leaves the terminal repair/failure lifecycle unchanged after projection. Original failure members referenced by an applied repair are exempt from payload and failure metadata retention so unchanged transaction members remain available to this resolver. A future rebuild must call this same resolver rather than create a parallel correction mechanism.

Planning

The UI selects canonical failure transaction IDs. Selecting one member always selects its complete transaction.

The planner:

1. loads complete immutable failure transactions;
2. loads any explicitly selected, approved repairs;
3. verifies original and corrected payload digests and availability;
4. rejects excluded event types;
5. derives graph, aggregate, or transaction-only scopes;
6. adds required failed dependency transactions;
7. deterministically orders the dependency graph;
8. records projection preconditions and isolation scope;
9. creates an immutable plan hash and expiry.

Supported selection strategies are:

• EXACT: use exactly the selected complete transactions when no earlier dependency is missing;
• DEPENDENCY_CLOSURE: add required failed predecessors automatically.

There is no unbounded Replay All operation. Bulk selection is bounded by application defaults and always produces a preview before approval.

Execution rejects a stale plan when failure content, dependency state, projection versions, repair approval, schema version, or payload digests change after planning. The planner cannot accept inline corrected data.

Plan expiry continues after approval. APPROVED may transition to EXPIRED, and execute compares the current time with the immutable expiresAt before scheduling. Pausing execution does not extend the TTL; an expired approved plan requires a new plan and approval instead of executing unexpectedly after a long pause.

Approval and Authorization

Light Gateway is the role-based authorization boundary for all replay service IDs. Its endpoint rules map deployment-defined role names to the JWT role claim. Replay code does not hard-code admin, host-admin, replay-admin, or any other role name.

The minimum authorization model is one authorized role and two distinct users:

• user A creates the replay plan;
• user B, authorized for the same host, approves the exact plan hash;
• an authorized user requests execution after approval.

The early-development deployment therefore assumes that two test identities can be created in the host. There is no single-user or development-mode bypass: such a bypass would make the same artifact behave differently when promoted and would weaken the audit evidence this feature exists to provide.

Existing admin or host-admin roles may be assigned to every replay endpoint, or a deployment may create one dedicated role. host-admin remains host scoped; endpoint permission never bypasses token-host validation.

The built-in state machine records requester, approver, executor, reason, plan hash, and timestamps. It rejects requester-as-approver. It does not require light-workflow or create a manual task. A future workflow integration may drive the same transitions without weakening these invariants.

API Contract

Replay remains in the existing user-query and user-command services:

All request bodies are host-scoped and bounded. Host and actor identity come from trusted token/audit context, not caller-supplied authorization fields. The approve-repair command accepts APPROVE or REJECT. CANCELLED is a system transition when the target failure reaches another terminal outcome; there is no separate repair-cancel endpoint, so the public contract remains exactly twelve endpoints.

Waiver remains a two-person operation without adding a thirteenth endpoint. The requester first calls waiveEventReplayFailure with the exact failure IDs; the response is AWAITING_APPROVAL and includes a waiverRequestId plus the computed downstream impact. A different user approves by calling the same endpoint with that waiverRequestId, the exact failure IDs, and the expected downstream blocked failure IDs. Neither step advances projection metadata.

This is an intentional v2 narrowing of the former waiver surface. Deployments upgrading from v1 must not expect previously permitted ordered-failure waivers to remain available: any still-open ordered failure now requires exact replay or an approved repair.

V2 inheritance is closed, not catch-all. Only sections named in inheritsFrom.inheritedSections carry forward from v1. The shared LIVE/REPLAY execution modes, validation-mode semantics, Kafka DLQ evidence contract, replay policies, and failure/barrier/audit state remain inherited. V1 featureGates, mandatory encryption, required objectStore, operator-facing limits, and fixed retentionDays are explicitly superseded and must not be merged into v2.

Isolation and Execution

Replay must not race with newer live work for the same ordered scope.

Preferred barriers are:

• GRAPH_ROOT for graph-revision events;
• AGGREGATE for aggregate-version events;
• TRANSACTION_ONLY isolation when the transaction has no stronger ordering scope.

A complete transaction may touch several aggregates or graph roots. Planning derives the union of every member’s ordering scopes, checks dependency continuity for each scope, sorts the lock keys canonically, and acquires all scope locks before executing any member. A gap or exclusion in one scope makes the whole transaction non-executable; replay never applies the transaction to only the unaffected scopes. Live work intersecting any member scope is deferred as the same complete transaction. Canonical lock ordering prevents two cross-scope transactions from deadlocking each other.

Aggregate ordered-scope keys have one canonical encoding: aggregateType + ":" + aggregateId. Append validation uses the CloudEvent subject as aggregateId; canonical capture persists that same subject as event_failure_event_t.aggregate_id. Capture, dependency extraction, planning, barrier installation, and execution all call the shared encoder rather than reconstructing the key independently. This keeps the command guard and replay barrier byte-identical for the same aggregate.

The worker installs a fenced barrier, waits for current work in that scope to finish, and then applies the approved items through the shared executor. Live transactions intersecting the barrier are deferred as complete transactions; unrelated scopes continue normally. After repair, deferred transactions drain in source order before the barrier is removed.

Plain-payload deferred isolation

R6 closes the temporary R2 plain-codec limitation. The deferred table now has an exact-byte payload_plain representation whose SHA-256 digest and byte count are database constrained. A live transaction intersecting an active barrier is durably recorded as DEFERRED in the supported DATABASE_PLAIN representation before its source position advances. Unrelated later transactions can therefore continue, while deferred work still drains in source order before barrier release. Deferred bytes are immutable and direct column access is restricted in the same way as canonical plain failure bytes.

Replay requests use row locks, monotonic fencing tokens, leases, and advisory scope locks. A lease provides liveness and abandoned-work recovery; it never overrides a database lock or permits two workers to execute the same item.

Replay executes only from the explicit operator command. hybrid-command commits the INSTALLING_BARRIER intent, then calls the processEventReplay query service directly through HybridQueryClient with the same user bearer token. Query claims exactly (hostId, replayRequestId, planHash) and runs the existing fenced processor on that request thread. It never scans for unrelated eligible work.

There is no replay listener, polling scan, drain executor, automatic retry, or reserved listener connection. The outer browser request normally ends at the gateway’s five-second timeout while the internal command-to-query request keeps running. Event Admin resolves that ambiguity by polling the durable request every three seconds. An expired lease makes a request explicitly retryable; reenabling replay does not start it automatically.

Replay configuration follows the standard light-4j lazy reload contract used by AuditConfig. Command checks the circuit breaker before committing a new intent, and query checks it again before claiming the named request.

Direct replay operational status

Each hybrid-query replica exposes the following replica-local administrative endpoint:

GET /adm/event-replay/status

The endpoint returns application/json. It is an observation endpoint only: it does not list replay candidates, create or approve a plan, or start replay execution. Its purposes are to confirm that a replica loaded the expected execution-pause configuration, report direct-request mode, and expose blocked-scope incident state.

A healthy response has the following shape:

{
  "status": "UP",
  "effectiveEnabled": true,
  "configGeneration": "6b68d2...",
  "configReloadTimestamp": "2026-07-23T18:20:31.123Z",
  "processInstanceId": "019...",
  "executionMode": "DIRECT_REQUEST",
  "listenerConnectionRequirement": "NONE",
  "dedicatedListenerConnections": 0,
  "orderedScopeStatusAvailable": true,
  "blockedOrderedScopeCount": 0,
  "blockedScopeIncidentThresholdSeconds": 900,
  "blockedScopeIncident": false,
  "detail": "event replay executes only on an operator request"
}

The fields have these meanings:

For fleet-wide pause confirmation, an operator or aggregator polls every target query replica and verifies all of the following:

• every expected processInstanceId is represented by a fresh response;
• every response has effectiveEnabled=false;
• every response has the intended configGeneration; and
• there are no missing or stale replicas.

A missing or stale response never confirms a pause. The administrative status endpoint exposes no event payload or PII, but it reveals internal execution state, so the gateway or deployment ingress protects it with the same administrative access controls used for other /adm routes.

Request states are:

PLANNING -> READY -> AWAITING_APPROVAL -> APPROVED
         -> INSTALLING_BARRIER -> RUNNING -> SUCCEEDED
                                        \-> FAILED
READY/AWAITING_APPROVAL/APPROVED -> CANCELLED
READY/AWAITING_APPROVAL/APPROVED -> EXPIRED

Attempts are append-only. Success resolves the canonical failure in the same transaction as projection writes and attempt completion. A failed replay does not create a new DLQ loop; it records another attempt against the same failure.

Notification and UI

The Event Admin page provides:

• a list of open canonical replay candidates;
• transaction member count, event types, ordering scope, error, and failure time;
• explicit distinction between canonical candidates and legacy DLQ notifications;
• dependency-closure preview;
• plan hash, expiry, requester, approver, status, and attempts;
• approval, execution, cancellation, waiver, and quarantine controls according to gateway authorization.
• a separate Repair action when exact replay would repeat an invalid-data failure;
• a schema-driven repair form that exposes only authorized editable business fields and clearly states that the complete failed transaction is affected;
• repair status, changed field names, original/corrected digests, requester, approver, and linked replay plan without exposing payload values.

A legacy notification may remain visible in the lower notification table but must not be described as replayable. The empty candidate state explains that only newly captured canonical failures appear there.

Event Admin repair interaction

Event Admin presents two intentionally separate recovery paths. Replay original creates an exact/dependency replay plan after a processor defect is fixed. Repair creates an append-only amendment when replaying the persisted business data unchanged would fail again. Selecting Repair always identifies the complete failed transaction as the unit; the form may correct one or more members but cannot split transaction membership.

Repair forms are event-type and schema-version entries in Forms.json. Each entry mirrors the server repair schema and exposes only its declared business fields. The UI does not request original field values, so replacement inputs start blank. It has no generic JSON editor and cannot edit CloudEvent envelope, identity, ordering, key, header, or storage fields. A single repairable member uses SINGLE_EVENT_FIELDS; multiple compatible members use explicit event-ID-keyed PER_EVENT_FIELDS. The server still validates the pinned event policy, schema version, change shape, and every field; the UI schema is not a security boundary.

The form identifies which event type and how many transaction members are editable. All other members remain byte-identical while still participating in the complete transaction replay. If more than one form definition matches a mixed transaction, Event Admin fails closed instead of choosing by catalog order; the operator must use a server-supported unambiguous repair schema or fix the processor and replay the original transaction.

The initial supported product contract deliberately permits exactly one repair schema version per failed transaction. UserUpdatedEvent schema version 1 is the first deployed typed-repair policy and binds USER_UPDATED_V1 to user-updated-repair-v1. A transaction may contain multiple compatible members and use PER_EVENT_FIELDS; members with a non-repair disposition remain byte-identical. If members resolve to two different SCHEMA_REPAIR versions, both the provider and UI reject the proposal with REPAIR_SCHEMA_VALIDATION_FAILED. Supporting that case later requires a versioned multi-schema contract, approval fingerprint, persistence model, executor, and UI; catalog order is never a selection rule.

USER_UPDATED_V1 append validation mirrors the already-published user-command updateUserByIdRequest schema; it must not add UUID, length, or userType restrictions that the command contract does not enforce. The repair schema may be narrower for operator-entered replacement values and currently limits userType to C or E plus the documented per-field lengths. Email, entity ID, user ID, and host ID are deliberately not editable repair fields. Corruption in those identity-adjacent values is not a schema-repair case: the source must be corrected through an authorized domain workflow or quarantined for explicit reconciliation.

PORTAL_OBJECT_V1 is intentionally only a structural JSON-object validator. It cannot qualify an event for typed repair. Every deployed SCHEMA_REPAIR entry must instead name a concrete data validator, a concrete repair schema, an exact editable-field set, and a matching schema-driven Forms.json entry. The synthetic contract fixture remains in contract and test resources only; it is absent from runtime policy and form catalogs.

After creation, the browser discards replacement values and displays metadata only: repair state, changed field names, original/corrected transaction fingerprints, per-member original/corrected digests, reason, requester, reviewer/approver, timestamps, and linked replay request. repairId is a host-scoped URL/local-state key so a second user can open the proposal. The repair query resolves the latest linked replay request and status from durable plan-item metadata; linkage never depends solely on one browser’s state. The requester is shown that another authorized user must approve or reject it, but the UI never interprets role names. Gateway endpoint policy plus backend host, actor, fingerprint, state, and requester-not-approver checks authorize every operation.

Repair approval and replay-plan approval are independent. Only an APPROVED repair can create a repair-bound replay plan; the resulting immutable plan hash must then be approved through the normal replay workflow before execution. CANCELLED, REJECTED, STALE_PLAN, and REPAIR_FINGERPRINT_MISMATCH are terminal for the displayed artifact and tell the operator to refresh/re-plan rather than blindly retry.

When a replay reports SUCCEEDED with projectionCommitted=true, Event Admin refreshes canonical candidates, repair metadata, and the legacy notification table. It also emits portal:event-replay-applied with only hostId, replayRequestId, and optional repairId; a mounted business view uses that event to invalidate and reload its projection query. No payload or changed field value is included in the event.

Raw payloads, complete event JSON, Kafka keys, headers, and database payload references are never returned to the browser. A repair endpoint may return only the explicitly authorized and redacted business fields declared by the repair schema.

Failure Handling

• Invalid command-side transaction: reject the command before event-store or outbox commit.
• Persisted event has permanently invalid data: once a validated repair proposal classifies the failure, reject exact planning with EVENT_REPAIR_REQUIRED; block later commands in the ordered scope with AGGREGATE_REPAIR_REQUIRED; require an approved repair.
• External Kafka transaction is incomplete: do not create an executable candidate and do not commit the source offset when canonical evidence cannot be persisted safely.
• Excluded unordered event type: retain diagnostic failure metadata and report EVENT_NOT_REPLAYABLE during planning. Reject ordered exclusions at policy/configuration validation.
• Payload missing or digest mismatch: reject planning or execution with PAYLOAD_UNAVAILABLE or PAYLOAD_DIGEST_MISMATCH.
• Dependency gap: report the exact missing aggregate version, graph revision, or transaction and keep the plan non-executable.
• Query process crashes before commit: database rollback leaves the item pending; after lease expiry an operator retry starts a new fenced attempt.
• Query process crashes after commit: projection result and attempt outcome are already atomic; durable status observes completion.
• Replay handler still fails: stop the ordered plan, record the attempt, and keep the affected scope quarantined.
• Plan expires or becomes stale: require a new plan and approval.
• Repair fails validation: store no executable repair and report bounded field errors without changing the original failure.
• Repair execution fails: retain the approved repair, record the attempt, and keep the ordered scope quarantined for retry or cancellation.
• Emergency execution pause: push event-replay.enabled=false to all command and query instances, verify their effective health/status, and stop new execute transitions and direct query claims while validation, capture, query APIs, planning, approval, and durable state remain available.

Security and Privacy

• Light Gateway authorizes every replay endpoint.
• Every query and mutation validates token host against the requested host.
• Requester and approver must be distinct users.
• Repair requester and repair approver must also be distinct users; approval of a replay plan does not implicitly approve a repair.
• The reason and immutable plan hash are audited.
• Payload digests are verified immediately before projection execution.
• Repair APIs enforce event-type-specific editable fields and never accept caller-controlled envelope, tenant, identity, or ordering metadata.
• Payloads, keys, headers, event JSON, and direct storage locations never appear in API lists, browser state, logs, metrics, or audit details.
• Canonical schema and upgrade assets revoke every plain payload column from PUBLIC. This is the current repository-managed baseline; table owners and roles with explicit table grants remain privileged.
• Dedicated non-owner database roles and column-scoped grants are production hardening, not part of the early-development activation contract. A production deployment that requires database-level service separation must provision those roles and credentials outside this schema before promotion.
• Application-level encryption is an optional production hardening control, not a prerequisite for development replay.

Observability

Recommended bounded-cardinality metrics are:

• canonical failures captured, open, resolved, repaired, and waived;
• capture failures by safe error code;
• replay plans and attempts by status;
• repair proposals and executions by safe status and event type;
• replay transaction and event counts;
• planning, approval-wait, execution, and barrier duration;
• stale plans, dependency gaps, excluded events, and payload mismatches;
• blocked ordered scopes by age bucket and safe failure code;
• capture rate, stored payload bytes, capacity-watermark state, and source backpressure activations;
• active barriers, deferred transactions, quarantined scopes, and abandoned attempts;
• direct execution attempts, expired leases, and fencing failures;
• active barriers, oldest blocked-scope age, and operator retries.

Logs include request ID, failure ID, projection, consumer group, attempt, counts, and safe result code. They exclude payloads and unbounded exception messages.

Always-on capture uses reviewed internal soft/hard capacity defaults even though they are not development-facing configuration. A failure storm crossing the soft threshold raises health/alerts. At the hard threshold, the affected source stops before advancing past a failure whose replay bytes cannot be stored; it does not discard payloads or silently downgrade to legacy DLQ-only behavior. Blocked-scope age and failure-rate alerts make the resulting availability impact visible during a bad deployment.

Deployment Behavior

The schema migration creates the canonical failure, repair, replay request, item, attempt, lease, barrier, deferred-work, and audit tables. After both the schema and updated services are present:

1. replay validation, capture, APIs, and direct execution are active;
2. hybrid-query starts canonical capture without a replay execution thread;
3. hybrid-command accepts plan and state-transition commands;
4. new failed transactions appear in Event Admin;
5. gateway endpoint roles determine who may operate them.

user-command and user-query are source repositories/modules; hybrid-command and hybrid-query are their deployed service bundles. This document uses the module names for code ownership and the hybrid names for runtime instances.

Deployment order remains schema, shared light-portal artifacts, hybrid-command, then hybrid-query. Command-side schema validation is active when command deploys, but the open-failure scope block is inert until the query deployment is capturing canonical failures. Operators must not claim the block is fleet-effective until capture health is green on every query replica.

Repair tables are required replay history, not an optional UI feature. Startup must verify them together with the canonical failure and replay tables.

event-replay.enabled defaults to true. Config-server reload applies changes without restart. hybrid-command enforces the current value at execute time; every hybrid-query direct request enforces it before claiming work. Health/status reports the effective value for each instance.

event_replay_ready remains harmless schema history but has no runtime producer or listener. A committed non-terminal intent is resumed only by an explicit operator retry after its lease expires.

Startup fails with a clear schema error when required replay tables are absent. It must not silently downgrade to a partially working mode.

There is no rollout table, rollout stage, source/host allowlist, legacy backfill job, or requirement to enable capture, planning, and execution separately.

The R8 convergence migration removes the former rollout-audit and legacy backfill checkpoint/issue tables from upgraded databases and fresh-install DDL. The old dated Phase 11 migration remains in source as historical evidence only; it is followed by the repeatable R8 removal migration and is not a supported runtime or installation surface. The compiled runtime likewise has no backfill entry point, rollout model, change-ticket check, PKCS12 key provider, encrypted payload codec, object-store client, or configurable worker identity. Existing encrypted/object payload columns remain schema history, but the runtime accepts only exact-byte DATABASE_PLAIN payloads. Those columns are retained solely as a non-destructive upgrade and rollback boundary; they are not a reserved or advertised future storage mode, and rows using DATABASE or OBJECT are not executable by the R8 runtime. During R9, deployment qualification must count every historical non-plain row. A later destructive migration may remove the columns and dead constraint branches only after that count is zero across every deployment and the pre-R8 rollback window has closed.

External Kafka DLQ compatibility remains available independently of replay execution. Kafka-source failures still publish through the normally configured Kafka producer to the code-owned topic template, retry, age, and acknowledgement defaults in EventReplayRuntimePolicy.KafkaDlq; no retired replay rollout, encryption, or object-store property controls it. PostgreSQL dead_letter_queue diagnostics and notification status also remain. The disabled durable external publication-outbox experiment is not required for canonical replay correctness; canonical capture remains authoritative.

Validation Plan

Append validation

• Reject missing transaction IDs, duplicate ordinals, inconsistent counts, non-contiguous membership, and cross-host members.
• Reject aggregate and graph events missing their required ordering metadata.
• Reject event data that fails its registered event-type/schema-version validator, and prove no event-store, outbox, notification, or failure row is committed.
• Prove event_store_t and outbox_message_t commit atomically.
• Prove a newly registered projection handler declares or derives a replay policy and satisfies the idempotency contract.
• Prove every policy declares a repair disposition and every SCHEMA_REPAIR policy resolves to a versioned schema.
• Prove a pinned registry/schema version remains usable for an already-appended event after a newer version is deployed, and referenced versions cannot be removed.
• Reject ordered NOT_REPLAYABLE policies and configuration that excludes a graph- or aggregate-ordered event.
• Smoke-test registry misconfiguration and prove portal/internal appends fail closed without reserving offsets or writing partial rows.

Failure capture

• Fail one PostgreSQL transaction and verify one complete canonical candidate is committed with the notification and source progress.
• Fail canonical persistence and prove PostgreSQL or Kafka source progress does not advance.
• Redeliver the same transaction and verify idempotent failure observation.
• Crash after PostgreSQL Kafka-failure capture commits but before Kafka offset commit; verify redelivery resolves to the same fingerprint and failure row.
• Round-trip equivalent JSON through JSONB with different key ordering and whitespace; verify replay still hashes the unchanged canonical BYTEA, not the re-serialized JSONB.
• Verify historical legacy DLQ rows are ignored by candidate queries.

Policy and planning

• Verify graph events derive GRAPH_ROOT.
• Verify events such as UserDeletedEvent derive AGGREGATE_VERSION.
• Verify ordinary complete unordered transactions derive TRANSACTION_ONLY.
• Verify an exact excluded unordered event type is rejected as EVENT_NOT_REPLAYABLE, and ordered exclusions are rejected at reload.
• Select one member and verify the complete transaction is planned.
• Verify dependency closure, deterministic order, stale-plan rejection, and payload-digest enforcement.
• Approve a plan, let its immutable TTL expire, and verify execute rejects it; pausing execution must not extend the TTL.
• Verify inline corrected data is rejected and only an approved immutable repair ID can change the materialized replay input.

Repair

• Fail an aggregate event at version N because of invalid data while its projection remains at N-1; after canonical capture commits, verify ordinary submission is blocked with AGGREGATE_REPAIR_REQUIRED. Separately race a command with asynchronous capture and document that an N+1 append may win before the block becomes visible.
• Verify a repair cannot change envelope, identity, transaction membership, aggregate version, graph revision, or fields absent from the repair schema.
• Verify original and corrected digests, changed field names, reason, requester, and distinct approver are durable while payload values remain out of API, audit, log, and metric output.
• Apply an approved repair and verify projection writes, ordering metadata, repair status, attempt completion, and failure status RESOLVED with resolution code RESOLVED_BY_REPAIR commit atomically.
• Re-run exact replay and a projection rebuild and verify both resolve the approved repair deterministically instead of processing the poison payload.
• Verify waiver does not apply the repair or advance projection metadata.
• Verify waiver cannot unblock an ordered scope while its projection metadata still has a version or revision gap.

Execution and concurrency

• Verify an intersecting barrier is durably DEFERRED with exact plain bytes before source progress and drains in source order after barrier release.
• Process the same transaction through LIVE and REPLAY and compare every projection and ordering row.
• Verify requester/approver separation and plan-hash binding.
• Verify unrelated scopes continue while a graph or aggregate barrier is active.
• Replay a complete transaction spanning multiple aggregate/root scopes; verify canonical lock ordering and that a gap in one scope prevents every member from executing.
• Verify deferred work drains in source order.
• Kill a direct query request before and after commit and prove one committed outcome; retry is unavailable until the lease expires.
• Invoke two direct requests for the same plan and prove fencing permits one active execution and rejects late stale writes.
• Leave the system idle and verify there is no replay listener, polling scan, execution executor, or permanently borrowed replay connection.
• Push event-replay.enabled=false and verify every command/query instance reports the effective value, config generation, reload timestamp, and instance ID without restart; verify the fleet aggregator refuses to confirm pause while a replica is missing or stale.
• Verify the execute endpoint returns REPLAY_EXECUTION_PAUSED and query stops new direct claims without stopping capture, planning, approval, status, or an in-flight transaction.
• Push event-replay.enabled=true and verify no work resumes until the operator retries it.
• Cross the internal capture soft and hard watermarks; verify alerting and that source progress stops before an uncaptured replay payload is lost.

Security and UI

• Verify gateway roles can be configured independently for all twelve endpoints.
• Verify a host-scoped user cannot inspect or mutate another host.
• Verify no payload, event JSON, key, header, or storage reference reaches the browser, log, metric, or audit record.
• Verify legacy DLQ notifications and canonical candidates have unambiguous UI wording.

Qualification and Deployment

Deploy in dependency order: portal-db, light-portal, user-command, user-query, gateway endpoint policy, portal-view, and finally the config mirrors. Run the R9 gate against a disposable PostgreSQL database before each environment promotion. The gate composes R0-R8, exercises the database and Kafka-source semantic paths, verifies the real repair policy and UI form, and builds the command, query, UI, and documentation artifacts.

Before promotion, run the read-only storage inventory against every target database. DATABASE_PLAIN and tombstoned DELETED failure rows are supported; deferred and corrected repair rows must be DATABASE_PLAIN. Any historical DATABASE or OBJECT row is recorded as non-executable rollback-boundary evidence. The inventory exits non-zero when such a row exists, does not rewrite it automatically, and the legacy columns must not be dropped until the count is zero everywhere and the pre-R8 rollback window is closed.

For PostgreSQL pub/sub, qualification proves capture, planning, approval, barrier installation, repair-aware execution, resolution, and deferred drain. For Kafka pub/sub, it additionally proves validation before projection, canonical capture before source-offset commit, idempotent redelivery, retained source coordinates, no live-topic republish, and no consumer-offset rewind. Gateway rules remain deployment-defined independently for all thirteen replay endpoints, with two distinct authenticated users used for repair approval and replay-plan approval.

The deployable evidence and exact commands are maintained in the R9 qualification record under the implementation repository. /adm/event-replay/status is polled on every query replica after deployment to confirm direct execution mode, effective execution state, config generation, reload timestamp, and instance identity; a missing or stale replica never confirms a fleet pause.

Future Production Hardening

The following may be added later as advanced capabilities without changing the core replay contract:

• application-level envelope encryption and key rotation;
• immutable object storage and payload lifecycle policies;
• configurable retention and legal holds;
• deployment-specific non-owner database roles and column-scoped payload grants;
• operator tuning of the mandatory baseline failure-storm capacity controls;
• staged rollout and canary scopes for an existing production deployment;
• legacy migration tooling if historical replay becomes a requirement;
• rollback dry-run support for handlers proven safe for transactional dry run;
• light-workflow manual approval tasks;
• production-specific limits and break-glass policy.

The non-negotiable contracts are complete transaction membership, deterministic ordering, immutable original and repair payload digests, code-level event validation, a planner that cannot edit data, shared live and replay projection behavior, host-scoped authorization, distinct-user approval, database-enforced fencing, durable repair history, and durable failure capture before source progress.