Arrival order is not business order
Distributed systems rarely deliver events in the order they were created. Network paths differ, queues retry independently, workers scale out, and a provider may batch or replay events after an outage. A subscription cancellation can arrive before the subscription creation event. A profile update from yesterday can arrive after a newer update from today. If a consumer treats arrival order as truth, a late message can move a resource backward in time and silently restore stale state.
The first design question is whether order matters for the business decision. Some events are independent facts and can be applied in any order. Others describe a state transition and require a version, sequence number, or source timestamp. Do not add a global ordering queue when only one customer or aggregate needs ordering. Define the smallest ordering key that matches the business object, then enforce it at that boundary.
Carry a monotonic version when you can
The cleanest ordering signal is a monotonic version supplied by the source for each aggregate. Store the last accepted version and reject or quarantine events with a lower version. If the source provides an event creation timestamp but not a sequence, use it carefully. Clocks can drift, timestamps can share the same precision, and a producer may update a record without updating the event time. A timestamp is a useful hint, not always a strict order.
If the source gives only an opaque event ID, you may not be able to prove order. Model the consumer so an event is a fact that can be reconciled rather than a command that assumes previous state. Fetch current source state when the event indicates a change, or store all events and derive a projection that can be rebuilt. A projection that cannot be rebuilt is a hidden dependency on delivery order.
Separate the event log from the projection
An event log records what the source told you; a projection records the convenient current view your application uses. Keeping those concepts separate gives you a recovery path. You can reprocess historical events with a corrected handler, compare a projection with the source, and explain why a value changed. The log should be append-oriented and deduplicated by event identity. The projection can be updated, indexed, and optimized for reads.
This does not require a large event-sourcing platform. A relational table with an event ID, aggregate ID, type, version, received time, source time, raw or redacted payload, and processing state can be enough. Add a unique constraint on the source event identity and indexes for the aggregate and processing state. Make retention explicit. If raw payloads contain sensitive data, keep a minimal event envelope after the raw body expires.
Handle late events intentionally
A late event can be ignored, applied, quarantined, or used to trigger a reconciliation. The right choice depends on whether the event is a complete state snapshot or a partial change. A snapshot with version 8 can replace version 7. A partial change that says a single field changed may need to be merged, and merging it after a newer update can be unsafe. Document the policy for each event type instead of applying one global rule.
Quarantine is valuable when the consumer cannot decide safely. Store the event with a reason, expose it to operators, and provide a replay path after the source state is understood. Do not silently discard events because they are inconvenient. A discarded event is a data-loss decision. If you intentionally ignore a stale event, record that decision and its version so an audit can distinguish it from a delivery failure.
Replay from a known starting point
A replay needs a boundary. You might replay one event, all events for an aggregate, a time range, or the entire projection. Each option has different risk. Replaying one event is fast but may rely on state that has changed. Replaying an aggregate rebuilds context but can be expensive. A full projection rebuild gives the strongest correctness story but requires a safe way to serve reads while the rebuild runs.
Make replay idempotent and observable. Show the selected range, number of events, estimated work, and destination before starting. Record the replay job, code version, operator, and outcome. Use a new projection or shadow table when possible, compare results, then switch reads atomically. A replay tool should never call an external side effect by default. If an event needs a side effect, model that action separately and require an explicit, audited choice.
Design for concurrency and leases
Multiple workers should be able to claim different events without processing the same event concurrently. A short lease with an owner ID and expiration gives a worker time to process while allowing recovery after a crash. Claim records with an atomic database operation and update the lease heartbeat only when needed. Avoid holding a database transaction open while waiting on an external API. Persist the claim, perform bounded work, and write the result with a conditional state transition.
A late event can race with a current event. Compare the event version in the same operation that updates the projection. If the incoming version is lower, record it as stale rather than overwriting the current view. If two events have the same version, use a deterministic tie-breaker or quarantine them. The goal is not to eliminate concurrency; it is to make concurrent outcomes explicit and repeatable.
Observe the shape of disorder
Measure more than processing throughput. Track duplicate rate, stale-event rate, maximum event age, version gaps, quarantine count, replay duration, and time from source creation to receipt. These metrics tell you whether the problem is occasional network jitter or a systemic ordering failure. A rising maximum event age may require a provider support ticket even when your queue is healthy.
Include aggregate ID and event version in diagnostic views, but redact customer data and payload secrets. An operator should be able to answer which version is current, which events are waiting, and whether a stale event was intentionally ignored. A payload inspector helps establish what arrived; the event log and projection explain what your system did with it. Together they make the pipeline debuggable without relying on memory.
Implementation example
Persist an event ID, source sequence, aggregate ID, and received time before handing work to consumers. Consumers should advance a durable cursor only after applying an event and should retain enough history to rebuild an aggregate. If a source cannot guarantee order, make the state transition conditional on the version it expects and route gaps to a replay queue.
select event_id, aggregate_version, payload
from events
where aggregate_id = $1 and aggregate_version > $2
order by aggregate_version asc;Verify and troubleshoot
Inject ordered, duplicated, delayed, and missing events into a test stream. Assert that a late event is held or reconciled instead of silently overwriting newer state. Measure cursor age, gap count, replay latency, duplicate rate, and the number of aggregates requiring manual repair. A consumer that reports success while its cursor skips a version is not healthy.
Operations and recovery
Keep a bounded event archive and a documented replay command that can target one aggregate or time range. Quarantine poison events with their failure reason and preserve the original bytes. During recovery, stop downstream side effects if ordering is uncertain, rebuild a projection from the source log, and compare the rebuilt state with the live projection before resuming writes.
References and further reading
Compare the design with event-sourcing, transactional outbox, and stream-processing documentation for the chosen broker. Define whether ordering is per key, partition, tenant, or entire stream; that boundary is part of the API contract.