Event-driven by default: design for the failure modes first

In an event-driven system on AWS, the failure modes — duplicate delivery, out-of-order events, downstream outages, poison messages — are not edge cases. They are the normal operating condition. Designing for them first, before the happy path, is what separates a serverless architecture that survives production from a demo that falls over under load.

The problem

The appeal of event-driven serverless — Lambda, SQS, SNS, EventBridge — is real: decoupled services and elastic scale, almost for free. The trap is that the happy path is trivial to build and demos beautifully, so teams build it first and file resilience under "we'll add error handling later."

But event-driven systems fail differently from request/response. At-least-once delivery means every consumer will see the same message twice — not rarely, routinely. Async fan-out means events arrive out of order relative to how they were produced. A downstream outage doesn't return a clean 500; it makes messages pile up until a naive consumer either drops them or reprocesses them into corruption. And a single malformed event can wedge a consumer that retries it forever. Under real load these aren't tail risks — they happen every day, and "later" arrives as an incident.

What we did

The decision that matters is design for the failure modes first, then make the happy path fast — the same principle behind our cloud architecture work. In practice that means four things are non-negotiable before the first consumer ships:

• Idempotency as a first-class requirement. Every event carries a stable id, and every consumer is keyed on it, so a re-delivered message is a no-op instead of a double-charge.
• A dead-letter queue on every queue and subscription, from day one, with a redrive path. A message that can't be processed parks for inspection instead of blocking the stream or vanishing silently.
• Order-tolerance by default; strict ordering only where the domain demands it. Global ordering is expensive (FIFO throughput limits, partition keys) and usually unnecessary — so most handlers are written to tolerate any order, and ordering is spent only where correctness truly needs it.
• Bounded retries with backoff, then DLQ. One poison message can't pin the consumer, and DLQ depth is alarmed so a growing backlog pages someone.

The alternative we explicitly rejected was happy-path first, resilience later. Retrofitting idempotency and dead-lettering after the consumers and the message schema already exist means touching every consumer — and usually the event contract itself. It is far more expensive than designing the event envelope (id, timestamp, version) up front, and it tends to get prioritized only after the first data-corruption incident. This is the kind of trade-off we settle in the design phase, before code, on the systems we've shipped.

The result

The artifact is the mapping we design against: each failure mode has a why and a standard response, decided once and applied everywhere.

Those responses aren't bolted onto the topology afterward; they are the topology. The producer publishes to a bus, each consumer has its own queue with a dead-letter queue behind it, and the redrive path is wired before any business logic exists.

Takeaway

In event-driven systems, resilience is not a layer you add — it is the shape of the design. Decide idempotency, dead-lettering, and ordering before the first consumer ships, and the happy path becomes the easy part.