Event-Driven Architecture for E-Commerce: Kafka & Webhooks

1. The Refund That Went Out Four Times

A flooring retailer I look after refunded the same £340 order four times on a Tuesday morning in March. The customer, to their credit, phoned to say so.

The mechanism was ordinary. Their order service published a RefundApproved message to SNS. A consumer picked it up and called Stripe. Stripe's API responded slowly that morning — around 22 seconds, well past the consumer's 15-second HTTP timeout — so the consumer threw, the message went back to the queue, and thirty seconds later another instance picked it up and called Stripe again. Four times in total before someone noticed the alarm. Every one of those calls succeeded at Stripe's end. The consumer never saw any of the responses.

Nobody wrote a bug. Every component did exactly what it was configured to do. SQS delivered at least once, as documented. The consumer retried on failure, as designed. Stripe processed each request, as asked. The system as a whole did something nobody wanted, and no single piece of it was wrong.

That is the characteristic failure of event-driven architecture, and it is why I get slightly irritated by material that presents it as a way to decouple services and stops there. Decoupling is what you buy. What you pay is that correctness stops being a property of code you can read in one place and becomes a property of interactions you have to reason about deliberately.

I have caused worse. On a distributor's build in 2021 I wrote a stock-adjustment consumer with no idempotency key at all, on the reasoning that the broker was configured for exactly-once and therefore I did not need one. A broker upgrade six months later changed a default, redelivery started happening, and inventory drifted by a few units a day for eleven days before the warehouse count caught it. Reconciling that against physical stock took two people the better part of a week.

So this piece is about the parts that actually bite. What separates an event from a command and why getting that wrong couples your services back together. What "at least once" really obliges you to build. The dual-write problem, which is the single most common correctness bug in ecommerce integrations and which almost nobody names. Schema evolution. And debugging a system that has no call stack, which is the part that makes engineers who are good at monoliths feel suddenly incompetent.

2. Events Versus Commands, and Why It Matters

Two messages, superficially similar, with opposite coupling properties.

A command is an instruction to a specific recipient to do a specific thing. ReserveStock. SendConfirmationEmail. ChargeCard. The sender knows who will handle it, knows what should happen, and generally wants to know whether it worked. There is exactly one logical handler.

An event is a statement that something has already happened. OrderPlaced. PaymentCaptured. StockLevelChanged. Past tense, factual, and the publisher has no opinion about who cares. Zero handlers is a valid outcome. So is nine.

The distinction is not pedantry. It determines who owns a business rule.

If the order service publishes SendConfirmationEmail, then the order service has decided that placing an order causes an email. Add SMS next quarter and you edit the order service. Add a loyalty accrual and you edit it again. The order service slowly becomes a list of every side effect in the business, and you have a monolith with a message broker in the middle of it, which is strictly worse than a monolith.

If the order service publishes OrderPlaced and the notification service subscribes, then adding SMS means deploying the notification service and nothing else. That is the decoupling people came for.

My rule: commands go point-to-point on a queue and may expect a reply; events go to a topic, fan out, and never expect one. Mixing those is where most trouble starts.

// COMMAND. One handler. Imperative. The sender cares about the result.
{
  "type": "ReserveStock",
  "target": "inventory-service",
  "commandId": "cmd_01HQ8V3XKPM",
  "payload": { "orderId": "ord_88213", "lines": [{ "sku": "PL-4402", "qty": 2 }] },
  "replyTo": "order-service-replies"
}

// EVENT. Past tense. No target, no replyTo. Anyone may listen.
{
  "type": "StockReserved",
  "eventId": "evt_01HQ8V4B2NW",
  "occurredAt": "2026-03-17T09:41:22.318Z",
  "aggregate": { "type": "Order", "id": "ord_88213" },
  "version": 3,
  "payload": { "orderId": "ord_88213", "lines": [{ "sku": "PL-4402", "qty": 2 }] }
}

Two fields in the event are worth pointing at. eventId is what makes consumers idempotent. version is what lets a consumer detect that it has missed something or received things out of order. Neither is optional and both are routinely omitted in the first version of a system, then added painfully later.

3. Naming, Which Is Half the Design

Event names are the public interface of your system. They outlive the code and they are extremely hard to change once three teams depend on them.

Past tense, always. OrderPlaced, not PlaceOrder and not OrderPlacing. If a name reads as an instruction, you have written a command and mislabelled it.

Name the business fact, not the technical operation. CustomerAddressChanged is a business fact. CustomerRowUpdated is a database detail leaking into your contract, and it will force every consumer to understand your schema.

Be specific enough to be useful. I have seen a system with a single OrderUpdated event carrying a changeType field, which meant every consumer subscribed to everything and immediately discarded 90% of it. Split it: OrderPaid, OrderShipped, OrderCancelled, OrderAddressCorrected. Routing then happens in the broker rather than in eleven consumers.

Include the aggregate identity in the name where ambiguity is possible. StockReserved and StockAdjusted are different facts with different consumers, even though both change the same number.

One convention I now insist on: a version number in the type itself once you are past the first release. order.placed.v2. It looks ugly. It also means a breaking change is a new topic rather than a coordinated deployment across five teams, which is the difference between a Tuesday and a fortnight.

4. Thin Events, Fat Events, and the Trade You Are Making

How much data goes in the payload is the design argument I have most often, and both sides are defensible.

A thin event carries identifiers and nothing else: { orderId: "ord_88213" }. Consumers call back to the source for detail. Small messages, no stale data, and a single source of truth.

The cost is that every event produces N callbacks. Publish OrderPlaced to six consumers and your order API takes six extra requests, synchronously, at exactly the moment it is busiest. I have watched that pattern turn a Black Friday order spike into an outage of the order service, caused entirely by its own consumers.

A fat event carries everything a consumer might plausibly need. No callbacks, consumers work while the source is down, and replaying old events gives you the data as it was at the time — which matters enormously for anything financial.

The cost is coupling to the payload shape, larger messages, and the awkward question of what happens when the data changes after publication.

What I do in practice, and would defend: fat enough to act, thin enough to stay true. Include the fields that were part of the fact at the moment it happened — the prices charged, the quantities ordered, the address it shipped to — because those are immutable properties of the event. Exclude fields that are current-state and may drift, like the customer's marketing preferences or their loyalty tier. For those, carry the id and let the consumer look them up.

{
  "type": "order.placed.v2",
  "eventId": "evt_01HQ8V4B2NW",
  "occurredAt": "2026-03-17T09:41:22.318Z",
  "payload": {
    "orderId": "ord_88213",
    "customerId": "cus_5512",

    // Immutable facts about this order. Safe to embed: they cannot
    // change without a new event being emitted.
    "currency": "GBP",
    "totals": { "goods": 28400, "shipping": 495, "tax": 5779, "grand": 34674 },
    "lines": [
      { "sku": "PL-4402", "qty": 2, "unitPrice": 14200, "taxRate": 0.20 }
    ],
    "shipTo": { "postcode": "BS1 4XE", "country": "GB" }

    // NOT embedded: customer.email, customer.loyaltyTier, product.stock.
    // Those are current state. A consumer that needs them fetches them,
    // and gets the value that is true when it acts, not when we published.
  }
}

Note that money is in minor units as integers. Floating-point currency in event payloads is a bug that will find you eventually, usually as a penny discrepancy in a reconciliation report six months after anyone remembers writing the code.

5. Delivery Guarantees, Stated Honestly

Three phrases get used loosely and the difference between them is the difference between a working system and the refund story above.

At most once. Fire and forget. Messages can be lost. Acceptable for analytics pings and nothing else in ecommerce.

At least once. The message will arrive, possibly more than once. This is what SQS, SNS, EventBridge, Kafka in its default configuration, and every webhook you will ever receive actually give you. It is what you should assume in all circumstances.

Exactly once. Does not exist across a network boundary in the general case. What Kafka calls exactly-once semantics is real, but it applies within Kafka — a transactional read-process-write where source and sink are both Kafka topics. The moment your consumer calls Stripe, or writes to MySQL, or sends an email, you are back to at-least-once and no broker setting changes that.

I want to be blunt about this because it is oversold. If a vendor tells you their queue gives you exactly-once delivery to an arbitrary external system, they are describing at-least-once delivery plus deduplication that they are performing on your behalf, and you should ask exactly where the dedupe window ends. SQS FIFO, for example, deduplicates within a five-minute window. Redelivery after six minutes is not deduplicated, and a consumer that took seven minutes to time out will see the message again.

The practical consequence is one sentence long: your consumers must be idempotent, and no broker configuration relieves you of that.

6. Ordering Is a Harder Problem Than Delivery

Delivery you can solve with idempotency. Ordering you often cannot solve at all, and the honest approach is to design so that you do not need it.

Consider a stock adjustment sequence: set to 100, decrement 2, set to 95. Applied in order you end at 95. Applied as set-95 then decrement-2 then set-100 you end at 100. Both orderings are possible on a topic with multiple partitions and parallel consumers, and the second one has your warehouse shipping stock you do not have.

There are three approaches and I use all three in different places.

Partition by key

Kafka guarantees ordering within a partition. Publish all events for a given SKU to the same partition by using the SKU as the partition key and you get ordering where it matters, with parallelism across keys.

This works well and has one trap: rebalancing. When a consumer joins or leaves the group, partitions get reassigned, and there is a window where the new owner can start processing while the old owner has an in-flight message. Ordering within a partition does not save you across a rebalance. Commit offsets carefully, and prefer cooperative sticky assignment which reduces but does not eliminate the window.

SQS FIFO does the same thing with MessageGroupId, at a throughput ceiling of 300 messages per second per group without batching, 3,000 with. For per-SKU ordering that is almost always plenty.

Version numbers and rejection

Every event carries a monotonically increasing version for its aggregate. The consumer stores the last version it applied and discards anything lower. Simple, works across any transport, and it turns an ordering problem into a deduplication problem you have already solved.

-- Apply an event only if it is newer than what we have already
-- applied for this aggregate. Out-of-order arrivals become no-ops.
INSERT INTO order_read_model (order_id, status, last_version, updated_at)
VALUES (?, ?, ?, NOW())
ON DUPLICATE KEY UPDATE
  status       = IF(VALUES(last_version) > last_version, VALUES(status), status),
  updated_at   = IF(VALUES(last_version) > last_version, NOW(), updated_at),
  last_version = GREATEST(last_version, VALUES(last_version));
-- One statement, atomic, no read-modify-write race between consumers.

Design the operations to commute

The best answer, when you can get it. If every stock event is a relative delta rather than an absolute set, order stops mattering — addition commutes. Decrement 2, decrement 3, increment 1 gives the same answer in any sequence.

This is why I push hard against absolute-value events for anything countable. StockAdjusted { delta: -2 } is order-independent. StockSet { value: 98 } is not, and the difference costs you nothing at design time and a great deal at 2am.

The mistake I made on that distributor build was accepting absolute stock values from the ERP because that is what the ERP emitted, and then not adding a version number because the events "obviously" arrived in order. They did, until a consumer restart replayed a window of them.

7. Idempotent Consumers, In Detail

Every consumer must produce the same result whether it processes a message once or five times. There are three ways to get there and they are not equally good.

Natural idempotency. The operation is already safe to repeat. Setting a status to shipped, writing a row with a deterministic primary key, upserting a read model. Free when you can arrange it, and worth restructuring an operation to get.

A processed-events table. Record every event id you have handled, and check before processing. The check and the write must be in the same transaction as the business change, or you have created a smaller version of the dual-write problem described below.

// Idempotent consumer. The dedupe insert and the business write share
// one transaction, so a crash between them is impossible.
async function handle(event) {
  const tx = await db.begin();
  try {
    // Unique key on event_id. A duplicate throws here and we stop.
    await tx.query(
      'INSERT INTO processed_events (event_id, consumer, handled_at) VALUES (?,?,NOW())',
      [event.eventId, 'loyalty-accrual'],
    );

    await applyLoyaltyPoints(tx, event.payload);

    await tx.commit();
  } catch (err) {
    await tx.rollback();
    if (err.code === 'ER_DUP_ENTRY') {
      // Already handled. Acknowledge and move on — this is success,
      // not failure. Throwing here would loop the message forever.
      return { status: 'duplicate' };
    }
    throw err;
  }
}

Two operational notes on that table. It grows without bound, so partition it by date and drop partitions older than your longest possible redelivery window plus a margin — I use 30 days. And the consumer column matters: the same event is legitimately processed by six different consumers and each needs its own record.

Idempotency keys at the boundary. For calls to external systems, pass a key the remote end honours. Stripe, Adyen and most modern payment APIs support this, and it is the correct fix for the refund incident that opens this article.

// The key must be derived from the event, not generated per attempt.
// crypto.randomUUID() here would defeat the entire mechanism.
await stripe.refunds.create(
  { payment_intent: event.payload.paymentIntentId, amount: event.payload.amountMinor },
  { idempotencyKey: `refund:${event.payload.refundId}` },
);
// Stripe returns the ORIGINAL response for a repeated key within 24h.
// Four retries now produce one refund and three replays of the receipt.

That one line, in place from the start, would have turned a four-times refund into a non-event. It costs nothing. I now treat a call to an external system without an idempotency key as a code review blocker in any consumer.

8. The Dual-Write Problem, Which Is Everywhere

Here is the bug in almost every first-generation event integration I have reviewed.

// This is broken. It looks completely reasonable.
async function placeOrder(cmd) {
  const order = await db.insert('orders', cmd);      // write 1
  await broker.publish('order.placed.v2', order);    // write 2
  return order;
}

Two writes to two systems with no shared transaction. Four things can happen and only one of them is what you want.

Both succeed: correct. The database write fails: nothing happened, correct. The publish fails after the database write succeeded: the order exists and nobody downstream knows. No confirmation email, no warehouse pick, no ERP record. And the process crashing between the two lines produces exactly the same outcome with no error anywhere.

Swapping the order does not help; it gives you a published event for an order that does not exist, which is worse because downstream systems act on it.

Wrapping them in a try/catch does not help either. If the publish fails you cannot roll back a committed database transaction, and if you delete the row you have created a new dual-write with the same problem.

The frequency of this bug is not low. On a typical broker with 99.9% availability, publishing a hundred thousand orders a month means roughly a hundred silent losses a month. Most merchants discover it as "sometimes an order doesn't reach the warehouse and we don't know why", which sits in a ticket queue for a year because it is not reproducible.

There are exactly two correct fixes and both work by reducing two writes to one.

9. The Outbox Pattern, Concretely

Write the event to a table in the same database, in the same transaction as the business data. One transaction, one system, atomic. A separate process reads the table and publishes.

CREATE TABLE outbox (
  id            BIGINT AUTO_INCREMENT PRIMARY KEY,
  aggregate_id  VARCHAR(64)  NOT NULL,
  event_type    VARCHAR(128) NOT NULL,
  payload       JSON         NOT NULL,
  created_at    DATETIME(6)  NOT NULL DEFAULT CURRENT_TIMESTAMP(6),
  published_at  DATETIME(6)  NULL,
  attempts      INT          NOT NULL DEFAULT 0,

  -- Partial-ish index: the relay only ever scans unpublished rows.
  -- Without this the relay table-scans and gets slower every day.
  KEY idx_unpublished (published_at, id)
) ENGINE=InnoDB;

// Now there is one write. If the transaction commits, the event WILL
// be published. If it rolls back, the event never existed.
async function placeOrder(cmd) {
  const tx = await db.begin();
  const order = await tx.insert('orders', cmd);
  await tx.insert('outbox', {
    aggregate_id: order.id,
    event_type: 'order.placed.v2',
    payload: JSON.stringify(toEventPayload(order)),
  });
  await tx.commit();
  return order;
}

The relay is the part people get wrong, so here it is with the details that matter.

// Outbox relay. Runs continuously. At-least-once by construction,
// which is fine because consumers are idempotent.
async function relay() {
  const tx = await db.begin();

  // SKIP LOCKED lets several relay instances run without contending
  // and without processing the same row twice. Without it you either
  // run a single instance (a bottleneck) or double-publish.
  const rows = await tx.query(`
    SELECT id, event_type, payload FROM outbox
    WHERE published_at IS NULL
    ORDER BY id
    LIMIT 200
    FOR UPDATE SKIP LOCKED
  `);

  for (const row of rows) {
    // Publish BEFORE marking published. If we crash here the row is
    // still unpublished and gets retried — a duplicate, not a loss.
    // The reverse order would lose events, which is unrecoverable.
    await broker.publish(row.event_type, JSON.parse(row.payload), {
      messageId: `outbox-${row.id}`,   // stable id => consumer dedupe works
    });
    await tx.query(
      'UPDATE outbox SET published_at = NOW(6), attempts = attempts + 1 WHERE id = ?',
      [row.id],
    );
  }

  await tx.commit();
  return rows.length;
}

Three details that are easy to miss and expensive to discover.

Publish before marking. Getting this backwards converts a duplicate — harmless, because consumers are idempotent — into a permanent loss.

SKIP LOCKED is what makes the relay horizontally scalable. It is available in MySQL 8.0+ and PostgreSQL 9.5+, and if you are on MySQL 5.7 you will be running a single relay instance and should plan capacity accordingly.

The message id must be stable across retries. Deriving it from the outbox row id gives you that for free and makes downstream deduplication work without any extra bookkeeping.

Then the operational half: alert on the age of the oldest unpublished row, not on the count. A backlog of 50,000 rows that is draining is fine. Three rows that have been stuck for twenty minutes is a broken relay, and a count-based alarm will never fire on it. Prune published rows on a schedule; an unpruned outbox on a busy store reaches tens of millions of rows within a year and the index stops fitting in memory.

10. Change Data Capture, and When I'd Use It Instead

The alternative to a relay reading your table is a tool reading the database's replication log. Debezium is the common choice, tailing the MySQL binlog or Postgres WAL and publishing every committed change to Kafka.

The appeal is that it requires no application change at all, which makes it the pragmatic option for getting events out of something you cannot modify — a legacy ERP, a vendor application, an old Magento install with heavy customisation.

The trade is that CDC emits row changes, not business events. You get sales_order_grid UPDATE, status: processing -> complete. Turning that back into OrderShipped means encoding your database schema into a transformation layer, and every schema migration becomes a potential break for every consumer. You have decoupled the services and coupled everything to your tables, which is the coupling you were trying to escape.

Where I land: outbox when you own the code, CDC when you do not. And if you use CDC, put a translation service immediately after it that converts row changes into named business events, so that exactly one component knows about your schema. I have seen a team wire eleven consumers directly to Debezium topics; a routine column rename took down five of them.

There is a hybrid worth knowing about: use CDC to tail the outbox table specifically. You get the outbox's clean business events with no relay process to run or scale. Debezium ships an outbox event router transform for exactly this. It is the setup I would choose for a new build on Kafka.

11. Schema Evolution Without Breaking Everyone

Your first event schema is wrong. Not possibly wrong — wrong, in ways you will discover in month four. The question is only whether changing it is a deployment or a project.

The governing rule is that producers must not break consumers they cannot see, and once an event is on a topic you genuinely cannot see all of them.

Safe changes, deployable any time: adding an optional field; adding a new event type; widening a numeric range; adding a value to an enum if consumers handle unknown values gracefully, which they only do if you made them.

Breaking changes, requiring a new version: removing a field; renaming a field; changing a type; making an optional field required; changing units or semantics while keeping the name.

That last one is the dangerous category because nothing detects it. Changing weight from kilograms to grams passes every schema check ever written and silently multiplies your shipping quotes by a thousand. If the meaning changes, the name changes. No exceptions.

I set Full compatibility on any topic crossing a team boundary and Backward on internal ones. Full is restrictive and the restriction is the point: it makes the breaking change obvious at build time rather than at 3am.

For a genuinely breaking change, run both versions. Publish order.placed.v1 and order.placed.v2 simultaneously from the same outbox transaction, let consumers migrate on their own schedule, then instrument v1 to find out who is still reading it. Instrument first — asking around does not work, because the team that owns the forgotten consumer left last year.

// Dual publishing during a migration. v1 is derived from v2 so there
// is one source of truth and no chance of the two drifting.
await tx.insert('outbox', [
  { event_type: 'order.placed.v2', payload: JSON.stringify(v2) },
  { event_type: 'order.placed.v1', payload: JSON.stringify(downgrade(v2)) },
]);

function downgrade(v2) {
  return {
    orderId: v2.orderId,
    customerId: v2.customerId,
    // v1 had a single flat total in pounds. v2 splits it and uses pence.
    total: v2.totals.grand / 100,
    items: v2.lines.map(l => ({ sku: l.sku, qty: l.qty })),
  };
}

Set an end date and publish it when you introduce v2. "We will stop publishing v1 on 30 September" written down at the start is the only thing I have found that actually gets consumers migrated. Without it both versions run for years.

12. Choosing the Transport

The broker decision gets more attention than it deserves and the wrong criteria dominate it. Almost every ecommerce business that chose Kafka would have been fine with SQS, and a meaningful minority regret the operational surface.

The last row is not a joke. A database table with SELECT ... FOR UPDATE SKIP LOCKED is a perfectly good queue at the volumes most ecommerce businesses operate at, it is transactional with your business data so the dual-write problem disappears entirely, and you already know how to back it up and monitor it. I have deployed exactly this for a store doing 40,000 orders a month and it has never been the bottleneck.

The honest case for Kafka is retention and replay. If you need to add a consumer next year and have it process two years of history to build its own view, only Kafka gives you that without a bespoke archive. That is a real requirement for analytics and event sourcing and it is not a requirement for a store that wants its warehouse to know about orders.

My default for a store on AWS: EventBridge for routing and fan-out, SQS for anything that needs a retry policy and a dead letter queue, and Kafka only when someone can articulate a replay requirement. If you are running functions rather than services, the shape of that decision interacts with how you drew your serverless function boundaries, because a queue between two functions is also a failure boundary.

13. Webhooks Are Events With Worse Guarantees

Every ecommerce system consumes webhooks — Shopify, Stripe, the 3PL, the marketplace. They are events over HTTP with none of the properties a broker gives you, and treating them as ordinary API calls is how integrations rot.

What you get: at-least-once delivery, no ordering, a retry policy you do not control, a timeout you do not control, and delivery from an IP range that changes without notice.

What you must therefore build. Verify the signature before parsing anything. Persist the raw body and return 200 within a couple of seconds; do the work asynchronously. Deduplicate on the provider's event id, which every serious provider supplies. Handle out-of-order arrival explicitly, because a shipment webhook overtaking an order webhook is routine.

export async function receive(req, res) {
  // 1. Verify first, on the RAW body. Parsing before verifying is how
  //    signature checks get bypassed by a body-parser that normalises.
  if (!verifyHmac(req.rawBody, req.headers['x-shopify-hmac-sha256'])) {
    return res.status(401).end();
  }

  // 2. Persist raw. If everything downstream is broken we can replay.
  //    The unique index on provider_event_id makes this the dedupe point.
  try {
    await db.query(
      `INSERT INTO webhook_inbox (provider, provider_event_id, topic, body)
       VALUES ('shopify', ?, ?, ?)`,
      [req.headers['x-shopify-webhook-id'], req.headers['x-shopify-topic'], req.rawBody],
    );
  } catch (e) {
    // Duplicate delivery. Already stored. 200 stops the provider retrying.
    if (e.code === 'ER_DUP_ENTRY') return res.status(200).end();
    throw e;
  }

  // 3. Acknowledge fast. Shopify's timeout is 5s; exceeding it triggers
  //    a retry and eventually gets the webhook subscription removed.
  res.status(200).end();
}

That webhook_inbox table has paid for itself on every build I have used it on. When a downstream consumer has a bug, you fix the bug and replay from the table. Without it, the events are gone and you are asking the provider to resend, which most will not do beyond a short window.

One trap specific to Shopify: their webhook delivery is explicitly not ordered, and their documentation says so. An orders/updated can arrive before the orders/create it updates. Consumers that assume otherwise fail rarely and confusingly, which is the worst failure frequency.

14. Retries, Backoff, and the Dead Letter Queue Nobody Reads

Retry policy is where a system either degrades gracefully or amplifies its own failure.

Classify errors before you retry them. A network timeout, a 503, a database deadlock — retry those. A validation failure, a 404 on a resource that will never exist, a conditional check failure — do not, because no number of attempts will change the outcome and each one costs you throughput. I see consumers retry a JSON parse error thirty times, which is thirty attempts to parse the same malformed string.

Exponential backoff with jitter, always. Without jitter, a downstream service that recovers gets hit by every retrying consumer simultaneously and falls over again. The retry storm that follows a brief outage is frequently more damaging than the outage.

// Full jitter. The random spread is the important part, not the base.
const delay = Math.random() * Math.min(30_000, 200 * 2 ** attempt);

// Retry only what can succeed on a second attempt.
const RETRYABLE = new Set([408, 429, 500, 502, 503, 504]);
function retryable(err) {
  if (err.name === 'TimeoutError' || err.code === 'ECONNRESET') return true;
  if (err.name === 'ValidationError') return false;   // never
  return RETRYABLE.has(err.statusCode);
}

Then the dead letter queue. Every consumer needs one, and a DLQ with no alarm and no owner is a folder where messages go to be forgotten. I have opened DLQs containing eighteen months of failed messages that nobody had looked at once — including, on one occasion, forty-one orders that had never reached the warehouse.

What I set up now, on every consumer without exception: an alarm on DLQ depth above zero, routed to a person rather than a channel; a runbook entry saying what the messages mean and how to replay them; and a redrive script that exists and has been tested before it is needed, because writing one during an incident is how you replay a message twice.

15. Debugging a System With No Call Stack

This is the part that surprises people, and the honest thing to say is that you are trading a debugging experience you know for one you have to build.

In a monolith, an exception gives you a stack trace: the whole causal chain, in order, in one place. In an event-driven system, a consumer fails and you get one frame. The event arrived. Something was wrong with it. You have no idea what published it, what else it triggered, or what the customer was doing.

Three things reconstruct that, and they must be in place before you need them.

Correlation and causation ids

A correlation id is minted at the true entry point — a customer request, a scheduled job — and copied into every event, message and log line that results, however many hops away. A causation id points at the immediate parent event. Together they give you a tree.

{
  "eventId": "evt_04",
  "type": "shipment.dispatched.v1",
  "correlationId": "cor_9f2a",   // same across the whole customer journey
  "causationId": "evt_03",       // the event that directly caused this one
  "occurredAt": "2026-03-17T11:02:04.771Z"
}

With those two fields, one query returns every event in the journey and the parent pointers let you draw the actual causal graph. Without them you are correlating on timestamps and order ids, and order ids only work while you are inside a flow that has one — which excludes exactly the failures where the order was never created.

Distributed tracing across the async boundary

OpenTelemetry propagates a trace context in HTTP headers automatically. It does not cross a queue unless you put the traceparent into the message and restore it on the consuming side. That is about fifteen lines of code and it is the difference between a trace that ends at the publish call and one that spans the whole flow.

// Producer: inject the active trace context into message attributes.
const carrier = {};
propagation.inject(context.active(), carrier);
await sqs.send(new SendMessageCommand({
  QueueUrl: url,
  MessageBody: JSON.stringify(event),
  MessageAttributes: {
    traceparent: { DataType: 'String', StringValue: carrier.traceparent },
  },
}));

// Consumer: extract it and run the handler inside that context, so the
// consumer span becomes a child of the producer's rather than a root.
const parent = propagation.extract(context.active(), {
  traceparent: record.messageAttributes?.traceparent?.stringValue,
});
await context.with(parent, () => handle(JSON.parse(record.body)));

An event store you can query

Keep every event, with its ids, somewhere queryable. Kafka with long retention, or an S3 archive with Athena over it, or a table if your volume is modest. The question you will actually ask during an incident is "show me everything that happened for correlation id X, in order", and answering it in ten seconds instead of an hour changes the character of the whole operation.

I would also say something uncomfortable: this observability layer is not optional overhead you can add later. It is a load-bearing part of the architecture. A team that adopts events without it will spend more time debugging than they saved by decoupling, conclude that event-driven architecture is a fad, and go back to synchronous calls. I have watched that happen twice and in both cases the technology was not the problem.

16. Testing Flows You Cannot Step Through

Unit tests on handlers are easy and cover less than you would like. The interesting bugs live in the interactions.

Contract tests are the highest-value thing here. The producer asserts that it emits events matching a published schema; each consumer asserts it can handle every example in that schema's fixture set. Pact does this well, and a schema registry with compatibility checking does most of it for free. Either way the goal is that a producer's breaking change fails the producer's build rather than the consumer's production.

Beyond that, three tests I write that most teams do not.

Replay the same message five times and assert the end state is identical. This is a two-line test that catches every idempotency bug you have. Run it in CI against every consumer.

Shuffle an event sequence and assert the end state converges. Generate a plausible sequence, permute it, apply each permutation to a clean fixture, compare. Where the results differ you have found an ordering dependency you did not know you had. On the distributor build this found three, two of which were real.

Kill the relay mid-batch. An integration test that publishes, kills the outbox relay process partway, restarts it, and asserts nothing was lost and duplicates were absorbed. It is fiddly to write and it is the test that proves the property the whole design exists to guarantee.

17. A Worked Example, Including the Part That Failed

The flooring retailer from the opening, over about four months. Magento 2.4.7, roughly 12,000 orders a month, a Netsuite ERP, a 3PL, Klaviyo, and a bespoke loyalty scheme. Before the work, all of it was synchronous calls from Magento observers, which meant a slow Netsuite response made checkout slow and a Netsuite outage made checkout fail.

Phase one — outbox and relay. An outbox table in the Magento database, written inside the existing order-placement transaction by a plugin on the order repository. A relay process publishing to EventBridge. No consumers yet; we ran it for two weeks publishing into a void, purely to confirm the relay kept up and the transaction boundary was right. It was not, initially — the first implementation wrote to the outbox after the transaction committed, which is the dual-write bug with extra steps, and a code review caught it before it shipped. Barely.

Phase two — one consumer. Klaviyo, chosen deliberately because it was the lowest-stakes: a duplicated marketing email is embarrassing, not expensive. This is where we found that our event ids were not stable across relay retries, because the first version generated a UUID at publish time rather than deriving it from the outbox row. Two weeks of duplicate emails at a low rate before anyone connected the two.

Phase three — Netsuite. The high-value change. Checkout stopped calling Netsuite entirely; the ERP consumer reads order.placed.v2 and pushes. Median checkout time went from 4.1 seconds to 1.3. During a four-hour Netsuite outage in month three, checkout was completely unaffected and 340 orders queued and flushed cleanly afterwards. That single incident justified the project to the board more effectively than anything I said.

Phase four — 3PL and loyalty. Straightforward by then, because the pattern was established and the observability existed.

What went wrong, properly. In month three we added a fulfilmentPreference field to the order event and, in the same deployment, tightened the consumer's schema validation to reject unknown fields. Standard hygiene, we thought. What we had not accounted for was the events already sitting in the queue, published by the previous producer version, which the new stricter consumer now rejected — including a batch of 61 orders that had queued during a deployment window. They went to the DLQ. The DLQ alarm existed and fired correctly. Nobody had assigned it an owner, so it went to a Slack channel that was muted over a weekend.

Those 61 orders sat unfulfilled for 51 hours. Roughly £9,000 of orders and, worse, 61 customers who had been charged and were waiting. Replaying them once we found the problem took four minutes; the damage was entirely in the detection delay.

Two changes came out of it. Consumers now ignore unknown fields rather than rejecting them, which is what forward compatibility means and which I had written into the design document and then not enforced in code. And every DLQ alarm routes to a person on a rota, with an acknowledgement requirement, not to a channel.

Where it settled. Checkout p50 1.3s against 4.1s before. ERP sync failures visible in a dashboard rather than discovered at month-end reconciliation. Adding the loyalty consumer took two days against an estimated three weeks under the old architecture, because it required no change to Magento at all. Infrastructure cost about $60 a month for EventBridge, SQS and the relay.

18. When I'd Tell You Not to Do This

Events are not free and the cost is paid in a currency most teams undercount: the number of things that can be true at once.

Do not use events for a request that needs a synchronous answer. "Is this coupon valid" is a question, not a fact. Making it an event means inventing a correlation mechanism and a timeout, which is a worse HTTP call.

Do not use events inside a single service to talk to itself. That is a function call with a message broker's failure modes attached.

Do not use events when the consistency requirement is strict and immediate. Eventual consistency means a window where the order exists and the stock has not moved. If that window is unacceptable — genuinely unacceptable, not just uncomfortable — you need a transaction, which means the data has to live in one place. Some of that is a data architecture question rather than a messaging one, and how you split and scale the database constrains what consistency you can offer in the first place.

Do not adopt events because a conference talk said microservices need them. The right size for a first event-driven system is two or three producers and a handful of consumers at the integration edges. A team that starts with fourteen services and no correlation ids will be back on synchronous calls within a year.

And be honest about the organisational precondition: this architecture assumes teams that can deploy independently and own their consumers. If one team owns everything, the decoupling buys you very little and the debugging cost is real.

19. Questions I Get Asked

"Kafka or SQS?" SQS unless you can state a replay requirement in one sentence. Kafka's retention and replay are genuinely different capabilities, and its operational cost — even on MSK — is genuinely higher. Most stores discover after eighteen months that they have used Kafka as an expensive queue.

"Do I need event sourcing?" Almost certainly not. Event-driven architecture means services communicate by events. Event sourcing means the event log is your database and current state is a projection. The second is a much larger commitment and it is only worth it where the history is the product — financial ledgers, audit-heavy domains. Using events for integration while keeping ordinary tables as your source of truth is a perfectly coherent position and it is where I land on most builds.

"How do I handle a consumer that needs data from three services?" Build it a read model. It subscribes to events from all three and maintains its own denormalised view, which it owns and can query locally with no runtime dependency. This feels wasteful and is the correct answer; the alternative is a synchronous fan-out that couples availability across four services.

"What if an event is published in error?" Publish a compensating event. OrderCancelled, RefundReversed. You cannot unpublish, and a system that tries to — by deleting from a topic — is a system whose consumers have already acted. Compensation is how the physical world works too, which is a reasonable sanity check on the design.

"Should Magento publish events directly?" Through an outbox in the same transaction, yes. Magento's own message queue framework can do this and is fine at modest volume; its consumers are cron-driven and the latency shows. For anything time-sensitive I write to an outbox table and relay externally, which keeps the transactional guarantee and gives you a broker with real retry semantics.

"How much latency should I expect end to end?" With an outbox polled at one second and a broker in the same region, 1–3 seconds from database commit to consumer action is typical and achievable. If you need under 500ms, use CDC on the outbox rather than polling. If you need under 50ms, you need a synchronous call and you should say so.

"Can I put the outbox in a different database from the business data?" No. The entire point is the shared transaction. An outbox in a separate database is the dual-write problem with an extra table.

20. What I'd Do First

In order, and the order is not arbitrary — each step makes the next one safe.

One. Find your dual writes. Grep for a database write followed by a publish, an HTTP call, or an email send outside a transaction. Every one is a silent data loss at a rate proportional to your volume. On most codebases this takes an afternoon and finds between three and ten.

Two. Add correlation and causation ids to everything before you add a second consumer. Retrofitting them means touching every producer and every message shape, and you will retrofit them, because the first cross-service incident makes it obvious.

Three. Build the outbox for your highest-value flow only. Order placement, usually. Run it publishing to nothing for a fortnight and watch the lag metric. Get the transaction boundary right before anything depends on it.

Four. Make your first consumer the one whose duplicate output is least expensive. Marketing email, analytics, a search index update. Your idempotency will be wrong the first time; find out where it costs nothing.

Five. Write the replay test — process the same event five times, assert identical end state — and put it in CI for every consumer before you write the second one.

Six. Set up dead letter queues with alarms routed to a named person on a rota. Not a channel. A person. This is the step I got wrong and it cost a client £9,000 of delayed orders and a weekend of goodwill.

Seven. Register your schemas and set compatibility to Full on anything crossing a team boundary. Do it while you have two events and it takes an hour; do it at forty and it is a project.

Eight. Only then move a flow that matters — the ERP sync, the fulfilment push. By this point you have the plumbing to see what is happening, and the change becomes routine rather than frightening.

The thing I would argue hardest against skipping is the second step. Correlation ids feel like housekeeping and they are the single reason a distributed failure takes ten minutes to diagnose instead of two days. Everything else on this list can be retrofitted at moderate cost. That one gets more expensive every week you defer it.