A practical guide to patterns, reliability, ordering, and trade-offs

Modern applications rarely complete a business operation inside a single service.

When a customer places an order, the system may need to:

• Reserve inventory
• Authorize payment
• Send a confirmation
• Update analytics
• Start fulfilment
• Notify fraud-monitoring systems

A synchronous design may connect these services through a chain of API calls:

Order Service
      ↓
Inventory Service
      ↓
Payment Service
      ↓
Notification Service

This approach is easy to understand initially. However, every additional dependency increases latency and creates another way for the complete request to fail.

If the Notification Service is unavailable, should order creation fail?

If Analytics becomes slow, should the customer wait longer at checkout?

Event-driven architecture provides another way to coordinate such systems.

Instead of directly calling every interested service, the producer publishes a business event. Independent consumers react to that event according to their own responsibilities.

The central distinction is:

A command asks a system to perform an action. An event records that something has already happened.

Examples:

Command: Authorize Payment
Event:   Payment Authorized

Command: Reserve Inventory
Event:   Inventory Reserved

Events describe completed business facts. They should be written in past tense and should remain meaningful outside the producer’s implementation.

1. Core Event-Driven Architecture

An event-driven system normally contains three primary roles:

• Event producer
• Event backbone
• Event consumer

The producer performs a business operation and publishes an event.

The event backbone stores or transports that event.

Consumers subscribe to relevant event types and perform their own local actions.

Figure 1: A producer publishes a business event to an event backbone, allowing independent consumers to react asynchronously.

Event producer

The producer owns the business capability that created the event.

For example, the Order Service may publish:

Order Placed

The producer should not need to know every consumer of that event.

It should not contain code such as:

Notify Inventory Service
Notify Payment Service
Notify Analytics Service
Notify Email Service

Instead, it publishes one durable business fact.

Event backbone

The event backbone may be implemented using:

• Kafka
• A message broker
• A cloud event bus
• A publish-subscribe platform
• A durable queueing system

Its responsibilities may include:

• Accepting events
• Persisting them
• Delivering them to consumers
• Retrying delivery
• Maintaining ordering within defined boundaries
• Supporting replay
• Distributing work across consumer instances

The technology itself does not make the system event-driven. The important architectural decisions concern event ownership, contracts, delivery, ordering, retries, and failure recovery.

Event consumer

A consumer subscribes to relevant events and performs a local action.

For example:

Order Placed
      ↓
Inventory Service reserves items

Another consumer may independently use the same event:

Order Placed
      ↓
Analytics Service updates sales metrics

Consumers should not assume that events arrive exactly once or that all event types arrive in a globally ordered sequence.

2. Event Fan-out

One of the strongest uses of event-driven architecture is fan-out.

Consider an order being created.

The Order Service publishes:

Order Placed

Several independent services can respond:

• Inventory Service reserves the requested items.
• Payment Service authorizes payment.
• Analytics Service updates sales data.
• Notification Service sends an order confirmation.

Figure 2: A single Order Placed event triggers several independent business reactions.

Place Figure 2 here, immediately after listing the services that react to the Order Placed event.

The Order Service does not wait for every consumer to finish.

This provides several advantages.

Independent scaling

The Inventory Service may require ten consumer instances while the Notification Service requires only two.

Each consumer scales according to its own workload.

Independent evolution

A new fraud-monitoring consumer can subscribe later without changing the Order Service.

Reduced synchronous latency

The customer-facing request does not need to wait for analytics, reporting, or notifications.

Failure isolation

A failure in one consumer does not necessarily stop every other consumer.

For example, an analytics failure should not prevent inventory reservation.

However, event fan-out also introduces important questions:

• Which consumers are business-critical?
• How does the system know whether the full workflow completed?
• What happens when one consumer repeatedly fails?
• How are duplicate events handled?
• How are event contracts changed safely?

Event-driven architecture reduces direct runtime coupling, but it does not remove business dependencies.

3. Event Notification and Event-Carried State

Not every event needs the same amount of information.

Two common approaches are event notification and event-carried state transfer.

Event notification

A small event announces that something changed.

Example:

Customer Address Changed
Customer ID: 4821

The consumer must call the Customer Service to retrieve the new address.

Advantages

• Smaller event payloads
• Less duplicated data
• The producer remains the source of truth
• Reduced exposure of sensitive fields

Trade-offs

• Consumers remain dependent on the producer at runtime
• Replaying an old event may retrieve current data rather than historical data
• Large event bursts can create API-call bursts
• The producer must remain available during consumer processing

Event-carried state transfer

The event contains enough state for consumers to act independently.

Example:

Customer Address Changed
Customer ID: 4821
Country: India
Postal Code: 577201
Address Version: 7

Advantages

• Consumers avoid synchronous callbacks
• Historical replay is more deterministic
• Independent read models are easier to build
• Consumer availability is less dependent on the producer

Trade-offs

• Data is duplicated
• Event schemas require stronger governance
• Sensitive information may spread to more systems
• Consumers may retain stale copies

A practical principle is:

Include the minimum stable business information legitimate consumers need to complete their work.

Do not publish full database rows merely because they are convenient.

4. Delivery Semantics

Distributed event systems must define what delivery guarantee they provide.

At-most-once

An event may be lost, but it is not intentionally delivered again.

This may be acceptable for:

• Non-critical telemetry
• Approximate analytics
• Disposable signals

It is generally unsuitable for important financial or fulfilment operations.

At-least-once

The system attempts not to lose events, but the same event may be delivered more than once.

This is common in production systems.

Consumers must therefore tolerate duplicates.

Exactly-once

Some platforms offer exactly-once processing within a carefully defined boundary.

However, when a workflow crosses:

• An event broker
• A service
• A relational database
• An external API
• A payment provider

the complete business operation still requires explicit idempotency and recovery design.

A practical assumption is:

Events may be duplicated, delayed, or redelivered after a failure.

5. Idempotent Consumer Processing

A consumer is idempotent when processing the same event more than once does not create additional business effects.

Suppose the Fulfilment Service receives Payment Authorized twice.

It must not ship the order twice.

A common process is:

1. Receive the event.
2. Check whether its event ID was already processed.
3. If already processed, return safely.
4. Otherwise, perform the local business operation.
5. Store a processing marker.
6. Acknowledge the event.

Figure 3: The consumer checks a durable processing record before applying the event’s business effect.

A processed-event record may contain:

Event ID
Consumer name
Business entity ID
Processing result
Processed time
Event version

Example:

Event ID: evt-7721
Consumer: fulfilment-service
Order ID: order-9014
Result: shipment-created
Processed At: 2026-07-31T10:42:00Z

Otherwise, the following failure can occur:

Business operation succeeds
      ↓
Consumer crashes before storing marker
      ↓
Event is delivered again
      ↓
Business operation is repeated

An event ID alone may not always define the correct idempotency boundary.

Some operations may use:

Business ID + Operation Type + Version

order-9014:create-shipment:v1

6. The Dual-Write Problem

A producer often needs to perform two writes:

1. Update its database.
2. Publish an event.

Example:

Update order status to CONFIRMED
Publish Order Confirmed

These operations usually belong to different systems.

The database update may succeed while event publication fails.

The result is:

Order is confirmed internally
      ↓
No event is published
      ↓
Inventory, fulfilment, and analytics never learn about it

Publishing first does not solve the problem.

The event may be published while the database transaction later fails, causing consumers to react to a state that was never committed.

This is known as the dual-write problem.

7. Transactional Outbox Pattern

The transactional outbox pattern stores the event intent in the same database transaction as the business update.

The process is:

1. Begin a local database transaction.
2. Update business state.
3. Insert an outbox record.
4. Commit both changes together.
5. An outbox relay reads unpublished records.
6. The relay publishes them to the event backbone.
7. The relay marks them as published or records publication progress.

Figure 4: Business state and event intent are committed atomically before a relay publishes the event.

An outbox record may contain:

Outbox ID
Aggregate type
Aggregate ID
Event type
Event payload
Created time
Publication status
Attempt count

Example:

{
  "outboxId": "outbox-5521",
  "aggregateType": "Order",
  "aggregateId": "order-9014",
  "eventType": "OrderConfirmed",
  "status": "PENDING"
}

Why the outbox works

The business state and outbox entry are committed in one local database transaction.

Therefore:

• Both succeed
• Or both fail

The relay can safely retry publishing pending outbox records.

Duplicate publication

The relay may publish an event successfully and crash before recording that success.

After restarting, it may publish the same event again.

The transactional outbox prevents event loss, but it does not eliminate duplicate delivery.

Consumers must still be idempotent.

Relay implementations

The relay may use:

• Database polling
• Change-data capture
• Transaction-log streaming
• Scheduled workers

The correct choice depends on throughput, latency, operational capabilities, and database technology.

8. Ordering

Many workflows require events to be processed in a meaningful sequence.

For one order:

Order Created
      ↓
Inventory Reserved
      ↓
Payment Authorized
      ↓
Order Shipped

If Order Shipped is processed before Order Created, a consumer may not have the required state.

However, global ordering across every event severely limits scalability and is usually unnecessary.

Most systems require ordering only for one business entity.

Examples:

• One order
• One payment
• One account
• One customer
• One inventory item

9. Partitioning and Business Keys

A stable business key can route related events to the same partition.

Examples:

orderId
paymentId
accountId
customerId

All events for order-9014 can be assigned to one partition.

Within that partition, the event backbone preserves their sequence.

Figure 5: Events with the same business key are routed to the same partition, preserving entity-level ordering while allowing parallelism across entities.

A topic may process many orders concurrently:

Partition 0 → order-202
Partition 1 → order-101
Partition 2 → order-303

Ordering exists within each partition, not across the entire topic.

Entity versions

Events should carry a business version where ordering matters.

Example:

Order ID: order-9014
Order Version: 8

A consumer that has already processed version 8 should reject a late version 7 event.

Versions can also help detect missing events.

If the consumer receives version 9 after version 7, it knows version 8 may be missing.

Hot partitions

A poor key can direct too much traffic to one partition.

Examples:

• One global tenant identifier
• One very large merchant
• One constant default key
• Missing keys routed together

Possible responses include:

• Choose a more granular key
• Isolate high-volume tenants
• Separate workloads
• Aggregate events before publishing
• Accept serialization when correctness requires it

Do not randomize keys when entity ordering is required.

10. Event Replay

A durable event stream can retain events after consumers process them.

Consumers can reset their position and replay history.

Replay supports:

• Rebuilding projections
• Backfilling a new service
• Recovering from a consumer defect
• Reprocessing after a schema fix
• Auditing
• Incident reconstruction

For example, a new reporting service can consume historical order events without asking the Order Service to reproduce every past state.

Replay must be designed carefully.

Questions include:

• Are consumer operations idempotent?
• Could replay resend customer notifications?
• Could replay repeat payment or fulfilment actions?
• Are old event schemas still understood?
• Can downstream systems handle replay traffic?
• How will progress be monitored?
• Can replay be paused safely?

A replay mechanism without guardrails can recreate old business effects.

11. Event Schema Design

Events often outlive the deployment that created them.

A useful event envelope may contain:

Event ID
Event type
Schema version
Business entity ID
Entity version
Occurred time
Published time
Producer
Correlation ID
Causation ID
Payload

Example:

{
  "eventId": "evt-7721",
  "eventType": "PaymentAuthorized",
  "schemaVersion": 2,
  "paymentId": "payment-4821",
  "paymentVersion": 6,
  "occurredAt": "2026-07-31T10:30:00Z",
  "producer": "payment-service"
}

Prefer additive evolution

Safer changes include:

• Adding optional fields
• Adding new event types
• Expanding metadata without changing old meaning

Riskier changes include:

• Renaming existing fields
• Changing field meaning
• Changing units
• Changing an identifier’s interpretation
• Removing fields immediately

Business events versus database events

Prefer:

Payment Authorized
Order Cancelled
Inventory Reserved

Avoid:

Payment Row Updated
Order Status Changed to 4
Inventory Table Inserted

A business event should remain understandable even if the producer changes its internal database.

12. Error Handling

Not every event-processing failure should be treated the same way.

Transient technical failure

Examples:

• Temporary database outage
• Network timeout
• Dependency unavailable
• Rate limit

Response:

• Retry with bounded exponential backoff
• Add jitter
• Preserve ordering requirements
• Stop after a defined retry budget

Permanent technical failure

Examples:

• Unsupported schema
• Corrupted payload
• Missing mandatory configuration
• Invalid event structure

Response:

• Stop automatic retries
• Quarantine the event
• Alert the owning team
• Repair and replay deliberately

Business rejection

Examples:

• Order already cancelled
• Refund is not permitted
• Inventory reservation has expired
• Account is blocked

Response:

• Record the business outcome
• Publish a rejection event where appropriate
• Do not retry indefinitely

Poison event

One event repeatedly fails and blocks a partition.

Possible responses include:

• Dead-letter topic
• Parking-lot topic
• Manual correction
• Controlled skip policy
• Side-channel retry

A dead-letter queue or topic needs:

• A clear owner
• Alerting
• Retention
• Security controls
• Repair instructions
• Replay tooling

Without those, it becomes a graveyard.

13. Observability

Event-driven systems must be observable as business workflows, not only as broker infrastructure.

Every event should include context that supports tracing.

Useful fields include:

Event ID
Correlation ID
Causation ID
Trace ID
Business entity ID
Producer
Schema version
Occurred time
Published time

Important technical metrics

Track:

• Publish latency
• Consumer lag
• Event age
• Processing latency
• Retry rate
• Duplicate rate
• Dead-letter volume
• Partition skew
• Outbox backlog
• Replay progress

Important business metrics

Track:

• Orders created versus fulfilled
• Payments authorized versus recorded
• Inventory reservations versus releases
• Events published versus projections updated
• End-to-end workflow completion time

A broker may be technically healthy while a business workflow is broken.

For example:

Order Placed events are being published
but
Fulfilment has stopped processing them

The broker dashboard may look healthy while customers receive no shipments.

14. Common Event-Driven Patterns

Event notification

Publish a small event and let consumers retrieve additional state.

Event-carried state transfer

Include enough data for consumers to act without calling the producer.

Transactional outbox

Commit the business change and event intent together.

Competing consumers

Several consumer instances share partitions or queued work.

Materialized view

A consumer builds a query-optimized representation from events.

Saga

A long-running business workflow is coordinated through local transactions and compensating actions.

Event sourcing

Events become the authoritative history from which current state is derived.

Event-driven architecture does not automatically require event sourcing.

15. Benefits and Trade-offs

Event-driven architecture provides important advantages, but those advantages come with operational and consistency costs.

Figure 6: Event-driven systems improve decoupling, scaling, fan-out, and replay while introducing eventual consistency, duplicate handling, ordering, and operational complexity.

Benefits

Loose runtime coupling

The producer does not need every consumer to be available when the event is created.

Independent scaling

Each consumer scales according to its own processing requirements.

Natural fan-out

Multiple consumers can react to one business fact.

Replay and recovery

Retained events can rebuild projections or repair processing defects.

Burst absorption

The event backbone separates producer throughput from consumer throughput.

Independent deployment

Consumers can be added or changed without modifying the producer.

Trade-offs

Eventual consistency

Different services may observe a change at different times.

Duplicate delivery

Consumers must be idempotent.

Ordering complexity

Ordering is normally guaranteed only within a partition or business key.

Harder debugging

One business workflow may span several services, events, retries, and delays.

Schema governance

Events become shared contracts that require ownership and compatibility management.

Operational tooling

Teams need visibility into lag, retries, dead letters, outbox state, replay, and business completion.

Event-driven architecture does not eliminate complexity.

It moves complexity from synchronous call chains into:

• Time
• Contracts
• Delivery guarantees
• Ordering
• Replay
• Failure recovery
• Operations

16. When Event-Driven Architecture Is a Good Fit

Use it when:

• Several systems react to one business fact
• Producers should not wait for every downstream operation
• Independent scaling is important
• Replay or audit history is useful
• Work arrives in bursts
• Read models are built asynchronously
• Long-running workflows are expected
• Services need reduced runtime dependency

Common use cases include:

• Order processing
• Payments and payouts
• Inventory
• Notifications
• Analytics
• Fraud detection
• Search indexing
• IoT telemetry
• Audit streams
• Data-platform ingestion

17. When It Is the Wrong Choice

Avoid it when:

• A simple synchronous call solves the problem
• The application is small and straightforward
• Every operation requires immediate strong consistency
• The team cannot operate asynchronous systems
• Event ownership is unclear
• Replay and schema evolution are not needed
• The infrastructure cost exceeds the business value

A synchronous API is not inferior.

For operations requiring an immediate answer, synchronous communication may be more appropriate.

Many mature systems use both:

Synchronous APIs
for immediate commands and queries

Events
for asynchronous reactions and integration

18. Design Review Checklist

Before approving an event-driven design, ask:

• Is this a business event or a disguised command?
• Who owns the event?
• Is the event name meaningful outside the producer?
• What information must the event contain?
• What delivery guarantee applies?
• How are duplicate events handled?
• What is the partition key?
• What ordering scope is required?
• How is the dual-write problem solved?
• Can consumers safely replay events?
• How are schemas evolved?
• What happens after retries are exhausted?
• Who owns dead-letter processing?
• Which metric represents business completion?
• Can operators reconstruct one workflow end to end?

Key Trade-offs

Final Perspective

A mature event-driven architecture is not defined by Kafka, queues, or cloud functions.

It is defined by explicit decisions about:

• Business ownership
• Event meaning
• Delivery
• Idempotency
• Ordering
• Replay
• Schema evolution
• Failure recovery
• Observability

The architecture succeeds when events become stable business facts that producers, consumers, and operators can trust.

The strongest design principle is:

Publish facts, preserve their history where needed, and assume every consumer will eventually face duplicates, delays, failures, and replay.

Back to Architecture Lab