A business transaction often spans several independently deployed services.

Consider a typical e-commerce order:

1. Create the order.
2. Reserve the required inventory.
3. Authorize the customer’s payment.
4. Schedule the shipment.

Each step appears to be part of one business transaction. However, in a microservices architecture, the steps are usually performed by separate services that own separate databases.

E-commerce workflow showing order creation, inventory reservation, payment authorization, and shipment scheduling:

Figure 1: A typical order-fulfilment workflow distributed across multiple services

In a monolith using one database, these operations might run inside one transaction.

In a microservices architecture, each service usually owns its own data. A single database transaction can no longer protect the full workflow.

The Saga pattern replaces one distributed transaction with a sequence of local transactions. If a later action fails, compensating actions move the system toward an acceptable outcome.

In a monolithic application using one database, these operations could run inside a single ACID transaction. If any step failed, the database could roll back the entire transaction.

That model no longer works when the Order, Inventory, Payment, and Shipping services maintain independent data stores.

A database transaction owned by the Order Service cannot automatically roll back a payment authorization performed by the Payment Service or release inventory held by the Inventory Service.

The Saga pattern addresses this problem by replacing one distributed transaction with a sequence of local transactions.

Each participating service commits its own change. If a later action fails, the Saga initiates compensating actions for the steps that have already completed.

What is a Saga?

A Saga is a long-running business transaction divided into a sequence of smaller local transactions.

Each local transaction:

• Is performed by one service
• Updates that service’s own database
• Produces an event or response
• Causes the next step to begin

For an order workflow, the forward path may look like this:

Create Order
    ↓
Reserve Inventory
    ↓
Authorize Payment
    ↓
Schedule Shipment

Suppose the Shipping Service cannot schedule delivery after the payment has already been authorized and the inventory has already been reserved.

The system cannot use a shared database rollback. Instead, it must perform business-level compensating actions:

Shipping failed
      ↓
Void payment authorization
      ↓
Release inventory
      ↓
Cancel order

Figure 2:   When a later step fails, the Saga executes compensating actions for previously completed steps

The objective is not to pretend that the earlier work never happened. The objective is to move the system into a valid and explainable business state.

A Saga is a durable state machine

A production Saga should not exist only as a chain of API calls held in application memory.

It should be modelled as a durable state machine.

The Saga needs to know:

• Which business operation is being processed
• Which steps have completed
• Which step is currently running
• How many times a step has been attempted
• Which deadlines have been reached
• Whether compensation has started
• Which compensating actions have completed
• Whether human intervention is required
• The final business outcome
• The version of the workflow being executed

Figure 3:A Saga persists both its workflow state and the information required for retry, recovery,
compensation, and audit.

A possible Saga record could contain:

Saga ID: saga-5832
Order ID: order-9014
Workflow version: 3
Current state: PAYMENT_AUTHORIZED
Completed steps:
  - ORDER_CREATED
  - INVENTORY_RESERVED
  - PAYMENT_AUTHORIZED
Next step:
  - SCHEDULE_SHIPMENT
Retry count: 1
Deadline: 2026-07-31T10:30:00Z
Compensation status: NOT_STARTED
Final outcome: PENDING

If the orchestrator process restarts, another instance should be able to reload this record and continue from the correct state.

Without durable state, a Saga is merely a chain of optimistic API calls.

Local transactions and compensation

Each Saga participant commits its own local transaction.

For the order workflow, the forward actions are:

T1: Create order
T2: Reserve inventory
T3: Authorize payment
T4: Schedule shipment

If T4 fails, the Saga may run the compensating actions in reverse business order:

A compensation is not necessarily a technical rollback.

For example:

• A payment authorization may be voided.
• A captured payment may require a refund.
• Reserved inventory may be released.
• An order may be marked as cancelled.
• A shipment already in transit may require a return workflow.
• A notification that has already been delivered cannot be unsent; a correction may need to be sent instead.

Compensation must reflect business reality rather than technical symmetry.

Compensations can also fail

A Saga design must account for failures during compensation.

For example:

1. Shipment scheduling fails.
2. The Saga requests a payment void.
3. The payment provider times out.
4. The system does not know whether the authorization was voided.

The workflow cannot simply mark the Saga as cancelled. It must retain a state such as:

COMPENSATION_PENDING

The system can then retry, query the provider by operation ID, or escalate the case for manual review.

Saga coordination models

Sagas are commonly coordinated using either:

• Choreography
• Orchestration

Both models divide the workflow into local transactions, but they differ in how the next action is selected.

Choreography

In choreography, there is no central Saga coordinator.

Each service reacts to an event and publishes another event after completing its local action.

A successful flow may look like this:

Order Placed
      ↓
Inventory Reserved
      ↓
Payment Authorized
      ↓
Shipment Scheduled

A failure flow may look like this:

Payment Declined
      ↓
Inventory Released
      ↓
Order Cancelled

Figure 5:  In choreography, services react to business events without a central workflow coordinator

For example:

1. The Order Service publishes Order Placed.
2. The Inventory Service consumes it and publishes Inventory Reserved.
3. The Payment Service consumes that event and publishes either Payment Authorized or Payment Declined.
4. The Shipping Service reacts to Payment Authorized.
5. The Order Service and Inventory Service react to failure events when compensation is required.

Advantages of choreography

Choreography works well when:

• The workflow has a small number of steps
• Events are meaningful outside the workflow
• Services should remain autonomous
• No central status view is required
• Additional passive consumers may be added later

For example, analytics and notification services can consume the same events without modifying the Saga flow.

Trade-offs of choreography

As the workflow grows, choreography can become difficult to understand.

The disadvantages include:

• The workflow is distributed across several codebases
• Dependencies are not visible in one place
• Event cycles can emerge
• End-to-end timeouts are harder to enforce
• Debugging requires correlating events across services
• The final business state may be difficult to query
• Changes to one event can affect unknown consumers

Choreography is not automatically more loosely coupled. Services may become semantically coupled through a web of events even when they are not synchronously coupled.

Orchestration

In orchestration, a central coordinator explicitly controls the workflow.

The orchestrator sends commands to participating services and records their results.

Orchestrator → Reserve Inventory
Inventory Service → Inventory Reserved

Orchestrator → Authorize Payment
Payment Service → Payment Authorized

Orchestrator → Schedule Shipment
Shipping Service → Shipment Scheduled

Figure 6: In orchestration, the coordinator determines the next command and records the result of each step.

The orchestrator does not need to perform the business operation itself.

Its responsibility is to:

• Track the current workflow state
• Determine the next step
• Send commands
• Process responses or events
• Enforce deadlines
• Retry safe operations
• Start compensation
• Record the final outcome

The participating services still own their data and business rules.

Advantages of orchestration

Orchestration provides:

• An explicit workflow definition
• Centralized retries and deadlines
• Easier auditability
• Better visibility into stuck transactions
• Clear handling of branches and loops
• Easier human intervention
• A natural place to manage compensation

Trade-offs of orchestration

The orchestrator becomes important infrastructure.

Risks include:

• Too much domain logic accumulating in the coordinator
• The coordinator becoming a runtime bottleneck
• Poor boundaries creating a distributed monolith
• Workflow-version management becoming complex
• Availability of the coordinator affecting progress
• Participants becoming overly dependent on orchestrator-specific commands

The orchestrator should coordinate the workflow, not absorb all business logic from participating services.

Choosing between choreography and orchestration

Neither coordination model is universally better.

Figure 7:Choreography is suited to shorter autonomous flows, while orchestration provides stronger control for
complex and long-running workflows.

Use choreography when

• The workflow is short
• There are few participants
• Events are independently valuable
• Services can react autonomously
• A central workflow status is unnecessary
• Failure paths are simple

Use orchestration when

• The workflow contains many steps or branches
• It can run for minutes, hours, or days
• Global deadlines must be enforced
• Manual intervention is possible
• Audit history is important
• Compensation requires explicit sequencing
• Operations teams need to query workflow status

Hybrid approaches

A hybrid design is often the most practical.

For example:

• The critical order transaction is orchestrated.
• Analytics consumes domain events through choreography.
• Notifications consume the same events independently.
• Fraud monitoring listens passively.
• The orchestrator remains responsible only for the business-critical completion path.

This avoids forcing every reaction into the central coordinator.

Idempotency

Saga commands and events may be delivered more than once.

A timeout does not prove that the remote operation failed. The service may have completed the action but failed to return the response.

Every Saga step should therefore have a stable operation identifier.

A useful format is:

Saga ID + Step Name

For example:

saga-5832:authorize-payment

The Payment Service stores the operation ID with the first successful result.

If it receives the same command again, it returns the stored result rather than repeating the authorization.

Conceptually:

Receive Authorize Payment command
          ↓
Look up operation ID
          ↓
Already completed?
      /            \
    Yes             No
     ↓               ↓
Return stored     Perform authorization
result            Store result

Idempotency protects the business effect, not merely the HTTP endpoint.

Unknown outcomes

Suppose the orchestrator sends an authorization request and receives a timeout.

Three outcomes are possible:

1. The Payment Service never received the request.
2. The Payment Service received it but failed before completing it.
3. The payment was authorized, but the response was lost.

The orchestrator should not immediately issue a new authorization.

It should query using the same operation ID.

Figure 8:After a timeout, the Saga queries the existing operation before deciding whether a retry is safe.

The decision flow is:

Send authorization
        ↓
      Timeout
        ↓
Query by operation ID
      /          \
Authorized     Not found
    ↓              ↓
Continue       Retry safely

An unknown outcome is different from a confirmed business failure.

Treating every timeout as failure can create duplicate payments. Treating every timeout as success can create unpaid orders.

Timeouts and deadlines

Silence must become an explicit state transition.

Examples include:

• Inventory reservation expires after 15 minutes.
• Payment authorization expires after a provider-defined period.
• Shipment scheduling is retried for 10 minutes.
• Manual review expires after 24 hours.
• A compensation step is escalated after five failed attempts.

Deadlines must be persisted as part of the Saga state.

Do not depend only on:

• In-memory timers
• One running process
• One HTTP connection
• One application instance

A durable scheduler, workflow engine, delayed queue, or database-driven polling mechanism can resume expired workflows.

Concurrent Sagas

Two Sagas may operate on the same business entity simultaneously.

Examples:

• Two customers attempt to reserve the final item.
• An order cancellation races with shipment scheduling.
• A refund races with a dispute.
• Two payout workflows select the same payable balance.
• A customer retries checkout while the first attempt is still running.

Saga sequencing controls one workflow. It does not automatically protect the system from competing workflows.

Possible controls include:

• Optimistic locking
• Entity-version checks
• Reservation records
• Uniqueness constraints
• Compare-and-set updates
• Scoped business locks
• Revalidation before irreversible operations
• Idempotency keys at domain boundaries

For example, the Inventory Service should atomically verify and reserve stock. The Saga orchestrator should not assume that inventory remains available merely because an earlier read reported it.

Failure categories

A mature Saga distinguishes different kinds of failure.

Transient technical failure

Examples:

• Network timeout
• Temporary database problem
• Broker outage
• Service unavailable
• Rate limit

Response:

• Retry with bounded exponential backoff
• Add randomized jitter
• Stop after a defined retry budget
• Preserve the Saga state between attempts

Permanent technical failure

Examples:

• Invalid message schema
• Unsupported workflow version
• Missing mandatory configuration
• Corrupted data
• Invalid command structure

Response:

• Stop automatic retries
• Quarantine the message or workflow
• Alert the owning team
• Repair the data or configuration
• Resume explicitly

Business rejection

Examples:

• Inventory unavailable
• Payment declined
• Account blocked
• Delivery address unsupported
• Refund not permitted

Response:

• Follow an explicit business failure path
• Compensate completed steps where necessary
• Record the reason
• Communicate the outcome clearly

A business rejection is not a system error and should not be retried indefinitely.

Unknown outcome

The remote action may have succeeded, but the caller did not receive a definitive response.

Response:

• Query by operation or idempotency ID
• Wait for an asynchronous event where appropriate
• Reconcile against the external system
• Retry only when the provider contract says it is safe
• Escalate if the state cannot be established

Compensation failure

A compensating action can also fail.

Examples:

• The payment provider rejects a void request.
• Inventory cannot be released because fulfilment already started.
• A refund remains pending.
• The compensation event is repeatedly rejected.

Response:

• Persist the compensation state
• Retry according to the action’s semantics
• Avoid repeating irreversible operations
• Escalate for manual handling
• Preserve a full audit trail

The Saga is not complete until its forward or compensating path reaches an acceptable business outcome.

Observability

A Saga should be observable as one business process rather than as disconnected service calls.

Every log, trace, command, and event should carry identifiers such as:

Saga ID
Business key
Order ID
Current step
Operation ID
Attempt number
Workflow version
Correlation ID
Causation ID

Important metrics include:

• Saga completion time
• Time spent in each state
• Step retry count
• Business rejection rate
• Compensation rate
• Compensation failure rate
• Unknown-outcome rate
• Stuck workflow count
• Manual-intervention volume
• Workflows approaching deadlines
• Outcomes grouped by failure reason

An operations dashboard should answer:

• Which orders are currently stuck?
• Which step creates the longest delay?
• Which service causes the most retries?
• Which Sagas are compensating?
• Which compensations require manual intervention?
• How many workflows are using an older version?

Testing a Saga

The happy path is only a small part of Saga testing.

Test scenarios should include:

• Failure before every forward step
• Failure after a remote action succeeds but before the response arrives
• Duplicate commands
• Duplicate events
• Out-of-order events
• Timeout followed by late success
• Orchestrator restart
• Participant-service restart
• Compensation failure
• Duplicate compensation command
• Cancellation racing with completion
• Old workflow versions still running after deployment
• Manual approval delayed beyond its deadline
• Event publication failure after a local commit

A Saga is trustworthy only when its failure and recovery paths have been tested deliberately.

When not to use the Saga pattern

Do not introduce a Saga simply because the application uses microservices.

Avoid it when:

• The operation belongs within one service boundary
• One local database transaction can solve the problem
• The workflow is short and synchronous
• Intermediate states are unacceptable
• No meaningful compensation exists
• The operational complexity is greater than the business value

Sometimes the correct solution is to redraw the service boundaries so that strongly related operations remain inside one service.

A distributed transaction problem may be evidence that the domain was divided at the wrong boundary.

Key trade-offs

Final perspective

The Saga pattern does not recreate a traditional distributed ACID transaction.

It provides a different and more realistic guarantee:
The system will continue making explicit, durable, and recoverable decisions until the business process reaches an acceptable outcome.

A well-designed Saga does not hide failure. It makes failure, retry, compensation, and intervention part of the workflow itself.

That is what allows independently deployed services to participate safely in one larger business transaction.

Back to Architecture Lab