How Do You Handle Failed Messages in Microservices?

When a consumer receives a message but cannot process it successfully, we normally do not simply delete or acknowledge it.

A production messaging system generally uses:

Retry
  ↓
Retry with delay/backoff
  ↓
Still failing?
  ↓
Dead-Letter Queue (DLQ)
  ↓
Monitor + Investigate
  ↓
Fix / Replay when appropriate

Failed messages are typically handled using retries, dead-lettering, monitoring, and idempotent processing.

1. What Is a Failed Message?

Suppose Inventory Service receives:

{
  "eventId": "EVT-1001",
  "orderId": 5001,
  "productId": 100,
  "quantity": 5
}

The consumer tries:

OrderCreated
     ↓
Inventory Consumer
     ↓
Update SQL Server

But SQL Server is temporarily unavailable:

Update SQL Server
      ↓
Timeout ❌

The message has been received, but processing has failed.

2. Why Can Message Processing Fail?

Failures generally fall into two categories.

Transient Failure

SQL Server temporarily unavailable
Network timeout
External API returns 503
Service temporarily unavailable

These may succeed later.

Permanent Failure

Invalid JSON
Required field missing
Unsupported event version
Invalid business data
Unknown event type

Repeating the same operation may never succeed.

3. Basic Failure-Handling Flow

A common design is:

Message Broker
      │
      ▼
   Consumer
      │
      ▼
Process Message
      │
   ┌──┴─────┐
   │        │
Success   Failure
   │        │
   ▼        ▼
 ACK      Retry
            │
            ▼
       Still fails?
            │
            ▼
           DLQ

Three important concepts are:

1. Retry
2. Dead-Letter Queue
3. Idempotency

4. Don't ACK a Failed Message

Suppose:

Receive Message
      ↓
Update Inventory
      ↓
FAILED

If you acknowledge it anyway:

FAILED
   ↓
ACK
   ↓
Broker considers message finished
   ↓
Message gone ❌

The business operation never happened.

Therefore:

ACK/Complete only after successful processing.

5. Azure Service Bus Example

With Azure Service Bus:

private async Task ProcessMessageAsync(
    ProcessMessageEventArgs args)
{
    try
    {
        var json =
            args.Message.Body.ToString();

        var message =
            JsonSerializer.Deserialize<OrderCreatedEvent>(
                json);

        if (message is null)
        {
            throw new Exception(
                "Invalid message.");
        }

        await ProcessOrderAsync(message);

        await args.CompleteMessageAsync(
            args.Message);
    }
    catch (Exception ex)
    {
        Console.WriteLine(
            $"Processing failed: {ex.Message}");

        throw;
    }
}

Conceptually:

ProcessOrderAsync()
       ↓
Success?
   ┌───┴───┐
  YES      NO
   │        │
Complete   Exception
            ↓
       Message can be
       redelivered

Exact retry/redelivery behavior depends on your broker and receiver configuration.

6. Retry

Suppose SQL Server is unavailable for five seconds.

Immediately sending the message to the DLQ would be unnecessary.

Instead:

Attempt 1
   ↓
DB Timeout
   ↓
Retry
   ↓
Attempt 2
   ↓
DB available
   ↓
Success ✓

Retries are appropriate for transient failures.

7. Immediate Retry vs Delayed Retry

There are two important approaches.

Immediate Retry

Attempt 1 → Fail
Attempt 2 → Fail
Attempt 3 → Success

Useful for very short transient problems.

But if SQL Server will be unavailable for 30 seconds:

Retry
Retry
Retry
Retry
Retry
Retry...

you may just overload the failing dependency.

Delayed Retry

Better:

Attempt 1
   ↓
Fail
   ↓
Wait 2 sec

Attempt 2
   ↓
Fail
   ↓
Wait 5 sec

Attempt 3
   ↓
Fail
   ↓
Wait 10 sec

This is backoff.

8. Exponential Backoff

A common retry strategy is:

Attempt 1 → Fail
            ↓
          Wait 1 sec

Attempt 2 → Fail
            ↓
          Wait 2 sec

Attempt 3 → Fail
            ↓
          Wait 4 sec

Attempt 4 → Fail
            ↓
          Wait 8 sec

The delay increases after each failure.

Conceptually:

Delay ≈ baseDelay × 2^attempt

Usually some randomness called jitter is also added so hundreds of consumers don't retry at exactly the same moment.

9. Why Not Retry Forever?

Imagine the message contains invalid data:

{
  "orderId": "INVALID",
  "quantity": -100
}

Retrying:

Attempt 1 → Fail
Attempt 2 → Fail
Attempt 3 → Fail
Attempt 4 → Fail
...
Attempt 1000 → Fail

doesn't help.

It wastes:

• CPU
• Network resources
• Database connections
• Consumer capacity
• Broker operations

Eventually the message should be isolated.

That's where the Dead-Letter Queue comes in.

10. What Is a Dead-Letter Queue?

A Dead-Letter Queue (DLQ) is a special location for messages that cannot be processed normally.

Main Queue / Subscription
          │
          ▼
       Consumer
          │
          ▼
        Failure
          │
          ▼
        Retry
          │
          ▼
     Repeated Failure
          │
          ▼
   Dead-Letter Queue

Instead of losing the message:

Failed Message
     ↓
DLQ

we preserve it for investigation.

11. Example

Suppose:

OrderCreated
EventId = EVT-1001

fails repeatedly.

Delivery 1 → Failed
Delivery 2 → Failed
Delivery 3 → Failed
...

Eventually:

Main Subscription
        ↓
      DLQ
        ↓
EVT-1001

Now operations/developers can investigate it without continuously blocking normal processing.

12. Azure Service Bus Dead-Letter Queue

Azure Service Bus queues and subscriptions have an associated DLQ.

Conceptually:

orders-topic
     │
     ▼
inventory-subscription
     │
     ├── Active Messages
     │
     └── Dead-Letter Messages

The DLQ is associated with the queue/subscription rather than being an unrelated application queue you must invent.

13. Explicitly Dead-Lettering a Message

Sometimes you know immediately that retrying is pointless.

For example:

Unsupported Event Version
Invalid Payload
Required Information Missing

You can explicitly dead-letter it:

await args.DeadLetterMessageAsync(
    args.Message,
    "InvalidMessage",
    "Required ProductId was missing.");

This is better than retrying a permanently invalid message many times.

14. Retryable vs Non-Retryable Errors

This distinction is very important.

Retryable

Database timeout
Network timeout
HTTP 503
Temporary service outage
Temporary broker/dependency issue
Rate limiting

Flow:

Failure
  ↓
Retry with backoff

Non-Retryable

Malformed JSON
Unsupported schema
Invalid event type
Missing mandatory field
Impossible business data

Flow:

Failure
  ↓
DLQ

So don't use the same handling for every exception.

15. Example Error Classification

try
{
    await ProcessMessageAsync(message);
}
catch (SqlException)
{
    // Usually treat appropriate transient SQL failures
    // according to your resilience strategy.

    throw;
}
catch (InvalidMessageException ex)
{
    await args.DeadLetterMessageAsync(
        args.Message,
        "InvalidMessage",
        ex.Message);
}

In a real application, classify exceptions more precisely rather than assuming every SqlException is transient.

16. Poison Message

A message that repeatedly fails processing is commonly called a poison message.

Example:

Message A → Success
Message B → Success
Message C → FAIL
Message C → FAIL
Message C → FAIL
Message C → FAIL

Message C may be a poison message.

Without appropriate handling, it can repeatedly consume resources.

With DLQ:

Message C
   ↓
Repeated failures
   ↓
DLQ

Normal processing can continue.

17. What Information Should Be Logged?

When a message fails, log useful context such as:

MessageId
EventType
CorrelationId
ConsumerName
DeliveryCount
Exception type
Exception message
Timestamp

Example:

MessageId     : EVT-1001
EventType     : OrderCreated
CorrelationId : CORR-5001
Consumer      : InventoryConsumer
DeliveryCount : 4
Error         : SQL timeout

Be careful not to log sensitive payload data unnecessarily.

18. Correlation ID

Suppose:

Order Service
     ↓
OrderCreated
     ↓
Inventory Service
     ↓
InventoryReserved
     ↓
Payment Service

Use:

CorrelationId = CORR-5001

across the workflow.

Then if Payment processing fails:

CORR-5001
   │
   ├── Order Service ✓
   ├── Inventory Service ✓
   └── Payment Service ❌

Tracing the failure becomes much easier.

19. Idempotency Is Required with Retry

Retries and duplicate delivery are closely related.

Suppose:

Consumer
   ↓
Update Inventory ✓
   ↓
Network failure before ACK ❌

Broker redelivers:

Same message
     ↓
Consumer

If the consumer isn't idempotent:

Stock = 100
   ↓
95
   ↓
90 ❌

With idempotency:

EVT-1001
   ↓
Already processed?
   ↓
YES
   ↓
Skip business operation

Stock remains 95 ✓

Therefore:

Retry + Idempotency belong together.

20. Failed Message vs Failed Consumer

These are different.

Message Processing Failure

Consumer UP
    ↓
Receives message
    ↓
Processing fails

Use:

Retry → DLQ

Consumer Completely Down

Consumer DOWN

With durable messaging:

Broker
  ↓
Retains pending messages
  ↓
Consumer returns
  ↓
Processes backlog

The second case is not necessarily a poison-message situation.

21. Don't Block the Queue Forever

Suppose:

Message 1 → Success
Message 2 → INVALID
Message 3 → Waiting
Message 4 → Waiting
Message 5 → Waiting

If Message 2 is retried indefinitely:

Message 2
 ↓
Retry
 ↓
Retry
 ↓
Retry...

it can create operational problems, particularly where strict ordering/session/partition semantics are involved.

After an appropriate retry policy:

Message 2 → DLQ

then the system can deal with it separately, subject to your ordering requirements.

22. What Do We Do with DLQ Messages?

Putting messages in DLQ is not the end of the process.

You need an operational procedure:

DLQ
 ↓
Monitor / Alert
 ↓
Investigate
 ↓
Find Root Cause
 ↓
Fix Application or Data
 ↓
Decide whether replay is safe
 ↓
Replay / Resubmit
 ↓
Verify

A DLQ that nobody monitors is just a place where failures accumulate.

23. Can We Replay a DLQ Message?

Often yes, but only after understanding why it failed.

Suppose:

OrderCreated
    ↓
DLQ

because Inventory Service had a bug.

After fixing the bug:

Fix deployed
     ↓
DLQ message
     ↓
Resubmit / Replay
     ↓
Inventory Consumer
     ↓
Success

But remember:

The consumer should be idempotent before replaying messages.

Otherwise replaying may duplicate a business operation that partially succeeded earlier.

24. Retry Architecture

A stronger production architecture often looks like:

                   Message
                      │
                      ▼
                   Consumer
                      │
             ┌────────┴────────┐
             │                 │
          Success            Failure
             │                 │
             ▼                 ▼
        ACK/Complete      Classify Error
                               │
                    ┌──────────┴──────────┐
                    │                     │
                Transient             Permanent
                    │                     │
                    ▼                     ▼
             Delayed Retry               DLQ
                    │
                    ▼
             Retry Limit?
                    │
              ┌─────┴─────┐
              │           │
             NO          YES
              │           │
              ▼           ▼
            Retry        DLQ

25. Example with Azure Service Bus

A simplified consumer:

private async Task HandleMessageAsync(
    ProcessMessageEventArgs args)
{
    try
    {
        var json = args.Message.Body.ToString();

        var order =
            JsonSerializer.Deserialize<OrderCreatedEvent>(
                json);

        if (order is null)
        {
            await args.DeadLetterMessageAsync(
                args.Message,
                "DeserializationFailed",
                "Unable to deserialize OrderCreatedEvent.");

            return;
        }

        if (order.ProductId <= 0)
        {
            await args.DeadLetterMessageAsync(
                args.Message,
                "InvalidProductId",
                "ProductId must be greater than zero.");

            return;
        }

        await _handler.HandleAsync(order);

        await args.CompleteMessageAsync(
            args.Message);
    }
    catch (Exception ex)
    {
        _logger.LogError(
            ex,
            "Failed processing message {MessageId}",
            args.Message.MessageId);

        // Don't complete the message.
        // Let the configured failure/redelivery policy apply.
        throw;
    }
}

The key principle is:

Success
   ↓
Complete

Permanent invalid message
   ↓
Dead-letter

Transient/unexpected failure
   ↓
Retry/redelivery according to policy

26. Retry Must Have Limits

Never blindly implement:

while (true)
{
    try
    {
        await ProcessAsync();
        break;
    }
    catch
    {
        // retry forever
    }
}

This can cause:

• Infinite loops
• Resource exhaustion
• Dependency overload
• Huge logs
• Consumer starvation

Instead define:

Maximum attempts
Backoff strategy
Maximum delay
Timeout
DLQ destination
Alerting

27. Circuit Breaker

Suppose 1,000 messages all call the same unavailable downstream API.

Without protection:

Message 1 → API → Fail
Message 2 → API → Fail
Message 3 → API → Fail
...
Message 1000 → API → Fail

A Circuit Breaker can temporarily stop calls to a dependency known to be unhealthy.

Failures exceed threshold
        ↓
Circuit OPEN
        ↓
Stop calling dependency temporarily
        ↓
Wait
        ↓
Probe / Half-Open
        ↓
Dependency recovered?
     ┌───────┴───────┐
    YES              NO
     │                │
   Close            Open

This prevents hammering an unhealthy dependency.

It complements retry; it doesn't replace DLQ or idempotency.

28. Retry + Circuit Breaker + DLQ

These mechanisms solve different problems.

29. Retry vs DLQ

A simple rule:

Retry

Use when:

"This operation may succeed if I try again later."

Examples:

Timeout
503
Temporary database outage
Rate limit

DLQ

Use when:

"Retrying this exact message is unlikely to help, or the allowed retry attempts are exhausted."

Examples:

Invalid JSON
Unsupported version
Invalid required field
Repeated failure after retry limit

30. Retry vs Idempotency

These are also different.

Retry

Try processing again

Idempotency

If processing happens again,
don't duplicate the business effect

Therefore:

Reliable Consumer
      =
Retry
   +
Idempotency
   +
DLQ
   +
Monitoring

31. Outbox Does Not Solve Consumer Failures

Recall the Transactional Outbox:

Producer
   ↓
Database + Outbox
   ↓
Broker

Outbox solves:

"How do I reliably publish my outgoing event?"

It does not solve:

"What if the consumer cannot process that event?"

Consumer-side handling still requires:

Idempotency
Retry
DLQ
Monitoring

32. Inbox / Idempotent Consumer

On the consumer side:

Broker
  ↓
Consumer
  ↓
Inbox / ProcessedMessages
  ↓
Business Operation

This protects against duplicate effects during retry/redelivery.

A robust end-to-end design can therefore be:

Producer
   │
   ▼
Transactional Outbox
   │
   ▼
Message Broker
   │
   ▼
Consumer
   │
   ├── Retry
   ├── Inbox / Idempotency
   ├── DLQ
   └── Monitoring

33. Industry-Style Processing Flow

For an important message, think in this order:

1. Receive message
        ↓
2. Validate envelope/schema
        ↓
3. Check idempotency
        ↓
4. Process business logic
        ↓
5. Commit local transaction
        ↓
6. ACK message

If something fails:

Transient?
    │
    ├── YES → Retry/backoff
    │
    └── NO  → DLQ

After retry limit:

DLQ
 ↓
Alert
 ↓
Investigate
 ↓
Fix
 ↓
Replay safely

34. Monitoring and Alerts

Production systems should monitor:

Failed message count
Retry count
DLQ count
Queue depth
Oldest message age
Processing latency
Consumer health
Exception rate

For example:

inventory-subscription

Active Messages : 25
DLQ Messages    : 0
Consumers       : 3

Normal.

But:

Active Messages : 50,000
DLQ Messages    : 4,500
Consumers       : 0

requires investigation.

35. Advantages of Proper Failure Handling

It provides:

• Better reliability
• Fault tolerance
• No silent message loss
• Recovery from transient failures
• Isolation of poison messages
• Safer message replay
• Better troubleshooting
• Better observability

36. Disadvantages / Complexity

It introduces:

• Retry configuration
• DLQ management
• Idempotency storage
• Monitoring requirements
• Operational procedures
• More complicated debugging
• Replay tooling
• Error classification complexity

But these are standard concerns in reliable event-driven systems.

37. Key Points

For interview preparation:

• Don't ACK a message before successful processing.
• Transient failures → retry.
• Prefer bounded retry with backoff.
• Add jitter when appropriate to avoid retry storms.
• Permanent failures → DLQ.
• Repeatedly failing messages are often called poison messages.
• Never blindly retry forever.
• Consumers should be idempotent because retry/redelivery can create duplicates.
• Use a unique MessageId / EventId.
• Monitor the DLQ; don't treat it as permanent storage.
• Log MessageId, CorrelationId, event type and useful failure information.
• Replay DLQ messages only after fixing/understanding the problem.
• Circuit Breaker can protect repeatedly failing downstream dependencies.
• Transactional Outbox handles producer-side reliable publishing; it doesn't replace consumer failure handling.
• Inbox/Idempotent Consumer helps make consumer-side retries safe.

38. Interview Questions and Answers

Q1. How do you handle failed messages?

Answer:

I classify the failure as transient or permanent. Transient failures are retried using a bounded retry policy with backoff. Permanently invalid messages, or messages that exceed the retry limit, are moved to a Dead-Letter Queue. Consumers should also be idempotent so retries don't create duplicate business effects.

Q2. What is a Dead-Letter Queue?

Answer:

A DLQ stores messages that could not be successfully processed through the normal path so they can be investigated and potentially replayed later.

Q3. What is a poison message?

Answer:

A poison message is a message that repeatedly fails processing, often because of invalid data, incompatible schema, or some persistent business/technical problem.

Q4. Should every failure be retried?

Answer:

No.

Temporary failures such as timeouts may be retried. Permanently invalid messages should normally be dead-lettered rather than repeatedly retried.

Q5. Why use exponential backoff?

Answer:

It increases the delay between retry attempts so an unhealthy dependency has time to recover and is not overwhelmed by continuous retries.

Q6. Why is idempotency important during retry?

Answer:

A message may have partially or fully completed its business operation before a failure or acknowledgement problem occurs. Retrying could therefore repeat the operation. An idempotent consumer prevents duplicate business effects.

Q7. When should the message be acknowledged?

Answer:

After the important business processing and local transaction have successfully completed.

Process
   ↓
Commit
   ↓
ACK

Q8. What should happen to messages in the DLQ?

Answer:

They should be monitored and investigated. After the root cause is fixed, they can be corrected/replayed when doing so is safe and appropriate.

Interview-Friendly Answer

Failed messages are handled using a combination of retry, backoff, dead-lettering, idempotency, and monitoring. If the failure is transient, such as a database timeout or temporary service outage, the message is retried with a bounded backoff policy. If the message is permanently invalid or exceeds the retry limit, it is moved to a Dead-Letter Queue for investigation. The consumer should be idempotent so retries and redelivery do not create duplicate business effects, and the message should only be acknowledged after successful processing and transaction commit.