How Do You Handle Duplicate Messages in Microservices?

Duplicate messages are a normal possibility in distributed messaging systems.

The standard approach is to make the consumer idempotent.

Idempotent Consumer = processing the same message multiple times produces the business effect only once.

For example:

PaymentCompleted Event
        ↓
Received first time
        ↓
Update Order → Paid ✓

Same event received again
        ↓
Already processed?
        ↓
YES
        ↓
Ignore duplicate ✓

1. Why Do Duplicate Messages Happen?

Consider this flow:

Message Broker
      │
      │ OrderCreated
      ▼
Consumer
      │
      ├── Update database ✓
      │
      └── ACK message

Normally everything works.

But imagine:

Broker
   │
   │ EventId = ABC123
   ▼
Consumer
   │
   ├── Update database ✓
   │
   ├── CRASH / network failure ❌
   │
   └── ACK never reaches broker

The broker doesn't know whether processing completed.

So it may deliver the message again:

Broker
   │
   │ EventId = ABC123
   ▼
Consumer
   │
   └── Process again ❌

This is a common source of duplicate processing.

2. Why Are Duplicates Dangerous?

Some operations are relatively harmless if repeated.

For example:

UPDATE Orders
SET Status = 'Paid'
WHERE Id = 1001;

Running this twice may still result in:

Status = Paid

But consider:

Charge Credit Card
Add Loyalty Points
Reduce Inventory
Create Invoice
Send Email
Transfer Money

Running those twice could cause:

Customer charged twice ❌

Inventory reduced twice ❌

100 points added twice ❌

Two invoices created ❌

So consumers must be designed carefully.

3. Main Solution — Idempotent Consumer

Every event/message should have a unique identifier.

Example:

{
  "eventId": "91d8ef64-a342-4ed5-b6e5-9cb23a7ef513",
  "eventType": "OrderCreated",
  "orderId": 1001,
  "amount": 2500
}

The consumer records the EventId after processing.

For example:

ProcessedMessages

EventId
------------------------------------
ABC123
DEF456
GHI789

When a message arrives:

Receive Event
     ↓
Check EventId
     ↓
Already processed?
     │
   ┌─┴─┐
  YES  NO
   │    │
   │    ▼
   │  Process
   │    ↓
   │  Record EventId
   │
   ▼
 Ignore

4. Basic Algorithm

Conceptually:

if (await IsProcessedAsync(message.EventId))
{
    return;
}

await ProcessMessageAsync(message);

await MarkAsProcessedAsync(message.EventId);

This is called the Idempotent Consumer Pattern.

However, this simple code still has an important problem that we'll fix shortly.

5. SQL Server Table for Processed Messages

For an ASP.NET Core + SQL Server application, create a table such as:

CREATE TABLE ProcessedMessages
(
    MessageId UNIQUEIDENTIFIER NOT NULL,
    ConsumerName NVARCHAR(200) NOT NULL,
    ProcessedAtUtc DATETIME2 NOT NULL,

    CONSTRAINT PK_ProcessedMessages
        PRIMARY KEY (MessageId, ConsumerName)
);

Why include ConsumerName?

Because the same event may legitimately be processed by several different subscribers:

Event ABC123
     │
     ├── Inventory Consumer
     ├── Notification Consumer
     └── Analytics Consumer

All three should process it once.

Therefore:

MessageId + ConsumerName

is a useful deduplication key.

6. Example Data

MessageId   ConsumerName             ProcessedAt
----------------------------------------------------------
ABC123      InventoryConsumer        10:00
ABC123      NotificationConsumer     10:01
DEF456      InventoryConsumer        10:02

If Inventory Consumer receives ABC123 again:

ABC123 + InventoryConsumer
             ↓
Already exists
             ↓
Duplicate
             ↓
Ignore

But Notification Consumer can still process the same event independently.

7. EF Core Entity

public class ProcessedMessage
{
    public Guid MessageId { get; set; }

    public string ConsumerName { get; set; }
        = string.Empty;

    public DateTime ProcessedAtUtc { get; set; }
}

Configure composite key:

protected override void OnModelCreating(
    ModelBuilder modelBuilder)
{
    modelBuilder.Entity<ProcessedMessage>()
        .HasKey(x => new
        {
            x.MessageId,
            x.ConsumerName
        });
}

8. DbContext

public class InventoryDbContext : DbContext
{
    public InventoryDbContext(
        DbContextOptions<InventoryDbContext> options)
        : base(options)
    {
    }

    public DbSet<ProcessedMessage> ProcessedMessages
        => Set<ProcessedMessage>();

    public DbSet<Product> Products
        => Set<Product>();

    protected override void OnModelCreating(
        ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<ProcessedMessage>()
            .HasKey(x => new
            {
                x.MessageId,
                x.ConsumerName
            });
    }
}

9. Event Contract

public class OrderCreatedIntegrationEvent
{
    public Guid EventId { get; set; }

    public int OrderId { get; set; }

    public int ProductId { get; set; }

    public int Quantity { get; set; }
}

The important property is:

public Guid EventId { get; set; }

Every logical event should get a stable unique ID.

10. Consumer Example

Suppose Inventory Service consumes OrderCreated.

public async Task HandleAsync(
    OrderCreatedIntegrationEvent message)
{
    const string consumerName =
        "InventoryOrderCreatedConsumer";

    bool alreadyProcessed =
        await _dbContext.ProcessedMessages
            .AnyAsync(x =>
                x.MessageId == message.EventId &&
                x.ConsumerName == consumerName);

    if (alreadyProcessed)
    {
        Console.WriteLine(
            $"Duplicate ignored: {message.EventId}");

        return;
    }

    var product =
        await _dbContext.Products
            .SingleAsync(x =>
                x.Id == message.ProductId);

    product.Stock -= message.Quantity;

    _dbContext.ProcessedMessages.Add(
        new ProcessedMessage
        {
            MessageId = message.EventId,
            ConsumerName = consumerName,
            ProcessedAtUtc = DateTime.UtcNow
        });

    await _dbContext.SaveChangesAsync();
}

Now:

ABC123 arrives
     ↓
Not found
     ↓
Stock 100 → 95
     ↓
Record ABC123


ABC123 arrives again
     ↓
Found
     ↓
Ignore
     ↓
Stock remains 95 ✓

11. But There Is Still a Race Condition

The previous example checks:

AnyAsync(...)

before processing.

Suppose two instances receive the same message nearly simultaneously:

Consumer Instance A
       │
       └── ABC123 not found

Consumer Instance B
       │
       └── ABC123 not found

Both could try processing.

Therefore:

Do not rely only on "check first" logic.

The database should enforce uniqueness too.

That's why this is important:

PRIMARY KEY (MessageId, ConsumerName)

or a unique index.

The database becomes the final guard against duplicate processed-message records.

12. Most Important Issue — Atomicity

Consider this:

await UpdateInventoryAsync(message);

await MarkAsProcessedAsync(message.EventId);

What happens if:

Update Inventory
      ↓
SUCCESS ✓

Application crashes
      ↓
MarkAsProcessed never happens ❌

The message gets redelivered:

Same Message
      ↓
No ProcessedMessages record
      ↓
Update Inventory AGAIN ❌

So this is not sufficiently reliable.

13. Use a Database Transaction

The business operation and the processed-message record should be committed atomically when they use the same transactional database.

BEGIN TRANSACTION
       │
       ├── Update Inventory
       │
       └── Insert ProcessedMessage
       │
       ▼
     COMMIT

Now either:

Inventory Update ✓
ProcessedMessage ✓

or:

Inventory Update ✗
ProcessedMessage ✗

14. Transaction Example

public async Task HandleAsync(
    OrderCreatedIntegrationEvent message)
{
    const string consumerName =
        "InventoryOrderCreatedConsumer";

    await using var transaction =
        await _dbContext.Database
            .BeginTransactionAsync();

    try
    {
        bool alreadyProcessed =
            await _dbContext.ProcessedMessages
                .AnyAsync(x =>
                    x.MessageId == message.EventId &&
                    x.ConsumerName == consumerName);

        if (alreadyProcessed)
        {
            await transaction.RollbackAsync();
            return;
        }

        var product =
            await _dbContext.Products
                .SingleAsync(x =>
                    x.Id == message.ProductId);

        product.Stock -= message.Quantity;

        _dbContext.ProcessedMessages.Add(
            new ProcessedMessage
            {
                MessageId = message.EventId,
                ConsumerName = consumerName,
                ProcessedAtUtc = DateTime.UtcNow
            });

        await _dbContext.SaveChangesAsync();

        await transaction.CommitAsync();
    }
    catch
    {
        await transaction.RollbackAsync();
        throw;
    }
}

The key idea is not the exact code—it is:

Business Change
      +
Processed Message Record
      ↓
Same Transaction

15. When Should We ACK the Broker Message?

Only after the database transaction has successfully committed.

Correct order:

Receive Message
      ↓
Check duplicate
      ↓
BEGIN TRANSACTION
      ↓
Business Operation
      ↓
Insert ProcessedMessage
      ↓
COMMIT
      ↓
ACK / Complete Message

For Azure Service Bus:

await handler.HandleAsync(message);

await args.CompleteMessageAsync(
    args.Message);

So:

Database commit
      ↓
CompleteMessageAsync()

16. What If ACK Fails?

This is important.

Suppose:

Process message       ✓
Commit DB             ✓
CompleteMessageAsync  ✗

The broker may redeliver the message.

But now:

MessageId = ABC123
       ↓
ProcessedMessages
       ↓
Already exists
       ↓
Ignore business operation
       ↓
Complete message

That's exactly why idempotency is needed.

17. Full Failure-Safe Flow

                 Message Broker
                       │
                       │ ABC123
                       ▼
                    Consumer
                       │
                       ▼
             Check ProcessedMessages
                       │
                ┌──────┴──────┐
                │             │
              Found        Not Found
                │             │
                ▼             ▼
             Ignore      BEGIN TRANSACTION
                │             │
                │       Business Operation
                │             │
                │       Insert ABC123
                │             │
                │           COMMIT
                │             │
                └──────┬──────┘
                       ▼
                  ACK / Complete

This is a common robust pattern.

18. Azure Service Bus Example

With Azure Service Bus:

private async Task ProcessMessageAsync(
    ProcessMessageEventArgs args)
{
    var json =
        args.Message.Body.ToString();

    var message =
        JsonSerializer.Deserialize<
            OrderCreatedIntegrationEvent>(json);

    if (message is null)
    {
        await args.DeadLetterMessageAsync(
            args.Message,
            "InvalidMessage",
            "Unable to deserialize message.");

        return;
    }

    await _handler.HandleAsync(message);

    await args.CompleteMessageAsync(
        args.Message);
}

The handler performs the idempotency check and transaction.

19. Broker Duplicate Detection vs Consumer Idempotency

Some brokers provide duplicate-detection features.

For example, Azure Service Bus can perform broker-side duplicate detection when configured.

Conceptually:

Publisher
   │
   │ MessageId = ABC123
   ▼
Broker

ABC123 first time  → Accept
ABC123 again       → Detect duplicate

This is useful.

But:

Broker-side duplicate detection should not replace consumer idempotency for critical business operations.

Why?

Because duplicates can arise during consumer processing/redelivery, not only when producers send the same message twice.

20. Producer Duplicate vs Consumer Duplicate

There are two related problems.

Producer sends twice

Publisher
   ├── ABC123
   └── ABC123

Broker duplicate detection may help.

Consumer processes twice

Broker
   ↓
ABC123
   ↓
Consumer
   ↓
Business update ✓
   ↓
ACK fails
   ↓
ABC123 redelivered

Consumer idempotency protects this case.

21. Natural Idempotency

Not every operation requires a ProcessedMessages table.

Some operations are naturally idempotent.

For example:

UPDATE Orders
SET Status = 'Paid'
WHERE Id = 1001;

Repeating:

Paid → Paid → Paid

may not cause an incorrect business effect.

But this:

UPDATE Products
SET Stock = Stock - 5
WHERE Id = 10;

is not naturally idempotent.

Run twice:

100
 ↓
95
 ↓
90 ❌

Similarly:

Balance = Balance - ₹500

is dangerous when duplicated.

22. Business-Key Idempotency

Sometimes the business operation itself has a unique key.

For example:

PaymentId = PAY-1001

Database:

CREATE UNIQUE INDEX UX_Payments_PaymentId
ON Payments(PaymentId);

If the same payment request arrives again:

PAY-1001
   ↓
Already exists
   ↓
Do not charge/create again

This can be stronger than purely technical message-level deduplication.

23. Message ID vs Business Idempotency Key

They solve slightly different problems.

For critical operations such as payment, a business idempotency key is especially useful.

24. Redis for Duplicate Detection?

You could store processed IDs in Redis:

processed:Inventory:ABC123

and check:

Exists?
  ↓
YES → Ignore
NO  → Process

This can be useful for performance or short-lived deduplication.

However, there's an important limitation.

If:

Business data → SQL Server
Processed IDs → Redis

you cannot easily commit both as one normal local SQL transaction.

You may end up with:

SQL update succeeded
Redis write failed

Therefore, when the business operation is in SQL Server, storing the idempotency record in the same SQL database/transaction is often much easier to reason about for correctness-critical operations.

Redis can still be useful when its tradeoffs fit the requirement.

25. Inbox Pattern

The pattern of storing incoming messages before/while processing is often called the Inbox Pattern.

Conceptually:

Message Broker
      ↓
Consumer
      ↓
Inbox / ProcessedMessages
      ↓
Business Processing

The Inbox provides durable tracking of received/processed messages.

You may encounter:

Producer side
    ↓
Transactional Outbox

Consumer side
    ↓
Inbox / Idempotent Consumer

These patterns complement each other.

26. Outbox + Inbox

A robust architecture can look like:

Order Service
     │
     ├── Orders
     └── OutboxMessages
            │
            ▼
       Outbox Worker
            │
            ▼
      Message Broker
            │
            ▼
    Inventory Service
            │
            ├── Inbox / ProcessedMessages
            │
            └── Inventory

Producer

Business Data
     +
Outbox Event
     ↓
Same Transaction

Consumer

Business Update
     +
Processed Message
     ↓
Same Transaction

This gives you a much more reliable end-to-end design.

27. Don't Try to Guarantee "Exactly Once" Too Casually

You'll often hear:

Exactly-once processing

In distributed systems, end-to-end exactly-once business effects are difficult.

A practical design is often:

At-least-once delivery
        +
Idempotent consumer
        ↓
Effectively-once business effect

Meaning:

The broker may deliver:

ABC123
ABC123
ABC123

but your application produces:

Business effect
     ↓
ONCE

That's usually the important business requirement.

28. Example: Payment Service

Imagine:

ProcessPayment

PaymentId = PAY-1001
Amount    = ₹5,000

The message arrives twice.

Without idempotency:

PAY-1001
   ↓
Charge ₹5,000

PAY-1001 again
   ↓
Charge ₹5,000 again ❌

With idempotency:

PAY-1001
   ↓
Already processed?
   ↓
NO
   ↓
Charge
   ↓
Save PAY-1001


PAY-1001 again
   ↓
Already processed?
   ↓
YES
   ↓
Return previous result / ignore

For payment-like operations, business-level idempotency is essential.

29. Example: Inventory

Event:

EventId  = EVT-5001
ProductId = 100
Quantity  = 5

Initial:

Stock = 100

First processing:

100 - 5
   ↓
95

Save EVT-5001

Duplicate:

EVT-5001
    ↓
Already processed
    ↓
Do not subtract again

Stock remains 95 ✓

Without idempotency:

100 → 95 → 90 ❌

30. Cleanup of Processed Messages

The ProcessedMessages table can grow indefinitely.

1 million
10 million
100 million
...

Therefore, define a retention policy based on:

• Maximum possible redelivery window
• Broker retention
• Audit requirements
• Business requirements
• Regulatory requirements
• Storage cost

For example, old processed-message records might be archived or deleted after an appropriate period.

Don't arbitrarily delete them too soon, because an old message could potentially be redelivered or replayed depending on the system.

31. Advantages

Idempotent consumers provide:

• Protection against duplicate processing
• Safer message redelivery
• Reliable retries
• Better fault tolerance
• Safer financial/inventory operations
• Better recovery after crashes
• Support for at-least-once delivery models

32. Disadvantages

There is additional complexity:

• Extra database table/storage
• Extra lookup/write per message
• Transaction management
• Cleanup/retention requirements
• Concurrency/race-condition handling
• Unique constraints required
• Business-specific idempotency design

But for important side effects, this complexity is usually justified.

33. Key Points

For interview purposes, remember:

• Duplicate messages are expected possibilities in distributed messaging.
• The main solution is an Idempotent Consumer.
• Every logical message/event should have a stable unique MessageId or EventId.
• Store processed IDs where appropriate.
• Check whether a message was already processed.
• Use a unique constraint as a database-level guard.
• Business operation + processed-message record should ideally be committed atomically when possible.
• ACK/Complete only after successful business processing/commit.
• A crash after DB commit but before ACK can cause redelivery.
• Idempotency makes that redelivery safe.
• Broker duplicate detection is useful but does not replace consumer idempotency.
• Natural idempotency can sometimes avoid explicit deduplication records.
• Business idempotency keys are important for operations such as payments.
• Inbox Pattern is commonly associated with reliable consumer-side processing.
• Transactional Outbox addresses producer-side reliable publication.
• Outbox + Inbox/Idempotent Consumer is a strong production combination.
• Be cautious with claims of true end-to-end "exactly once."

34. Interview Questions and Answers

Q1. How do you handle duplicate messages?

Answer:

Use an idempotent consumer. Assign each message a unique ID, record successfully processed IDs, and check that record before applying the business operation again.

For critical operations, the business update and processed-message record should be committed atomically.

Q2. Why can duplicate messages occur?

Answer:

A common scenario is that the consumer successfully performs the business operation but crashes before acknowledging the message. The broker then redelivers it because it cannot know whether processing completed successfully.

Q3. What is an idempotent consumer?

Answer:

An idempotent consumer can process the same message multiple times without causing duplicate business effects.

ABC123 → Process ✓
ABC123 → Ignore duplicate
ABC123 → Ignore duplicate

Q4. Why isn't checking MessageId enough?

Answer:

A simple check can have race conditions, and the application can crash between performing the business operation and recording the MessageId.

A stronger design uses:

Unique constraint
+
Database transaction

so the business change and deduplication record are consistent.

Q5. When should the consumer ACK the message?

Answer:

After the important business processing has successfully completed and its transaction has committed.

Process
   ↓
Commit
   ↓
ACK

Q6. What if ACK fails after the database commits?

Answer:

The broker may redeliver the message. The idempotency record allows the consumer to recognize it as already processed and avoid repeating the business operation.

Q7. What is the Inbox Pattern?

Answer:

The Inbox Pattern stores/tracks incoming message IDs or messages so the consumer can safely detect duplicate processing and coordinate message handling with its local business transaction.

Q8. Outbox vs Inbox?

Answer:

Q9. Can Redis be used for duplicate detection?

Answer:

Yes, particularly for fast or temporary deduplication. However, if your business data is stored in SQL Server, coordinating a Redis idempotency marker atomically with a SQL business update is harder than storing both in the same SQL transaction.

Q10. Does Azure Service Bus duplicate detection solve everything?

Answer:

No. Broker duplicate detection can help prevent repeated sends with the same message identity, but consumer redelivery can still occur because of processing or acknowledgement failures.

Consumer idempotency is still important for critical side effects.

Interview-Friendly Final Answer

Duplicate messages are normally handled using the Idempotent Consumer Pattern. Each message has a unique MessageId or EventId. Before applying the business operation, the consumer checks whether that message was already processed. For critical operations, the business update and processed-message record are stored in the same database transaction, protected by a unique constraint. The broker message is acknowledged only after that transaction commits. Therefore, if the consumer crashes after committing but before acknowledgement and the broker redelivers the message, the consumer recognizes the duplicate and does not repeat the business operation. On the producer side, Transactional Outbox handles reliable publication; on the consumer side, Inbox/Idempotent Consumer handles duplicate processing.