Load Balancing for ASP.NET Core APIs

A load balancer distributes incoming requests across multiple running instances of the same application.

For an ASP.NET Core Web API, you normally run several copies of the API and place a load balancer in front of them.

1. Why do we need a load balancer?

Suppose one API instance receives thousands of requests. It may become overloaded.

You can run additional instances and distribute requests between them.

An instance means one running copy of your application. The copies can run on separate servers, containers, or processes.

2. How does it work?

Assume your Product API runs on two instances:

The client always calls:

GET http://localhost:5000/products

The load balancer selects one instance for each request and forwards its response to the client.

With Round Robin, requests cycle between instances:

The starting instance may differ. The important behavior is that requests are distributed cyclically.

3. Common load-balancing algorithms

Not every load balancer supports every algorithm. Ocelot supports Round Robin and a LeastConnection option that tracks outstanding requests within that gateway instance. ocelot.readthedocs.io

4. Where do we implement this in .NET?

Usually, the selection logic is outside your business API.

Common choices include:

Azure distinguishes application-layer routing from network-layer load balancing. Microsoft Learn

Since we just covered Ocelot, let’s use it for a complete local example.

5. Full simple program

We need two projects, but we will run three processes:

The example targets .NET 8 and uses the same fixed Ocelot version as the previous example.

Step 1: Create the projects

dotnet new web -n ProductApi -f net8.0

dotnet new web -n LoadBalancerGateway -f net8.0

dotnet add LoadBalancerGateway/LoadBalancerGateway.csproj package Ocelot --version 23.4.3

Step 2: Product API — Program.cs

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHealthChecks();

var app = builder.Build();

app.MapGet("/api/products", (HttpContext context) =>
{
    return Results.Ok(new
    {
        servedBy = $"Product API on port {context.Connection.LocalPort}",

        products = new[]
        {
            new { Id = 1, Name = "Laptop", Price = 55000 },
            new { Id = 2, Name = "Mouse", Price = 800 }
        }
    });
});

app.MapHealthChecks("/health");

app.Run();

The servedBy property lets us see which instance handled the request.

You can use controller endpoints instead; load balancing works the same way.

Step 3: Gateway — ocelot.json

Create this file in the LoadBalancerGateway project root:

{
  "Routes": [
    {
      "UpstreamPathTemplate": "/products",
      "UpstreamHttpMethod": [ "GET" ],

      "DownstreamPathTemplate": "/api/products",
      "DownstreamScheme": "http",

      "DownstreamHostAndPorts": [
        {
          "Host": "localhost",
          "Port": 5001
        },
        {
          "Host": "localhost",
          "Port": 5002
        }
      ],

      "LoadBalancerOptions": {
        "Type": "RoundRobin"
      }
    }
  ],
  "GlobalConfiguration": {
    "BaseUrl": "http://localhost:5000"
  }
}

This configures two destinations for the same Product API route.

Step 4: Gateway — Program.cs

using Ocelot.DependencyInjection;
using Ocelot.Middleware;

var builder = WebApplication.CreateBuilder(args);

builder.Configuration.AddJsonFile(
    "ocelot.json",
    optional: false,
    reloadOnChange: true);

builder.Services.AddOcelot(builder.Configuration);

var app = builder.Build();

await app.UseOcelot();

await app.RunAsync();

Step 5: Build Product API once

From the parent directory:

dotnet build ProductApi

Step 6: Run the first API instance

In terminal 1:

dotnet run --project ProductApi --no-build --no-launch-profile --urls http://localhost:5001

Step 7: Run the second API instance

In terminal 2:

dotnet run --project ProductApi --no-build --no-launch-profile --urls http://localhost:5002

The same API code is running twice. You do not need a duplicate project.

Step 8: Run the gateway

In terminal 3:

dotnet run --project LoadBalancerGateway --no-launch-profile --urls http://localhost:5000

HTTP is used for local demonstration. Use HTTPS for public production traffic.

6. Test the load balancer

Call this URL repeatedly using Postman:

GET http://localhost:5000/products

One response:

{
  "servedBy": "Product API on port 5001",
  "products": [
    {
      "id": 1,
      "name": "Laptop",
      "price": 55000
    },
    {
      "id": 2,
      "name": "Mouse",
      "price": 800
    }
  ]
}

Another response:

{
  "servedBy": "Product API on port 5002",
  "products": [
    {
      "id": 1,
      "name": "Laptop",
      "price": 55000
    },
    {
      "id": 2,
      "name": "Mouse",
      "price": 800
    }
  ]
}

The changing port shows that different instances are handling requests.

7. Explanation of the configuration

This is the key part:

"DownstreamHostAndPorts": [
  {
    "Host": "localhost",
    "Port": 5001
  },
  {
    "Host": "localhost",
    "Port": 5002
  }
],
"LoadBalancerOptions": {
  "Type": "RoundRobin"
}

Ocelot cycles between those destinations. Its Round Robin selection state is local to each gateway process. ocelot.readthedocs.io

8. What happens if one API instance stops?

This simple Ocelot configuration distributes requests, but it does not automatically poll /health and remove failed instances.

If instance 1 stops:

• Requests sent to instance 2 can succeed.
• Requests selected for instance 1 can fail.

We added /health to make the API probeable, but creating a health endpoint does not configure health monitoring.

For production, configure a health-aware load balancer or a service-discovery setup that removes unhealthy instances.

A typical health-based process is:

1. Probe each instance periodically.
2. Mark an instance unhealthy after the configured failure threshold.
3. Stop sending new traffic to it.
4. Restore traffic after successful recovery checks.

Existing requests can still fail during an outage or before the failure is detected.

9. What changes when the API runs on multiple instances?

This is a critical design consideration.

Example:

If instance 1 stores a value in IMemoryCache, instance 2 cannot automatically read it.

Load balancing distributes traffic. It does not synchronize application memory or data.

10. Load Balancer vs API Gateway

Ocelot can perform both roles: route /products to Product API, then select one Product API instance.

11. Production approach and key points

• Deploy the same API to multiple instances.
• Configure health checks and traffic draining.
• Keep instances stateless where practical.
• Use shared storage for data that must be available across instances.
• Scale the gateway itself so it does not become a single point of failure.
• Load balancing and autoscaling are different: load balancing distributes traffic; autoscaling changes the number of instances.
• More API instances do not fix a slow database automatically.
• Do not blindly retry payment or order POST requests; use idempotency where retries are required.
• The local example demonstrates distribution only. Two processes on one computer do not provide protection against that computer failing.
• In hosted environments, prefer the platform’s load-balancing capabilities where they meet your requirements.

In Azure App Service, Ocelot usually does not track individual instances. It calls one stable App Service URL, and Azure manages the instances behind that URL.

Your earlier localhost:5001 and localhost:5002 example used manually configured instances. App Service handles this differently.

1. One URL, multiple instances

Suppose your Product API is deployed at:

https://my-product-api.azurewebsites.net

When Azure scales it from two instances to four, the application URL remains the same. Azure distributes traffic across the running instances. azure.microsoft.com

You do not receive a new public URL for each scaled instance.

2. What should Ocelot configuration contain?

Configure the App Service hostname, rather than individual instance addresses:

{
  "Routes": [
    {
      "UpstreamPathTemplate": "/products",
      "UpstreamHttpMethod": [ "GET" ],

      "DownstreamPathTemplate": "/api/products",
      "DownstreamScheme": "https",

      "DownstreamHostAndPorts": [
        {
          "Host": "my-product-api.azurewebsites.net",
          "Port": 443
        }
      ]
    }
  ]
}

Here:

• Ocelot routes /products to the Product API application.
• App Service distributes that request to an instance.
• Ocelot does not need RoundRobin to balance the instances inside this App Service.

3. What happens during scale-out?

Assume two instances are running and traffic increases.

1. Azure’s scaling mechanism decides to add capacity.
2. App Service starts additional instances of your application.
3. Azure manages their participation in its internal routing.
4. New requests can be served by the added instances.
5. Ocelot continues calling the same hostname.

Azure App Service supports automatic scale-out and scale-in based on its configured scaling mechanism. Microsoft Learn

There is no Ocelot JSON update required.

4. What happens when an instance is removed?

When Azure scales in:

1. Azure selects an instance to remove.
2. App Service manages its removal from traffic routing and shutdown.
3. Subsequent requests are routed to the remaining instances.
4. The public hostname stays unchanged.

No URL is removed from Ocelot, because Ocelot never stored that instance’s address.

Requests already running during shutdown need appropriate graceful-shutdown handling; scaling should not be treated as a guarantee that every in-flight request completes.

5. What if an instance is running but unhealthy?

This is different from scale-in.

Configure App Service Health check to call your API’s health endpoint, for example:

/health

In your API:

builder.Services.AddHealthChecks();

app.MapHealthChecks("/health");

Then enable Health check in the App Service configuration and supply /health as the path.

Azure probes the instances and can remove unhealthy instances from rotation after the configured failure threshold, subject to platform limits. It can return recovered instances to rotation. A single unhealthy instance is not removed if doing so would leave the application with no active instance. Microsoft Learn

The basic endpoint only confirms that the application responds. Add dependency checks if readiness requires database or other service availability.

6. When does the gateway need service discovery?

When the gateway connects directly to changing backend addresses, it needs a way to discover them.

Ocelot supports service discovery integrations. With a suitable registry setup, added and removed service registrations change the destinations available to Ocelot. Registration and health-based deregistration must also be configured. ocelot.readthedocs.io

For your App Service scenario: keep one App Service hostname in Ocelot. Azure handles adding and removing the application instances behind it.