1. How Does Load Balancing Work with Microservices?

What is Load Balancing?

Load balancing is the process of distributing incoming requests across multiple instances of the same microservice.

Suppose ProductService has three instances:

ProductService

Instance 1 → 10.0.1.11
Instance 2 → 10.0.1.12
Instance 3 → 10.0.1.13

If 3,000 requests arrive, we don't want all requests going to Instance 1.

Instead:

                    Load Balancer
                         |
             +-----------+-----------+
             |           |           |
             v           v           v
         Product-1   Product-2   Product-3

The load balancer distributes requests among the healthy, eligible instances.

Why Do We Need Load Balancing?

Microservices are commonly deployed with multiple instances for:

• scalability
• high availability
• fault tolerance
• rolling deployments
• handling traffic spikes
• avoiding a single overloaded instance

Without load balancing:

                1,000 Requests
                      |
                      v
                 Product-1
                  OVERLOAD

while:

Product-2 → Idle
Product-3 → Idle

With load balancing:

                1,000 Requests
                      |
                Load Balancer
                      |
          +-----------+-----------+
          |           |           |
          v           v           v
       Product-1   Product-2   Product-3

The workload can be distributed across the available instances.

Purpose

The major purposes are:

1. Traffic distribution
2. Scalability
3. High availability
4. Fault tolerance
5. Health-aware routing
6. Supporting horizontal scaling

Vertical Scaling vs Horizontal Scaling

Load balancing becomes particularly important with horizontal scaling.

Vertical Scaling

Increase the resources of one server:

Before
4 CPU
8 GB RAM

       ↓

After
16 CPU
32 GB RAM

Horizontal Scaling

Add more instances:

Before

Product-1

After

Product-1
Product-2
Product-3
Product-4

Load balancing distributes traffic across those instances.

Basic Microservices Architecture

Suppose we have:

Product Service
Order Service
Payment Service

Each has multiple instances:

                     Client
                       |
                       v
                  API Gateway
                       |
          +------------+------------+
          |            |            |
          v            v            v
       Product       Order        Payment
         LB            LB            LB
       / | \          / | \          / \
      P1 P2 P3       O1 O2 O3       M1 M2

The gateway determines which service should receive the request.

Load balancing determines which instance of that service should receive it.

That distinction is important.

Example

Client sends:

GET /products/100

API Gateway determines:

/products/* → ProductService

ProductService has:

P1
P2
P3

The load-balancing layer selects:

P2

Final flow:

Client
   ↓
API Gateway
   ↓
Product Service
   ↓
Load Balancing
   ↓
P2

How Does the Load Balancer Know the Instances?

This is where Service Discovery becomes important.

Suppose the registry/platform knows:

ProductService

P1 → 10.0.1.11
P2 → 10.0.1.12
P3 → 10.0.1.13

The routing/load-balancing infrastructure uses this information to identify available backends.

Conceptually:

Service Discovery
       |
       | Available Product instances
       v
Load Balancer
       |
       | Select one
       v
Product Instance

Without dynamic discovery, the load balancer would need some other mechanism or static configuration to know where the instances are.

Common Load-Balancing Algorithms

1. Round Robin

Requests are distributed sequentially.

Request 1 → P1
Request 2 → P2
Request 3 → P3
Request 4 → P1
Request 5 → P2
Request 6 → P3

Simple and commonly understood.

2. Random

An available instance is selected randomly.

Request 1 → P3
Request 2 → P1
Request 3 → P3
Request 4 → P2

3. Least Connections

Send the new request/connection to the instance with fewer active connections.

Example:

P1 → 100 connections
P2 → 20 connections
P3 → 70 connections

The load balancer may select:

P2

This can be useful when request/connection durations vary substantially.

4. Weighted Load Balancing

More powerful instances can receive a larger share of traffic.

For example:

P1 → Weight 50
P2 → Weight 30
P3 → Weight 20

Conceptually:

P1 → ~50%
P2 → ~30%
P3 → ~20%

The exact interpretation depends on the load-balancing implementation.

5. IP Hash / Consistent Affinity Techniques

A value derived from the client/request can influence backend selection.

Conceptually:

Client A → P1
Client B → P3
Client C → P2

This can be used for affinity scenarios, although stateless microservices generally reduce the need for sticky sessions.

Client-Side vs Server-Side Load Balancing

There are two major approaches.

Client-Side Load Balancing

The calling service obtains available instances and chooses one.

OrderService
     |
     | Discover ProductService
     v
Service Registry
     |
     | P1, P2, P3
     v
OrderService
     |
     | Choose P2
     v
Product P2

The client participates in:

Discovery
+
Load Balancing

Advantages

• Can make direct service-to-service calls
• No mandatory centralized load-balancing hop
• Client can implement custom selection

Disadvantages

• More complexity in client applications
• Discovery/load-balancing libraries may be required
• Harder in polyglot environments

Server-Side Load Balancing

The calling service sends traffic to a stable intermediary or service endpoint.

OrderService
      |
      v
Load Balancer / Service
      |
   +--+--+
   |  |  |
   v  v  v
  P1 P2 P3

The infrastructure chooses the backend.

Advantages

• Simpler clients
• Centralized/infrastructure-controlled routing
• Easier for different programming languages

Disadvantages

• Additional infrastructure
• Potential additional network hop
• Routing infrastructure must be highly available

Load Balancing in Kubernetes

From our previous example:

OrderService
      |
      | http://product-service
      v
Kubernetes Service
      |
      +--------+--------+
      |        |        |
      v        v        v
     P1       P2       P3

OrderService doesn't need to know:

P1 = 10.244.1.10
P2 = 10.244.2.15
P3 = 10.244.3.20

It calls:

product-service

The Kubernetes Service and cluster networking infrastructure route connections to eligible backend endpoints.

What Happens If an Instance Fails?

Suppose:

P1 → Healthy
P2 → Failed
P3 → Healthy

The failed/unready instance should be removed from the eligible backend set.

Then:

             Load Balancing
                  |
             +----+----+
             |         |
             v         v
            P1        P3

New normal traffic should no longer be routed to P2.

This is why health checking/readiness and load balancing work closely together.

What Happens During Autoscaling?

Initially:

P1
P2

Traffic increases.

Autoscaling creates:

P3
P4
P5

Now:

Service Discovery
      ↓
P1 P2 P3 P4 P5
      ↓
Load Balancing
      ↓
Traffic distributed across eligible instances

When traffic decreases:

P4 removed
P5 removed

Discovery/routing state changes accordingly:

P1 P2 P3

No application URL change should be necessary.

Advantages of Load Balancing

1. Scalability

Multiple service instances can process traffic.

2. High Availability

Failure of one instance does not necessarily make the entire service unavailable.

3. Fault Tolerance

Traffic can be directed away from unhealthy instances.

4. Better Resource Utilization

Requests can be spread across available compute resources.

5. Supports Autoscaling

New instances can be incorporated dynamically.

6. Supports Rolling Deployment

Traffic can gradually move across old/new instances depending on the deployment platform.

Disadvantages

1. Additional Complexity

Another networking component or platform capability must be operated.

2. Additional Hop

Some server-side architectures add an intermediary.

3. Configuration Complexity

Health checks, timeouts, connection handling and routing policies must be configured correctly.

4. Load Balancer Failure

A single load-balancer instance can itself become a single point of failure unless the load-balancing tier is highly available.

5. Stateful Applications Are Harder

If session state exists only in one service instance:

Request 1 → P1
Request 2 → P2

P2 may not know P1's local session state.

Microservices therefore generally benefit from stateless service instances, with required shared state stored externally where appropriate.

2. How Do API Gateway, Service Discovery, and Load Balancer Work Together?

This is an important microservices architecture question.

First understand that they solve different problems.

A useful mental model is:

API Gateway
     ↓
Which Service?

Service Discovery
     ↓
Where are its instances?

Load Balancer
     ↓
Which instance?

API Gateway

What is it?

An API Gateway is commonly the entry point for external API clients in a microservices architecture.

For example:

Mobile App
Web App
Partner Application
       |
       v
   API Gateway
       |
   Microservices

Instead of exposing:

https://product-server/api/products
https://order-server/api/orders
https://payment-server/api/payments

clients can use one public entry point:

https://api.mycompany.com

API Gateway Responsibilities

Depending on the gateway/product, it can handle:

Routing
Authentication
Authorization
Rate Limiting
TLS termination
Request transformation
Response transformation
Logging
Tracing
Caching
Request aggregation
Retry/timeout policies

Not every gateway should necessarily perform every responsibility.

Service Discovery

Service Discovery answers:

Where is ProductService currently available?

For example:

ProductService
      ↓
10.0.1.11
10.0.1.12
10.0.1.13

If autoscaling adds:

10.0.1.14

the discovery/platform state should reflect it.

Load Balancer

The load balancer answers:

Which eligible ProductService instance should receive this particular connection/request?

For example:

ProductService

P1
P2
P3

It selects:

P2

Complete Flow

Suppose a client sends:

GET https://api.mycompany.com/products/100

Architecture:

                         Client
                           |
                           | GET /products/100
                           v
                     API Gateway
                           |
                           | Route:
                           | /products → ProductService
                           v
                   Service Discovery
                           |
                           | ProductService endpoints
                           v
                     Load Balancer
                           |
               +-----------+-----------+
               |           |           |
               v           v           v
              P1          P2          P3

Conceptually, the responsibilities are:

Gateway:
"Request belongs to ProductService."

Discovery:
"These ProductService instances are available."

Load Balancer:
"I will route this traffic to P2."

Step-by-Step Flow

Step 1 — Client Sends Request

Client
   |
   v
https://api.company.com/products/10

Step 2 — API Gateway Receives It

The gateway examines:

/products/10

Its routing configuration might contain:

/products/* → ProductService
/orders/*   → OrderService
/payments/* → PaymentService

Therefore:

/products/10
       ↓
ProductService

Step 3 — Gateway/Infrastructure Needs a Destination

The gateway knows the logical service, but it should not necessarily hard-code a particular instance:

10.0.1.11

because instances can change.

Instead, it can route through a stable service endpoint or integrate with service-discovery infrastructure.

Step 4 — Service Discovery Provides Current Location Information

Conceptually:

Find ProductService

       ↓

P1 → 10.0.1.11
P2 → 10.0.1.12
P3 → 10.0.1.13

Depending on the architecture, the gateway itself, a proxy, Kubernetes Service, service mesh, DNS, or load balancer may consume this discovery information.

Step 5 — Load Balancing Selects an Instance

Suppose:

P1 → Healthy
P2 → Healthy
P3 → Healthy

The load balancer selects:

P2

Step 6 — Request Reaches ProductService

Client
   ↓
API Gateway
   ↓
ProductService routing/load balancing
   ↓
P2

P2 processes:

GET /products/10

Step 7 — Response Returns

Product P2
    ↑
    |
Load Balancer / Service
    ↑
    |
API Gateway
    ↑
    |
  Client

The client doesn't need to know that P2 processed the request.

What Happens When a New Instance Is Added?

Initially:

ProductService

P1
P2
P3

Autoscaling adds:

P4

The sequence is conceptually:

P4 Created
    ↓
P4 becomes healthy/ready
    ↓
Discovery/platform endpoint state updated
    ↓
Load-balancing infrastructure sees P4
    ↓
P4 becomes eligible for traffic

The API Gateway still routes to:

ProductService

It doesn't need a new public URL.

What Happens When an Instance Is Removed?

Suppose:

P2

is removed.

P1
P2 ← Removed
P3
P4

Then:

Instance removed/unhealthy
        ↓
Discovery/endpoint state updated
        ↓
Load-balancing backend set updated
        ↓
P2 no longer receives new normal traffic

The client still calls:

https://api.company.com/products/10

Nothing changes for the client.

Very Important: They Are Logical Responsibilities

Do not assume that every architecture physically contains three separate servers:

API Gateway Server
Service Discovery Server
Load Balancer Server

These are responsibilities that can be implemented differently.

For example, Kubernetes combines several networking/discovery capabilities through:

Services
CoreDNS
EndpointSlices
Service networking
Ingress/Gateway implementations

A cloud platform may similarly provide managed routing and load-balancing capabilities.

Example with Kubernetes

Consider:

Internet
   |
   v
API Gateway / Ingress
   |
   | /products
   v
Kubernetes Service
product-service
   |
   +-----------+-----------+
   |           |           |
   v           v           v
  P1          P2          P3

Here:

Gateway/Ingress

Handles external routing:

/products → product-service
/orders   → order-service

Kubernetes Service + DNS

Provides stable service identity/discovery:

product-service

Kubernetes service networking

Routes/load-distributes connections to eligible backend Pods:

P1
P2
P3

Example with Ocelot

Suppose ASP.NET Core Ocelot is the API Gateway.

A simple static configuration could look like:

{
  "Routes": [
    {
      "DownstreamPathTemplate": "/api/products/{everything}",
      "DownstreamScheme": "http",

      "DownstreamHostAndPorts": [
        {
          "Host": "product-service",
          "Port": 80
        }
      ],

      "UpstreamPathTemplate": "/products/{everything}",
      "UpstreamHttpMethod": [ "GET", "POST", "PUT", "DELETE" ]
    }
  ]
}

In Kubernetes, product-service can itself be the stable Kubernetes Service name.

So Ocelot does not need to know every Pod:

P1
P2
P3
P4

It routes to:

product-service

and Kubernetes handles the backend membership/routing.

Ocelot with Multiple Static Hosts

Ocelot can also be configured with multiple downstream hosts.

Conceptually:

{
  "DownstreamHostAndPorts": [
    {
      "Host": "product1",
      "Port": 8080
    },
    {
      "Host": "product2",
      "Port": 8080
    }
  ],
  "LoadBalancerOptions": {
    "Type": "RoundRobin"
  }
}

This illustrates gateway-side load balancing.

However, in dynamic environments, manually maintaining every instance in gateway configuration is usually undesirable.

A platform/discovery mechanism is generally preferable.

Why Service Discovery Is Important Here

Imagine configuring:

{
  "Host": "10.0.1.11",
  "Port": 8080
}

Then autoscaling removes that instance.

Your gateway configuration becomes stale.

Dynamic discovery avoids this problem:

Gateway
   ↓
Logical Service
   ↓
Discovery/platform
   ↓
Current healthy endpoints

API Gateway vs Load Balancer

These are often confused.

API Gateway

Primarily understands API/application-level routing and policies.

For example:

/products → ProductService
/orders   → OrderService

It may also perform:

Authentication
Authorization
Rate limiting
Transformation
API aggregation

Load Balancer

Primarily distributes traffic among instances:

ProductService
     |
     +---- P1
     +---- P2
     +---- P3

Simplified:

API Gateway
    ↓
Which microservice?

Load Balancer
    ↓
Which instance?

Service Discovery vs Load Balancer

They are also different.

Service Discovery

"Which ProductService instances/endpoints currently exist?"

Result:

P1
P2
P3

Load Balancer

"Which of those eligible instances should receive this traffic?"

Result:

P2

Therefore:

Discovery finds available destinations; load balancing selects/distributes traffic among them.

Complete Production Concept

A simplified architecture could look like:

                       INTERNET
                           |
                           v
                    +-------------+
                    | API Gateway |
                    +-------------+
                           |
            +--------------+--------------+
            |              |              |
            v              v              v
         Product          Order         Payment
         Service          Service        Service
            |              |              |
       Load Balancing Load Balancing Load Balancing
            |              |              |
       +----+----+      +--+--+       +---+---+
       |    |    |      |     |       |       |
       v    v    v      v     v       v       v
      P1   P2   P3     O1    O2      M1      M2

                ^
                |
         Service Discovery
         / Platform State

The exact physical architecture varies by platform.

Advantages of Combining Them

Dynamic scalability

Instances can be added without clients knowing their addresses.

High availability

Failed instances can be removed from the eligible backend pool.

Single client entry point

Clients interact through the gateway.

Centralized API policies

Authentication, rate limiting and routing can be handled centrally where appropriate.

Better separation of responsibilities

Gateway   → API policies/routing
Discovery → Dynamic location
LB        → Traffic distribution

Disadvantages / Challenges

Infrastructure complexity

More moving parts must be configured and monitored.

Additional latency

Gateway/proxy/load-balancing layers can add network processing.

Failure dependencies

Critical gateway and discovery infrastructure must be highly available.

Troubleshooting

A failed request may require checking:

Client
↓
Gateway
↓
DNS / Discovery
↓
Load Balancer
↓
Network
↓
Microservice

Misconfiguration

Incorrect routes, health checks, timeouts or service names can cause outages even when the application itself is healthy.

Key Points

1. Load balancing distributes traffic across multiple service instances.
2. It is essential for horizontal scaling and high availability.
3. Service Discovery identifies the currently available service endpoints.
4. The Load Balancer selects/distributes traffic among eligible instances.
5. The API Gateway determines where an API request should logically be routed and may apply cross-cutting API policies.
6. Gateway and Load Balancer are not the same thing, although some products can perform both functions.
7. Service Discovery and Load Balancing are not the same thing.
8. Autoscaling works well with discovery because backend membership can change dynamically.
9. Failed/unready instances should stop receiving new normal traffic.
10. Kubernetes can provide service discovery and internal load balancing without ASP.NET Core knowing individual Pod addresses.
11. Avoid hard-coding dynamically changing service-instance IP addresses.
12. Keep microservice instances as stateless as practical so requests can safely reach different replicas.

Interview Questions & Answers

Q1. What is load balancing in Microservices?

Answer:
Load balancing distributes requests across multiple instances of the same microservice to improve scalability, availability and resource utilization.

Q2. Why do Microservices need load balancing?

Because production microservices often run multiple replicas. Load balancing prevents all requests from reaching a single instance and allows the system to scale horizontally.

Q3. What are common load-balancing algorithms?

Examples include:

• Round Robin
• Random
• Least Connections
• Weighted balancing
• Affinity/hash-based approaches

The algorithms available depend on the load-balancing product.

Q4. What is client-side load balancing?

The calling service obtains available service instances and selects one itself, usually through a client library.

Client → Discovery → Select Instance → Service

Q5. What is server-side load balancing?

The client sends traffic to a stable intermediary, and infrastructure selects the backend instance.

Client → Load Balancer → Service Instance

Q6. How do API Gateway, Service Discovery and Load Balancer work together?

Answer:

The API Gateway identifies the target microservice based on the request. Service Discovery provides information about currently available service endpoints, and the Load Balancer distributes traffic among eligible instances.

Client
  ↓
API Gateway
  ↓
Target Service
  ↓
Service Discovery
  ↓
Load Balancer
  ↓
Service Instance

The actual ordering/implementation can vary because discovery may be consumed internally by the gateway, load balancer, proxy, or orchestration platform.

Q7. What is the difference between API Gateway and Load Balancer?

Answer:

An API Gateway primarily handles API-level concerns such as routing, authentication, rate limiting and transformations.

A Load Balancer primarily distributes traffic among multiple backend instances.

Q8. What is the difference between Service Discovery and Load Balancing?

Answer:

Service Discovery answers:

Where are the available instances?

Load Balancing answers:

Which available instance should receive the traffic?

Q9. What happens when a new service instance is created?

Once it is registered/discovered and considered healthy or ready, it can be added to the eligible backend pool and begin receiving traffic.

Q10. What happens when an instance fails?

Health/discovery mechanisms identify that the instance is unavailable, and it should be removed from the eligible load-balancing pool so new traffic goes to healthy instances.

Q11. Can an API Gateway also perform load balancing?

Yes.

Some API Gateway products can perform load balancing themselves. Alternatively, they can route to another load-balancing layer or a platform Service such as a Kubernetes Service.

Q12. Do we always need separate API Gateway, Service Registry and Load Balancer servers?

No.

They represent different architectural responsibilities, but modern platforms may combine or manage several of them.

For example, Kubernetes provides service discovery and internal service routing/load balancing through its Service, DNS, EndpointSlice, and networking mechanisms.

Interview Summary

In Microservices, the API Gateway provides the client-facing entry point and routes a request toward the appropriate microservice. Service Discovery determines the currently available locations of that service, while Load Balancing distributes traffic among its healthy instances. When autoscaling adds or removes instances, discovery and routing information changes dynamically, so clients and gateways do not need to maintain individual instance IP addresses.

A useful way to remember it is:

API Gateway
   ↓
"Which service?"

Service Discovery
   ↓
"Where is the service?"

Load Balancer
   ↓
"Which instance?"

Microservice Instance