Showing posts with label Ingress. Show all posts
Showing posts with label Ingress. Show all posts

Thursday, 2 July 2026

Mitigating Load Balancer Routing Clamping




In computing and networking, routing clumping (also known as traffic clumping or clustering) refers to the phenomenon where a load balancer or router unevenly sends bursts of traffic to the same backend servers instead of distributing requests evenly across the entire available server pool.

While a load balancer is designed to act as a traffic cop, routing clumping defeats this purpose, momentarily overloading specific servers while leaving others idle.

Primary Causes of Routing Clumping


  • Hashing Bias: Algorithms like 5-tuple hashing (which use source IP, destination IP, ports, and protocols) calculate a fixed value to select a routing path. If thousands of users share a proxy or exit node, the load balancer may calculate the same hash and clump all their traffic onto a single backend server.
  • Sticky Sessions: If you use session persistence (sticky sessions) based on a client's IP, multiple requests from the same office or network range will clump to a single server to maintain session continuity.
  • Connection Multiplexing: Load balancers maintain long-lived backend connections to save resources. When a new batch of requests comes in, the load balancer routes them through existing open connections, causing requests to cluster on specific nodes rather than opening new connections to idle servers.
  • Inconsistent Pod or Instance Count: In environments like Kubernetes, if traffic is processed node-by-node, unequal distribution of application pods across nodes or restrictive node affinities can cause traffic to clump in specific zones or servers.

How to Prevent and Mitigate Clumping


  • Use Round Robin or Least Connections: Switch from IP-hashing or fixed-hash methods to AWS Least Connections or round-robin algorithms, which evaluate server availability in real-time rather than relying on mathematical routing properties.
  • Enable Cross-Zone Load Balancing: For cloud architectures (like AWS Elastic Load Balancing), enabling cross-zone load balancing forces your load balancers to distribute traffic evenly across all registered targets in all enabled Availability Zones.
  • Use Layer 7 Intelligent Routing: Instead of Layer 4 packet forwarding, use a Layer 7 Application Load Balancer to inspect HTTP headers and distribute traffic based on request context, allowing for finer-grained distribution.
  • Deploy Hierarchical Splitting: For massive, hyperscale systems, implementing a hierarchical L4 + L7 structure (ultra-fast L4 hardware balancers routing to a software-based L7 pool) helps prevent clumping while maintaining high raw throughput.


Why round robin routing didn't work well in this scenario so we need to use least connections routing? 



Good question — the answer is about the kind of workload service3 is. Round robin isn't broken in general; it's the wrong fit for this specific traffic shape.

  What each algorithm does 
  • Round robin: hands each new request to the next target in rotation (pod A, B, C, A, B, C…), purely by count — blind to how busy each target currently is.
  • Least outstanding requests (LOR): sends each new request to the target with the fewest in-flight requests right now — i.e., the least-busy one.

Why round robin fails here

Round robin distributes by count, which only equals distributing by load when requests are uniform and short. service3 is the opposite — it has the toxic combination:

1. Highly variable, long service times — a spaCy extraction is anywhere from ~0.3s to multi-second depending on text length/entity count. One heavy request ties a pod up for seconds.
2. Low per-pod concurrency — each pod does ~1–2 concurrent extractions (2 workers, GIL-bound). It saturates almost immediately.

Under those conditions, round robin keeps handing new requests to a pod that's still grinding a heavy multi-second extraction, just because it's "next in rotation" — while a pod that just finished sits idle. The new request then queues head-of-line behind the heavy one for seconds, even though free capacity exists elsewhere. Over a burst this compounds: unlucky pods accumulate a backlog while others drain and idle → exactly the clumping / 33%-CPU-but-3s-latency pattern we measured.

The analogy: round robin is a supermarket sending shoppers to checkout lanes strictly 1‑2‑3‑1‑2‑3, ignoring that lane 2 has someone with a giant cart. You get stuck behind the big cart while lane 3 is empty. LOR is "go to the lane with the fewest items in it right now."

Why LOR fixes it

LOR tracks in-flight requests per target and skips the busy pod — a pod mid-heavy-extraction has a high outstanding count, so new requests go to the idle/draining pods instead. It self-corrects for variance (whichever pod is busy, for any reason, automatically gets fewer new requests until it catches up) and it reaches the idle capacity round robin was stranding.

The catch — why we also needed IP mode

LOR only helps if the ALB is choosing among the right units. In instance mode, the ALB's targets are nodes, so it'd balance outstanding requests across nodes — then kube-proxy randomly picks a pod within the node, re-scrambling the per-pod balance. LOR literally can't see per-pod busyness in instance mode.

IP mode makes the targets pods, so LOR's "least outstanding" is measured per pod → it can actually steer to the least-busy pod. That's why the fix is ip mode + LOR together: ip mode gives LOR the right granularity; LOR gives ip mode the right decision rule. Either alone wouldn't do it.

Two footnotes

- The other 5 services keep round robin — deliberately. auth/data/game/webhook are fast (<0.5s), uniform, high-concurrency — round robin distributes them fine, and there's no reason to touch them. The algorithm choice is workload-specific: LOR for
  slow/variable/expensive backends, round robin for fast/uniform ones.
- Terminology: you said "least connections" — the ALB (L7) equivalent is least outstanding requests, which counts in-flight HTTP requests, not TCP connections. That distinction matters here: the ALB reuses keep-alive connections (one connection carries many sequential requests), so "least connections" would be misleading — LOR counts the actual requests, which is the right signal for this.



In which cases round-robin doesn't help so we need to use least outstanding requests routing?


While Round-Robin works perfectly for uniform traffic and identical servers, it fails when the system experiences variations in request complexity or server capacity. You must switch from Round-Robin to Least Outstanding Requests (LOR) (also known as Least Connections) in the following four specific engineering scenarios:

1. Varying Request Processing Times (Asymmetric Workloads)


  • The Problem: In many applications, some API calls take 5 milliseconds (e.g., fetching a profile cached in memory), while others take 5 seconds (e.g., generating a heavy PDF report or running a complex database query).Why Round-Robin Fails: Round-Robin blindly hands out requests in a strict, alternating sequence. If Server A randomly receives a consecutive string of heavy PDF requests while Server B receives fast cache requests, Server A's queue will spike, causing high latency or timeouts, while Server B sits mostly idle.
  • How LOR Helps: LOR actively tracks the active connection count. It will notice Server A is backed up with pending work and will divert all new incoming traffic to Server B until Server A finishes its heavy processing.

2. Heterogeneous Server Capacities (Mixed Server Sizes)


  • The Problem: Production clusters often use mixed hardware. For example, during an auto-scaling event, you might temporarily mix older 4-core virtual machines with newer, high-performance 16-core instances.Why Round-Robin Fails: Round-Robin treats every backend target as equal. It sends exactly 1,000 requests to the weak 4-core machine and exactly 1,000 requests to the powerful 16-core machine. The weaker machine will quickly choke, run out of memory, or drop packets, while the stronger machine remains underutilised.
  • How LOR Helps: The faster, more powerful server processes and closes connections much quicker than the weaker server. Because its outstanding connection count drops rapidly, LOR naturally funnels a significantly higher volume of traffic to the stronger hardware without needing manual weight configurations.

3. Persistent and Long-Lived Connections (WebSocket & gRPC)


  • The Problem: Modern applications rely heavily on persistent connections like WebSockets, HTTP/2 multiplexing, server-sent events (SSE), or gRPC streams. These connections stay open for minutes or hours.Why Round-Robin Fails: Round-Robin only counts the initial connection establishment. If Server A hosts 50 clients who stay connected for 3 hours chatting, and Server B hosts 50 clients who disconnect after 30 seconds, Round-Robin will continue to feed new connections to both servers equally. Server A will slowly crush under the cumulative weight of long-lived active sessions.
  • How LOR Helps: LOR continuously monitors active, open connections rather than connection arrival rates. It will see that Server A has 50 active outstanding connections and Server B has 0, immediately routing all new clients to Server B.

4. Unpredictable Backend "Cold Starts" and Drifts


The Problem: When a new application container or server boots up (e.g., during a Kubernetes deployment or AWS auto-scaling event), it often suffers from a "cold start" where it runs slowly while warming up caches or compiling code just-in-time (JIT).Why Round-Robin Fails: Round-Robin immediately floods the newly booted server with its full share of production traffic. Because the server is not yet performing at 100% capacity, this sudden burst easily overwhelms it, causing it to crash immediately after launching.
How LOR Helps: LOR naturally throttles traffic to the warming server. Since the cold server processes its first few requests slowly, its outstanding request count will naturally rise, signaling the load balancer to back off and route traffic elsewhere until the server catches up.


Algorithm Comparison Table


  • Scenario
  • Round-Robin Behavior
  • Least Outstanding Requests Behavior
  • All requests take equal time 
  • Perfect, completely equal distribution.
  • Excellent, identical result to Round-Robin.
  • Mix of short & long requests
  • Overloads random servers (creates traffic clumps).
  • Dynamically balances the processing queue.
  • Mixed server capacities
  • Overwhelms smaller, weaker backend servers.
  • Directs more traffic to faster instances automatically.
  • Long-lived WebSockets / gRPC
  • Ignores session duration; causes massive load imbalance.
  • Tracks real-time active sessions perfectly.

If you are using a cloud platform, you can learn more about implementing this via the AWS Least Outstanding Requests Documentation or the NGINX Least Connections Guide.



k8s ingress - AWS ALB


To implement Least Outstanding Requests (LOR) routing with an AWS Application Load Balancer (ALB) inside a Kubernetes cluster, you must configure it using the AWS Load Balancer Controller.

By default, the AWS ALB uses Round-Robin routing. You can change this behavior by applying a specific routing annotation to your Kubernetes TargetGroupBinding or Ingress resource.


1. Ingress Configuration Example


Apply the alb.ingress.kubernetes.io/target-group-attributes annotation to your Ingress manifest. This tells the AWS controller to configure the underlying target groups to use the load_balancing.algorithm.type=least_outstanding_requests setting.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  namespace: production
  annotations:
    # Essential ALB Controller configurations
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    
    # Enable Least Outstanding Requests Routing
    alb.ingress.kubernetes.io/target-group-attributes: load_balancing.algorithm.type=least_outstanding_requests
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app-service
                port:
                  number: 80



2. Crucial Step: Use "IP" Target Type


When running AWS ALB in Kubernetes, you should always set alb.ingress.kubernetes.io/target-type: ip instead of instance mode. 

  • Why? In instance mode, the ALB routes traffic to NodePorts on the EC2 worker nodes. The node then uses kube-proxy (iptables or IPVS) to randomly route the packet to a pod. This completely breaks the math behind Least Outstanding Requests because the ALB can only see outstanding connections to the node, not the actual pods.
  • The Fix: Using ip mode configures the ALB to bypass the node network entirely and route traffic directly to the Pod IPs. This gives the ALB true visibility into the exact outstanding request count for each application container.

3. Verify the Configuration


After applying the manifest via kubectl apply -f ingress.yaml, you can verify the changes are active in AWS:
  • Open the AWS EC2 Console and navigate to Target Groups.
  • Select the target group automatically generated by your Kubernetes Ingress.
  • Click on the Attributes tab.
  • Verify that Routing algorithm is set to Least outstanding requests.

Alternatively, check the AWS Load Balancer Controller logs to ensure the modification was successfully reconciled:

kubectl logs -n kube-system deployment/aws-load-balancer-controller

---

Monday, 29 July 2024

Kubernetes Ingress Service


Ingress is a more flexible and powerful solution for managing external access to services within a Kubernetes cluster than Kubernetes LoadBalancer Service

It provides:
  • load balancing
  • SSL termination
  • name-based virtual hosting

Ingress controllers can be configured to handle traffic more efficiently and securely.

A full Ingress deployment manifest in Kubernetes defines how external traffic is routed to services within the cluster. It uses an Ingress controller (like NGINX) to manage the rules, and the manifest specifies which services should be exposed, based on hostnames, paths, or other criteria. The manifest typically includes details about the Ingress class, rules, and the services it exposes. 

Manifest example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  namespace: default
spec:
  ingressClassName: nginx # Or gce, aws, etc.
  rules:
  - host: example.com # or myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-service
            port:
              number: 80
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-service
            port:
              number: 80
  - host: myapp2.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-service2
            port:
              number: 80

Here is the meaning of all fields in the manifest:
  • apiVersion: Kubernetes API version
  • kind: Kubernetes object type (Ingress)
  • metadata: Metadata about the Ingress, including its name, namespace, and labels. 
  • spec: The heart of the manifest, defining the Ingress rules and behavior. 
    • ingressClassName: Specifies the Ingress controller to use (e.g., nginx, gce, aws).
    • rules: An array of rules that define how traffic should be routed. Each rule includes: 
      • host: The hostname to match (e.g., example.com). 
      • http: A section defining HTTP traffic routing.
        • paths: An array of paths to match within the HTTP request.
          • backend: The service to forward traffic to based on the path.
            • serviceName: The name of the service.
            • servicePort: The port of the service.



host

Removing the host (host-specific rules) from the ingress will make it accept traffic for any hostname, potentially exposing the service more broadly than intended. These changes could lead to unintended access to services if the ALB receives requests with different or no host headers.

serviceName

What is this service and what are its requirements so it can serve the traffic?


In Kubernetes (K8s), an Ingress manifest by itself does not directly create an AWS load balancer — but it can indirectly result in one being created depending on how your cluster is configured.

Ingress is just a K8s object that defines HTTP routing rules (e.g., "requests to /foo go to Service A"). It requires an Ingress Controller to actually implement those rules and expose the traffic. If we just apply an Ingress manifest and no Ingress Controller is installed, nothing will happen — no load balancer will be created.

To get an AWS Load Balancer via Ingress we need to install and use an Ingress Controller that integrates with AWS, such as:

  • AWS ALB Ingress Controller (now called AWS Load Balancer Controller)
  • Nginx Ingress Controller (but then you'd still need a Service of type LoadBalancer to expose it)

If we're using the AWS Load Balancer Controller and define an Ingress object with proper annotations, then it will create an AWS Application Load Balancer (ALB) and configure it with listeners and target groups according to the Ingress rules. This is the most direct way to have an Ingress lead to the creation of an AWS load balancer.

NOTE: LB creation takes some time e.g. 30 seconds.


Example: Ingress manifest that works with AWS Load Balancer Controller

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  namespace: default
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}]'
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-service
                port:
                  number: 80


Annotations

For full list of AWS ALB annotations, see Annotations - AWS Load Balancer Controller.

alb.ingress.kubernetes.io/scheme


The annotation alb.ingress.kubernetes.io/scheme in AWS Load Balancer Controller specifies the visibility and network exposure of the Application Load Balancer (ALB) created for your Kubernetes Ingress.

Possible values:
  • internet-facing
    • ALB is publicly accessible over the internet.
    • Use for services that need to be accessed publicly — e.g., websites, APIs, dashboards
  • internal
    • ALB is private, only accessible within your VPC (e.g., internal apps).
    • Use for internal microservices, admin interfaces, or when you only want access within a private network (e.g., from a VPN or other VPC resources).
Example (internet facing):

metadata:
  name: my-ingress
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing


This will create a public-facing ALB with a public DNS name like:

my-ingress-1234567890.us-west-2.elb.amazonaws.com


The subnets in your cluster must be correctly tagged:
  • For internet-facing: subnets must be public and tagged with: kubernetes.io/role/elb = 1
  • For internal: subnets must be private and tagged with: kubernetes.io/role/internal-elb = 1
If you don't set the scheme explicitly, it defaults to internet-facing.


Example: Internal Ingress 

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: internal-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/subnets: subnet-abc123,subnet-def456  # Optional: for precise control
    alb.ingress.kubernetes.io/group.name: internal-apps
    alb.ingress.kubernetes.io/tags: "env=dev,app=my-internal-service"
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-service
                port:
                  number: 80


This configuration instructs the AWS Load Balancer Controller to:
  • Create an internal ALB (i.e., not exposed to the public internet).
  • Place the ALB in private subnets (must be tagged: kubernetes.io/role/internal-elb = 1).
  • The ALB is reachable only within the VPC or from connected networks (like via VPN or Direct Connect).

The ALB will get a DNS name that looks like this:

internal-k8s-<name>-<hash>.<region>.elb.amazonaws.com

e.g.

internal-k8s-internalingr-abc123456789.us-west-2.elb.amazonaws.com

The internal- prefix on the DNS name indicates it is not public. It will not resolve outside the VPC (e.g., from your home network or the public internet).

We can verify the internal ALB:

  • AWS Console:
            Go to EC2 → Load Balancers → Look at the Scheme column (internal).
  • CLI:
     aws elbv2 describe-load-balancers \
         --names internal-k8s-internalingr-abc123456789


alb.ingress.kubernetes.io/listen-ports


The annotation alb.ingress.kubernetes.io/listen-ports in AWS Load Balancer Controller is used to customize the listener ports that are created on the Application Load Balancer (ALB) for a Kubernetes Ingress resource.

By default, the AWS Load Balancer Controller creates listeners for port 80 (HTTP) and/or port 443 (HTTPS) depending on your Ingress TLS settings.
If you need to override this behavior—for example, to support only HTTPS or use non-default ports—you can use this annotation.

Example: Default Behavior (HTTP and HTTPS):

alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'


Creates both:
  • an HTTP listener on port 80
  • an HTTPS listener on port 443 (requires TLS configuration)

It must be a JSON array of objects, where each object specifies a protocol and port.


Example: HTTPS Only (no HTTP listener):

alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'


Useful if you want to force HTTPS and avoid exposing port 80. Make sure you also configure tls: section in the Ingress and associate an ACM certificate using:

alb.ingress.kubernetes.io/certificate-arn


Example: Custom Ports (e.g., HTTP on 8080):

alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 8080}]'

ALB will listen on port 8080 for HTTP traffic instead of the default 80.

We can only use ports supported by ALB, which are typically: 1–65535, but port 80 and 443 are standard and expected by most clients.. Custom ports may not work well unless you control the client environment.


Example: Internal Service with Only HTTPS

annotations:
  alb.ingress.kubernetes.io/scheme: internal
  alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
  alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:...

If we specify only HTTPS, you must provide a valid certificate via alb.ingress.kubernetes.io/certificate-arn.

If we omit this annotation, the controller decides based on TLS block:
  • If TLS is configured, both HTTP and HTTPS are created
  • If not, only HTTP on port 80 is created



alb.ingress.kubernetes.io/tags


Specifies which tags will be applied onto Application Load Balancer that will be created by LB Controller. Example:

alb.ingress.kubernetes.io/tags: "kubernetes.io/ingress-name=${local.ingress_name},Environment=${local.workspace},Team=${local.team}"

This means that LB will have these tags:
  • kubernetes.io/ingress-name, set to value ${local.ingress_name}
  • Environment set to value ${local.workspace}
  • Team, set to value ${local.team}

Note that by default, AWS LB Controller applies the following tags:
  • ingress.k8s.aws/resource = LoadBalancer
  • ingress.k8s.aws/stack = <namespace>/<ingress_name>



alb.ingress.kubernetes.io/group.name

One of the possible annotations is alb.ingress.kubernetes.io/group.name. It is used in AWS Load Balancer Controller. It specifies the target group name for grouping multiple Ingress resources under a single Application Load Balancer (ALB). 

By default, each Ingress resource gets its own ALB, which can be costly or unnecessary. Setting the alb.ingress.kubernetes.io/group.name annotation allows you to share a single ALB across multiple Ingress resources.

Example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    alb.ingress.kubernetes.io/group.name: my-apps
    alb.ingress.kubernetes.io/scheme: internet-facing
spec:
  rules:
    - host: app1.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app1-service
                port:
                  number: 80


If we have another Ingress resource with the same group.name, it will be attached to the same ALB.

Benefits of using group.name:
  • Cost efficiency: Fewer ALBs to manage and pay for.
  • Simplified architecture: Group related services behind one ALB.
  • Custom routing: Combine multiple paths or hostnames on one load balancer.

All Ingresses in the same group must share the same ALB settings, such as:
  • alb.ingress.kubernetes.io/scheme (e.g. internet-facing or internal)
  • alb.ingress.kubernetes.io/load-balancer-name (optional but can be used)

The ALB is shared, but each rule/path is managed separately based on the Ingress definitions.



external-dns.alpha.kubernetes.io/ingress-hostname-source


 annotations-only: This annotation configures External DNS to only use explicit hostname annotations when determining the DNS records to create


When do we need to use Ingress?


Whether you need one depends entirely on who needs to reach the service/application.

We DO NOT need an Ingress (most common) if:
  • Service runs inside the cluster
  • Service clients run inside the cluster
  • Everything talks via Kubernetes service DNS e.g. http://tempo.grafana-tempo.svc.cluster.local

Then Service should stay ClusterIP only.

Why We Generally Should NOT Expose Application/Service?

If application/service is:
  • Not an end-user UI
  • Not meant for browser access
  • Not something external systems should hit directly
  • Not something we want publicly reachable
Exposing it increases attack surface unnecessarily.

We only need an Ingress if:

1) External write clients send requests directly to cluster

Example:
  • Apps outside cluster
  • On-prem services
  • External environments

Then they need: https://my-app.company.com


In that case, yes — create an Ingress or LoadBalancer.

2) External read clients runs outside the cluster

If App UI is external and querying App inside Kubernetes,
then we must expose App via:
  • Ingress
  • LoadBalancer
  • Private networking

Example: Grafana backend service has this URL:  http://tempo.grafana-tempo.svc.cluster.local:3200

That is:
  • Internal Kubernetes DNS
  • ClusterIP service
  • Port 3200 (Tempo HTTP API)
  • Namespace: grafana-tempo

So our Grafana Tempo is internal-only, and being queried from inside the cluster.

That means: we do NOT need an Ingress.
---


How can multiple ingresses share the same ALB?                                                                                                                                                

It's a feature of the AWS Load Balancer Controller called IngressGroup.
  
Default: one Ingress = one ALB

Out of the box, the controller provisions a dedicated ALB per Ingress resource. That's wasteful — an ALB is ~$16/mo + LCUs, and you'd get one per service.

IngressGroup: the group.name annotation


Ingresses that declare the same alb.ingress.kubernetes.io/group.name are merged by the controller into a single ALB:

# staging-ingress      → group.name: example-staging-default-k8s
# service3-staging     → group.name: example-staging-default-k8s   ← same group

The controller watches all Ingresses, buckets them by group.name, and for each bucket provisions one ALB whose listener rules are the union of every member Ingress's rules. That's exactly why, after our apply, kubectl get ingress -n staging showed both service3-staging-ingress and staging-ingress with the same ADDRESS (k8s-examplestagingdefau-74e6ea742a-…) — same ALB.

How routing works on the shared ALB


- Each backend service referenced by any member Ingress gets its own target group.
- Each host/path rule becomes a listener rule on the shared listener (443), with a host condition forwarding to that service's target group:

  service1.staging.example.com            → service1-api TG      (instance / round_robin)
  service2.staging.example.com            → service2-api TG      (instance / round_robin)
  service3.staging.example.com            → service3 TG            (ip / least_outstanding_requests)

- DNS: external-dns points every host at the one shared ALB DNS name.

What's shared vs what's per-Ingress — the key to our split


This is the crucial distinction that let us change only service3:
  • Scope: ALB-level (shared, must be consistent across the group) 
  • Examples: scheme, listener ports, certificates, security groups, WAF ACL, access-logs/idle-timeout, subnets
  • In our change: Replicated identically on the new Ingress so the group stays consistent (conflicting values make the controller error)   
  • Scope: Per-Ingress → per-target-group (can differ)
  • Examples: target-type (instance/ip), target-group-attributes (LOR/round_robin, stickiness), healthcheck, backend-protocol   
  • In our change: service3 TG gets ip + LOR; the other services' TGs stay instance + round_robin — same ALB

So the ALB is one shared object, but each service's target group is independent — which is precisely how we gave service3 ip-mode + least_outstanding_requests without touching the other 5 services. The controller confirmed it merged cleanly with SuccessfullyReconciled and no annotation conflict.

Why do it (and the trade-off you already met)


Benefits: one ALB instead of N → big cost saving, one DNS/cert/WAF/log stream to manage, consolidated metrics.

Trade-off: a shared ALB means shared metrics and a shared alarm. The target-response-time alarm is ALB-wide (request-weighted across all 6 services), so service3's latency drags the shared average and trips the alarm even though the other 5 are healthy. A dedicated ALB would isolate that — but as we discussed, it costs ~$25–35/mo and doesn't fix the latency, so we kept the shared ALB and fixed the routing instead.

Two operational notes

  • Rule ordering: when member Ingresses could have overlapping host/path rules, alb.ingress.kubernetes.io/group.order (an integer, lower = higher priority) controls listener-rule precedence. For distinct hosts (our case) there's no conflict, so we didn't need it.
  • ALB lifecycle is tied to the group, not any one Ingress: removing an Ingress just drops its rules/TG from the shared ALB (what we did — service3 moved from the shared Ingress's rules to its own); the ALB is only deprovisioned when the last member Ingress in the group is deleted.


Does k8s ingress gives us a fine grained control over alb target groups?


Yes, exactly. The Kubernetes Ingress resource—driven by the AWS Load Balancer Controller—provides immense control over underlying target group attributes natively through its annotation ecosystem. 

However, it is important to distinguish where the boundaries of this control lie and when a TargetGroupBinding custom resource (CRD) becomes the better choice.

What Ingress Can Control Natively


By using the alb.ingress.kubernetes.io/target-group-attributes annotation, you can pass comma-separated strings to adjust almost any native AWS Target Group feature directly from your YAML:
  • Routing Algorithms: Forcing load_balancing.algorithm.type=least_outstanding_requests or changing the algorithm on the fly.
  • Connection Draining: Adjusting deregulation delays via deregistration_delay.timeout_seconds=30.
  • Sticky Sessions: Configuring cookie-based persistence using stickiness.enabled=true,stickiness.type=lb_cookie.
  • Slow Start Mode: Gracefully warming up targets with slow_start.duration_seconds=30.

The "Catch" with Ingress-Driven Configuration


While highly granular, configuring everything inside the Ingress manifest ties the lifecycle of the AWS infrastructure strictly to the lifecycle of your Kubernetes objects.

  • Destructive Updates: If someone deletes or misconfigures the Ingress manifest, the AWS Load Balancer Controller will actively delete or alter the actual AWS Target Groups and listeners.
  • Monolithic Manifests: If you have dozens of backend services, your Ingress file becomes a massive block of text crammed with annotations, introducing formatting risk.
  • Blast Radius (Shared ALBs): If you use group.name to combine multiple Ingresses onto a single ALB, a bad change by one team can inadvertently overwrite shared settings or cause priority conflicts.

Moving to Ultimate Control: TargetGroupBinding

If your platform requires strict infrastructure isolation, you should decouple the load balancer from the Ingress object entirely using the TargetGroupBinding custom resource.

With this pattern, your cloud/Terraform team provisions the ALB, listeners, and Target Groups natively inside AWS (applying Least Outstanding Requests, certificates, and strict IAM rules there). You then hand the Target Group ARN to the Kubernetes team.

apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: decoupled-app-binding
  namespace: production
spec:
  # The controller watches this and syncs your Pod IPs straight to this AWS resource
  targetGroupARN: arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/my-pre-made-tg/abcdef123456
  targetType: ip
  serviceRef:
    name: my-app-service
    port: 80


Why Use TargetGroupBinding?

  • Zero Lifecycle Risk: Deleting the Kubernetes deployment or service will never delete the actual AWS infrastructure. The controller only manages the IP target list.
  • Multi-Cluster Routing: You can apply the exact same TargetGroupBinding file in Cluster A and Cluster B, allowing a single AWS ALB target group to seamlessly balance traffic across two entirely separate Kubernetes clusters.
  • Clean GitOps separation: Network teams manage AWS infrastructure via Terraform, while application developers handle standard K8s objects without touching infrastructure code.

How to implement blue-green deployment routing using TargetGroupBindings?


To implement a blue-green deployment using TargetGroupBindings, you leverage the absolute decoupling of the data plane.

Instead of updating Kubernetes manifests or re-routing traffic inside the cluster, you provision two independent AWS Target Groups (Blue and Green) via Terraform or the AWS Console. Your active traffic flip happens at the AWS ALB Listener rule level, completely safely, while the Kubernetes controller handles updating the actual pod IPs in the background.

Here is the step-by-step architecture and implementation pattern.

Step 1: Deploy Two TargetGroupBindings in Kubernetes


You deploy both your Blue and Green application deployments and services concurrently. Then, you map each Kubernetes service directly to its dedicated AWS Target Group ARN using two separate TargetGroupBinding manifests.

apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: app-blue-binding
  namespace: production
spec:
  # Points to the BLUE target group in AWS
  targetGroupARN: arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/app-blue-tg/111111111111
  targetType: ip
  serviceRef:
    name: app-blue-service
    port: 80
---
apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: app-green-binding
  namespace: production
spec:
  # Points to the GREEN target group in AWS
  targetGroupARN: arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/app-green-tg/222222222222
  targetType: ip
  serviceRef:
    name: app-green-service
    port: 80


At this point, the AWS Load Balancer Controller populates both target groups with the direct IPs of your pods.  AWS handles health checks for both environments independently.

Step 2: The Infrastructure State (Steady State)


Your AWS Application Load Balancer listener rule is configured to route 100% of live traffic to the Blue target group.

AWS ALB Listener (Port 443) 
     │
     └───► [Rule 1]: Forward to:
              ├──► app-blue-tg  (Weight: 100) ◄─── LIVE TRAFFIC
              └──► app-green-tg (Weight: 0)   ◄─── IDLE (Deploying New Code)


Step 3: Executing the Canary/Weighted Shift

When you want to roll out a new version, you deploy it to your app-green pods in Kubernetes. Once you verify the Green pods are healthy inside Kubernetes, you update your AWS ALB Listener Rule (using Terraform, AWS CLI, or an external progressive delivery tool like Argo Rollouts).You can execute a 10% canary test purely at the AWS level:

AWS ALB Listener (Port 443)
     │
     └───► [Rule 1]: Forward to:
              ├──► app-blue-tg  (Weight: 90)
              └──► app-green-tg (Weight: 10)  ◄─── Testing new release


Because both target groups inherit your Least Outstanding Requests (LOR) setting independently, the ALB smoothly evaluates traffic distribution across the two pools based on active requests without crashing your new containers.


Step 4: The Final Promotion (The "Flip")

If metrics, error rates, and latencies remain healthy on the green environment, you update the AWS ALB listener weights one final time to shift all users over:

AWS ALB Listener (Port 443)
     │
     └───► [Rule 1]: Forward to:
              ├──► app-blue-tg  (Weight: 0)   ◄─── OLD VERSION (Hold for rollback)
              └──► app-green-tg (Weight: 100) ◄─── NEW LIVE PRODUCTION


Why this pattern beats traditional Kubernetes Ingress flips:Zero DNS or K8s API Lag: Modifying an AWS ALB listener rule happens almost instantly at the cloud provider level. You do not have to wait for Kubernetes API reconciliation loops or internal routing tables to update.Instant Rollback: If the new Green release starts throwing 500 errors, you instantly change the AWS ALB weight back to app-blue-tg: 100. The rollback takes seconds and drops zero active connections.True Out-of-Band Testing: You can create a private AWS ALB listener (e.g., port 8080) that routes 100% to app-green-tg. This allows QA teams to run end-to-end integration tests on the new infrastructure using the live load balancer before production traffic ever sees it.If you want, I can:Provide the Terraform code to manage these weighted AWS ALB listener rulesShow how Argo Rollouts automates this specific TargetGroupBinding logic


Here is the complete architectural layout showing how requests flow through your infrastructure, alongside how the AWS Load Balancer Controller handles control plane configurations behind the scenes.


┌────────────────────────────────────────────────────────────────────────┐
│                          1. REQUEST DATA PLANE                         │
└────────────────────────────────────────────────────────────────────────┘

    [ External Clients ]               [ Internal Clients (VPC/VPN) ]
             │                                        │
             │ (HTTPS Request)                        │ (HTTP/HTTPS Request)
             ▼                                        ▼
   ┌──────────────────────────────────────────────────────────────────────┐
   │ ──► [ Route 53 DNS Lookup ] ──────────────────────────────────────── │
   │     Resolves domain name to ALB DNS name / IP addresses              │
   └──────────────────────────────────────────────────────────────────────┘
                                      │
                                      ▼
   ┌──────────────────────────────────────────────────────────────────────┐
   │ ──► [ AWS Application Load Balancer (ALB) ] ──────────────────────── │
   │     - Terminates TLS/SSL                                             │
   │     - Evaluates Listener Rules (Weights, Paths, Host headers)        │
   └──────────────────────────────────────────────────────────────────────┘
                                      │
                 ┌────────────────────┴────────────────────┐
                 │ (Routes based on Rule Weights / LOR)    │
                 ▼                                         ▼
   ┌──────────────────────────────┐          ┌──────────────────────────────┐
   │ [ AWS Target Group: BLUE ]   │          │ [ AWS Target Group: GREEN ]  │
   │ (Algorithm: LOR)             │          │ (Algorithm: LOR)             │
   └──────────────────────────────┘          └──────────────────────────────┘
                 │                                         │
                 │ (Direct Pod IP Routing Bypass)          │
                 ▼                                         ▼
   ┌──────────────────────────────────────────────────────────────────────┐
   │ [ AWS EC2 Worker Nodes ]                                             │
   │                                                                      │
   │   Pod A (Blue Application)               Pod C (Green Application)   │
   │   [IP: 10.0.1.45] ◄─────────┐            [IP: 10.0.1.92] ◄────────┐  │
   │                             │                                     │  │
   │   Pod B (Blue Application)  │            Pod D (Green Application)│  │
   │   [IP: 10.0.2.11] ◄─────────┘            [IP: 10.0.2.84] ◄────────┘  │
   │                                                                      │
   └──────────────────────────────────────────────────────────────────────┘


┌────────────────────────────────────────────────────────────────────────┐
│                        2. K8s KINETIC CONTROL PLANE                    │
└────────────────────────────────────────────────────────────────────────┘

   ┌──────────────────────────────────────────────────────────────────────┐
   │ [ Kubernetes API Server ]                                            │
   │                                                                      │
   │   Manifests: [ Ingress / TargetGroupBinding ]                        │
   │       Contains configuration properties like:                        │
   │       - load_balancing.algorithm.type=least_outstanding_requests    │
   └──────────────────────────────────────────────────────────────────────┘
             ▲
             │ Watches resources for changes
             ▼
   ┌──────────────────────────────────────────────────────────────────────┐
   │ [ AWS Load Balancer Controller Pod ] (Runs inside EKS/EC2 cluster)   │
   └──────────────────────────────────────────────────────────────────────┘
             │
             │ Reconciles state via AWS APIs
             ▼
   ┌──────────────────────────────────────────────────────────────────────┐
   │ [ AWS Cloud Control Plane ]                                          │
   │ Modifies Listener Rules, Target Group Attributes, and Target IPs     │
   └──────────────────────────────────────────────────────────────────────┘


Architectural Highlights

Unified Entrance Network: Both External Clients coming from the public internet and Internal Clients originating from inside your corporate VPC use Amazon Route 53 to find the entry point. Route 53 directs them straight to the single AWS Application Load Balancer (ALB).Separation of Concerns: Notice that the Kubernetes Ingress / TargetGroupBinding manifests and the AWS Load Balancer Controller do not handle actual application traffic. They operate strictly on the Control Plane to build and update the AWS components.Bypassing Node-Port Constraints (The IP Target Type): When traffic leaves the ALB Target Groups, it routes directly to individual Pod IPs inside the EC2 Worker Nodes. By utilizing the AWS VPC CNI network plugin, the traffic bypasses the Kubernetes kube-proxy layer entirely. This bypass ensures the ALB retains distinct visibility into the active connection metrics of each pod, keeping Least Outstanding Requests (LOR) calculations precise.