Showing posts with label Autoscaling. Show all posts
Showing posts with label Autoscaling. Show all posts

Friday, 6 February 2026

Kubernetes Cluster Autoscaler


Kubernetes Cluster Autoscaler (CAS):
  • Designed to automatically adjust the number of nodes (EC2 instances) in our cluster based on the resource requests of the workloads running in the cluster
  • Kubernetes project, supported on EKS: https://github.com/kubernetes/autoscaler

Key Features:

  • Node Scaling: It adds or removes nodes based on the pending pods that cannot be scheduled due to insufficient resources.
  • Pod Scheduling: Ensures that all pending pods are scheduled by scaling the cluster up.

It works with EKS Managed Node Groups backed by AWS Auto Scaling Groups. In node group, if we provide specific settings (like custom block_device_mappings), EKS creates an EC2 Launch Template under the hood.


Cluster Autoscaler and kube-scheduler


kube-scheduler is the default control plane component in Kubernetes responsible for deciding which Node a newly created or unscheduled Pod should run on. It essentially matches pods to the most suitable available machines based on resource requirements and specific constraints.

Cluster Autoscaler and kube-scheduler components DO NOT directly communicate with the other. Instead, the Cluster Autoscaler (CA) watches the kube-scheduler by monitoring the state of pods in the cluster. 

They work in an indirect loop via the Kubernetes API server: 
  • Kube-scheduler: Attempts to place pods on existing nodes. If it cannot find a node with sufficient capacity, it marks the pod as Pending with an Unschedulable status.
  • Cluster Autoscaler: Monitors the cluster for these Unschedulable pods.
  • Action: When CA detects a pending pod, it triggers a scale-up by adding a node.
  • Completion: Once the new node joins, the kube-scheduler notices the new capacity and schedules the pending pod. 

Key Takeaways:
  • The Autoscaler watches the Scheduler: The autoscaler reacts to the decisions (or failed attempts) of the scheduler.
  • No Direct Connection: They are "blissfully unaware" of each other and interact only through Kubernetes API objects.
  • Not Resource Based: The Cluster Autoscaler does not directly monitor node CPU/memory usage; it only cares if the scheduler cannot place a pod

This indirect workflow ensures that new nodes are only provisioned when necessary to satisfy pod scheduling constraints. 



How to check if it's installed and enabled?


(1) Look for its deployment


Cluster Autoscaler usually runs as a Deployment in kube-system namespace so we can look for that deployment:

% kubectl get deployments -n kube-system | grep -i cluster-autoscaler

cluster-autoscaler-aws-cluster-autoscaler   2/2     2            2           296d

We can also list pods directly:

% kubectl get pods -n kube-system | grep -i cluster-autoscaler

cluster-autoscaler-aws-cluster-autoscaler-7cbb844455-q2lxv 1/1 Running 0 206d
cluster-autoscaler-aws-cluster-autoscaler-7cbb844455-vhbsw 1/1 Running 0 206d

If we see a pod running, it’s installed.

Typical names:
  • cluster-autoscaler
  • cluster-autoscaler-aws-clustername
  • cluster-autoscaler-eks-...

(2) Inspect the Deployment 


Confirm it’s enabled & configured.

% kubectl describe deployment cluster-autoscaler -n kube-system

Name:                   cluster-autoscaler-aws-cluster-autoscaler
Namespace:              kube-system
CreationTimestamp:      Wed, 16 Apr 2025 12:25:38 +0100
Labels:                 app.kubernetes.io/instance=cluster-autoscaler
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=aws-cluster-autoscaler
                        helm.sh/chart=cluster-autoscaler-9.46.6
Annotations:            deployment.kubernetes.io/revision: 1
                        meta.helm.sh/release-name: cluster-autoscaler
                        meta.helm.sh/release-namespace: kube-system
Selector:               app.kubernetes.io/instance=cluster-autoscaler,app.kubernetes.io/name=aws-cluster-autoscaler
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/instance=cluster-autoscaler
                    app.kubernetes.io/name=aws-cluster-autoscaler
  Service Account:  cluster-autoscaler-aws-cluster-autoscaler
  Containers:
   aws-cluster-autoscaler:
    Image:      registry.k8s.io/autoscaling/cluster-autoscaler:v1.32.0
    Port:       8085/TCP
    Host Port:  0/TCP
    Command:
      ./cluster-autoscaler
      --cloud-provider=aws
      --namespace=kube-system
      --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/mycorp-prod-mycluster
      --logtostderr=true
      --stderrthreshold=info
      --v=4
    Liveness:  http-get http://:8085/health-check delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAMESPACE:     (v1:metadata.namespace)
      SERVICE_ACCOUNT:   (v1:spec.serviceAccountName)
      AWS_REGION:       us-east-1
    Mounts:             <none>
  Volumes:              <none>
  Priority Class Name:  system-cluster-critical
  Node-Selectors:       <none>
  Tolerations:          <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   cluster-autoscaler-aws-cluster-autoscaler-7cbb844455 (2/2 replicas created)
Events:          <none>


Key things to look for:
  • Replicas ≥ 1
  • No crash loops
  • Command args like:
    • --cloud-provider=aws
    • --nodes=1:10:nodegroup-name
    • --balance-similar-node-groups

If replicas are 0, it’s installed but effectively disabled.

(3) Check logs


Is it actively scaling?

This confirms it’s working, not just running.

kubectl logs deployment/cluster-autoscaler -n kube-system 

or find pods:

kubectl get pods \
    -l app.kubernetes.io/name=cluster-autoscaler \
    -n kube-system

Then check logs:

kubectl logs \
    -l app.kubernetes.io/name=cluster-autoscaler \
    -n kube-system \
    | grep "Standard-Autoscaler" 

Healthy / active signs:
  • scale up
  • scale down
  • Unschedulable pods
  • Node group ... increase size
  • If you see messages like Refresher: resolving ASGs, it will list the names of the ASGs it is currently monitoring.

Red flags:
  • AccessDenied
  • no node groups found
  • failed to get ASG

(4) Check for unschedulable pods trigger


If CA is working, it reacts to pods stuck in Pending.

% kubectl get pods -A | grep Pending

If pods are pending and CA logs mention them → CA is enabled and reacting.

(5) AWS EKS-specific checks (very common)


a) Check IAM permissions (classic failure mode)

Cluster Autoscaler must run with an IAM role that can talk to ASGs.

% kubectl -n kube-system get sa | grep autoscaler

cluster-autoscaler-aws-cluster-autoscaler     0         296d
horizontal-pod-autoscaler                     0         296d

Let's inspect cluster-autoscaler-aws-cluster-autoscaler service accont:

% kubectl -n kube-system get sa cluster-autoscaler-aws-cluster-autoscaler  -o yaml

apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxx:role/mycorp-prod-mycluster-cluster-autoscaler
    meta.helm.sh/release-name: cluster-autoscaler
    meta.helm.sh/release-namespace: kube-system
  creationTimestamp: "2026-04-16T11:25:37Z"
  labels:
    app.kubernetes.io/instance: cluster-autoscaler
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: aws-cluster-autoscaler
    helm.sh/chart: cluster-autoscaler-9.46.6
  name: cluster-autoscaler-aws-cluster-autoscaler
  namespace: kube-system
  resourceVersion: "15768"
  uid: 0a7da521-1bf5-5a5f-a155-8801e876ea7b


Look for:

eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/ClusterAutoscalerRole

If missing → CA may exist but cannot scale.

b) Check Auto Scaling Group tags

Our node group ASGs must be tagged:

k8s.io/cluster-autoscaler/enabled = true
k8s.io/cluster-autoscaler/<cluster-name> = owned

Without these → CA runs but does nothing. If those tags are missing, Cluster Autoscaler will ignore that ASG entirely.

(6) Check Helm (if installed via Helm)

Let's list all Helm releases across every namespace in a Kubernetes cluster and look for cluster autoscaler:

% helm list -A
NAME                  NAMESPACE      REVISION UPDATED      
cluster-autoscaler    kube-system    1           2025-04-16 12:25:30.389073326 +0100BST

STATUS   CHART                                APP VERSION
deployed cluster-autoscaler-9.46.6          1.32.0     


Then:

helm status cluster-autoscaler -n kube-system

The command helm list -A (or its alias helm ls -A) is used to list all Helm releases across every namespace in a Kubernetes cluster. Helm identifies your cluster and authenticates through the same mechanism as kubectl: the kubeconfig file. It uses the standard Kubernetes configuration file, typically located at ~/.kube/config, to determine which cluster to target.


(7) Double-check it’s not replaced by Karpenter


Many newer EKS clusters don’t use Cluster Autoscaler anymore.

% kubectl get pods -A | grep -i karpenter

kube-system karpenter-6f67b8c97b-lbq8p 1/1 Running     0       206d
kube-system karpenter-6f67b8c97b-wmprj 1/1 Running     0       206d


If Karpenter is installed, Cluster Autoscaler usually isn’t (or shouldn’t be).

Quick decision table

-----------------------------------------------------------------
Symptom                         Meaning
-----------------------------------------------------------------
No CA pod                         Not installed
Pod running, replicas=0         Installed but disabled
Logs show AccessDenied Broken IAM
Pods Pending, no scale-up ASG tags / config issue
Karpenter present                 CA likely not used
-----------------------------------------------------------------


(8) Check the "Status" ConfigMap


Cluster Autoscaler maintains a ConfigMap that shows which groups it is managing and if they are at their max/min size:


% kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml

apiVersion: v1
data:
  status: |
    time: 2026-03-21 08:52:07.308206626 +0000 UTC
    autoscalerStatus: Running
    clusterWide:
      health:
        status: Healthy
        nodeCounts:
          registered:
            total: 6
            ready: 6
            notStarted: 0
          longUnregistered: 0
          unregistered: 0
        lastProbeTime: "2026-03-21T08:52:07.308206626Z"
        lastTransitionTime: "2026-03-20T16:30:07.460032826Z"
      scaleUp:
        status: NoActivity
        lastProbeTime: "2026-03-21T08:52:07.308206626Z"
        lastTransitionTime: "2026-03-20T16:30:07.460032826Z"
      scaleDown:
        status: NoCandidates
        lastProbeTime: "2026-03-21T08:52:07.308206626Z"
        lastTransitionTime: "2026-03-20T16:30:07.460032826Z"
    nodeGroups:
    - name: eks-mycorp-env-app-k8s-v1_33-202603...04-2cc...dc8
      health:
        status: Healthy
        nodeCounts:
          registered:
            total: 2
            ready: 2
            notStarted: 0
          longUnregistered: 0
          unregistered: 0
        cloudProviderTarget: 2
        minSize: 2
        maxSize: 10
        lastProbeTime: "2026-03-21T08:52:07.308206626Z"
        lastTransitionTime: "2026-03-20T16:30:07.460032826Z"
      scaleUp:
        status: NoActivity
        lastProbeTime: "2026-03-21T08:52:07.308206626Z"
        lastTransitionTime: "2026-03-20T16:30:07.460032826Z"
      scaleDown:
        status: NoCandidates
        lastProbeTime: "2026-03-21T08:52:07.308206626Z"
        lastTransitionTime: "2026-03-20T16:30:07.460032826Z"
kind: ConfigMap
metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/last-updated: 2026-03-21 08:52:07.308206626 +0000
      UTC
  creationTimestamp: "2026-03-20T16:29:56Z"
  name: cluster-autoscaler-status
  namespace: kube-system
  resourceVersion: "18...78"
  uid: 17b...0af

Installation and Setup:


To use the Cluster Autoscaler in the EKS cluster we need to deploy it using a Helm chart or a pre-configured YAML manifest.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml


In Terraform:

resource "helm_release" "cluster_autoscaler" {
  name = "cluster-autoscaler"

  repository = "https://kubernetes.github.io/autoscaler"
  chart      = "cluster-autoscaler"
  version    = "9.46.6"
  namespace  = "kube-system"

  set {
    name  = "autoDiscovery.clusterName"
    value = local.cluster_name
  }

  set {
    name  = "awsRegion"
    value = local.aws_region
  }

  set {
    name  = "rbac.serviceAccount.create"
    value = "false"
  }

  set {
    name  = "rbac.serviceAccount.name"
    value = local.service_account_name
  }
}

Configuration:

  • Ensure the --nodes flag in the deployment specifies the min and max nodes for your node group.
  • Annotate your node groups with the k8s.io/cluster-autoscaler tags to enable autoscaler to manage them.

How to know if node was provisioned by Cluster Autoscaler?


Cluster Autoscaler applies labels on nodes it provisions so let's check labels:

% kubectl get nodes --show-labels

If label like eks.amazonaws.com/nodegroup exists, node was launched by and belongs to EKS Managed Node Group as Cluster Autoscaler launched the node.

Example:

% kubectl get nodes --show-labels

NAME                                     STATUS ROLES  AGE  VERSION            
ip-10-2-1-244.us-east-1.compute.internal Ready  <none> 206d v1.32.3-eks-473151a 

LABELS
Environment=prod,
beta.kubernetes.io/arch=amd64,
beta.kubernetes.io/instance-type=m5.xlarge,
beta.kubernetes.io/os=linux,
eks.amazonaws.com/capacityType=ON_DEMAND,
eks.amazonaws.com/nodegroup-image=ami-07fa6c030f5802c74,
eks.amazonaws.com/nodegroup=mycorp-prod-mycluster-20260714151819635800000002,
eks.amazonaws.com/sourceLaunchTemplateId=lt-0edc7a2b08ea82a28,
eks.amazonaws.com/sourceLaunchTemplateVersion=1,
failure-domain.beta.kubernetes.io/region=us-east-1,
failure-domain.beta.kubernetes.io/zone=us-east-1a,
mycorp;/node-type=default,
k8s.io/cloud-provider-aws=12b0e11196b7091c737cf66015f19720,
kubernetes.io/arch=amd64,
kubernetes.io/hostname=ip-10-2-1-244.us-east-1.compute.internal,
kubernetes.io/os=linux,
node.kubernetes.io/instance-type=m5.xlarge,
topology.ebs.csi.aws.com/zone=us-east-1a,
topology.k8s.aws/zone-id=use1-az1,
topology.kubernetes.io/region=us-east-1,
topology.kubernetes.io/zone=us-east-1a


If we list all nodegroups in the cluster, the one above is listed:

% aws eks list-nodegroups \
    --cluster-name mycorp-env-app-k8s \
    --profile my_profile
{
    "nodegroups": [
        "mycorp-env-app-k8s-20260714151819635800000002"
    ]
}

To inspect the nodegroup, including its labels, use:

% aws eks describe-nodegroup \
    --cluster-name mycorp-env-app-k8s \
    --nodegroup-name mycorp-env-app-k8s-v1_33-20260...03 \
    --region us-east-2 \
    --profile my_profile \
    --output json
{
    "nodegroup": {
        "nodegroupName": "mycorp-env-app-k8s-v1_33-20260...003",
        "nodegroupArn": "arn:aws:eks:us-east-2:xxxx:nodegroup/mycorp-env-app-k8s/mycorp-env-app-k8s-v1_33-202....03/2cce8....c7",
        "clusterName": "mycorp-env-app-k8s",
        "version": "1.33",
        "releaseVersion": "1.33.8-20260317",
        "createdAt": "2026-03-20T12:41:07.961000+00:00",
        "modifiedAt": "2026-03-22T09:21:58.892000+00:00",
        "status": "ACTIVE",
        "capacityType": "ON_DEMAND",
        "scalingConfig": {
            "minSize": 2,
            "maxSize": 10,
            "desiredSize": 2
        },
        "instanceTypes": [
            "m5.large"
        ],
        "subnets": [
            "subnet-02xxx",
            "subnet-00xxx",
            "subnet-04xxx"
        ],
        "amiType": "AL2023_x86_64_STANDARD",
        "nodeRole": "arn:aws:iam::xxx:role/mycorp-env-app-k8s-v1_33-eks-node-group",
        "labels": {
            "Environment": "prod",
            "mycorp/node-type": "v1.33"
        },
        "resources": {
            "autoScalingGroups": [
                {
                    "name": "mycorp-env-app-k8s-v1_33-202603...dc7"
                }
            ]
        },
        "health": {
            "issues": []
        },
        "updateConfig": {
            "maxUnavailablePercentage": 33
        },
        "launchTemplate": {
            "name": "mycorp-env-app-k8s-v1_33-202...0001",
            "version": "1",
            "id": "lt-xxx"
        },
        "tags": {
            "ClusterName": "mycorp-env-app-k8s",
            "Environment": "prod",
            "terraform-aws-modules": "eks",
            "Terraform": "true",
            "Name": "mycorp-env-app-k8s-v1_33"
        }
    }
}


If we are using terraform-aws-modules/eks/aws to provision EKS cluster and within it we define EKS-managed node groups, this module will create AWS Autoscaling Group (ASG) for each of them. Their names can be read from module's output variable eks_managed_node_groups_autoscaling_group_names.

If we inspect the labels on one such ASG, we can see that this Terraform module attached labels on it, so CAS can discover it and manage its parameters (usually just desired_size which is used for increasing or decreasing the number of current EC2 instances):

% aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names eks-mycorp-env-app-k8s-v1_33-20260320...4-2c...8 \
  --query "AutoScalingGroups[].Tags[?starts_with(Key, 'k8s.io/cluster-autoscaler')]" \
  --region us-east-2 \
  --profile my_profile
[
    [
        {
            "ResourceId": "eks-mycorp-env-app-k8s-v1_33-20260320...4-2c...8",
            "ResourceType": "auto-scaling-group",
            "Key": "k8s.io/cluster-autoscaler/enabled",
            "Value": "true",
            "PropagateAtLaunch": true
        },
        {
            "ResourceId": "eks-mycorp-env-app-k8s-v1_33-20260320...4-2c...8",
            "ResourceType": "auto-scaling-group",
            "Key": "k8s.io/cluster-autoscaler/mycorp-env-app-k8s",
            "Value": "owned",
            "PropagateAtLaunch": true
        }
    ]
]

In the example above, mycorp-env-app-k8s is the name of the cluster.



If cluster is overprovisioned, why Cluster Autoscaler doesn't scale nodes down automatically?


If Cluster Autoscaler is running but not shrinking the cluster, it's usually because:
  • System Pods: Pods like kube-dns or metrics-server don't have PDBs (Pod Disruption Budgets) and CA is afraid to move them.

  • Local Storage: A pod is using emptyDir or local storage.

  • Annotation: A pod has the "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" annotation.

  • Manual Overrides: Check if someone manually updated the Auto Scaling Group (ASG) or the EKS Managed Node Group settings in the AWS Console. Terraform won't automatically "downgrade" those nodes until the next terraform apply or a node recycle.

  • If nodes are very old, they are "frozen" in time. Even if you changed your Terraform to smaller EC2 instances recently, EKS Managed Node Groups do not automatically replace existing nodes just because the configuration changed. They wait for a triggered update or a manual recycling of the nodes.


How to fix this overprovisioning?


Since your current Terraform state says you want e.g. 2 nodes of m5.large, but the reality is e.g. 4 nodes of m5.xlarge, you need to force a sync.

Step 1: Check for Drift

Run a terraform plan. It will likely show that it wants to update the Launch Template or the Node Group version to switch from xlarge back to large.

Step 2: Trigger a Rolling Update

If you apply the Terraform and nothing happens to the existing nodes, you need to tell EKS to recycle them. You can do this via the AWS CLI:

aws eks update-nodegroup-version \
    --cluster-name <your-cluster-name> \
    --nodegroup-name <your-nodegroup-name> \
    --force

Note: This will gracefully terminate nodes one by one and replace them with the new m5.large type defined in your TF.



Cluster Autoscaler VS Karpenter


CAS (Cluster Autoscaler) and Karpenter are Kubernetes tools for adjusting node capacity based on workload, with CAS relying on fixed node groups and slow, infrastructure-driven scaling. Karpenter is a faster, modern, open-source, workload-driven node provisioner that directly interacts with cloud APIs, improving efficiency and cost-optimization.

Cluster Autoscaler (CAS): Operates by adjusting the size of specific, pre-defined node groups (e.g., autoscaling groups). It is generally better suited for smaller, predictable, or steady-state workloads where strict node group management is preferred.

Karpenter: Evaluates pending pods and launches optimally sized nodes directly, bypassing the need for manual node group management. It is ideal for high-churn, highly dynamic, and cost-sensitive, large-scale production environments.

While both tools scale Kubernetes nodes to meet pod demand, they use fundamentally different approaches. Cluster Autoscaler (CA) is the traditional, "group-based" tool that adds nodes to existing pools, whereas Karpenter is a "provisioning" tool that directly creates the specific instances your applications need. 


Quick Feature Comparison Table


Scaling Logic
  • Cluster Autoscaler (CA): Scales pre-defined node groups (ASGs)
  • Karpenter: Directly provisions individual EC2 instances.

Speed
  • Cluster Autoscaler (CA): Slower; waits for cloud provider group updates
  • Karpenter: Faster; provisions nodes in seconds via direct APIs; better for rapid, "spiky" traffic.

Cost Control
  • Cluster Autoscaler (CA): Limited; uses fixed node sizes in groups.
  • Karpenter: High; picks the cheapest/optimal instance for the pod. It has built-in node consolidation, which intelligently reduces costs by binpacking, or packing, pods onto fewer, more efficient nodes.

Complexity
  • Cluster Autoscaler (CA): Higher; must manage multiple node groups.
  • Karpenter: Lower; one provisioner can handle many pod types. 

Flexibility
  • Karpenter: supports diverse instance types and, while commonly used with AWS, it can be used with other providers.

Configuration
  • Karpenter uses Kubernetes-native YAML for defining node pools and node classes.

Key Differences


Infrastructure Model:
  • CA asks, "How many more of these pre-configured nodes do I need?". 
  • Karpenter asks, "What specific resources (CPU, RAM, GPU) does this pending pod need right now?" and builds a node to match.

Node Groups: 
  • CA requires you to manually define and maintain Auto Scaling Groups (ASGs) for different instance types or zones. 
  • Karpenter bypasses ASGs entirely, allowing it to "mix and match" instance types dynamically in a single cluster.

Consolidation: 
  • Karpenter actively monitors the cluster to see if it can move pods to fewer or cheaper nodes to save money (bin-packing). 
  • While CA has a "scale-down" feature, it is less aggressive at optimizing for cost.

Spot Instance Management: 
  • Karpenter handles Spot interruptions and price changes more natively, selecting the most stable and cost-efficient Spot instances in real-time.

Which should you choose?


Use Cluster Autoscaler if you need a stable, battle-tested solution that works across multiple cloud providers (GCP, Azure) or if your workloads are very predictable and don't require rapid scaling.

Use Karpenter if you are on AWS EKS, need to scale up hundreds of nodes quickly, want to heavily use Spot instances, or want to reduce the operational burden of managing dozens of node groups.

Disable Cluster Autoscaler if you plan to use Karpenter. Having both leads to race conditions and wasted cost.

When to Run Both Together

It's generally not recommended to run Cluster Autoscaler and Karpenter together in the same cluster. However, there are specific scenarios where it might be acceptable:

Valid use cases for running both:
  • Migration period: Transitioning from Cluster Autoscaler to Karpenter, where you temporarily run both while gradually moving workloads
  • Hybrid node management: Managing distinct, non-overlapping node groups where Cluster Autoscaler handles some node groups and Karpenter handles others (though this adds complexity)

When It's Not Recommended (and Why)

Primary reasons to avoid running both:

Conflicting decisions: Both tools make independent scaling decisions, which can lead to:
  • Race conditions where both try to provision nodes simultaneously
  • Inefficient resource allocation
  • Unpredictable scaling behavior
  • One tool removing nodes the other just provisioned

Increased operational complexity:
  • Two systems to monitor, troubleshoot, and maintain
  • Doubled configuration overhead
  • More difficult to understand which tool made which scaling decision

Resource contention: Both tools consume cluster resources and API server capacity, adding unnecessary load.

No significant benefits: Karpenter can handle everything Cluster Autoscaler does, often more efficiently, so there's rarely a technical need for both.

EKS-Specific Considerations

The same principles apply to AWS EKS clusters, with some additional context:

EKS particularities:
  • Karpenter was designed specifically for AWS/EKS and integrates deeply with EC2 APIs
  • Karpenter typically provides better performance on EKS (faster provisioning, better bin-packing)
  • If you're on EKS, the general recommendation is to choose Karpenter over Cluster Autoscaler for new deployments

Migration best practice for EKS: If migrating from Cluster Autoscaler to Karpenter on EKS, ensure they manage completely separate node groups, and complete the migration as quickly as feasible to minimize the period of running both.


How to migrate pods from nodes deployed by Cluster Autoscaler to those deployed by Karpenter?


If you'd rather use Karpenter for everything, you should eventually set your min_size, max_size, and desired_size to 0 in this node group and let Karpenter handle the provisioning instead.

---

Kubernetes Metrics Server

 


Kubernetes Metrics Server is a foundational component required by several other critical cluster modules and tools: 

1. Horizontal Pod Autoscaler (HPA)

2. Vertical Pod Autoscaler (VPA) 
  • Purpose: While HPA adds more pods, the Vertical Pod Autoscaler (VPA) adjusts the CPU and memory requests/limits of existing pods.
  • Dependency: VPA relies on Metrics Server for the real-time resource data it uses to recommend or apply these resource changes. 

2. Native CLI Observability (kubectl top) 
  • Purpose: Commands used for ad-hoc debugging and performance monitoring.
  • Dependency: Both kubectl top pods and kubectl top nodes query the Metrics API directly. Without the server, these commands will return an error. 

3. Kubernetes Dashboard 
  • Purpose: A web-based UI for managing and troubleshooting clusters.
  • Dependency: The Kubernetes Dashboard uses Metrics Server to display resource usage graphs and live statistics for nodes and pods. 

4. Third-Party Monitoring Tools & Adapters
  • Custom Metrics Adapters: Some adapters that bridge external sources (like CloudWatch or Datadog) to Kubernetes may use the standard Metrics API for fallback or basic resource data.
  • Resource Management Tools: Operational tools such as Goldilocks, which suggests "just right" resource requests, often depend on the baseline metrics provided by this server. 

Key Distinction


While the Metrics Server is essential for these control loops (HPA, VPA), it is not a replacement for a full observability stack like Prometheus. It only stores a short-term, in-memory snapshot and does not provide historical data

How to to install the Metrics Server as an EKS Community Add-on to enable these features?


In March 2025, AWS introduced a new catalog of community add-ons that includes the Metrics Server. This allows you to manage it directly through EKS-native tools like any other AWS-managed add-on (e.g., VPC CNI or CoreDNS). 

Method 1: Using the AWS Management Console


The easiest way to install it is through the EKS console: 
  • Navigate to your EKS cluster in the AWS Console.
  • Select the Add-ons tab and click Get more add-ons.
  • Scroll down to the Community add-ons section.
  • Find Metrics Server, select it, and click Next.
  • Choose the desired version (usually the latest recommended) and click Create. 

Method 2: Using the AWS CLI


You can also install the community add-on via the command line:

aws eks create-addon \
  --cluster-name <YOUR_CLUSTER_NAME> \
  --addon-name metrics-server

Verification


Once the installation status moves to Active, verify that the pods are running in the kube-system namespace: 

kubectl get deployment metrics-server -n kube-system

Finally, test that the Metrics API is responding:

kubectl top nodes

Note: If you are using AWS Fargate, you may need to update the containerPort from 10250 to 10251 in the deployment configuration to ensure compatibility with Fargate's networking constraints. 


Metrics Server Configuration



To configure custom resource limits for the Metrics Server EKS community add-on, you can use Configuration Values during installation or update. This is essential for high-pod-count clusters where the default allocation may lead to OOMKilled errors. 

1. Scaling Recommendations


The Metrics Server's resource consumption scales linearly with your cluster's size. Baseline recommendations include: 
  • CPU: Approximately 1 millicore per node in the cluster.
  • Memory: Approximately 2 MB of memory per node.
  • Large Clusters: If your cluster exceeds 100 nodes, it is recommended to double these defaults and monitor performance. 

2. How to Apply Custom Limits


You can provide a JSON or YAML configuration block via the AWS EKS Add-ons API. 

Via AWS CLI


Use the configuration-values flag to pass your resource overrides:

aws eks create-addon \
  --cluster-name <YOUR_CLUSTER_NAME> \
  --addon-name metrics-server \
  --configuration-values '{
    "resources": {
      "requests": { "cpu": "100m", "memory": "200Mi" },
      "limits": { "cpu": "200m", "memory": "500Mi" }
    }
  }'


Via AWS Console

  • Go to the Add-ons tab in your EKS cluster.
  • Click Edit on the metrics-server add-on.
  • Expand the Optional configuration settings.
  • Paste the JSON configuration into the Configuration values text box. 

3. Critical Configuration for High Traffic


In addition to resource limits, you may want to adjust the scraping frequency to make HPA more responsive.

  • Metric Resolution: The default is 60s. For faster scaling, add --metric-resolution=15s to the container arguments via the same configuration block.
  • High Availability: The community add-on defaults to 2 replicas to prevent downtime during scaling events. 



Thursday, 5 February 2026

Horizontal Pod Autoscaler (HPA)

 


The Horizontal Pod Autoscaler (HPA) serves to automatically align your application's capacity with its real-time demand by adjusting the number of pod replicas. Its operation depends on several critical components and configurations within an EKS or Kubernetes cluster. 

Kubernetes Horizontal Pod Autoscaler (HPA) is a built-in Kubernetes controller
  • Standard Kubernetes autoscaling mechanism
  • HPA API is available out of the box. It is a part of the Core Kubernetes and does not need installing third-party controllers or addons.
    • But it is NOT fully operational "by default" in a standard Amazon EKS cluster. API definitions for HPA resources exist within Kubernetes, but they require a Metrics Server to function—which AWS does not install for you automatically during cluster creation.

It's "standard" in the sense that the feature is built into the Kubernetes control plane, but it isn't "automatic" in the sense that it guesses which of your apps need scaling.

Think of it like a Thermostat: The thermostat (HPA Controller) is already installed on the wall (EKS Control Plane), but it won't turn on the AC until you tell it what the Target Temperature (CPU/Memory threshold) is and which room (Deployment) to monitor.

Here is why a manifest is required for every app:

1. The Controller vs. The Resource


The Controller (The "How"): This is a loop running inside the EKS Control Plane. It is always active, waiting for instructions. Kubernetes HPA Documentation explains this loop.
The Resource (The "What"): The HPA Manifest is that instruction. It tells the controller: "Watch Deployment X, keep CPU at 50%, and don't go above 10 pods."

2. Manual Intent


Kubernetes follows a Declarative Model. It never assumes you want to scale. If it scaled every pod automatically, a single bug in your code (like an infinite loop) could scale your cluster to 1,000 nodes and drain your AWS budget instantly. You must explicitly opt-in by creating the HPA resource.

3. Unique Criteria for Every App


Not all apps scale the same way:
  • Web API: Might scale when CPU hits 70%.
  • Background Worker: Might scale based on Memory usage.
  • Data Processor: Might scale based on a Custom Metric like SQS queue depth.

Summary: What is "Standard"?


What is standard is the API definition and the Controller. What is not standard is your specific application's scaling logic.

To see what the HPA Controller is looking for, you can check your Deployment's resource requests via kubectl:

kubectl get deployment <name> -o yaml | grep resources -A 5


Key Features:

  • Pod Scaling: Adjusts the number of pod replicas to match the demand.
  • Automatically scales up/down the number of pods in a deployment, replication controller, or replica set based on observed CPU utilization, memory or other selected custom/external metrics.

Purpose

  • Dynamic Scalability: Automatically adds pods during traffic surges to maintain performance and removes them during low-traffic periods to reduce waste.
  • Cost Optimisation: Ensures you only pay for the compute resources currently needed rather than over-provisioning for peak loads.
  • Resilience & Availability: Prevents application crashes and outages by proactively scaling out before resources are fully exhausted.
  • Operational Efficiency: Replaces manual intervention with "architectural definition," allowing infrastructure to manage itself based on predefined performance rules. 

Dependencies


HPA cannot function on its own; it requires the following "links" and infrastructure: 

  • Metrics Server (The Aggregator): This is the most critical infrastructure dependency. The HPA controller queries the Metrics API (typically provided by the Metrics Server) to get real-time CPU and memory usage data.
    • Both HPA and VPA rely on the metrics.k8s.io API to retrieve CPU and memory data. Because EKS is a managed control plane, AWS keeps it "lean" by leaving the choice of metrics provider to you.
    • HPA: Without Metrics Server, HPA will show a status of <unknown> for its targets.
    • VPA: Without Metrics Server, the VPA Recommender cannot analyze resource usage to suggest changes
  • Resource Requests (The Baseline): For the HPA to calculate percentage-based utilization (e.g., "scale at 50% CPU"), the target Deployment must have resources.requests defined. Without these, the HPA has no 100% baseline to measure against and will show an unknown status.
  • Controller Manager: The HPA logic runs as a control loop within the Kubernetes kube-controller-manager, which periodically (every 15 seconds by default) evaluates the metrics and updates the desired replica count.
  • Scalable Target: The HPA must be linked to a resource that supports scaling, such as a Deployment, ReplicaSet, or StatefulSet.
  • Cluster Capacity (Node Scaling): While HPA scales pods, it depends on an underlying node scaler (like Karpenter or Cluster Autoscaler) to provide new EC2 instances if the cluster runs out of physical space for the additional pods. 

Installation and Configuration


To use HPA ensure the Metrics Server is installed in your cluster to provide resource metrics.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml


Once the Metrics Server is installed, HPA is ready to go. It is a core Kubernetes controller, so you don't need to install any additional software beyond the metrics provider. We can simply create an HPA resource for our deployment:

kubectl autoscale deployment our-deployment \
    --cpu=50% \
    --min=1 \
    --max=10

We can also use the manifest:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50



How to check if HPA is enabled in the cluster?


% kubectl api-resources -o wide | grep autoscaling

NAME SHORTNAMES APIVERSION NAMESPACED KIND
VERBS CATEGORIES
...
horizontalpodautoscalers  hpa  autoscaling/v2 true HorizontalPodAutoscaler  create,delete,deletecollection,get,list,patch,update,watch all


In which namespace do HorizontalPodAutoscalers reside in?


In AWS EKS, HorizontalPodAutoscalers (HPA) are namespaced resources, meaning they belong in the same namespace as the workload (e.g., Deployment or StatefulSet) they are intended to scale. 

While there is no single "HPA namespace," here is how they are distributed and where related components live:

1. The HPA Resource Namespace 

  • Application Namespace: When you create an HPA, you define it within the specific namespace where your application is running (e.g., default, production, or demo).
  • Constraint: An HPA can only scale a target resource (like a Deployment) that exists in that same namespace. 

2. Infrastructure & Metrics Namespaces 

While the HPA resource lives with your app, the supporting infrastructure often resides in system namespaces: 

  • Metrics Server: This is a mandatory prerequisite for HPA on EKS. It is typically deployed in the kube-system namespace.
  • Custom Metrics Adapters: If you are scaling based on custom metrics (like Prometheus or CloudWatch), components like the prometheus-adapter or k8s-cloudwatch-adapter may be installed in kube-system or a dedicated namespace like custom-metrics.
  • Cluster Autoscaler: Often confused with HPA, the Cluster Autoscaler (which scales EC2 nodes rather than pods) also typically resides in the kube-system namespace. 

To find all HPAs across your entire EKS cluster, you can run:

kubectl get hpa -A

We might have an output like this:

% kubectl get horizontalpodautoscalers -A    
No resources found

It is possible to get "No resources found" for several reasons, despite the resource being namespaced. This usually means that while the API type exists, no actual instances of that resource have been created in your EKS cluster yet.

Why you see "No resources found":
  • HPA is not yet createdBy default, EKS clusters do not come with any HorizontalPodAutoscalers pre-configured. You must explicitly create one for your application.
  • Metrics Server Missing: HPAs rely on the Kubernetes Metrics Server to function. While the HPA object can be created without it, it will show a status of <unknown> and may not appear if you are looking for active scaling.
  • Namespace Context: Even with -A (all namespaces), if no user or system service has defined an HPA resource, the list will be empty. 

How to Verify and Fix:
  • Check if Metrics Server is running:
    • Run kubectl get deployment metrics-server -n kube-system. If it’s missing, you can install it via the AWS EKS Add-ons in the console or via kubectl apply.
        kubectl get all -A | grep metrics-server 
  • Check API availability:
    • Run kubectl api-resources | grep hpa to confirm the cluster recognizes the resource type.
  • Create a test HPA:
    • If you have a deployment named my-app, try creating one: 
        kubectl autoscale deployment my-app \
            --cpu=50% \
            --min=1 \
            --max=10


Note: If you are using a newer version of EKS (like 1.31) with Auto Mode, some autoscaling is handled automatically by the control plane, but standard HPAs still need to be manually defined if you want pod-level scaling based on custom metrics. 

% kubectl get all -A | grep metrics-server 

default pod/metrics-server-5db5f64c66-sjd2p        1/1     Running     0          205d
default service/metrics-server  ClusterIP       172.21.76.224    <none>  443/TCP  95d
default deployment.apps/metrics-server             1/1     1            1         295d
default replicaset.apps/metrics-server-5db5f64c66   1         1         1         295d

This behavior occurs because no instances of HorizontalPodAutoscaler (HPA) have been created yet, even though the supporting infrastructure (Metrics Server) and API are active. 

In Kubernetes, the presence of the metrics-server and the autoscaling/v2 API resource does not mean an HPA is automatically running for your appsYou must manually define an HPA for each deployment you want to scale. 

Why kubectl get hpa -A is empty
  • Workloads are not yet auto-scaled: By default, EKS (and Kubernetes) does not apply HPAs to your deployments. You must explicitly create an HPA object that references your target Deployment or StatefulSet.
  • kubectl get all exclusion: Standard kubectl get all does not include HPAs in its output, which is why your previous command didn't show them even if they existed.
  • Namespace Location: While your metrics-server is in the default namespace (though typically it's in kube-system), HPAs must be created in the same namespace as the app they are scaling. 

How to create your first HPA


If you have a deployment (e.g., named my-deployment) in the default namespace, you can create an HPA for it using this command:

kubectl autoscale deployment my-deployment \
    --cpu=60%
    --min=1 \
    --max=10

--cpu string
Target CPU utilization over all the pods. When specified as a percentage (e.g."70%" for 70% of requested CPU) it will target average utilization. When specified as quantity (e.g."500m" for 500 milliCPU) it will target average value. Value without units is treated as a quantity with miliCPU being the unit (e.g."500" is "500m").

--memory string
Target memory utilization over all the pods. When specified as a percentage (e.g."60%" for 60% of requested memory) it will target average utilization. When specified as quantity (e.g."200Mi" for 200 MiB, "1Gi" for 1 GiB) it will target average value. Value without units is treated as a quantity with mebibytes being the unit (e.g."200" is "200Mi").

Thursday, 4 July 2024

Google Cloud Compute Engine

Identity and API access This article extends my notes from Coursera course Google Cloud Fundamentals: Core Infrastructure | Coursera






Compute Engine

  • example of IaaS
  • like AWS EC2
  • with it, users can create and run virtual machines on Google infrastructure
  • no upfront investments
  • thousands of virtual CPUs can run on a system that’s designed to be fast and to offer consistent performance


Each virtual machine contains the power and functionality of a full-fledged operating system. This means a virtual machine can be configured much like a physical server, by specifying required:

  • amount of CPU power (virtual CPUs)
  • memory
  • amount and type of storage
  • operating system

We can use machine types which are:
  • predefined
  • custom, created by us
    • We pay for what we need with custom machine types.
    • We can in fact configure very large VMs, which are great for workloads such as in-memory databases and CPU-intensive analytics
    • most Google Cloud customers start off with scaling out (autoscaling - see below), not up
    • The maximum number of CPUs per VM is tied to its machine family and is also constrained by the quota available to the user, which is zone-dependent (see cloud.google.com/compute/docs/machine-types)


A virtual machine instance can be created via:

  • Google Cloud console, which is a web-based tool to manage Google Cloud projects and resources
  • Google Cloud CLI
  • Compute Engine API


The instance can run:

  • Linux and Windows Server images provided by Google or any customized versions of these images
  • images of other operating systems that we build


Cloud Marketplace

  • Offers software packages e.g. Bitnami LAMP stack (Bitnami package for LAMP – Marketplace – Google Cloud console)
  • A quick way to get started with Google Cloud
  • Offers solutions from both Google and third-party vendors
  • With these solutions, there’s no need to manually configure the software, virtual machine instances, storage, or network settings, although many of them can be modified before launch if that’s required
  • Most software packages are available at no additional charge beyond the normal usage fees for Google Cloud resources


Compute Engine’s pricing and billing structure:

  • use of virtual machines is billed by the second with a one-minute minimum
  • sustained-use discounts start to apply automatically to virtual machines the longer they run
    •  for each VM that runs for more than 25% of a month, Compute Engine automatically applies a discount for every additional minute
  • also offers committed-use discounts: for stable and predictable workloads, a specific amount of vCPUs and memory can be purchased with a discount in return for committing to a usage term of one year or three years. 
  • if we have a workload that doesn’t require a human to sit and wait for it to finish–such as a batch job analyzing a large dataset we can save money, in some cases up to 90%, by choosing Preemptible or Spot VMs to run the job. 
    • Preemptible or Spot VM is different from an ordinary Compute Engine VM in only one respect: Compute Engine has permission to terminate a job if its resources are needed elsewhere. Although savings are possible with preemptible or spot VMs, we'll need to ensure that our job can be stopped and restarted. 
    • Spot VMs differ from Preemptible VMs by offering more features
      • preemptible VMs can only run for up to 24 hours at a time
      • Spot VMs do not have a maximum runtime


Storage

  • high throughput between processing and persistent disks is default


Autoscaling

  • Compute Engine feature
  • VMs can be added to or subtracted from an application based on load metrics

Load Balancing

  • balancing the incoming traffic among the VM
  • Google’s Virtual Private Cloud (VPC) supports several different kinds of load balancing



Provisioning 



When we create a new Compute Engine VM Instance we can specify:

  • Name
  • Region
  • Zone
  • Machine type e.g. Series E2, N2 etc...
  • Boot disk image e.g. Debian GNU/Linux 11 (bullseye)
    • boot disk's architecture (x86/64 or ARM) must be compatible with the selected machine type
  • Identity and API access
    • Service Account: e.g. Compute Engine default service account
        • Requires the Service Account User role (roles/iam.serviceAccountUser) to be set for users who want to access VMs with this service account.
      • Applications running on the VM use the service account to call Google Cloud APIs. Use Permissions on the console menu to create a service account or use the default service account if available.
    • Access scopes: Select the type and level of API access to grant the VM. Default: read-only access to Storage and Service Management, write access to Stackdriver Logging and Monitoring, read/write access to Service Control.
      • Default access
      • Full access to Google APIs
      • Set access for each API
  • Firewall. By default all incoming traffic from outside a network is blocked. Select the type of network traffic you want to allow.  Add tags and firewall rules to allow specific network traffic from the Internet
    • Allow HTTP traffic
    • Allow HTTPS traffic
    • Allow Load Balancer Health Checks
  • Advanced options
    • Networking - Hostname and network interfaces
    • Disks - Additional disks
    • Security - Shielded VM and SSH keys
    • Management - Description, deletion protection, reservations, and automation
      • Automation  >> Startup script - You can choose to specify a startup script that will run when your instance boots up or restarts. Startup scripts can be used to install software and updates, and to ensure that services are running within the virtual machine.
Startup script can be e.g. for setting LAMP stack:

apt-get update
apt-get install apache2 php php-mysql -y
service apache2 restart

click to see the original image



Once instance is running we're able to see internal (e.g. 10.164.0.2) and external IP (e.g. 34.90.152.162) addresses assigned to it:




SSH-ing to VM:







References:


Google Cloud Fundamentals: Core Infrastructure | Coursera.


Disclaimer: The course content rights belong to course creators (Google Cloud Training).