Sunday 5 May 2024

Kubernetes Controllers

These are custom notes that extend my notes from an Udemy course "Kubernetes for the Absolute Beginners - Hands-on". All course content rights belong to course creators. 

This is the third article in the series, the previous one being Managing Pods In A Minikube Cluster | My Public Notepad.

Controllers are the processes that monitor Kubernetes objects and respond accordingly. One of them is the Replication Controller.

Let's consider a case where we have a single pod running our application. We'll show here two reasons why we want to have pod replicas running.

1) High Availability

If application crashes and the pod fails, users will no longer be able to access our application. To prevent this we need to have more than one instance of pod (pod replicas) running at the same time. If one fails, we still have our application running on the other one.

Replication Controller helps us run multiple instances of a single pod in the Kubernetes cluster, thus providing high availability. Even if we want to have a single pod, the Replication Controller can help by automatically bringing up a new pod when the existing one fails.

Replication Controller ensures that the specified number of pods are running at all times, no matter if it's one or more pods.

2) Load Balancing 

So we have a single pod serving a set of users. When the number of users increase, we deploy an additional pod to share and balance the load across the two pods. If the demand further increases and if we were to run out of resources on the first node, we could deploy additional pods across the other nodes in the cluster.

Replication Controller spans across multiple nodes in the cluster. It helps us balance the load across multiple pods on different nodes as well as scale our application when the demand increases.

Replication Controller vs Replica Set

Replication Controller and Replica Set are similar terms, both have the same purpose, but they are not the same.

Replication Controller is the older technology that is being replaced by Replica Set which is the new recommended way to set up replication. There are minor differences in the way each work.

How to create a Replication Controller?

Just as we used definition files to create pod objects (see Managing Pods In A Minikube Cluster | My Public Notepad), we will use a Replication Controller definition file which, as any Kubernetes definition file, has four sections: apiVersion, kind, metadata and spec.


apiVersion: v1
kind: ReplicationController
  name: myapp-rc
    app: myapp
    type: front-end
      name: myapp-pod
        app: myapp
        type: frontend
      - name: nginx-container
        image: nginx
  replicas: 3 

Replication Controller is supported in Kubernetes API version v1.

In any Kubernetes definition file the spec section defines what's inside the object we are creating.

Replication Controller needs to create multiple instances of a pod. We need to specify a blueprint of those pod instances so we'll create a template section under spec to provide a pod template to be used by the Replication Controller to create replicas.

For pod template definition we can reuse the contents of the pod definition file pod-definition.yml that we created in Managing Pods In A Minikube Cluster | My Public Notepad so we'll simply move all the contents of the pod definition file into the template section of the replication controller, except for the apiVersion (it is already specified) and kind (no need to specify kind as it can only be a pod). Make sure that metadata and spec are children of the template and are properly indented.

To specify how many replicas we want to have we use property replicas under the spec (at the same level as template).

To create this Kubernetes object let's first start minikube (which a single cluster on the local host and configures kubectl to talk to that minikube cluster):

$ minikube start
😄  minikube v1.33.0 on Ubuntu 22.04
✨  Using the docker driver based on existing profile
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.43 ...
🏃  Updating the running docker "minikube" container ...
🐳  Preparing Kubernetes v1.30.0 on Docker 26.0.1 ...
🔎  Verifying Kubernetes components...
    ▪ Using image
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

We can check kubectl config:

$ cat  ~/.kube/config 
apiVersion: v1
- cluster:
    certificate-authority: /home/bojan/.minikube/ca.crt
    - extension:
        last-update: Sun, 05 May 2024 12:54:52 BST
        version: v1.33.0
      name: cluster_info
  name: minikube
- context:
    cluster: minikube
    - extension:
        last-update: Sun, 05 May 2024 12:54:52 BST
        version: v1.33.0
      name: context_info
    namespace: default
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
- name: minikube
    client-certificate: /home/bojan/.minikube/profiles/minikube/client.crt
    client-key: /home/bojan/.minikube/profiles/minikube/client.key

To find more details about ReplicaSet Kubernetes object we can use:

$ kubectl explain replicaset
GROUP:      apps
KIND:       ReplicaSet
VERSION:    v1

    ReplicaSet ensures that a specified number of pod replicas are running at
    any given time.
  apiVersion    <string>
    APIVersion defines the versioned schema of this representation of an object.
    Servers should convert recognized schemas to the latest internal value, and
    may reject unrecognized values. More info:

  kind  <string>
    Kind is a string value representing the REST resource this object
    represents. Servers may infer this from the endpoint the client submits
    requests to. Cannot be updated. In CamelCase. More info:

  metadata      <ObjectMeta>
    If the Labels of a ReplicaSet are empty, they are defaulted to be the same
    as the Pod(s) that the ReplicaSet manages. Standard object's metadata. More

  spec  <ReplicaSetSpec>
    Spec defines the specification of the desired behavior of the ReplicaSet.
    More info:

  status        <ReplicaSetStatus>
    Status is the most recently observed status of the ReplicaSet. This data may
    be out of date by some window of time. Populated by the system. Read-only.
    More info:

Let's now create replication controller based on its definition file:

$ kubectl create -f rc-definition.yaml 
replicationcontroller/myapp-rc created

When the replication controller is created, it first creates the pods using the pod definition template, as many as required, which is 3 in our case.

To view the list of created replication controllers:

$ kubectl get replicationcontrollers
myapp-rc   3         3         3       6m51s

The output also shows the number of desired, current and ready pod replicas.

To see all the pods and among them the pods that were created by the replication controller:

$ kubectl get pods
myapp-rc-4mcjh   1/1     Running   0          9m58s
myapp-rc-5jrm2   1/1     Running   0          9m58s
myapp-rc-rg6jm   1/1     Running   0          9m58s

Unlike standalone, independent pods, all pods created automatically by the replication controller have names that start with the name of the replication controller, which is myapp-rc in our case. 

RC, indicating that they are all created automatically by the replication controller.

To delete objects defined in a definition file:

$ kubectl delete -f ./minikube/rc-definition.yaml 
replicationcontroller "myapp-rc" deleted

$ kubectl get pods
No resources found in default namespace.

How to create a Replica Set?

Let's create a definition file, which is very similar to the one above, for replication controller:


apiVersion: apps/v1
kind: ReplicaSet
  name: myapp-replicaset
    app: myapp
    type: front-end
      name: myapp-pod
        app: myapp
        type: front-end
      - name: nginx-container
        image: nginx
  replicas: 3
      type: front-end

Kubernetes API version v1 has no support for Replica Sets but app/v1 does. ReplicaSets moved to apps/v1 in 1.9 version of Kubernetes.

One major difference between replication controller and replica set: replica set requires a selector definition.

The selector section helps the replica set identify what pods fall under it. We need to specify it although we specify what pods fall under it via the template because replica set can also manage pods that were not created as part of the replica set creation.

There might be pods created before the creation of the replica set that match labels specified in the selector. The replica set will also take (the number of) those pods into consideration when creating the replicas.

The selector is not a required field in case of a replication controller, but it is still available. When we skip it, as we did in the rc-definition.yaml, Kubernetes assumes it to be the same as the labels provided in the pod definition file.

In case of replica set, selector needs explicitly to be stated in the definition file.

matchLabels selector simply matches the labels specified under it to the labels on the pod.

The replica set selector also provides many other options for matching labels that were not available in a replication controller.

To create a replica set and then verify it and also number of pods:

$ kubectl create -f replicaset-definition.yaml 
replicaset.apps/myapp-replicaset created

$ kubectl get replicasets
myapp-replicaset   3         3         3       9s

There is also a shorter version of the same command, using alias rs instead of replicaset(s):

$ kubectl get rs
$ kubeclt delete rs my-replicaset
Let's check the pods:

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
myapp-replicaset-47xr4   1/1     Running   0          13s
myapp-replicaset-9bq9l   1/1     Running   0          13s
myapp-replicaset-p6q7m   1/1     Running   0          13s

To find out more details about ReplicaSets e.g. the name of the container running in pods and which Docker image has been used for pod creation we can use:

$ kubectl get replicasets -o wide
new-replica-set   4         4         0       57s   busybox-container   busybox777   name=busybox-pod

(The example above is not related to the execution of previous commands, it's just a generic example)

In the example above, we see that there are 4 current pods but none of them is READY. To find out the reason we can check pods events with:

$ kubectl describe pods

or, if we want to check the specific pod:

$ kubectl get pods
NAME                    READY   STATUS             RESTARTS   AGE
new-replica-set-2kp8n   0/1     ImagePullBackOff   0          6m19s
new-replica-set-4mrf4   0/1     ImagePullBackOff   0          6m19s
new-replica-set-nxggw   0/1     ImagePullBackOff   0          6m19s
new-replica-set-hh4j9   0/1     ImagePullBackOff   0          6m19s

$ kubectl describe pod new-replica-set-2kp8n
Name:             new-replica-set-2kp8n
Namespace:        default
Priority:         0
Service Account:  default
Node:             controlplane/
Start Time:       Tue, 07 May 2024 11:05:48 +0000
Labels:           name=busybox-pod
Annotations:      <none>
Status:           Pending
Controlled By:  ReplicaSet/new-replica-set
    Container ID:  
    Image:         busybox777
    Image ID:      
    Port:          <none>
    Host Port:     <none>
      echo Hello Kubernetes! && sleep 3600
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
      /var/run/secrets/ from kube-api-access-8lgtt (ro)
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:        op=Exists for 300s
                    op=Exists for 300s
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  13m                   default-scheduler  Successfully assigned default/new-replica-set-2kp8n to controlplane
  Normal   Pulling    11m (x4 over 13m)     kubelet            Pulling image "busybox777"
  Warning  Failed     11m (x4 over 13m)     kubelet            Failed to pull image "busybox777": failed to pull and unpack image "": failed to resolve reference "": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed     11m (x4 over 13m)     kubelet            Error: ErrImagePull
  Warning  Failed     11m (x6 over 13m)     kubelet            Error: ImagePullBackOff
  Normal   BackOff    3m22s (x42 over 13m)  kubelet            Back-off pulling image "busybox777"

In this case, the specified Docker image was wrong, non-existing. 

To fix the replica set so it uses the correct Docker image we need to update ReplicaSet definition after which we can: 
  • either delete and recreate the ReplicaSet 
  • or delete all pods, so new ones with the correct image will be created
If we fix/edit ReplicaSet definition, new pods, using the updated information, will not be created automatically.

To check events for all ReplicaSets:

$ kubectl describe replicasets

or, for specific one:

$ kubectl describe replicaset new-replica-set
Name:         new-replica-set
Namespace:    default
Selector:     name=busybox-pod
Labels:       <none>
Annotations:  <none>
Replicas:     4 current / 4 desired
Pods Status:  0 Running / 4 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  name=busybox-pod
    Image:      busybox777
    Port:       <none>
    Host Port:  <none>
      echo Hello Kubernetes! && sleep 3600
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
  Type    Reason            Age    From                   Message
  ----    ------            ----   ----                   -------
  Normal  SuccessfulCreate  7m46s  replicaset-controller  Created pod: new-replica-set-hh4j9
  Normal  SuccessfulCreate  7m46s  replicaset-controller  Created pod: new-replica-set-2kp8n
  Normal  SuccessfulCreate  7m46s  replicaset-controller  Created pod: new-replica-set-nxggw
  Normal  SuccessfulCreate  7m46s  replicaset-controller  Created pod: new-replica-set-4mrf4

Scaling In Action

Case #1: There are less pods than specified in replicas

Let's delete one of the pods that are managed by ReplicaSet:

$ kubectl delete pod myapp-replicaset-47xr4
pod "myapp-replicaset-47xr4" deleted

As a side note, we can delete multiple pods with:

$ kubectl delete pods my-replica-set-4mrf4 my-replica-set-hh4j9 my-replica-set-nxggw my-replica-set-tgs9d 
pod "my-replica-set-4mrf4" deleted
pod "my-replica-set-hh4j9" deleted
pod "my-replica-set-nxggw" deleted
pod "my-replica-set-tgs9d" deleted

But let's assume we only deleted that one pod.

If we now check pods, we'll see that we still have 3 pods, but one has a new name. That is the pod that ReplicaSet created once it realized that one pod went down:

kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
myapp-replicaset-ne54m   1/1     Running   0          12s
myapp-replicaset-9bq9l   1/1     Running   0          1m23s
myapp-replicaset-p6q7m   1/1     Running   0          1m233s

All events related to the specific ReplicaSet can be seen in the output of this command:

$ kubectl describe replicaset myapp-replicaset
Name:         myapp-replicaset
Namespace:    default
Selector:     env=production
Labels:       app=myapp
Annotations:  <none>
Replicas:     3 current / 3 desired
Pods Status:  3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  env=production
    Image:         nginx
    Port:          <none>
    Host Port:     <none>
    Environment:   <none>
    Mounts:        <none>
  Volumes:         <none>
  Node-Selectors:  <none>
  Tolerations:     <none>
  Type    Reason            Age    From                   Message
  ----    ------            ----   ----                   -------
  Normal  SuccessfulCreate  17m    replicaset-controller  Created pod: myapp-replicaset-47xr4
  Normal  SuccessfulCreate  17m    replicaset-controller  Created pod: myapp-replicaset-9bq9l
  Normal  SuccessfulCreate  17m    replicaset-controller  Created pod: myapp-replicaset-p6q7m
  Normal  SuccessfulCreate  6m33s  replicaset-controller  Created pod: myapp-replicaset-ne54m

Case #2: There are more pods than specified in replicas

Let's consider situation when we have more pods, all with the same label as specified in ReplicaSet's matchLabels selector. Let's  test what happens if we have one extra such pod:

$ kubectl create -f nginx.yaml 
pod/nginx created

We would expect now to have 4 pods running but we actually have 3 as ReplicaSet terminated one extra pod:

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
myapp-replicaset-jnm6z   1/1     Running   0          28m
myapp-replicaset-nk5n9   1/1     Running   0          18m
myapp-replicaset-nwwmh   1/1     Running   0          28m

If we executed the last command as soon as new pod was created, we would have seen all four pods here but this last one would be in Terminating state.

Labels and Selectors

What is the use case for labels and selectors? Why do we label our pods and objects in Kubernetes?

Let's assume we deployed three instances of our front end web application as three pods. We would like to create a replication controller or replica set to ensure that we have three active pods at any time.

We can use replica set to monitor these existing pods if we have them already created as it is in this example. In case they were not created, the replica set will create them for us.

The role of the replica set is to monitor the pods and if any of them were to fail deploy new ones.
The replica set is in fact a process that monitors the pods.

Now how does the replica set know what pods to monitor? There could be hundreds of other pods in the cluster running different applications. This is where labelling our pods during creation comes in handy. We could now provide these labels as a filter for replica set. Under the selector section we use the matchLabels filter and provide the same label that we used while creating the pods. This way the replica set knows which pods to monitor.

What happens if label in template does not match the label in the matchLabels? kubectl create fails and reports an error related to mismatched labels.

Why labels are not enough and why we need templates?

Let's assume we have three existing pods that were created already and we need to create a replica set to monitor the pods to ensure there are a minimum of three running at all times. When the replica set is created, it is not going to deploy a new instance of POD as three of them with matching labels are already created. We need to provide a template section in the replica set specification although we are not expecting the replica set to create a new port on deployment because in case one of the pods were to fail in the future, the replica set needs to create a new one to maintain the desired number of pods. And for the replica set to create a new pod, the template definition section is required.

How to scale the Replica Set?

Let's assume we started with three replicas and then we decide to scale to six. There are multiple ways to update our replica set to scale to six replicas.

1) Update the number of replicas in the definition file to six:


  replicas: 6

 Then run:

$ kubectl replace -f replicaset-definition.yaml

2) Run kubectl scale command by specifying replica set definition file:

$ kubectl scale --replicas=6 -f replicaset-definition.yaml

or by specifying object type name (replicaset) and object name (myapp-replicaset):

$ kubectl scale --replicas=6 replicaset myapp-replicaset
replicaset.apps/myapp-replicaset scaled

Using the file name as input will not result in the number of replicas being updated automatically in the file. The number of replicas in the replica set definition file will still be three, even though we scaled our replica set to have six replicas using the kubectl scale command and the file as input.

3) Run:

$ kubectl edit replicaset myapp-replicaset

This commands opens a copy of the ReplicaSet definition in a text editor (e.g. vi, vim):

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
apiVersion: apps/v1
kind: ReplicaSet
  creationTimestamp: "2024-05-05T22:47:22Z"
  generation: 1
    app: myapp
    type: front-end
  name: myapp-replicaset
  namespace: default
  resourceVersion: "138523"
  uid: 6237c310-29f9-4c13-8f03-34fa70aa4307
  replicas: 3
      type: front-end
      creationTimestamp: null
        type: front-end
      name: nginx
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  availableReplicas: 3
  fullyLabeledReplicas: 3
  observedGeneration: 1
  readyReplicas: 3
  replicas: 3
"/tmp/kubectl-edit-352613877.yaml" 45L, 1135B    

As we can see that file is stored in /tmp directory. Changes in this file are applied to ReplicaSet as soon as this file is saved.

If we change number of replicas from 3 to 4 and save this file:

$ kubectl edit replicaset myapp-replicaset
replicaset.apps/myapp-replicaset edited

...number of pods will automatically be increased to 4:

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
myapp-replicaset-5r8vj   1/1     Running   0          4s
myapp-replicaset-jnm6z   1/1     Running   0          7h25m
myapp-replicaset-nk5n9   1/1     Running   0          7h15m
myapp-replicaset-nwwmh   1/1     Running   0  

The same approach can be used to scale down.

We can use kubectl edit replicaset to change any of the attributes in its definition, not just replicas number. That can be e.g. name of the Docker image in the template. 

There are also options available for automatically scaling the replica set based on load.

How to delete Replica Set?

To delete replica set (and underlying pods):

$ kubectl delete replicaset myapp-replicaset

It is possible to specify multiple ReplicaSets for deletion:

$ kubectl delete replicaset replicaset-1 replicaset-2
replicaset.apps "replicaset-1" deleted
replicaset.apps "replicaset-2" deleted

No comments: