Sunday 26 May 2024

Introduction to Minikube

These are custom notes that extend my notes from an Udemy course "Kubernetes for the Absolute Beginners - Hands-on". All course content rights belong to course creators. 



For starting with Kubernetes and local tests we can use VMs running on a local machine so our choice is Minikube.

In Introduction to Kubernetes | My Public Notepad we listed components that need to be installed on the master and worker nodes. Minikube blends together master and worker into a single node which is run on a local machine, in VM. Minikube packages together all these components into a single ISO image which can be downloaded and run in VM.



A Kubernetes cluster can be deployed on either physical or virtual machines. To get started with Kubernetes development, you can use Minikube. Minikube is a lightweight Kubernetes implementation that creates a VM on your local machine and deploys a simple cluster containing only one node. Minikube is available for Linux, macOS, and Windows systems. The Minikube CLI provides basic bootstrapping operations for working with your cluster, including start, stop, status, and delete. [Using Minikube to Create a Cluster | Kubernetes]
  • Our goal is to install / set up a a basic Kubernetes  cluster on our local machine using the minikube utility
  • minikube creates one Node cluster, where the master and worker processes are on the same machine
  • minikube creates a non-production sandbox environment that we can use to test out Kubernetes
  • minikube is used for quick tests within the single cluster, in a single host on the local machine before launching the full blown Kubernetes multi-node cluster.

Before installing minikube:

1) We must can install the kubectl utility locally
    • kubectl command line tool is what we will use to manage our Kubernetes resources and our cluster after it is set up using minikube
    • Installing the kubectl utility before installing minikube will allow minikube to configure the kubectl utility to work with the cluster when it provisions it
    • kubectl utility can work with multiple clusters, local or remote, at the same time
    • if kubectl is not installed locally, minikube already includes kubectl which can be used like this: minikube kubectl -- <kubectl commands>
    • if kubectl is already installed, minikube will automatically take care of configuring it when it starts, when it provisions a Kubernetes cluster
    • To install kubectl (on Linux), follow instructions from here: Install and Set Up kubectl on Linux | Kubernetes
kubectl needs to be configured so it knows to which cluster it needs to talk to (to which master node). $KUBECONFIG environment variable contains the path to the directory that contains minikube configuration and which is usually ~/.kube/.

Upon fresh kubectl installation, before it's configured, ~/.kube/ is not created and KUBECONFIG is not specified.

Default kubectl config file is ~/.kube/config. It initially does not exist and before we use kubectl to talk to Kubernetes cluster we need to create it or it can be created by Minikube on its startup (be careful as Minikube might overwrite existing kubeconfig file!)

To learn more about kubeconfig file: What is a kubeconfig file? | Enable Sysadmin

2) We need to make sure that virtualization is enabled.
On Linux, it is enabled if the following command returns non-empty output: 

$ grep -E --color 'vmx|svm' /proc/cpuinfo

If it's not enabled, there should be an option in BIOS to enable virtualization.


3) We need to install a virtual machine manager (hypervisor)  or container manager/engine such as: Docker, QEMU, Hyperkit, Hyper-V, KVM, Parallels, Podman, VirtualBox, or VMware Fusion/Workstation. 

We can use e.g. VirtualBox virtualization solution but minikube can also run without a hypervisor, directly on the host using Docker.
The Docker driver allows you to install Kubernetes into an existing Docker install. On Linux, this does not require virtualization to be enabled. [docker | minikube
VirtualBox is minikube’s original driver. It may not provide the fastest start-up time, but it is the most stable driver available for users of Microsoft Windows Home. [virtualbox | minikube]

VirtualBox installation: Linux_Downloads – Oracle VM VirtualBox

When we provision a cluster using minikube, it will automatically create a virtual machine as required.

Minikube installation: minikube start | minikube

Upon minikube installation, we have a new directory that minikube uses for state/configuration: .minikube. By default, it is created in home directory (~/) and it contains several sub-directories, all of which are initially empty:

$ ls -la ~/.minikube/
total 36
drwxr-xr-x  .
drwxr-xr-x  ..
drwxr-xr-x  addons
drwxr-xr-x  cache
drwxr-xr-x  certs
drwxr-xr-x  config
drwxr-xr-x  files
drwxr-xr-x  logs
drwxr-xr-x  machines 

MINIKUBE_HOME environment variable contains the path to the .minikube directory but if it's not specified, it defaults to ~/.minikube. For more details, see: Configuration | minikube.

Let's explore minikube CLI:

minikube --help
minikube provisions and manages local Kubernetes clusters optimized for development workflows.

Basic Commands:
  start            Starts a local Kubernetes cluster
  status           Gets the status of a local Kubernetes cluster
  stop             Stops a running local Kubernetes cluster
  delete           Deletes a local Kubernetes cluster
  dashboard        Access the Kubernetes dashboard running within 
                   the minikube cluster
  pause            pause Kubernetes
  unpause          unpause Kubernetes

Images Commands:
  docker-env       Provides instructions to point your terminal's                       docker-cli to the Docker Engine inside minikube.
                   (Useful for building docker images directly                           inside minikube)
  podman-env       Configure environment to use minikube's Podman                       service
  cache            Manage cache for images
  image            Manage images

Configuration and Management Commands:
  addons           Enable or disable a minikube addon
  config           Modify persistent configuration values
  profile          Get or list the current profiles (clusters)
  update-context   Update kubeconfig in case of an IP or port change

Networking and Connectivity Commands:
  service          Returns a URL to connect to a service
  tunnel           Connect to LoadBalancer services

Advanced Commands:
  mount            Mounts the specified directory into minikube
  ssh              Log into the minikube environment (for debugging)
  kubectl          Run a kubectl binary matching the cluster version
  node             Add, remove, or list additional nodes
  cp               Copy the specified file into minikube

Troubleshooting Commands:
  ssh-key          Retrieve the ssh identity key path of the specified node
  ssh-host         Retrieve the ssh host key of the specified node
  ip               Retrieves the IP address of the specified node
  logs             Returns logs to debug a local Kubernetes cluster
  update-check     Print current and latest version number
  version          Print the version of minikube
  options          Show a list of global command-line options (applies to all commands).

Other Commands:
  completion       Generate command completion for a shell
  license          Outputs the licenses of dependencies to a directory

Use "minikube <command> --help" for more information about a given command.


To test minikube installation we can provision a Kubernetes cluster:

minikube start
😄  minikube v1.33.0 on Ubuntu 22.04
🆕  Kubernetes 1.30.0 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.30.0
✨  Using the virtualbox driver based on existing profile
💿  Downloading VM boot image ...
    > minikube-v1.33.0-amd64.iso....:  65 B / 65 B [---------] 100.00% ? p/s 0s
    > minikube-v1.33.0-amd64.iso:  314.16 MiB / 314.16 MiB  100.00% 3.72 MiB p/
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔄  Restarting existing virtualbox VM for "minikube" ...
❗  Image was not built for the current minikube version. To resolve this you can delete and recreate your minikube cluster using the latest images. Expected minikube version: v1.32.0 -> Actual minikube version: v1.33.0
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...| Bad local forwarding specification '0:localhost:8443'

🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass

❗  /usr/local/bin/kubectl is version 1.30.0, which may have incompatibilities with Kubernetes 1.28.3.
    ▪ Want kubectl v1.28.3? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default


We can also explicitly specify the driver (virtualization tool) to be used e.g. VirtualBox:

minikube start --driver=virtualbox

minikube downloaded the minikube ISO image for mini cube. This image is then used to provision a VM on VirtualBox. It downloaded Kubernetes version 1.28.3 and any other required binaries.

We can open VirtualBox UI we can see that a virtual machine by the name minikube has been created and it is in a running state:



Note the last line in the minikube start output. Minikube set up ~/.kube/config to point to  minikube cluster. kubectl utility is now configured to use the Kubernetes cluster provisioned using minikube.

So, before we run kubectl for the first time and we want to use Minikube cluster, we first need to run minikube start.

To check kubectl configuration:

kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /home/bojan/.minikube/ca.crt
    extensions:
    - extension:
        last-update: Sun, 12 May 2024 22:31:20 BST
        provider: minikube.sigs.k8s.io
        version: v1.33.0
      name: cluster_info
    server: https://192.168.59.100:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    extensions:
    - extension:
        last-update: Sun, 12 May 2024 22:31:20 BST
        provider: minikube.sigs.k8s.io
        version: v1.33.0
      name: context_info
    namespace: default
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: /home/bojan/.minikube/profiles/minikube/client.crt
    client-key: /home/bojan/.minikube/profiles/minikube/client.key

If we run some kubectl command but we didn't start minikube we'll get the error:

$ kubectl get deployment
E0513 12:37:29.927893    8136 memcache.go:265] couldn't get current server API group list: Get "https://192.168.59.100:8443/api?timeout=32s": dial tcp 192.168.59.100:8443: i/o timeout

192.168.59.100 is the IP address of the apiserver node (a singe node acting both as master and worker) in case of Minikube) and 8443 is the default listening port on it.

Solution here is to run minikube start.

On one of the subsequent runs of minikube start I got the following output:

minikube start
😄  minikube v1.33.0 on Ubuntu 22.04
🆕  Kubernetes 1.30.0 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.30.0
✨  Using the virtualbox driver based on existing profile
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔄  Restarting existing virtualbox VM for "minikube" ...
❗  Image was not built for the current minikube version. To resolve this you can delete and recreate your minikube cluster using the latest images. Expected minikube version: v1.32.0 -> Actual minikube version: v1.33.0
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...- Bad local forwarding specification '0:localhost:8443'

🔗  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image docker.io/kubernetesui/dashboard:v2.7.0
    ▪ Using image docker.io/kubernetesui/metrics-scraper:v1.0.8
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
╭───────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                   │
│    You have selected "virtualbox" driver, but there are better options !                          │
│    For better performance and support consider using a different driver:                          │
│            - kvm2                                                                                 │
│            - qemu2                                                                                │
│                                                                                                   │
│    To turn off this warning run:                                                                  │
│                                                                                                   │
│            $ minikube config set WantVirtualBoxDriverWarning false                                │
│                                                                                                   │
│                                                                                                   │
│    To learn more about on minikube drivers checkout https://minikube.sigs.k8s.io/docs/drivers/    │
│    To see benchmarks checkout https://minikube.sigs.k8s.io/docs/benchmarks/cpuusage/              │
│                                                                                                   │
╰───────────────────────────────────────────────────────────────────────────────────────────────────╯
🔎  Verifying Kubernetes components...
💡  Some dashboard features require the metrics-server addon. To enable all features please run:

        minikube addons enable metrics-server

🌟  Enabled addons: storage-provisioner, default-storageclass, dashboard

❗  /usr/local/bin/kubectl is version 1.30.0, which may have incompatibilities with Kubernetes 1.28.3.
    ▪ Want kubectl v1.28.3? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

If we already have kubeconfig file set for some non-Minikube cluster, we can make Minikube create its own kubeconfig at some different location or with different name by changing the value of KUBECONFIG env variable:

$ KUBECONFIG=/home/user/.kube/minikube-config minikube start

To ensure that everything has been set up correctly we'll run:

minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured


Our cluster is now set up. We will deploy some applications on the cluster in order to make sure it's working as expected. 

To check if kubectl commands are working and also check how many nodes are in the cluster we can run:

kubectl get node
NAME       STATUS   ROLES           AGE   VERSION
minikube   Ready    control-plane   10d   v1.28.3


We can see that it is a single node cluster with node running Kubernetes v1.28.3.

ROLES can also have a value master.

kubectl get node
NAME           STATUS   ROLES                  AGE   VERSION
controlplane   Ready    control-plane,master   19m   v1.29.0+k3s1


Some more examples:

kubectl get nodes
NAME           STATUS   ROLES                  AGE   VERSION
controlplane   Ready    control-plane,master   30m   v1.29.0+k3s1

If we specify output format via -o (--output) argument as wide, we'll get a bit more info about the nodes:

kubectl get nodes -o wide
NAME           STATUS   ROLES                  AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
controlplane   Ready    control-plane,master   30m   v1.29.0+k3s1   192.10.75.9   <none>        Alpine Linux v3.16   5.4.0-1106-gcp   containerd://1.7.11-k3s2


To get the more details about the cluster:

kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

To get more detailed information about cluster (and here we filter out info about node(s)):

kubectl cluster-info dump | grep node
                    "node-role.kubernetes.io/control-plane": "true",
                    "node-role.kubernetes.io/master": "true",
                    "node.kubernetes.io/instance-type": "k3s"
                    "alpha.kubernetes.io/provided-node-ip": "192.10.75.9",
                    "k3s.io/node-args": "[\"server\",\"--advertise-address\",\"192.10.75.9\",\"--tls-san\",\"controlplane\",\"--flannel-iface\",\"eth0\"]",
                    "k3s.io/node-config-hash": "QSE3CODHPKSU74VVY6TG2XX6OZXX5OMS37YI54QW2G4MHIETIL2Q====",
                    "k3s.io/node-env": "{\"K3S_DATA_DIR\":\"/var/lib/rancher/k3s/data/e5efd5aeb0cb8e4c1d802582bd7085968576e89d8f34ca22a450b4f4ae4d4c15\"}",
                    "node.alpha.kubernetes.io/ttl": "0",
                    "wrangler.cattle.io/node"
                "nodeInfo": {
                        "nodePort": 30812
                        "nodePort": 31811
...

To get more information about the node of interest:

kubectl describe node controlplane
Name:               controlplane
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=k3s
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=controlplane
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/master=true
                    node.kubernetes.io/instance-type=k3s
Annotations:        alpha.kubernetes.io/provided-node-ip: 192.10.75.9
                    flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"02:0e:eb:5a:ab:97"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.10.75.9
                    k3s.io/hostname: controlplane
                    k3s.io/internal-ip: 192.10.75.9
                    k3s.io/node-args: ["server","--advertise-address","192.10.75.9","--tls-san","controlplane","--flannel-iface","eth0"]
                    k3s.io/node-config-hash: QSE3CODHPKSU74VVY6TG2XX6OZXX5OMS37YI54QW2G4MHIETIL2Q====
                    k3s.io/node-env: {"K3S_DATA_DIR":"/var/lib/rancher/k3s/data/e5efd5aeb0cb8e4c1d802582bd7085968576e89d8f34ca22a450b4f4ae4d4c15"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Mon, 06 May 2024 06:19:39 +0000
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  controlplane
  AcquireTime:     <unset>
  RenewTime:       Mon, 06 May 2024 06:45:50 +0000
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Mon, 06 May 2024 06:45:11 +0000   Mon, 06 May 2024 06:19:39 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Mon, 06 May 2024 06:45:11 +0000   Mon, 06 May 2024 06:19:39 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Mon, 06 May 2024 06:45:11 +0000   Mon, 06 May 2024 06:19:39 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Mon, 06 May 2024 06:45:11 +0000   Mon, 06 May 2024 06:19:39 +0000   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.10.75.9
  Hostname:    controlplane
Capacity:
  cpu:                36
  ephemeral-storage:  1016057248Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             214587056Ki
  pods:               110
Allocatable:
  cpu:                36
  ephemeral-storage:  988420490080
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             214587056Ki
  pods:               110
System Info:
  Machine ID:                 
  System UUID:                82ad48cc-cf0c-c0bf-7b55-2c21495826a1
  Boot ID:                    6914c0b2-ef4d-4ff1-9180-75737388e9af
  Kernel Version:             5.4.0-1106-gcp
  OS Image:                   Alpine Linux v3.16
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.11-k3s2
  Kubelet Version:            v1.29.0+k3s1
  Kube-Proxy Version:         v1.29.0+k3s1
PodCIDR:                      10.42.0.0/24
PodCIDRs:                     10.42.0.0/24
ProviderID:                   k3s://controlplane
Non-terminated Pods:          (5 in total)
  Namespace                   Name                                       CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                       ------------  ----------  ---------------  -------------  ---
  kube-system                 coredns-6799fbcd5-hl6zl                    100m (0%)     0 (0%)      70Mi (0%)        170Mi (0%)     26m
  kube-system                 local-path-provisioner-84db5d44d9-qgvm2    0 (0%)        0 (0%)      0 (0%)           0 (0%)         26m
  kube-system                 svclb-traefik-f39a148c-87mtw               0 (0%)        0 (0%)      0 (0%)           0 (0%)         25m
  kube-system                 metrics-server-67c658944b-5dsjt            100m (0%)     0 (0%)      70Mi (0%)        0 (0%)         26m
  kube-system                 traefik-f4564c4f4-4qvgx                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         25m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                200m (0%)   0 (0%)
  memory             140Mi (0%)  170Mi (0%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:
  Type     Reason                          Age                From                   Message
  ----     ------                          ----               ----                   -------
  Normal   Starting                        26m                kube-proxy             
  Normal   NodePasswordValidationComplete  26m                k3s-supervisor         Deferred node password secret validation complete
  Normal   Synced                          26m                cloud-node-controller  Node synced successfully
  Normal   Starting                        26m                kubelet                Starting kubelet.
  Warning  InvalidDiskCapacity             26m                kubelet                invalid capacity 0 on image filesystem
  Normal   NodeHasSufficientMemory         26m (x2 over 26m)  kubelet                Node controlplane status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure           26m (x2 over 26m)  kubelet                Node controlplane status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID            26m (x2 over 26m)  kubelet                Node controlplane status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced         26m                kubelet                Updated Node Allocatable limit across pods
  Normal   NodeReady                       26m                kubelet                Node controlplane status is now: NodeReady
  Normal   RegisteredNode                  26m                node-controller        Node controlplane event: Registered Node controlplane in Controller




To create a deployment using this cluster:

kubectl create deployment hello-minikube --image=kicbase/echo-server:1.0
deployment.apps/hello-minikube created

To check it:

kubectl get deployments
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
hello-minikube   1/1     1            1           68s

To expose this deployment as a service on port 8080:

$ kubectl expose deployment hello-minikube --type=NodePort --port=8080
service/hello-minikube exposed

To get info on the service:

kubectl get services hello-minikube
NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
hello-minikube   NodePort   10.104.92.181   <none>        8080:30235/TCP   4m5s

To get the URL of the exposed service:

minikube service hello-minikube --url
http://192.168.59.100:30235

If we copy this URL and paste it into a browser:


Cleanup:

kubectl delete services hello-minikube
service "hello-minikube" deleted

kubectl delete deployment hello-minikube
deployment.apps "hello-minikube" deleted

kubectl get pods
No resources found in default namespace.


To continue your Kubernetes learning journey, read the next article in these series: Managing Pods In A Minikube ClusterMy Public Notepad | 

No comments: