Introduction to Vagrant

 

Vagrant:

• a modular framework for working with Virtual Machines (VMs) and containers 
• enables the creation and configuration of lightweight, reproducible, and portable development environments (Vagrant Machines)
• cross-platform (Win, MacOS, Linux)
• has networking with VMs set up out of the box
• directory which contains Vagrantfile automatically gets synced to VM
• important concept: Provisioners and Providers
• they can be mixed and matched
• designed to work with various virtualization platforms, such as VirtualBox, VMware, Hyper-V, and more, making it a versatile tool for developers

Vagrant Machine

• lightweight, reproducible, and portable development environment
• any virtual machine that was created by Vagrant as the result of an initial vagrant up command
• different from a standard/full virtual machine in that it is run in a terminal of the host computer (but under the bonnet, it does run the VM e.g. VirtualBox VM which can be verified by looking the VirtualBox UI)
• we write programs on the host OS but run them inside a Vagrant machine that runs different OS but can work with our files and folders on the host OS
• we need to install virtualization provider e.g. VirtualBox and Vagrant (and on Windows also Cygwin)
• we can then download a Vagrant machine with desired guest OS and either fill it with our custom software we can download a ready-made machine
• A Vagrant machine is compiled from a box. It can be a virtual machine, a container or a remote server from a cloud service.

Box

• a package that can be used to create Vagrant machines
• a file that we can download from a registry (e.g. Hashicorp Vagrant Cloud - https://app.vagrantup.com/boxes/search) and install on our host to get access to a complete computer (VM) running another OS 
• We can download boxes from app.vagrantup.com, or we can build a new box from a Vagrantfile.
• A box can be used as a base for another box. 
• The base boxes are usually operating system boxes downloaded from app.vagrantup.com.
• Vagrant Boxes | Vagrant | HashiCorp Developer

Providers

• provide virtualization support
• responsible for providing the virtualization technology that will run our machine.
• they bring VMs into existence
• 2 types:
• Local
• VirtualBox, libvirt, VMWare, Docker
• Remote
• OpenStack, Digital Ocean, AWS

Provisioners

• Provision the VM
• responsible for installing and configuring the necessary software on a newly created Vagrant machine.
• they do shared, repeatable configuration against the VM
• types:
• Simple on-ramp
• Shell (inline commands, shell scripts)
• More powerful
• Ansible, Puppet, Chef, Salt

Vagrantfiles

• a file that describes how to create one or more Vagrant machines. Vagrantfiles use the Ruby language, as well as objects provided by Vagrant itself.

How to start with Vagrant?

Let's create a test directory for Vagrant:

$ mkdir vagrant-test 

$ cd vagrant-test

Let's check out the options:

$ vagrant --help

Usage: vagrant [options] <command> [<args>]

    -h, --help                       Print this help.

Common commands:

     autocomplete    manages autocomplete installation on host

     box             manages boxes: installation, removal, etc.

     cloud           manages everything related to Vagrant Cloud

     destroy         stops and deletes all traces of the vagrant machine

     global-status   outputs status Vagrant environments for this user

     halt            stops the vagrant machine

     help            shows the help for a subcommand

     init            initializes a new Vagrant environment by creating a Vagrantfile

     login           

     package         packages a running vagrant environment into a box

     plugin          manages plugins: install, uninstall, update, etc.

     port            displays information about guest port mappings

     powershell      connects to machine via powershell remoting

     provision       provisions the vagrant machine

     push            deploys code in this environment to a configured destination

     rdp             connects to machine via RDP

     reload          restarts vagrant machine, loads new Vagrantfile configuration

     resume          resume a suspended vagrant machine

     serve           start Vagrant server

     snapshot        manages snapshots: saving, restoring, etc.

     ssh             connects to machine via SSH

     ssh-config      outputs OpenSSH valid configuration to connect to the machine

     status          outputs status of the vagrant machine

     suspend         suspends the machine

     up              starts and provisions the vagrant environment

     upload          upload to machine via communicator

     validate        validates the Vagrantfile

     version         prints current and latest Vagrant version

     winrm           executes commands on a machine via WinRM

     winrm-config    outputs WinRM configuration to connect to the machine

For help on any individual command run `vagrant COMMAND -h`

Additional subcommands are available, but are either more advanced

or not commonly used. To see all subcommands, run the command

`vagrant list-commands`.

        --[no-]color                 Enable or disable color output

        --machine-readable           Enable machine readable output

    -v, --version                    Display Vagrant version

        --debug                      Enable debug output

        --timestamp                  Enable timestamps on log output

        --debug-timestamp            Enable debug output with timestamps

        --no-tty                     Enable non-interactive output

Let's try to see what's the current status of the vagrant machine (which does not exist as we haven't created Vagrantfile and we haven't started/provisioned a new Vagrant environment from it):

$ vagrant status
A Vagrant environment or target machine is required to run this
command. Run `vagrant init` to create a new Vagrant environment. Or,
get an ID of a target machine from `vagrant global-status` to run
this command on. A final option is to change to a directory with a
Vagrantfile and to try again.

Let's check out the options of vagrant init [vagrant init - Command-Line Interface | Vagrant | HashiCorp Developer]:

$ vagrant init --help

Usage: vagrant init [options] [name [url]]

Options:

        --box-version VERSION        Version of the box to add

    -f, --force                      Overwrite existing Vagrantfile

    -m, --minimal                    Use minimal Vagrantfile template (no help comments). Ignored with --template

        --output FILE                Output path for the box. '-' for stdout

        --template FILE              Path to custom Vagrantfile template

        --[no-]color                 Enable or disable color output

        --machine-readable           Enable machine readable output

    -v, --version                    Display Vagrant version

        --debug                      Enable debug output

        --timestamp                  Enable timestamps on log output

        --debug-timestamp            Enable debug output with timestamps

        --no-tty                     Enable non-interactive output

    -h, --help                       Print this help

Let's run vagrant init with no options:

$ vagrant init

A `Vagrantfile` has been placed in this directory. You are now

ready to `vagrant up` your first virtual environment! Please read

the comments in the Vagrantfile as well as documentation on

`vagrantup.com` for more information on using Vagrant.

Let's check out the created Vagrantfile:

$ cat Vagrantfile 

# -*- mode: ruby -*-

# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure

# configures the configuration version (we support older styles for

# backwards compatibility). Please don't change it unless you know what

# you're doing.

Vagrant.configure("2") do |config|

  # The most common configuration options are documented and commented below.

  # For a complete reference, please see the online documentation at

  # https://docs.vagrantup.com.

  # Every Vagrant development environment requires a box. You can search for

  # boxes at https://vagrantcloud.com/search.

  config.vm.box = "base"

  # Disable automatic box update checking. If you disable this, then

  # boxes will only be checked for updates when the user runs

  # `vagrant box outdated`. This is not recommended.

  # config.vm.box_check_update = false

  # Create a forwarded port mapping which allows access to a specific port

  # within the machine from a port on the host machine. In the example below,

  # accessing "localhost:8080" will access port 80 on the guest machine.

  # NOTE: This will enable public access to the opened port

  # config.vm.network "forwarded_port", guest: 80, host: 8080

  # Create a forwarded port mapping which allows access to a specific port

  # within the machine from a port on the host machine and only allow access

  # via 127.0.0.1 to disable public access

  # config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"

  # Create a private network, which allows host-only access to the machine

  # using a specific IP.

  # config.vm.network "private_network", ip: "192.168.33.10"

  # Create a public network, which generally matched to bridged network.

  # Bridged networks make the machine appear as another physical device on

  # your network.

  # config.vm.network "public_network"

  # Share an additional folder to the guest VM. The first argument is

  # the path on the host to the actual folder. The second argument is

  # the path on the guest to mount the folder. And the optional third

  # argument is a set of non-required options.

  # config.vm.synced_folder "../data", "/vagrant_data"

  # Disable the default share of the current code directory. Doing this

  # provides improved isolation between the vagrant box and your host

  # by making sure your Vagrantfile isn't accessible to the vagrant box.

  # If you use this you may want to enable additional shared subfolders as

  # shown above.

  # config.vm.synced_folder ".", "/vagrant", disabled: true

  # Provider-specific configuration so you can fine-tune various

  # backing providers for Vagrant. These expose provider-specific options.

  # Example for VirtualBox:

  #

  # config.vm.provider "virtualbox" do |vb|

  #   # Display the VirtualBox GUI when booting the machine

  #   vb.gui = true

  #

  #   # Customize the amount of memory on the VM:

  #   vb.memory = "1024"

  # end

  #

  # View the documentation for the provider you are using for more

  # information on available options.

  # Enable provisioning with a shell script. Additional provisioners such as

  # Ansible, Chef, Docker, Puppet and Salt are also available. Please see the

  # documentation for more information about their specific syntax and use.

  # config.vm.provision "shell", inline: <<-SHELL

  #   apt-get update

  #   apt-get install -y apache2

  # SHELL

end

If we try to initialize Vagrant in a directory which already contains Vagrantfile, we'll get an error:

$ vagrant init

`Vagrantfile` already exists in this directory. Remove it before

running `vagrant init`.

Now when we have Vagrantfile, let's run vagrant up command which creates and configures guest machines according to the Vagrantfile [see vagrant up - Command-Line Interface | Vagrant | HashiCorp Developer]. Before that, let's check the vagrant up format:

$ vagrant up --help

Usage: vagrant up [options] [name|id]

Options:

        --[no-]provision             Enable or disable provisioning

        --provision-with x,y,z       Enable only certain provisioners, by type or by name.

        --[no-]destroy-on-error      Destroy machine if any fatal error happens (default to true)

        --[no-]parallel              Enable or disable parallelism if provider supports it

        --provider PROVIDER          Back the machine with a specific provider

        --[no-]install-provider      If possible, install the provider if it isn't installed

        --[no-]color                 Enable or disable color output

        --machine-readable           Enable machine readable output

    -v, --version                    Display Vagrant version

        --debug                      Enable debug output

        --timestamp                  Enable timestamps on log output

        --debug-timestamp            Enable debug output with timestamps

        --no-tty                     Enable non-interactive output

    -h, --help                       Print this help

name - Name of machine defined in Vagrantfile. Using name to specify the Vagrant machine to act on must be done from within a Vagrant project (directory where the Vagrantfile exists).

Let's see what happens if we don't specify the name: 

$ vagrant up

Bringing machine 'default' up with 'virtualbox' provider...

==> default: Box 'base' could not be found. Attempting to find and install...

    default: Box Provider: virtualbox

    default: Box Version: >= 0

==> default: Box file was not detected as metadata. Adding it directly...

==> default: Adding box 'base' (v0) for provider: virtualbox

    default: Downloading: base

An error occurred while downloading the remote file. The error

message, if any, is reproduced below. Please fix this error and try

again.

Couldn't open file /home/bojan/vagrant-test/base

Let's check vagrant box [vagrant box - Command-Line Interface | Vagrant | HashiCorp Developer] format:

$ vagrant box --help

Usage: vagrant box <subcommand> [<args>]

Available subcommands:

     add

     list

     outdated

     prune

     remove

     repackage

     update

For help on any individual subcommand run `vagrant box <subcommand> -h`

        --[no-]color                 Enable or disable color output

        --machine-readable           Enable machine readable output

    -v, --version                    Display Vagrant version

        --debug                      Enable debug output

        --timestamp                  Enable timestamps on log output

        --debug-timestamp            Enable debug output with timestamps

        --no-tty                     Enable non-interactive output

Let's list installed boxes:

$ vagrant box list

There are no installed boxes! Use `vagrant box add` to add some.

Let's go to https://app.vagrantup.com/boxes/search and pick some box to download and use:

$ vagrant box add centos/7

==> box: Loading metadata for box 'centos/7'

    box: URL: https://vagrantcloud.com/api/v2/vagrant/centos/7

This box can work with multiple providers! The providers that it

can work with are listed below. Please review the list and choose

the provider you will be working with.

1) hyperv

2) libvirt

3) virtualbox

4) vmware_desktop

Enter your choice: 3

==> box: Adding box 'centos/7' (v2004.01) for provider: virtualbox

    box: Downloading: https://vagrantcloud.com/centos/boxes/7/versions/2004.01/providers/virtualbox/unknown/vagrant.box

Download redirected to host: cloud.centos.org

    box: Calculating and comparing box checksum...

==> box: Successfully added box 'centos/7' (v2004.01) for 'virtualbox'!

$ vagrant box list 

centos/7 (virtualbox, 2004.01)

$ vi Vagrantfile 

...

config.vm.box = "centos/7"

...

Let's boot centos/7 VM:

$ vagrant up 

Bringing machine 'default' up with 'virtualbox' provider...

==> default: Importing base box 'centos/7'...

==> default: Matching MAC address for NAT networking...

==> default: Checking if box 'centos/7' version '2004.01' is up to date...

==> default: Setting the name of the VM: vagrant-test_default_1716141840424_31412

==> default: Clearing any previously set network interfaces...

==> default: Preparing network interfaces based on configuration...

    default: Adapter 1: nat

==> default: Forwarding ports...

    default: 22 (guest) => 2222 (host) (adapter 1)

==> default: Booting VM...

==> default: Waiting for machine to boot. This may take a few minutes...

    default: SSH address: 127.0.0.1:2222

    default: SSH username: vagrant

    default: SSH auth method: private key

    default: 

    default: Vagrant insecure key detected. Vagrant will automatically replace

    default: this with a newly generated keypair for better security.

    default: 

    default: Inserting generated public key within guest...

    default: Removing insecure key from the guest if it's present...

    default: Key inserted! Disconnecting and reconnecting using new SSH key...

==> default: Machine booted and ready!

==> default: Checking for guest additions in VM...

    default: No guest additions were detected on the base box for this VM! Guest

    default: additions are required for forwarded ports, shared folders, host only

    default: networking, and more. If SSH fails on this machine, please install

    default: the guest additions and repackage the box to continue.

    default: 

    default: This is not an error message; everything may continue to work properly,

    default: in which case you may ignore this message.

==> default: Rsyncing folder: /home/bojan/vagrant-test/ => /vagrant

vagrant up triggers the following:

• base box gets downloaded unless it's cached
• VM boots
• cross-OS networking gets set up
• cross-OS shared directory gets set up

Let's now ssh to the guest VM:

bojan@host:~/vagrant-test$ vagrant ssh

[vagrant@localhost ~]$ pwd

/home/vagrant

[vagrant@localhost ~]$ cat /etc/os-release

NAME="CentOS Linux"

VERSION="7 (Core)"

ID="centos"

ID_LIKE="rhel fedora"

VERSION_ID="7"

PRETTY_NAME="CentOS Linux 7 (Core)"

ANSI_COLOR="0;31"

CPE_NAME="cpe:/o:centos:centos:7"

HOME_URL="https://www.centos.org/"

BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"

CENTOS_MANTISBT_PROJECT_VERSION="7"

REDHAT_SUPPORT_PRODUCT="centos"

REDHAT_SUPPORT_PRODUCT_VERSION="7"

[vagrant@localhost ~]$ whoami

vagrant

[vagrant@localhost ~]$ uname -a

Linux localhost.localdomain 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[vagrant@localhost ~]$ pwd

/home/vagrant

[vagrant@localhost ~]$ exit

logout

bojan@host:~/vagrant-test$ ls 

We were in the terminal of the Vagrant machine (VM)!

Let's check the status of the Vagrant machine:

$ vagrant status

Current machine states:

default                   running (virtualbox)

The VM is running. To stop this VM, you can run `vagrant halt` to

shut it down forcefully, or you can run `vagrant suspend` to simply

suspend the virtual machine. In either case, to restart it again,

simply run `vagrant up`.

$ vagrant halt --help

Usage: vagrant halt [options] [name|id]

Options:

    -f, --force                      Force shut down (equivalent of pulling power)

        --[no-]color                 Enable or disable color output

        --machine-readable           Enable machine readable output

    -v, --version                    Display Vagrant version

        --debug                      Enable debug output

        --timestamp                  Enable timestamps on log output

        --debug-timestamp            Enable debug output with timestamps

        --no-tty                     Enable non-interactive output

    -h, --help                       Print this help

Let's stop this VM:

$ vagrant halt default

==> default: Attempting graceful shutdown of VM...

Let's now verify that it's stopped:

$ vagrant status

Current machine states:

default                   poweroff (virtualbox)

The VM is powered off. To restart the VM, simply run `vagrant up`

How to use Shell as Provisioner

Let's create another Vagrant machine on the same host. Let's use some other base OS, Fedora for example and let's use some shell command to perform provisioning e.g. installing some package e.g. mediawriter.

We'll create another test directory and in it a new Vagrantfile:

$ mkdir vagrant-test-2

$ cd vagrant-test-2

Let's create Vagrant file:

$ vagrant init

A `Vagrantfile` has been placed in this directory. You are now

ready to `vagrant up` your first virtual environment! Please read

the comments in the Vagrantfile as well as documentation on

`vagrantup.com` for more information on using Vagrant.

Let's edit the Vagrantfile:

$ vi Vagrantfile 

...

config.vm.box = boxcutter/fedora22

...

config.vm.provision "shell", inline: "dnf install -y mediawriter"

$ vagrant up

Bringing machine 'default' up with 'virtualbox' provider...

==> default: Box 'boxcutter/fedora22' could not be found. Attempting to find and install...

    default: Box Provider: virtualbox

    default: Box Version: >= 0

The box 'boxcutter/fedora22' could not be found or

could not be accessed in the remote catalog. If this is a private

box on HashiCorp's Vagrant Cloud, please verify you're logged in via

`vagrant login`. Also, please double-check the name. The expanded

URL and error message are shown below:

URL: ["https://vagrantcloud.com/boxcutter/fedora22"]

Error: The requested URL returned error: 404

Let's use some more recent Fedora version:

$ vi Vagrantfile 

...

config.vm.box = generic/fedora28

...

Let's try again to start the machine:

$ vagrant up

Bringing machine 'default' up with 'virtualbox' provider...

==> default: Box 'generic/fedora28' could not be found. Attempting to find and install...

    default: Box Provider: virtualbox

    default: Box Version: >= 0

==> default: Loading metadata for box 'generic/fedora28'

    default: URL: https://vagrantcloud.com/api/v2/vagrant/generic/fedora28

==> default: Adding box 'generic/fedora28' (v4.3.12) for provider: virtualbox (amd64)

    default: Downloading: https://vagrantcloud.com/generic/boxes/fedora28/versions/4.3.12/providers/virtualbox/amd64/vagrant.box

    default: Calculating and comparing box checksum...

==> default: Successfully added box 'generic/fedora28' (v4.3.12) for 'virtualbox (amd64)'!

==> default: Importing base box 'generic/fedora28'...

==> default: Matching MAC address for NAT networking...

==> default: Checking if box 'generic/fedora28' version '4.3.12' is up to date...

==> default: Setting the name of the VM: vagrant-test-2_default_1716153560668_498

==> default: Clearing any previously set network interfaces...

==> default: Preparing network interfaces based on configuration...

    default: Adapter 1: nat

==> default: Forwarding ports...

    default: 22 (guest) => 2222 (host) (adapter 1)

==> default: Running 'pre-boot' VM customizations...

==> default: Booting VM...

==> default: Waiting for machine to boot. This may take a few minutes...

    default: SSH address: 127.0.0.1:2222

    default: SSH username: vagrant

    default: SSH auth method: private key

    default: 

    default: Vagrant insecure key detected. Vagrant will automatically replace

    default: this with a newly generated keypair for better security.

    default: 

    default: Inserting generated public key within guest...

    default: Removing insecure key from the guest if it's present...

    default: Key inserted! Disconnecting and reconnecting using new SSH key...

==> default: Machine booted and ready!

==> default: Checking for guest additions in VM...

    default: The guest additions on this VM do not match the installed version of

    default: VirtualBox! In most cases this is fine, but in rare cases it can

    default: prevent things such as shared folders from working properly. If you see

    default: shared folder errors, please make sure the guest additions within the

    default: virtual machine match the version of VirtualBox you have installed on

    default: your host and reload your VM.

    default: 

    default: Guest Additions Version: 6.0.6

    default: VirtualBox Version: 7.0

==> default: Running provisioner: shell...

    default: Running: inline script

    default: Fedora 28 - x86_64 - Updates                    3.4 MB/s |  30 MB     00:08

    default: Fedora 28 - x86_64                              2.3 MB/s |  60 MB     00:26

    default: Last metadata expiration check: 0:00:18 ago on Sun 19 May 2024 09:20:16 PM UTC.

    default: Dependencies resolved.

    default: ================================================================================

    default:  Package                       Arch    Version                   Repository

    default:                                                                            Size

    default: ================================================================================

    default: Installing:

    default:  mediawriter                   x86_64  4.1.4-1.fc28              updates  3.6 M

    ...

    default: Installing weak dependencies:

    default:  mesa-dri-drivers              x86_64  18.0.5-4.fc28             updates   12 M

    default:  ntfs-3g-system-compression    x86_64  1.0-1.fc28                updates   27 k

    default: 

    default: Transaction Summary

    default: ================================================================================

    default: Install  75 Packages

    default: 

    default: Total download size: 65 M

    default: Installed size: 209 M

    default: Downloading Packages:

    default: (1/75): qt5-qtbase-common-5.11.3-1.fc28.noarch. 135 kB/s |  39 kB     00:00

    ...

    default: (74/75): mesa-dri-drivers-18.0.5-4.fc28.x86_64. 2.0 MB/s |  12 MB     00:06

    default: (75/75): llvm-libs-6.0.1-8.fc28.x86_64.rpm      2.5 MB/s |  15 MB     00:06

    default: --------------------------------------------------------------------------------

    default: Total                                           2.9 MB/s |  65 MB     00:22

    default: Running transaction check

    default: Transaction check succeeded.

    default: Running transaction test

    default: Transaction test succeeded.

    default: Running transaction

    default:   Preparing        :                                                        1/1

    default:   Installing       : libblockdev-utils-2.16-2.fc28.x86_64                  1/75

    ...

    default:   Verifying        : ntfs-3g-system-compression-1.0-1.fc28.x86_64         74/75

    default:   Verifying        : llvm-libs-6.0.1-8.fc28.x86_64                        75/75

    default: 

    default: Installed:

    default:   mediawriter.x86_64 4.1.4-1.fc28

    default:   mesa-dri-drivers.x86_64 18.0.5-4.fc28

    ...

    default:   xcb-util-renderutil.x86_64 0.3.9-10.fc28

    default:   xcb-util-wm.x86_64 0.4.1-12.fc28

    default: 

    default: Complete!

When we run vagrant up with shell provisioner specified, the following steps are triggered:

• networking set up
• mount points set up
• shell provisioner runs, some system dependencies are installed

When Vagrant is running a machine, it is actually a running a VM through a selected provider. In our case that is VirtualBox. We can actually see that VM in VirtualBox UI :

If we click "Show" button, we'll get the terminal of the guest OS where we can type and execute commands:

It is possible to define and provision multiple Vagrant machines in a single Vagrantfile. Example: https://github.com/kodekloudhub/certified-kubernetes-administrator-course/blob/master/kubeadm-clusters/virtualbox/Vagrantfile

Their status can be for example like here:

$ vagrant status

Current machine states:

controlplane              running (virtualbox)

node01                    running (virtualbox)

node02                    running (virtualbox)

This environment represents multiple VMs. The VMs are all listed

above with their current state. For more information about a specific

VM, run `vagrant status NAME`.

If we want to ssh to some of these machines, we need to specify its name: 

../certified-kubernetes-administrator-course/kubeadm-clusters/virtualbox$ vagrant ssh controlplane

Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-107-generic x86_64)

 * Documentation:  https://help.ubuntu.com

 * Management:     https://landscape.canonical.com

 * Support:        https://ubuntu.com/pro

 System information as of Sun May 19 23:17:23 UTC 2024

  System load:  0.0               Processes:               100

  Usage of /:   3.7% of 38.70GB   Users logged in:         0

  Memory usage: 10%               IPv4 address for enp0s3: 10.0.2.15

  Swap usage:   0%

Expanded Security Maintenance for Applications is not enabled.

0 updates can be applied immediately.

Enable ESM Apps to receive additional future security updates.

See https://ubuntu.com/esm or run: sudo pro status

Last login: Sun May 19 22:59:32 2024 from 10.0.2.2

vagrant@controlplane:~$ 

References:

Technical topics

Installation of a Vagrant machine

Vagrant Overview for MariaDB Users - MariaDB Knowledge Base

Creating a Vagrantfile - MariaDB Knowledge Base

(3) Fun with Vagrant VM | LinkedIn

https://developer.hashicorp.com/vagrant/docs/cli/up

▷ Vagrant vs Docker | Which one is better? in 2024

Introduction to Kubernetes

These are custom notes that extend my notes from an Udemy course "Kubernetes for the Absolute Beginners - Hands-on". All course content rights belong to course creators. 

Introduction

Web Applications are nowadays developed with containerisation on mind as containers contain everything that is needed to run the application: code, runtime, databases, system libraries, etc.

Kubernetes (k8s) is:

• an open-source container orchestration platform designed to automate deploying, scaling, and operating containerized applications. It’s essential for managing large-scale, distributed applications efficiently.
• Platform for managing application containers (containerised applications, container-oriented applications, containerized workloads and services) across multiple hosts (one or more host clusters)
• The easiest and the most recommended way to manage containers in production
• It makes it easy to orchestrate many containers on many hosts, scale them as microservices, and easily deploy, rollouts and rollbacks
• a set of APIs that we can use to deploy containers on a set of nodes called a cluster 
• We can describe a set of applications and how they should interact with each other, and Kubernetes determines how to make that happen
• Workload scheduler with focus on containerized applications.
• Container orchestration technology -  system for automating the operations and management of application containers in complex, multi-container workloads:
• Container creation
• Container deployment
• Rolling deployment *)
• Auto-scaling
• Load balancing
• Container health monitoring
• Compute resource management
• Volume management
• Persistent storage
• Networking
• High availability by cluster federation
• Open-source
• Originally designed by Google, based upon their running of containers in production. Now maintained by the Cloud Native Computing Foundation
• Supports hosting enhanced and complex applications on various kinds of architectures; it is designed to run anywhere:
• on a bare metal
• in our data center
• on the public cloud - supported on any cloud platform; it is platform-agnostic and integrates with a number of different cloud providers, allowing us to pick the platform that best suits our needs
• on the hybrid cloud
• 2 steps involved in scheduling container on a Kubernetes cluster:
• Provisioning somewhere the Kubernetes cluster with all its components
• Defining the Kubernetes resources, such as Deployments, Services, etc.
• With Kubernetes:
• we can decide when our containers should run
• increase, or decrease the size of application containers
• check the resource consumption of our application deployments
• To save time and effort when scaling applications and workloads, Kubernetes can be bootstrapped using:
• Amazon Elastic Kubernetes Service (EKS)
• Google Kubernetes engine (GKE)

*) Rolling deployment:

• A deployment strategy that slowly replaces previous versions of an application with new versions of an application by completely replacing the infrastructure on which the application is running;
• It is renowned for its ability to update applications without downtime. Incrementally updating nodes or replicas ensures that the service remains available to users throughout the deployment process)
• Rolling deployments use the concept of a window size—this is the number of servers that are updated at any given time. For example, if a Kubernetes cluster is running 10 instances of an application (10 pods), and you want to update two of them at a time, you can perform a rolling deployment with a window size of 2.

Core concepts:

• Cluster and its components:
• Nodes
• Pods
• Services
• Deployments

Cluster

A Kubernetes cluster is a set of node machines for running containerized applications, managed by the Kubernetes control plane. 

Its components are:

• Nodes
• Pods
• Services
• Deployments

Let's explain briefly each of them.

Nodes

• Definition: 
• Nodes are the worker machines in a Kubernetes cluster. They can be physical machines or virtual machines.
• Types of Nodes:
• Master Node(s) (Control Plane): Manages the Kubernetes cluster. It runs the control plane components:
• API Server (kube-apiserver): 
• Responsible for orchestrating all operations within the cluster.
• Exposes the Kubernetes API, acting as the entry point for all administrative tasks. It is used by:
• external users to perform management operations on the cluster
• various controllers to monitor the state of the cluster and make necessary changes as required and by the worker nodes to communicate with the server.
• It helps all the components listed below to communicate with each other and manages them at high level. 
• etcd: A distributed key-value store used to store all cluster data e.g. which containers are on which nodes, when were they loaded etc...
• Scheduler (kube-scheduler): Assigns work (pods) to the nodes in the cluster based on resource availability. It identifies which container needs to go onto which node, depending on the assigned characteristics of the node. It identifies the right node to place a container on based on the containers resource requirements, the worker nodes capacity or any other policies or constraints, such as taints and tolerations or node affinity rules that are on them.
• Controller Manager: Runs various controllers that regulate the state of the cluster
• Node Controller - takes care of onboarding new nodes in the cluster, handling situations when nodes become unavailable or get destroyed
• Replication Controller ensures that the desired number of containers are running at all times in a replication group
• Worker Node: Runs application workloads.
• Kubelet: An agent that runs on each node in a cluster and ensures containers are running in a pod.  It listens for instructions from the Kube API server and deploys or destroys containers on the nodes as required. The Kube API server periodically fetches status reports from the kubelet to monitor the status of nodes and containers on them.
• Kube-proxy: Manages network rules and facilitates communication between pods and services. Maintains network rules on nodes. It handles network communication to your pods from network sessions inside or outside of the cluster. Applications running on the worker nodes need to be able to communicate with each other. For example, you might have a web server running in one container on one of the nodes and a database server running on another container on another node. How would the web server reach the database server on the other node? Communication between worker nodes are enabled by another component that runs on the worker node known as the Kube-proxy service. The Kube-proxy service ensures that the necessary rules are in place on the worker nodes to allow the containers running on them to reach each other.
• Container Runtime Engine: The software responsible for running containers (e.g., Docker, containerd). The software that runs containers, such as Docker, containerd, or CRI-O. Our applications are in the form of containers. The different components that form the entire management system on the master node could be hosted in the form of containers. The DNS service networking solution can all be deployed in the form of containers. So we need these software that can run containers and that's the container runtime engine, a popular one being Docker. So we need Docker, or it's supported equivalent installed on all the nodes in the cluster, including the master nodes, if you wish to host the controlling components as containers. It doesn't always have to be Docker. Kubernetes supports other runtime engines as well like ContainerD, a Rocket.

Pods

• Definition: 
• A pod is the smallest and simplest Kubernetes object. It represents a single instance of a running process in your cluster.
• The smallest deployable units in Kubernetes, encapsulating one or more containers that share the same storage and network resources.
• Components: 
• Each pod contains one or more containers (usually Docker containers) that are tightly coupled and share the same network namespace and storage volumes.
• Lifecycle: 
• Pods are ephemeral. They are created, assigned a unique IP address, and scheduled on a node. If a pod dies, it won’t be resurrected; instead, a new pod will be created.
• Use Case: 
• Pods are used to run a single instance of an application or a part of an application. For example, a web server container and a helper container that pulls data from a database can run in the same pod.

Deploying containers on nodes by using a wrapper around one or more containers is what defines a pod. A pod is the smallest unit in Kubernetes that you can create or deploy. It represents a running process on your cluster as either a component of your application or an entire app. Generally, you only have one container per pod, but if you have multiple containers with a hard dependency, you can package them into a single pod and share networking and storage resources between them. The pod provides a unique network IP and set of ports for your containers, and configurable options that govern how your containers should run. 

One way to run a container in a pod in Kubernetes is to use the kubectl run command, which starts a deployment with a container running inside a pod. 

Services

• Definition: 
• A service is an abstraction that defines a logical set of pods and a policy by which to access them, such as a microservice within an application.
• Abstract a logical set of pods and provide a stable endpoint for accessing these pods, facilitating load balancing and service discovery.
• Components:
• ClusterIP: Exposes the service on an internal IP in the cluster. This type makes the service only reachable from within the cluster.
• NodePort: Exposes the service on the same port of each selected node in the cluster using NAT.
• LoadBalancer: Exposes the service externally using a cloud provider’s load balancer.
• Ingress: 
• Use Case: 
• Services provide a stable endpoint (IP and DNS name) for clients to access a set of pods. They handle load balancing and service discovery.

Kubernetes creates a service with a fixed IP address for our pods. And a controller says, I need to attach an external load balancer with a public IP address to that service so others outside the cluster can access it. 

In GKE, the load balancer is created as a network load balancer. Any client that reaches that IP address will be routed to a pod behind the service. 

A service is an abstraction which defines a logical set of pods and a policy by which to access them. As deployments create and destroy pods, pods will be assigned their own IP addresses, but those addresses don't remain stable over time. 

A service group is a set of pods and provides a stable endpoint or fixed IP address for them. For example, if you create two sets of pods called frontend and backend, and put them behind their own services, the backend pods might change, but frontend pods are not aware of this. They simply refer to the backend service. 

To scale a deployment run the kubectl scale command. In this example, three pods are created in your deployment, and they're placed behind the service and share one fixed IP address. You could also use autoscaling with other kinds of parameters. For example, you can specify that the number of pods should increase when CPU utilization reaches a certain limit. 

So far, we've seen how to run imperative commands like expose and scale. This works well to learn and test Kubernetes step by step. But the real strength of Kubernetes comes when you work in a declarative way. 

Instead of issuing commands, you provide a configuration file that tells Kubernetes what you want your desired state to look like, and Kubernetes determines how to do it. You accomplish this by using a deployment config file. You can check your deployment to make sure the proper number of replicas is running, by using either kubectl get deployments or kubectl describe deployments. To run five replicas instead of three, al you do is update the deployment config file and run the kubectl apply command to use the updated config file. 

You can still reach your endpoint as before by using kubectl get services to get the external IP of the service and reach the public IP address from a client. 

The last question is, what happens when you want to update a new version of your app? Well, you want to update your container to get new code in front of users, but rolling out all those changes at one time would be risky. So in this case, you would use kubectl rollout or change your deployment configuration file and then apply the change using kubectl apply. New pods will then be created according to your new update strategy. Here's an example configuration that will create new version pods individually and wait for a new pod to be available before destroying one of the old pods.

Deployments

• Definition: 
• A deployment provides declarative updates to applications. It describes an application’s lifecycle, such as which images to use for the app, the number of pod replicas, and how to update them.
• Declarative updates to applications, managing the lifecycle of pods and ensuring the desired state is maintained.
• Components:
• ReplicaSet: Ensures a specified number of pod replicas are running at any given time. Deployments manage ReplicaSets.
• Rolling Updates: Gradually replace instances of the old version of an application with a new version.
• Rollback: Revert back to a previous version of the deployment if the current version is not stable.
• Use Case: Deployments manage stateless applications. They ensure the desired state of an application is always maintained, handle scaling, and provide self-healing capabilities.

A deployment represents a group of replicas of the same pod and keeps your pods running even when the nodes they run on fail. A deployment could represent a component of an application or even an entire app. 

To see a list of the running pods in your project, run the command, kubectl get pods. 

Architecture Overview

The Kubernetes architecture is designed for flexibility, scalability, and robustness:

• Master Node (Control Plane)
•  Uses a set of components known as Control Plane components to manage the entire cluster:
• Schedules workloads - plans which containers go onto which nodes
• Monitors nodes
• Maintains the desired state of the cluster
• Worker Nodes:
• Host applications as containers
• Run the application workloads, managing the lifecycle of pods based on the instructions from the control plane.

How a Kubernetes Cluster Works

• User Interaction:
• Users interact with the cluster via the Kubernetes API, typically using kubectl or other client tools.
• API Server:
• The API server processes requests, validates them, and updates the cluster state in etcd.
• Scheduler:
• The scheduler watches for newly created pods that do not have an assigned node, selecting nodes for them based on resource requirements.
• Controller Manager:
• Controllers constantly monitor the cluster state and make adjustments to ensure the desired state matches the current state.
• Kubelet:
• The kubelet on each worker node ensures the containers described in the pod specs are running and healthy.
• Kube-proxy:
• Manages network routing and load balancing to ensure seamless communication between pods and services.

Benefits of a Kubernetes Cluster

• Scalability: Automatically scales applications up and down based on demand.
• High Availability: Ensures application availability even if some nodes fail.
• Resource Efficiency: Optimizes resource usage and balances workloads across nodes.
• Flexibility: Supports a wide range of workloads, from microservices to batch processing.
• Portability: Runs on various environments, including on-premises, cloud, and hybrid setups.