Pages

Monday, 28 March 2022

Introduction to Terraform


 

Benefits of using Terraform:
  • Automated way to achieve repeatable and scalable infrastructure. If we manually build infrastructure and then need to repeat the process, a) it would take double the time and b) chance of making an error / applying a different setting are high. If we use IaC tool like Terraform, we define infrastructure with all its parameters once, in a code, and then provision it and destroy it as many times as we wish.
  • Shorter deployment times. No manual work is required.
  • Lower costs. Defining infrastructure is done only once. 
  • All resources are defined in the code => 
  • From Terraform for Site24x7 | Online Help Site24x7:
    • Ensure error-free deployments
    • Handle heterogeneous cloud deployments for fault tolerance
    • Collaborate, obtain specific versions, and create your own environment
    • Spawn and remove testing environments easily
    • Improve deployment speeds
    • Build, version, or change infrastructure safely and efficiently
    • Trace and correct configuration-level issues
  • Cross-platform
  • Human-readable
  • Open source. We can create our own Terraform providers for our specific functionality.
  • Has both a free version (Terraform) and an enterprise version (Terraform Cloud) with no functionality lost between the versions.
  • From Kubernetes: Getting Started with Kubernetes provider | Guides | hashicorp/kubernetes | Terraform | Terraform Registry:
    • drift detection - terraform plan will always present you the difference between reality at a given time and config you intend to apply.
    • full lifecycle management - Terraform doesn't just initially create resources, but offers a single command for creation, update, and deletion of tracked resources without needing to inspect the API to identify those resources.
    • synchronous feedback - While asynchronous behaviour is often useful, sometimes it's counter-productive as the job of identifying operation result (failures or details of created resource) is left to the user. e.g. you don't have IP/hostname of load balancer until it has finished provisioning, hence you can't create any DNS record pointing to it.
    • graph of relationships - Terraform understands relationships between resources which may help in scheduling - e.g. if a Persistent Volume Claim claims space from a particular Persistent Volume Terraform won't even attempt to create the PVC if creation of the PV has failed.

Terraform uses configuration files with extension .tf to define resources that need to be deployed.

General format of tf file content blocks is:
 
<block><parameters> {
    key1 = value1
    key2 = value2
}

block contains information about infrastructure platform and a set of resources within that platform that we want to create.
 
resource Block - A resource block declares a resource of a given type with a given local name. The name is used to refer to this resource from elsewhere in the same Terraform module, but has no significance outside of the scope of a module.

Example: Create a file on a local host where Terraform is installed. 


# https://www.terraform.io/language/expressions/references#filesystem-and-workspace-info
resource "local_file" "foo" {
    filename = "${path.cwd}/temp/foo.txt"
    content = "This is a text content of the foo file!"
}

resource is the reserved word and is the block name. Each resource is uniquely defined by the combination of the resource type and name. In our example:

  • "local_file" is a predefined string which denotes the resource type:
    • local - resource provider (by convention, resource type names start with their provider's preferred local name)
    • file - resource type
  • "foo" is the logical resource name; this is a local name that can be used from the same Terraform module

Inside the curly braces (resource block) are arguments for resource. They are specific for the given resource type. In this example, we are using local_file | Resources | hashicorp/local | Terraform Registry.

Here are some resource providers:
  • local
  • aws
  • azure
  • gcp
  • ...

Each resource type has its own set of arguments.


Example: Create Amazon Machine Image (AMI)

aws-ec2.tf:

resource "aws_instance" "my_webserver" {
    ami = "ami-0123456789abcdef" 
    instance_type = "t2.micro"
}


Example: S3

aws-s3.tf:

resource "aws_s3_bucket" "my_data" {
    bucket = "webserver-bucket-org-1234" 
    acl = "private"
}



Simple Terraform workflow:
  • write .tf file
  • (optional) run terraform fmt command which scans .tf files in current working directory and formats them in the canonical format. This improves the readability of configuration files.
  • run terraform init command. It:
    • checks the .tf config file
    • initializes the working directory containing .tf file
    • from .tf file it finds out what type of resource(s) we want to create
    • it initializes specified providers - it downloads plugin(s) required in order to work on these resources
    • is a safe command. It can be many times without affecting the infrastructure that is being deployed.
  • (optional) run terraform validate to validate configuration file syntax
  • run terraform plan command in order to review the execution plan 
  • run terraform apply command to apply changes
  • run terraform show command in order to see the changes made; it prints the current state of the infrastructure; it prints all resources and values of their attributes. Use -json flag to print it in JSON format.
  • run terraform destroy to destroy created resources
 

terraform init

 
Let's first initialize working directory where .tf file resides:
 
/demo-terraform/local_resources_demo$ ls -a
.  ..  create_local_file_demo.tf
 
/demo-terraform/local_resources_demo$ cat create_local_file_demo.tf
resource "local_file" "foo" {
    filename = "${path.cwd}/temp/foo.txt"
    content = "This is a text content of the foo file!"
}
 
/demo-terraform/local_resources_demo$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/local...
- Installing hashicorp/local v2.2.2...
- Installed hashicorp/local v2.2.2 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

 
 
From the output above, we can see that Terraform used resource type information ("local_file") to deduct that it needs to download and install hashicorp/local plugin.

hashicorp/local is an example of the source address. It is the identifier used by Terraform to locate and download the plugin from the registry. The first value (e.g. hashicorp) represents the organisational namespace and the second one is type which is the name of provider (e.g. local, aws, google etc...). Source address can also optionally start with the hostname which is the name of the registry where plugin is located:

registry.terraform.io/hashicorp/local
 
By default, Terraform downloads the latest version of plugins but it is possible to specify desired version as well, if we want to prevent using the (latest) version which introduces breaking changes.
 
If we look at the directory, we can see that terraform init created one directory and one file:
 
/demo-terraform/local_resources_demo$ ls -a
.  ..  create_local_file_demo.tf  .terraform  .terraform.lock.hcl

 
 
.terraform directory contains downloaded plugins (executable binaries) for providers that are specified in .tf file. In our case, provider is named local:

/demo-terraform/local_resources_demo/.terraform$ ls -la providers/registry.terraform.io/hashicorp/local/2.2.2/linux_amd64/terraform-provider-local_v2.2.2_x5
-rwxr-xr-x 1 bojan bojan 13864960 Mar 28 13:27 providers/registry.terraform.io/hashicorp/local/2.2.2/linux_amd64/terraform-provider-local_v2.2.2_x5

.hcl file contains list of providers with their details. This is something like package-lock.json file in Node projects:

/demo-terraform/local_resources_demo$ cat .terraform.lock.hcl
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/local" {
  version = "2.2.2"
  hashes = [
    "h1:5UYW2wJ320IggrzLt8tLD6MowePqycWtH1b2RInHZkE=",
    "zh:027e4873c69da214e2fed131666d5de92089732a11d096b68257da54d30b6f9d",
    "zh:0ba2216e16cfb72538d76a4c4945b4567a76f7edbfef926b1c5a08d7bba2a043",
    "zh:1fee8f6aae1833c27caa96e156cf99a681b6f085e476d7e1b77d285e21d182c1",
    "zh:2e8a3e72e877003df1c390a231e0d8e827eba9f788606e643f8e061218750360",
    "zh:719008f9e262aa1523a6f9132adbe9eee93c648c2981f8359ce41a40e6425433",
    "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
    "zh:9a70fdbe6ef955c4919a4519caca116f34c19c7ddedd77990fbe4f80fe66dc84",
    "zh:abc412423d670cbb6264827fa80e1ffdc4a74aff3f19ba6a239dd87b85b15bec",
    "zh:ae953a62c94d2a2a0822e5717fafc54e454af57bd6ed02cd301b9786765c1dd3",
    "zh:be0910bdf46698560f9e86f51a4ff795c62c02f8dc82b2b1dab77a0b3a93f61e",
    "zh:e58f9083b7971919b95f553227adaa7abe864fce976f0166cf4d65fc17257ff2",
    "zh:ff4f77cbdbb22cc98182821c7ef84dce16298ab0e997d5c7fae97247f7a4bcb0",
  ]
}

 

terraform validate


It requires terraform init to have been executed. If not, if configuration directory is not initialized, we might get the error like:

$ terraform validate
Error: Missing required provider
│ 
│ This configuration requires provider registry.terraform.io/hashicorp/local, but that provider isn't available. You may be able to
│ install it automatically by running:
│   terraform init

Error: Missing required provider
│ 
│ This configuration requires provider registry.terraform.io/hashicorp/tls, but that provider isn't available. You may be able to
│ install it automatically by running:
│   terraform init

terraform apply might fail in spite of terraform validate reporting no errors. This is because the validate command only carries out a general verification of the configuration. It validates the resource block and the argument syntax but not the values the arguments expect for a specific resource.


terraform plan


Before creating resource we can see the execution plan that will be carried out by Terraform. It is always good to review these actions before they get executed. terraform plan shows actions that will be used to create resources. The output is similar to the one of git diff:

/demo-terraform/local_resources_demo$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # local_file.foo will be created
  + resource "local_file" "foo" {
      + content              = "This is a text content of the foo file!"
      + directory_permission = "0777"
      + file_permission      = "0777"
      + filename             = "/home/bojan/dev/github/demo-terraform/local_resources_demo/temp/foo.txt"
      + id                   = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.


+ symbol denotes resources that will be created.
Resource is listed with all its attributes showing default values for those that we didn't specify explicitly in our .tf file.

In rare occasions, when creation of one module requires another module to be created, we need to do plan (and apply) in stages where we target specific resources. Example:

The kubernetes_manifest resource needs access to the API server of the cluster during planning. This is because, in order to support CRDs in Terraform ecosystem, we need to pull the schema information for each manifest resource at runtime (during planing).

AFAICS, every person who reported seeing similar issues above, configures the attributes of the provider "kubernetes" {..} block using references to other resources' attributes. You can only do this if the referenced resource (the cluster in this case) IS ALREADY PRESENT before you start planing / applying any of the kubernetes_manifest resources. You can achieve this as simply as using the -target CLI argument to Terraform to limit the scope of the first apply to just the cluster and it's direct dependencies. Then you follow up with a second apply without a -target argument and this constructs the rest of the resources (manifest & others). You will end up with a single state file and subsequent updates no longer require this two-step approach as long as the cluster resource is present.

This limitation is stemming from Terraform itself, and the provider tries to push things as far as it can, but there is no way around needing access to schema from the API (Terraform is fundamentally a strongly-typed / schema based system).

$ terraform plan -target module.nginx-virtual-network -target module.nginx-eks-cluster
...

Plan: 31 to add, 0 to change, 0 to destroy.
│ Warning: Resource targeting is in effect
│ 
│ You are creating a plan with the -target option, which means that the result of this plan may not represent all of the changes requested by the current configuration.
│ 
│ The -target option is not for routine use, and is provided only for exceptional situations such as recovering from errors or mistakes, or when Terraform specifically
│ suggests to use it as part of an error message.

$ terraform plan -target module.nginx-eks-cluster-karpenter
...

terraform apply



This command creates resources but it also shows the plan first and asks us to confirm to go ahead with it:
 
/demo-terraform/local_resources_demo$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # local_file.foo will be created
  + resource "local_file" "foo" {
      + content              = "This is a text content of the foo file!"
      + directory_permission = "0777"
      + file_permission      = "0777"
      + filename             = "/home/bojan/dev/github/demo-terraform/local_resources_demo/temp/foo.txt"
      + id                   = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

local_file.foo: Creating...
local_file.foo: Creation complete after 0s [id=db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
 
File is created indeed:
 
/demo-terraform/local_resources_demo$ cat ./temp/foo.txt
This is a text content of the foo file!
 
terraform apply also creates terraform.tfstate file which contains details of the created resource:
 
/demo-terraform/local_resources_demo$ cat terraform.tfstate
{
  "version": 4,
  "terraform_version": "1.1.7",
  "serial": 1,
  "lineage": "921cfa36-9ad5-9a2e-66db-d2ef62e7509b",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "local_file",
      "name": "foo",
      "provider": "provider[\"registry.terraform.io/hashicorp/local\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "content": "This is a text content of the foo file!",
            "content_base64": null,
            "directory_permission": "0777",
            "file_permission": "0777",
            "filename": "/home/bojan/dev/github/demo-terraform/local_resources_demo/temp/foo.txt",
            "id": "db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e",
            "sensitive_content": null,
            "source": null
          },
          "sensitive_attributes": [],
          "private": "bnVsbA=="
        }
      ]
    }
  ]
}

 

If we want to prevent terraform plan|apply commands to print out the file content in their output in terminal, we can use sensitive_content argument (instead of content):
 
sensitive_content = "This is a text content of the foo file!"
 
Execution plan in terminal will now contain this line:
 
    + sensitive_content    = (sensitive value)
 
 

terraform show

 
 To see details of the resource created we can use terraform show command:
 
/demo-terraform/local_resources_demo$ terraform show
# local_file.foo:
resource "local_file" "foo" {
    content              = "This is a text content of the foo file!"
    directory_permission = "0777"
    file_permission      = "0777"
    filename             = "/home/bojan/dev/github/demo-terraform/local_resources_demo/temp/foo.txt"
    id                   = "db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e"
}

 
 

Updating already created resources

 
 
In the example above, we created a file with permissions 0777. Let's say we want to change this to permissions 0700.

We first need to edit our original .tf file:
 
/demo-terraform/local_resources_demo$ cat create_local_file_demo.tf
resource "local_file" "foo" {
    filename = "${path.cwd}/temp/foo.txt"
    content = "This is a text content of the foo file!"
    file_permission = "0700"
}
 

Let's review the changes:

/demo-terraform/local_resources_demo$ terraform plan
local_file.foo: Refreshing state... [id=db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # local_file.foo must be replaced
-/+ resource "local_file" "foo" {
      ~ file_permission      = "0777" -> "0700" # forces replacement
      ~ id                   = "db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e" -> (known after apply)
        # (3 unchanged attributes hidden)
    }


Plan: 1 to add, 0 to change, 1 to destroy.

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

-/+ means that resource will be deleted and then recreated. 
 
Line with # forces replacement comment shows the attribute whose change actually forces the resource replacement.
 
This type of infrastructure is called an immutable infrastructure: change of a single attribute requires re-creation of the entire resource.

demo-terraform/local_resources_demo$  terraform apply
local_file.foo: Refreshing state... [id=db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # local_file.foo must be replaced
-/+ resource "local_file" "foo" {
      ~ file_permission      = "0777" -> "0700" # forces replacement
      ~ id                   = "db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

local_file.foo: Destroying... [id=db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e]
local_file.foo: Destruction complete after 0s
local_file.foo: Creating...
local_file.foo: Creation complete after 0s [id=db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e]

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

 
Let's check new permissions on the file:

 /demo-terraform/local_resources_demo$ ls -la ./temp/
total 12
-rwx------ 1 bojan bojan   39 Mar 28 16:17 foo.txt


Deleting resources



To delete created resources we need to use terraform destroy command:


$ terraform destroy
local_file.foo: Refreshing state... [id=db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # local_file.foo will be destroyed
  - resource "local_file" "foo" {
     
- content              = "This is a text content of the foo file!" -> null
     
- directory_permission = "0777" -> null
     
- file_permission      = "0700" -> null
     
- filename             = "/home/bojan/dev/github/demo-terraform/local_resources_demo/temp/foo.txt" -> null
     
- id                   = "db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e" -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

local_file.foo: Destroying... [id=db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e]
local_file.foo: Destruction complete after 0s

Destroy complete! Resources: 1 destroyed.


We can verify that temp folder does not contain the test file. Note that temp folder itself remained, although it was created in process of creating a file resource.



Configuration Directory


Configuration directory is the one which contains one or more configuration (.tf) files. Each configuration file can specify one or more resources (resource blocks required to provision the infrastructure). 

We can combine all resources from multiple (smaller) configuration file into a single (large) configuration file.  The common convention is to name that file main.tf

Configuration files can be organised as here:
  • main.tf - main config file containing definition of resources
  • variables.tf - contains variable declarations
  • outputs.tf - contains outputs from resources
  • provider.tf - contains provider definitions

It is possible to use multiple providers (e.g. local, random etc...) in the same configuration directory. 

Whenever we add a resource for a provider that has not been used so far in the configuration directory, we have to initialize the directory by running terraform init command.


Debugging (Logging)


To enable logging we need to set TF_LOG environment variable to a desired log level which can be one of following:
  • INFO
  • WARNING
  • ERROR
  • DEBUG
  • TRACE (most verbose)
Example:

$ export TF_LOG=TRACE

To persist log into a file we need to:
  • set TF_LOG_PATH environment variable to a path to the log file
  • make sure that TF_LOG is set
Example:

$ export TF_LOG_PATH=/tmp/terraform.log

To disable logging into a file:

$ unset TF_LOG_PATH

---





Monday, 21 March 2022

How to install Terraform on Ubuntu



We can install Terraform on Ubuntu in two ways:
  • downloading and installing a binary file
  • via package manager (apt)

Downloading and Installing a Binary


Terraform provides binaries that can be downloaded from https://www.terraform.io/downloads. Linux binaries are pre-compiled for the following architectures: 
  • 386
  • Amd64
  • Arm
  • Arm64
 
To find out your architecture use: 

$ uname -m
x86_64


This means we're running on Amd64 architecture and so the binary we'll download is https://releases.hashicorp.com/terraform/M.m.r/terraform_M.m.r_linux_amd64.zip where M.m.r is the latest version. This version can be extracted from Terraform git repository tag:
 
$ TER_VER=`curl -s https://api.github.com/repos/hashicorp/terraform/releases/latest | grep tag_name | cut -d: -f2 | tr -d \"\,\v | awk '{$1=$1};1'`
 
This is the latest Terraform version at the time of writing:

$ echo $TER_VER
1.1.7
 
We can use this variable in the following commands which download and unpack the archive and move the binary into bin directory accessible to all users on this machine:
 
$ wget https://releases.hashicorp.com/terraform/${TER_VER}/terraform_${TER_VER}_linux_amd64.zip
$ unzip terraform_${TER_VER}_linux_amd64.zip
$ sudo mv terraform /usr/local/bin/

To test the installation:

$ terraform -v
Terraform v1.1.7
on linux_amd64


We can now delete the archive:
 
$ rm terraform_${TER_VER}_linux_amd64.zip
 
To uninstall Terraform:

$ sudo rm -rf /usr/local/bin/terraform

Installing Terraform via Package Manager

To install Terraform via apt package manager we first need to add Hashicorp package source to our local package source list, update apt packages and then install terraform package. 


apt installs Terraform binary in this directory:

$ which terraform 
/usr/bin/terraform

To uninstall Terraform:

$ sudo apt remove terraform


Resources:

 

Unix Filesystem Hierarchy



Run man hier to see extensive list of directories and description of the filesystem hierarchy.

/srv

  • owner is root
  • contains site-specific data which is served by this system
  • place for your workspace - your software development tree.
  • code repository (possibly in /srv/sourcerepo or something), and then developers would check out their own working copies into their home directories.
  • place for my source code (though I usually use /srv/vcs/sourcerepo)


/usr

  • the location where Distribution-based items are placed 


/usr/src
  • owner is root
  • Source files for different parts of the system, included with some packages for reference purposes.  Don't work here with your own projects, as files below /usr should be read-only except when installing software (optional).
  • meant for source code for the binaries that come with your system's installation. For example, it could contain the source code for your kernel, tools such as ls, passwd, cp, mv etc, which are all installed BY YOUR DISTRIBUTION. If you upgrade your OS from source, all the source code would go here, when you rebuild your system. You DON'T want to put any software that you install BY YOURSELF in here, because they may get overwritten when you upgrade your system. In general, files that are in /, /usr/bin, /usr/sbin, /bin, /sbin etc. have their source code in /usr/src.
  • is a system directory. You should not change the owner from root for security reasons
  • contains the linux headers and source code of the kernel. Since the system compiles the kernel from there, it IS a security breach to change the ownership to anything other than root
  • if you wanted to recompile an Ubuntu package from source, their package manager would place the source for package in /usr/src/{package dir}


/usr/local

  • owner is root
  • location where you'd place your own localized changesod the Distribution (/usr/local will be empty after a base install)
  • contains the following subdirectories: 
    • bin  
    • etc  
    • games  
    • include  
    • lib  
    • man  
    • sbin 
    • share  
    • src
  • this directory tree is meant to be used for software that you install by yourself, without using the distribution CD(s). For example, /usr/local/bin, /usr/local/sbin are for the binaries that are installed by you separately, /usr/local/etc is for config files etc. Thus /usr/local/src is for source files that you yourself downloaded. Example: go (binary distribution) gets installed in /usr/local/go.
  • If you upgrade your system, files under the /usr tree get overwritten, such as /usr/bin, /usr/sbin, /usr/src etc. However, anything under /usr/local will not be touched. This is why all the software you installed separately should go to /usr/local tree.
  • a place to install files built by the administrator, typically by using the make command (e.g., ./configure; make; make install). The idea is to avoid clashes with files that are part of the operating system, which would either be overwritten or overwrite the local ones otherwise (e.g., /usr/bin/foo is part of the OS while /usr/local/bin/foo is a local alternative).
  • for self, inhouse, compiled and maintained software. 
  • reserved for software installed locally by the sysadmin
  • place where you want to install software along with source files (for other programs to use or for people to look at)
  • It's not meant, however, to be a workspace. Since it is local, you can do whatever you want, of course, but this isn't designed to be the place to put your software development tree.
  • for use by the system administrator when installing software locally.
    It needs to be safe from being overwritten when the system software is updated. 

 

/usr/local/bin

  • for programs that a normal user may run
  • binaries at this path are accessible to all user accounts

 


/usr/local/src

  • owner is root
  • Source code for locally installed software
  • If you downloaded a program not managed by your distribution and wanted to compile/install it, FHS dictates that you do that in /usr/local/src.
  • a good place for downloading third party source code (eg for patching and rebuilding packages), not my own source code


/opt

  • This directory is reserved for all the software and add-on packages that are not part of the default installation. All third party applications should be installed in this directory. (Linux Filesystem Hierarchy: /opt)
  • a directory for installing unbundled packages (i.e. packages not part of the Operating System distribution, but provided by an independent source), each one in its own subdirectory. They are already built whole packages provided by an independent third party software distributor. Unlike /usr/local stuff, these packages follow the directory conventions (or at least they should). For example, someapp would be installed in /opt/someapp, with one of its command being /opt/someapp/bin/foo, its configuration file would be in /etc/opt/someapp/foo.conf, and its log files in /var/opt/someapp/logs/foo.access. (What is the difference between /opt and /usr/local?)
  • for non-self, external, prepackaged binary/application bundle installation
  • directory where you can just toss things and see if they work makes a whole lot of sense. I know I'm not going to go through the effort of packaging things myself to try them out. If the app doesn't work out, you can simply rm the /opt/mytestapp directory and that application is history.
  • used for third-party software, which in the context of Ubuntu, means precompiled software that is not distributed via Debian packages 
  • A program that is installed in /opt is supposed to be self-contained.
  • The main reason for using /opt is to provide a common standard path where external software can be installed without interfering with the rest of the installed system. /opt does not appear in standard compiler or linker paths (gcc -print-search-dirs or /etc/ld.so.conf etc.), so headers and libraries installed there are somewhat isolated from the main system and shouldn't interfere with already-installed programs. (Why should I move everything into /opt?)

 

Resources:

executable - What is /usr/local/bin? - Unix & Linux Stack Exchange

Monday, 14 March 2022

Using Makefile in Docker projects

 

To wrap docker build and docker run commands which contain multiple arguments into simpler make commands we can add Makefile to Docker-based projects.

Makefile example:
 
DOCKER_IMAGE_NAME := my-app

build:
#    cp -R ~/.aws ./my_db/.aws
#    docker image rm $(DOCKER_IMAGE_NAME) || (echo "Image $(DOCKER_IMAGE_NAME) didn't exist so not removed."; exit 0)
    docker build -t $(DOCKER_IMAGE_NAME) .
run:
#   Usage: make run AWS_REGION=us-east-1 AWS_DEFAULT_OUTPUT=json AWS_ACCESS_KEY_ID=id AWS_SECRET_ACCESS_KEY=key
    docker run \
        -e AWS_REGION=$(AWS_REGION) \
        -e AWS_DEFAULT_OUTPUT=$(AWS_DEFAULT_OUTPUT) \
        -e AWS_ACCESS_KEY_ID=$(AWS_ACCESS_KEY_ID) \
        -e AWS_SECRET_ACCESS_KEY=$(AWS_SECRET_ACCESS_KEY) \
        -v "$(shell pwd)"/shared:/usr/src/my-app/out/ \
        --rm \
        --name $(DOCKER_IMAGE_NAME) \
        $(DOCKER_IMAGE_NAME) 

To build the Docker image: 

$ make build

To run it:

$ make run AWS_REGION=us-east-1 AWS_DEFAULT_OUTPUT=json AWS_ACCESS_KEY_ID=<id> AWS_SECRET_ACCESS_KEY=<key>


Thursday, 3 March 2022

Secure File Copy (scp) Tool

 scp is used to securely copy files and directories to or from remote machine.

 


$man scp

SCP(1)                                                                                 BSD General Commands Manual                                                                                SCP(1)

NAME
     scp — OpenSSH secure file copy

SYNOPSIS
     scp [-346BCpqrTv] [-c cipher] [-F ssh_config] [-i identity_file] [-J destination] [-l limit] [-o ssh_option] [-P port] [-S program] source ... target

DESCRIPTION
     scp copies files between hosts on a network.  It uses ssh(1) for data transfer, and uses the same authentication and provides the same security as ssh(1).  scp will ask for passwords or
     passphrases if they are needed for authentication.

     The source and target may be specified as a local pathname, a remote host with optional path in the form [user@]host:[path], or a URI in the form scp://[user@]host[:port][/path].  Local file
     names can be made explicit using absolute or relative pathnames to avoid scp treating file names containing ‘:’ as host specifiers.

     When copying between two remote hosts, if the URI format is used, a port may only be specified on the target if the -3 option is used.

     The options are as follows:

     -3      Copies between two remote hosts are transferred through the local host.  Without this option the data is copied directly between the two remote hosts.  Note that this option disables the
             progress meter.

     -4      Forces scp to use IPv4 addresses only.

     -6      Forces scp to use IPv6 addresses only.

     -B      Selects batch mode (prevents asking for passwords or passphrases).

     -C      Compression enable.  Passes the -C flag to ssh(1) to enable compression.

     -c cipher
             Selects the cipher to use for encrypting the data transfer.  This option is directly passed to ssh(1).

     -F ssh_config
             Specifies an alternative per-user configuration file for ssh.  This option is directly passed to ssh(1).

     -i identity_file
             Selects the file from which the identity (private key) for public key authentication is read.  This option is directly passed to ssh(1).


     -J destination
             Connect to the target host by first making an scp connection to the jump host described by destination and then establishing a TCP forwarding to the ultimate destination from there.  Mul‐
             tiple jump hops may be specified separated by comma characters.  This is a shortcut to specify a ProxyJump configuration directive.  This option is directly passed to ssh(1).

     -l limit
             Limits the used bandwidth, specified in Kbit/s.

     -o ssh_option
             Can be used to pass options to ssh in the format used in ssh_config(5).  This is useful for specifying options for which there is no separate scp command-line flag.  For full details of
             the options listed below, and their possible values, see ssh_config(5).

                   AddressFamily
                   BatchMode
                   BindAddress
                   BindInterface
                   CanonicalDomains
                   CanonicalizeFallbackLocal
                   CanonicalizeHostname
                   CanonicalizeMaxDots
                   CanonicalizePermittedCNAMEs
                   CASignatureAlgorithms
                   CertificateFile
                   ChallengeResponseAuthentication
                   CheckHostIP
                   Ciphers
                   Compression
                   ConnectionAttempts
                   ConnectTimeout
                   ControlMaster
                   ControlPath
                   ControlPersist
                   GlobalKnownHostsFile
                   GSSAPIAuthentication
                   GSSAPIDelegateCredentials
                   HashKnownHosts
                   Host
                   HostbasedAuthentication
                   HostbasedKeyTypes
                   HostKeyAlgorithms
                   HostKeyAlias
                   Hostname
                   IdentitiesOnly
                   IdentityAgent
                   IdentityFile
                   IPQoS
                   KbdInteractiveAuthentication
                   KbdInteractiveDevices
                   KexAlgorithms
                   LogLevel
                   MACs
                   NoHostAuthenticationForLocalhost
                   NumberOfPasswordPrompts
                   PasswordAuthentication
                   PKCS11Provider
                   Port
                   PreferredAuthentications
                   ProxyCommand
                   ProxyJump
                   PubkeyAcceptedKeyTypes
                   PubkeyAuthentication
                   RekeyLimit
                   SendEnv
                   ServerAliveInterval
                   ServerAliveCountMax
                   SetEnv
                   StrictHostKeyChecking
                   TCPKeepAlive
                   UpdateHostKeys
                   User
                   UserKnownHostsFile
                   VerifyHostKeyDNS

     -P port
             Specifies the port to connect to on the remote host.  Note that this option is written with a capital ‘P’, because -p is already reserved for preserving the times and modes of the file.

     -p      Preserves modification times, access times, and modes from the original file.

     -q      Quiet mode: disables the progress meter as well as warning and diagnostic messages from ssh(1).

     -r      Recursively copy entire directories.  Note that scp follows symbolic links encountered in the tree traversal.

     -S program
             Name of program to use for the encrypted connection.  The program must understand ssh(1) options.

     -T      Disable strict filename checking.  By default when copying files from a remote host to a local directory scp checks that the received filenames match those requested on the command-line
             to prevent the remote end from sending unexpected or unwanted files.  Because of differences in how various operating systems and shells interpret filename wildcards, these checks may
             cause wanted files to be rejected.  This option disables these checks at the expense of fully trusting that the server will not send unexpected filenames.

     -v      Verbose mode.  Causes scp and ssh(1) to print debugging messages about their progress.  This is helpful in debugging connection, authentication, and configuration problems.

EXIT STATUS
     The scp utility exits 0 on success, and >0 if an error occurs.

SEE ALSO
     sftp(1), ssh(1), ssh-add(1), ssh-agent(1), ssh-keygen(1), ssh_config(5), sshd(8)

HISTORY
     scp is based on the rcp program in BSD source code from the Regents of the University of California.

AUTHORS
     Timo Rinne <tri@iki.fi>
     Tatu Ylonen <ylo@cs.hut.fi>

BSD                                                                                         November 30, 2019 
    


To copy a directory recursively use -r flag:

$ scp -r nvidia@nvidia-nano:/usr/src/tensorrt/samples/python/uff_ssd ~/dev/
nvidia@nvidia-nano's password: 
inference.py                                  100%   12KB   3.3MB/s   00:00    
engine.py                                     100% 5750     4.5MB/s   00:00    
__init__.py                                   100%    0     0.0KB/s   00:00    
boxes.py                                      100% 6725     5.2MB/s   00:00    
coco.py                                       100% 4755     4.2MB/s   00:00    
...

shell - How do I copy a folder from remote to local using scp? - Stack Overflow


To copy file which has spaces in path and/or name use double backslash before space and wrap entire file path in double quotation marks:

~/Downloads$ scp nvidia@nvidia-nano:"/home/nvidia/Pictures/Object\\ Detection\\ -\\ SSD.png" ~/Pictures
nvidia@192.168.0.10's password: 
Object Detection - SSD.png                    100%  444KB   1.2MB/s   00:00 

 
To copy a directory from local to remote:

$ scp -r LocalDir user@19.168.0.61:/home/user/destination

This will create LocalDir in destination folder.
 
 
If public-private key (SSH keys) authentication is used (instead of username/password) we can specify the path to the private key. For the following example we've created key pair for accessing AWS EC2 VM instance, downloaded the private key (.pem) file and can use it to access that VM in order to copy a directory onto it:

~/.ssh$ scp \
-i "my-vm.pem" \
-pr /home/bojan/dev/my-app/ \
ec2-user@50.51.52.53:/home/ec2-user/my-app/

This is how to copy a file from remote (EC2 instance in this case) to a local host (into a current directory):

$ scp \
-i key-pair--ec2--bojan-temp.pem \
ec2-user@ec2-19-208-98-120.compute-1.amazonaws.com:/usr/bin/my_file \
./
 
To copy a single file from local to remote:
 
$ scp \
-i key-pair--ec2--bojan-temp.pem \
my_file \
ec2-user@ec2-19-208-99-120.compute-1.amazonaws.com:/home/ec2-user/
 
 
To copy all files from current directory, we can use *:
 
~/path/to/my-app$ scp \
-i ~/.ssh/my-vm.pem \
-pr * \
ec2-user@50.51.52.53:/home/ec2-user/my-app/
 
app.py                                                                                                                                                                      100% 4921    17.9KB/s   00:00    
Dockerfile                                                                                                                                                                  100%  152     0.9KB/s   00:00    
Makefile                                                                                                                                                                    100%  313     2.1KB/s   00:00    
README.md                                                                                                                                                                   100% 2179     8.1KB/s   00:00    
requirements.txt 

 

scp preserves file/directory attributes e.g. if it was hidden on the origin machine, it will also be hidden on the target machine.

 

Troubleshooting

If SSH server is not installed, not running or firewall blocks incoming connections on port 22 on the remote machine, scp command might fail with this error:

$ scp -r local_dir/ user@remote_host:~/Downloads/
ssh: connect to host remote_host port 22: Connection refused
lost connection

To fix this, we need to have access to the remote machine so we can check if the SSH server is installed and is up and running.
 
The following commands show the case when openssh server is not installed:

To check if sshd process is running:

$ ps -aux | grep sshd
 
To check if there is SSH server config file (which existence is mandatory for running SSH server):

$ cat /etc/ssh/sshd_config
cat: /etc/ssh/sshd_config: No such file or directory


To check which openssh application is installed (the output shows that only SSH client is present):

$ apt list --installed | grep openssh

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

openssh-client/jammy-updates,now 1:8.9p1-3ubuntu0.1 amd64 [installed,automatic]

 
To install SSH server:
 
$ sudo apt install openssh-server

We can now verify again all installed openssh packages:

$ apt list --installed | grep openssh

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

openssh-client/jammy-updates,now 1:8.9p1-3ubuntu0.1 amd64 [installed,automatic]
openssh-server/jammy-updates,now 1:8.9p1-3ubuntu0.1 amd64 [installed]
openssh-sftp-server/jammy-updates,now 1:8.9p1-3ubuntu0.1 amd64 [installed,automatic]

SSH server config is now present:

$ cat /etc/ssh/sshd_config

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

Include /etc/ssh/sshd_config.d/*.conf

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile    .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
KbdInteractiveAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the KbdInteractiveAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via KbdInteractiveAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and KbdInteractiveAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem    sftp    /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#    X11Forwarding no
#    AllowTcpForwarding no
#    PermitTTY no
#    ForceCommand cvs server


Installing openssh-server is enough. If we re-run now scp command, it will be successful.

We can also verify that ssh daemon is listening on port 22:

$ sudo lsof -i :22 
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd    1737 root    3u  IPv4  30631      0t0  TCP *:ssh (LISTEN)
sshd    1737 root    4u  IPv6  30633      0t0  TCP *:ssh (LISTEN)



How to enforce password-based authentication?


In the following case password authentication was not enabled on the server side so the client received the following error:

ssh user@host
Received disconnect from host port 22:2: Too many authentication failures
Disconnected from host port 22

We can debug it with:

$ ssh user@host -vvv


To enable password-based authentication, we need to edit ssh daemon config on the server side:

$ cat /etc/ssh/sshd_config
...
PasswordAuthentication yes


After that:

$ sudo /etc/init.d/ssh force_reload
$ sudo /etc/init.d/ssh restart


References:

amazon ec2 - How Can I Download a File from EC2 - Stack Overflow

12.04 - Why am I getting a "port 22: Connection refused" error? - Ask Ubuntu

ssh - scp connection refused error - Super User

How to Change SSH Port Number in Linux

ssh - Is it okay when ssh_config does not exist? - Server Fault