Friday, 13 May 2022

AWS Simple Storage Service (S3)


 
 
 
AWS S2 is an infinitely scalable storage solution which provides high data availability which is achieved by storing data across multiple servers, at multiple locations.

Data objects are stored within S3 buckets which are like containers for grouping stored objects. We can create as many buckets as we need. Everything inside S3 bucket is an object: flat files (text, binary, images, videos etc...) and folders.

Max allowed file size is 5 TB.

How to create a S3 bucket


One way is to use AWS Management Console. We need to choose a name and region.

S3 bucket name must be unique because AWS creates DNS name for each new bucket. It comes in form:

https://<bucket_name>.<region>.amazonaws.com

DNS name is publicly accessible. Name also needs to be DNS-compliant: no upper cases and underscores, between 3 and 63 characters long, and should not end with a dash character. 

Objects are accessed via urls like:

https://<bucket_name>.<region>.amazonaws.com/my-folder/my-file.txt

Every object in S3 bucket has:
  • data
    • Key - actual name of the object e.g. my-file.txt
    • Value - actual data
  • metadata
    • Owner
    • Size
    • Last Modified
By default, upon creating a bucket and uploading the objects, no one can access them apart from bucket owner. Access is controlled via bucket policies and access control lists. Bucket policies are at bucket level and access control lists are at the level of individual objects. 

Just like IAM policies, bucket policies are JSON documents. With them we can grant access to users, groups, users from other AWS accounts or public access.

Example: bucket policy which allows user adam to retrieve all objects in a bucket

{
    "Version": "2022-05-13",
    "Statement": [
        {
            "Action": [
                "s3: GetObject"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::my-bucket/*",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::1234567890:user/adam"
                ]
            }
    ]
}


---

Thursday, 12 May 2022

Managing AWS IAM using Terraform

 

 
 
 

Creating a user 


 
From resource type aws_iam_user TF extracts the following information:
  • provider: aws
  • resource: iam_user
 
main.tf:
 
resource "aws_iam_user" "admin-user" {
    name = "Adam"
    tags = {
        Description = "Technical Team Lead"
    }
}
 
We can initialize the directory as provider (aws) is deducted from the resource type:
 
$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v4.13.0...
- Installed hashicorp/aws v4.13.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
 
If we have AWS CLI installed and we've already configured it by specifying access key ID, secret key and region (so they are store in /home/<user>/.aws/config/credentials and this is RECOMMENDED approach), we can execute terraform plan and apply:
 
$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_iam_user.admin-user will be created
  + resource "aws_iam_user" "admin-user" {
      + arn           = (known after apply)
      + force_destroy = false
      + id            = (known after apply)
      + name          = "Adam"
      + path          = "/"
      + tags          = {
          + "Description" = "Technical Team Lead"
        }
      + tags_all      = {
          + "Description" = "Technical Team Lead"
        }
      + unique_id     = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
 
 
$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_iam_user.admin-user will be created
  + resource "aws_iam_user" "admin-user" {
      + arn           = (known after apply)
      + force_destroy = false
      + id            = (known after apply)
      + name          = "Adam"
      + path          = "/"
      + tags          = {
          + "Description" = "Technical Team Lead"
        }
      + tags_all      = {
          + "Description" = "Technical Team Lead"
        }
      + unique_id     = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_iam_user.admin-user: Creating...
aws_iam_user.admin-user: Creation complete after 2s [id=Adam]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
 
 
If we did't have AWS CLI configured, we would have had 2 options: to provide AWS configuration using environment variables or by using aws provider block in TF configuration file.

Approach with environment variables:

$ export AWS_ACCESS_KEY_ID="anaccesskey"
$ export AWS_SECRET_ACCESS_KEY="asecretkey"
$ export AWS_REGION="us-west-2"
$ terraform plan

 
 
Approach with aws block (NOT recommended!):
 
It contains AWS CLI configuration and needs to be added to TF config file (see Docs overview | hashicorp/aws | Terraform Registry):
 
Inside main.tf:
 
provider "aws" {
  region     = "us-west-2"
  access_key = "my-access-key"
  secret_key = "my-secret-key"
}
 
...
 
Note that it is NOT recommended to have hardcoded credentials in any file that gets checked in into repository and this includes .tf files.


Removing the added user



$ terraform destroy
aws_iam_user.admin-user: Refreshing state... [id=Adam]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # aws_iam_user.admin-user will be destroyed
  - resource "aws_iam_user" "admin-user" {
      - arn           = "arn:aws:iam::036201377220:user/Adam" -> null
      - force_destroy = false -> null
      - id            = "Adam" -> null
      - name          = "Adam" -> null
      - path          = "/" -> null
      - tags          = {
          - "Description" = "Technical Team Lead"
        } -> null
      - tags_all      = {
          - "Description" = "Technical Team Lead"
        } -> null
      - unique_id     = "AIDAQQ3OFFXCFCKBEV5KP" -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

aws_iam_user.admin-user: Destroying... [id=Adam]
aws_iam_user.admin-user: Destruction complete after 2s

Destroy complete! Resources: 1 destroyed.


When user is added to AWS, they have no permissions attached. To add permissions to user, we need to attach IAM policies to them. 

Let's see how to attach a policy to a user we named Adam, via Terraform. To get an idea of some AWS-managed policies, see here: AWS managed policies for job functions - AWS Identity and Access Management.

For example, we want to attach AdministratorAccess policy. We'll need first to get policy definition in JSON format. If we log in to AWS Management Console and go to policies we can copy its JSON definition. TF resource type that we need to use is aws_iam_policy:

resource "aws_iam_policy" "admin-user-policy" {
  name        = "admin-user-policy"
  path        = "/"
  description = "Admin user"

  # Terraform's "jsonencode" function converts a
  # Terraform expression result to valid JSON syntax.
  policy = jsonencode({
    Version = "2022-05-13"
    Statement = [
      {
        Action = "*"
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  })
}


Another way of embedding a JSON into .tf file is by using Here document (heredoc) which redirects a multiline string literal to the preceding command while preserving line breaks. Unix syntax for it is:

[command] <<DELIMITER
    First line.
    Second line.
    Third line.
    Fourth line.
DELIMITER

In our case:

resource "aws_iam_policy" "admin-user-policy" {
  ...
  policy = <<EOF
  {
    Version = "2022-05-13"
    Statement = [
      {
        Action = "*"
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  }
  EOF
}


Probably more readable and convenient way to embed JSON is .tf file is to keep JSON string in a separate .json file (e.g. admin-policy.json in our case) which is located in the configuration directory and then simply use function file() to read the file content:

resource "aws_iam_policy" "admin-user-policy" {
  ...
  policy =  file("admin-policy.json")
}

Now we've provisioned a managed IAM policy, we need to attach it to the provisioned user. We need to use either aws_iam_policy_attachment resource which exclusively attaches managed IAM policy to users, groups or roles or aws_iam_user_policy_attachment which attaches a managed IAM Policy to a user only. 

NOTE: exclusive attachment means that the referenced policy can only be attached to a single role across your entire AWS account.

resource "aws_iam_user_policy_attachment" "adam-admin-user" {
    user = aws_iam_user.admin-user
    policy_arn = aws_iam_policy.admin-user-policy.arn
}

So, main.tf will look like this:

resource "aws_iam_user" "admin-user" {
    ...
}

resource "aws_iam_policy" "admin-user-policy" {
    ...
}

resource "aws_iam_user_policy_attachment" "adam-admin-user" {
    user = aws_iam_user.admin-user
    policy_arn = aws_iam_policy.admin-user-policy.arn
}

We can now execute terraform plan and apply


---

Wednesday, 11 May 2022

AWS Identity and Access Management (IAM)


AWS users:
  • Root user - account owner that performs tasks requiring unrestricted access
    • created when you sign up for AWS for the first time
    • uses e-mail address and password
    • has complete admin privileges
    • can be used to manage any service within AWS but this is not recommended; this user is like root user on Unix systems or admin on Windows and should be used only for special tasks
    • used to log in to AWS Management console where it can create other users (IAM users); this is actually recommended use of root user
  • IAM user - user within an account that performs daily tasks 
    • 2 types of access can be configured for it:
      • Access to the AWS Management Console
        • requires username and password
      • Programmatic access used to interact programmatically in Terminal on Unix and PowerShell in Windows
        • requires Access Key ID and Secret Access Key
        • can't be used to log in to AWS Management Console

When user is created, AWS assigns to it the least privilege permissions. Permissions are assigned to users and user groups. Permissions define what user can and can't do. 

Permissions are defined within IAM policies. Permissions are not attached to users directly but via policies. Policies get attached to users and user groups. 

A policy defines the AWS permissions that can be assigned to a user, group or role.

IAM policies can be:
  • AWS-managed
  • custom (customer-managed)
Policy Example: AdministratorAccess policy allows admin access to all resources and services. It is managed by AWS.

IAM policies are defined in JSON format: IAM JSON policy elements reference - AWS Identity and Access Management. IAM policies can be created and managed in visual editor and using JSON.

AdministratorAccess policy:

{
    "Version": "2022-05-11",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

Asterisk (*) means "all". Allow all actions on all resources.

There are many AWS-managed policies. Each of them is meant to be attached to a user with specific job role. Examples:
  • AdministratorAccess - administrator
  • Billing - view billing information, setup and manage payments
  • DatabaseAdministrator - DB admin
  • NetworkAdministrator - network admin
  • ViewOnlyAccess - view-only user


If multiple users need to have same permissions, they need to have attached multiple policies e.g. AmazonEC2FullAccess and AmazonS3FullAccess. Instead of attaching policies to each user, we can create IAM User Group, make these users its members and then attach these policies to this group. User groups allow managing policies of multiple users at once. Using groups is a best-practice way to manage users' permissions by job functions, AWS service access or your custom permissions. We can still attach policies to individual users.

How to manage permissions of services? E.g. what if EC2 instance needs to access S3 bucket. By default, just like users, resources don't have permissions to access other resources. Unlike users, we can't attach IAM policies to resources. We need to create IAM roles. IAM roles define access permissions for a resource. We need to create e.g. S3Access role and attach to it IAM policy AmazonS3FullAccess that we used for the user group. Then we attach this role to EC2 instance of interest.

IAM roles are a secure way to grant permissions to entities that we trust:
  • IAM users in another account
  • Application code running on an EC2 instance that needs to perform actions on AWS resources
  • An AWS service that needs to act on resources in your account to provide its features
  • Users from a corporate directory who use identity federation with SAML

IAM roles are used to provide access:
  • from one AWS service to another
  • to IAM user belonging to another AWS account
  • to applications to interact with services in AWS
  • to users managed outside AWS e.g. by Active Directory
Example of custom-made policy: the one which allows user to create and delete tags on EC2 instance:

CreateEC2TagsPolicy:

{
    "Version": "2022-05-11",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DeleteTags",
                "ec2:CreateTags"
            ],
            "Resource": "*"
        }
    ]
}


Creating a User and User Group in AWS Management Console


Log in to AWS Management Console with your root account.



Security, Identity and Compliance >> IAM
IAM service does not depend on the region so region is set to Global. Users, groups and roles are available in all regions.

Account Management >> Users >> Add User
We need to provide: 
  • User name
  • Desired Access type (both can be selected):
    • Programmatic access - enables access key ID and secret access key for the AWS API, CLI, SDK and other development tools
    • AWS Management Console access - enables a password that allows users to sign-in to the AWS Management Console
      • Require password reset - enable so user must create a new password at next sign-in. Users automatically get the IAMUserChangePassword policy to allow them to change their own password.
Next >> Permissions

Set permissions has 3 tabs:
  • Add user to group
  • Copy permissions from existing user
  • Attach existing policies directly
Next >> Tags

IAM Tags are key-value pairs you can add to your user. Tags can include user information such as an e-mail address, or can be descriptive, such as a job title. 

Next >> Review >> Create User

Add user - Success page: this is the only place and time when we can see/access/download the secret access key. We can download it within .csv file. 

Account Management >> Users: select user => Summary page
Summary page tabs:
  • Permissions
    • Add permissions (button); this can be done in 3 ways:
      • Add user to group
        • Create Group (button); shows list of all policies; we need to check those that we want to be attached to this group and then press button "Create group"
      • Copy permissions from existing user
      • Attach existing policies directly
        • check the desired policy 
    • Policies list:
      • for each policy we can see policy summary and JSON
  • Groups
  • Tags
  • Security Credentials
  • Access Advisor

Access Management >> Groups, select group => Summary page

Summary page has 3 tabs:
  • Users
  • Permissions
    • Attach Policy (button)
  • Access Advisor

Creating a Policy in AWS Management Console


Access Management >> Policies
Page shows:
  • Create Policy button - for creating a custom policy
    • Create Policy page has 2 tabs:
      • Visual Editor
        • We need to choose the service that the policy will be applicable for e.g. EC2
        • We then need to choose actions allowed in chosen service
          • Access level:
            • List
            • Read
            • Tagging
            • Write
            • Permissions management
        • We then need to choose resources this policy can be applied on
      • JSON
    • Review: 
      • set the name of the policy
      • set the description of the policy
  • list of all existing policies

Creating a Role in AWS Management Console


Access Management >> Roles; main page contains:
  • Create role (button)
    • Select type of trusted entity:
      • AWS Service (EC2, Lambda, ...) - allows AWS service to perform actions on our behalf e.g. EC2 or Lambda to call AWS services on our behalf
      • Another AWS account
      • Web identity (Cognito or OpenID provider)
      • SAML federation (corporate directory)
    • Add tags
    • Review
  • list of all roles

Programmatic Access


AWS CLI is an open source tool that allows interacting with AWS services using command line tools like Unix shell or Terminal and PowerShell in Windows. 

Installing AWS CLI on Mac



To verify installation:

% which aws
/usr/local/bin/aws

% aws --version
aws-cli/2.6.3 Python/3.9.11 Darwin/20.5.0 exe/x86_64 prompt/off


Configuring AWS CLI


% aws configure
AWS Access Key ID [None]: 
AWS Secret Access Key [None]: 
Default region name [None]: 
Default output format [None]: 

AWS Access Key ID and Secret Key are those that are downloaded when user was created. ID can be visible after that but secret key can be found only in downloaded file, it can be unveiled in AWS Management Console. 

For default region name we might want to put geographically closest region to our location. 

Default output format can be yaml, JSON, text or table.
 
$ ls  /home/bojan/.aws
config  credentials
 
$ cat  /home/bojan/.aws/config
[default]
region = eu-west-1
output = json

$ cat  /home/bojan/.aws/credentials
[default]
aws_access_key_id = ABC...DEF
aws_secret_access_key = DFCdfsg...sdceD

 
To check what region has been set:
 
$ aws configure get region 
eu-west-1
 
To check what Access Key ID and Secret Key have been set:
 
$ aws configure get aws_access_key_id

$ aws configure get aws_secret_access_key
 
 

Using AWS CLI


AWS CLI command syntax:

$ aws
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help



<command> is usually a service we want to interact with e.g. iam
<subcommand> specifies which operations to perform e.g. create-user
 

Example: creating a user


$ aws iam create-user --user-name test-user-1
{
    "User": {
        "Path": "/",
        "UserName": "test-user-1",
        "UserId": "AIDBBQ3OFFXCBT3AQWDK7",
        "Arn": "arn:aws:iam::136201378220:user/test-user-1",
        "CreateDate": "2022-05-12T11:55:25Z"
    }
}

Arn = Amazon Resource Name, a unique name assign to every resource in AWS

To see help for each command or subcommand:

$ aws iam help
$ aws iam create-user help

Example: list users

 
$ aws iam list-users
{
    "Users": [
          ...
         {
             "Path": "/",
             "UserName": "test-user-1",
             "UserId": "AIDBBQ3OFFXCBT3AQWDK7",
             "Arn": "arn:aws:iam::136201378220:user/test-user-1",
             "CreateDate": "2022-05-12T11:55:25Z"
        },
         ...
    ]
}
 

 
If using LocalStack:
 
$ aws --endpoint http://aws:4566 iam list-users
 

Example: deleting a user:

 
$ aws iam delete-user --user-name test-user-1

 

Example: attaching a policy to a user:

 
$ aws --endpoint http://aws:4566 iam attach-user-policy --user-name amelia --policy-arn arn:aws:iam::aws:policy/AdministratorAccess
 
 

Example: creating a group:

 
$ aws --endpoint http://aws:4566 iam create-group --group-name project-calabria-developers               {
    "Group": {
        "Path": "/",
        "GroupName": "project-calabria-developers",
        "GroupId": "c2tuoijto26m2cvuensk",
        "Arn": "arn:aws:iam::000000000000:group/project-calabria-developers",
        "CreateDate": "2022-05-12T13:53:47.627000+00:00"
    }
}

Example: adding a user to a group:


$ aws --endpoint http://aws:4566 iam add-user-to-group --group-name project-calabria-developers --user-name meridith
 
 
 

Example: check policies attached to the group: 

 
$ aws --endpoint http://aws:4566 iam list-attached-group-policies --group-name project-calabria-developers 
 
 

Example: check policies attached to user:

 
$ aws --endpoint http://aws:4566 iam list-attached-user-policies --user-name meridith


Example: attach a policy to a group

 
$ aws --endpoint http://aws:4566 iam attach-group-policy --group-name project-calabria-developers --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess 
 

---




Monday, 9 May 2022

Terraform Meta-Arguments

 

Terraform meta-arguments are special syntax constructs which help in specifying how we want resources to be managed.

There are 5 meta-arguments in Terraform:

  •     depends_on
  •     lifecycle
  •     count
  •     for_each
  •     provider

I've already covered depends_on and lifecycle in my previous articles Terraform Resource Dependencies | My Public Notepad and Terraform State | My Public Notepad.


 count

 

It specifies the number of infrastructure instances to be created. 


resource "local_file" "foo" {
    filename = "${path.cwd}/temp/foo.txt"
    content = "This is a text content of the foo file!"
    count = 3
}

Let's create multiple instances of local_file:

$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  + create

Terraform will perform the following actions:

  # local_file.foo[0] will be created
  + resource "local_file" "foo" {
      + content              = "This is a text content of the foo file!"
      + directory_permission = "0777"
      + file_permission      = "0777"
      + filename             = "/home/bojan/dev/github/demo-terraform/local_resources_demo/temp/foo.txt"
      + id                   = (known after apply)
    }

  # local_file.foo[1] will be created
  + resource "local_file" "foo" {
      + content              = "This is a text content of the foo file!"
      + directory_permission = "0777"
      + file_permission      = "0777"
      + filename             = "/home/bojan/dev/github/demo-terraform/local_resources_demo/temp/foo.txt"
      + id                   = (known after apply)
    }

  # local_file.foo[2] will be created
  + resource "local_file" "foo" {
      + content              = "This is a text content of the foo file!"
      + directory_permission = "0777"
      + file_permission      = "0777"
      + filename             = "/home/bojan/dev/github/demo-terraform/local_resources_demo/temp/foo.txt"
      + id                   = (known after apply)
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

local_file.foo[0]: Creating...
local_file.foo[2]: Creating...
local_file.foo[1]: Creating...
local_file.foo[1]: Creation complete after 0s [id=db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e]
local_file.foo[2]: Creation complete after 0s [id=db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e]
local_file.foo[0]: Creation complete after 0s [id=db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

 

Only one file is actually created (3 times) though as filename is the same:

$ ls ./temp/
foo.txt

To prevent this, we can interpolate count.index:

filename = "${path.cwd}/temp/foo${count.index}.txt"

After terraform apply we now have 3 files:
 
$ ls ./temp/
foo0.txt  foo1.txt  foo2.txt

 
If names are defined in a variable of a tuple type, we can access them by using indexing.
 
If variables.tf contains:
 
# var.filename is tuple with 3 elements
variable "filename" {
  default = [
    "foo.txt",
    "foo2.txt",
    "foo3.txt",
  ]
}
 
...then we can refer to each element via index which will actually be the current value of count's index attribute:
 
filename = "${path.cwd}/temp/${var.filename[count.index]}" 

count is set above to the static value but it can automatically pick up the length of the list/tuple:

count = length(var.filename)

 
A drawback of using count for iterating a list is that TF creates resources as lists.

If we delete element at index 0 from the var.filename and apply terraform apply, TF will re-create foo2 and foo3 and destroy foo. But ideally, only foo destruction should be performed. This is because TF uses only list index to resources, not vales of the elements (file names in this case).


for_each


If we use for_each for creating multiple resources, they will be created as elements of map, not a list which is the case when count is used.
 
variables.tf:
 
variable "filename" {
  # Use this in order to fix Error: Invalid for_each argument
  # type = set(string)

  type = list(string)

  default = [
    "foo.txt",
    "foo2.txt",
    "foo3.txt",
  ]
}

 
main.tf:
 
resource "local_file" "foo" {
  # use this if var.filename has type set
  # for_each = var.filename

  # toset() converts list into set. Use it if var.filename is a list.
  for_each = toset(var.filename)

  filename = "${path.cwd}/temp/${each.value}"
  content = "This is a text content of the foo file!"
}
 
We can see that resource is created as a map where keys are values based on the each.value current value during the iteration and values are resource instances:
 
$ terraform apply
...
local_file.foo["foo2.txt"]: Creating...
local_file.foo["foo3.txt"]: Creating...
local_file.foo["foo.txt"]: Creating...
...

$ terraform show
# local_file.foo["foo.txt"]:
resource "local_file" "foo" {
    content              = "This is a text content of the foo file!"
    directory_permission = "0777"
    file_permission      = "0777"
    filename             = "/home/bojan/dev/github/demo-terraform/meta_args_demo/for_each/temp/foo.txt"
    id                   = "db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e"
}

# local_file.foo["foo2.txt"]:
resource "local_file" "foo" {
    content              = "This is a text content of the foo file!"
    directory_permission = "0777"
    file_permission      = "0777"
    filename             = "/home/bojan/dev/github/demo-terraform/meta_args_demo/for_each/temp/foo2.txt"
    id                   = "db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e"
}

# local_file.foo["foo3.txt"]:
resource "local_file" "foo" {
    content              = "This is a text content of the foo file!"
    directory_permission = "0777"
    file_permission      = "0777"
    filename             = "/home/bojan/dev/github/demo-terraform/meta_args_demo/for_each/temp/foo3.txt"
    id                   = "db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e"
}


Outputs:

foo_files = (sensitive value)
 
$ terraform output foo_files
{
  "foo.txt" = {
    "content" = "This is a text content of the foo file!"
    "content_base64" = tostring(null)
    "directory_permission" = "0777"
    "file_permission" = "0777"
    "filename" = "/home/bojan/dev/github/demo-terraform/meta_args_demo/for_each/temp/foo.txt"
    "id" = "db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e"
    "sensitive_content" = tostring(null)
    "source" = tostring(null)
  }
  "foo2.txt" = {
    "content" = "This is a text content of the foo file!"
    "content_base64" = tostring(null)
    "directory_permission" = "0777"
    "file_permission" = "0777"
    "filename" = "/home/bojan/dev/github/demo-terraform/meta_args_demo/for_each/temp/foo2.txt"
    "id" = "db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e"
    "sensitive_content" = tostring(null)
    "source" = tostring(null)
  }
  "foo3.txt" = {
    "content" = "This is a text content of the foo file!"
    "content_base64" = tostring(null)
    "directory_permission" = "0777"
    "file_permission" = "0777"
    "filename" = "/home/bojan/dev/github/demo-terraform/meta_args_demo/for_each/temp/foo3.txt"
    "id" = "db5ca40b5588d44e9ec6c1b4005e11a6fd0c910e"
    "sensitive_content" = tostring(null)
    "source" = tostring(null)
  }
}

 
 
If we now remove an item from var.filename and execute terraform apply, TF will only destroy a single resource, the one based on that value and it will not re-create other resources.
 
---

 

Resources: 
 

Thursday, 5 May 2022

Terraform Providers

 



Terraform supports hundreds of providers via their plugins (terms plugins and providers are often used interchangeably). They are distributed by Hashicorp's Terraform registry. There are 3 tiers of providers:
  • official: owned and maintained by Hashicorp. 
    • aws
    • gcp
    • azure
    • local
  • verified: owned and maintained by 3rd party company that is Hashicorp's partner
    • bigip by F5Networks
    • heroku by Heroku
    • digitalocean by DigitalOcean
    • linode by Linode
  • community: published and maintained by individual contributors
 
linode provider is maintained by Linode and is thus a verified provider



Here are some useful commands related to providers:

terraform providers - prints all providers used in the configuration directory

Example:

$ terraform providers

Providers required by configuration:
.
├── provider[registry.terraform.io/hashicorp/local]
└── provider[registry.terraform.io/hashicorp/aws]

terraform providers mirror /path/to/destination/dir - to copy provider plugins needed for the current configuration directory to another directory
 
 
Version Constraints
 
By default TF will download the latest version of plugins required by resources specified in the configuration file (.tf file). To use some particular version of the plugin, we need to use a terraform block which is used to configure settings related to Terraform itself. Inside it, we need to use another block, called terraform_providers inside which we can list all providers and their settings:


terraform {
  required_providers {
    local = {
      source = "hashicorp/local"
      version = "2.2.2"
    }
  }
}


We can get this snippet for a given provider if we go to https://registry.terraform.io/providers, select desired provider and click on USE PROVIDER button in the upper right corner:

 

Version constraint can be specified in multiple ways:
 
version = "!= 2.0.0" - get the available version before 2.0.0 but not 2.0.0
version = "< 2.0.0"
version = "> 2.0.0"
version = "> 1.5.0, < 2.0.0, != 1.7.0" 
 
Pessimistic constraint operator:
 
version = "~> 1.5" - get the version 1.5, 1.6 ...up to 1.9
version = "~> 1.5.0" - get the version 1.5.0, 1.5.1...up to 1.5.9

---

Terraform Resource Dependencies

 


Terraform Resource Dependency types:

  • implicit
  • explicit


Resource Attribute Reference


Multiple resources are usually dependent on each other. E.g. output of one resource is used as and input for another. local_file resource might be using the output (id) of the random_pet. We can do that by using the interpolation sequence (${...}):

${resource_type.resource_name.attribute}

Example:

resource "random_pet" "my_random_text" {
    length = 2
    prefix = "my_rnd_txt_"
    separator = "."
}

resource "local_file" "foo" {
    filename = "${path.cwd}/temp/foo.txt"
    content = "Let's use the random_pet output which is id with value ${random_pet.my_random_text.id}"
}

Note that the type of the value in the output and input match in this case (string).
Interpolation takes place during the terraform apply command execution.

resource "time_static" "time_update" {
}

Resource block is empty because time_static does not need any arguments to be supplied to work.
When applied as it is, terraform creates a logical resource locally (similar to random_pet) with the current time. 

terraform apply output:

Initializing provider plugins...
- Finding latest version of hashicorp/time...
- Installing hashicorp/time v0.7.2...

terraform plan output (snippet):

Terraform will perform the following actions:

  # time_static.time_update will be created
  + resource "time_static" "time_update" {
      + day     = (known after apply)
      + hour    = (known after apply)
      + id      = (known after apply)
      + minute  = (known after apply)
      + month   = (known after apply)
      + rfc3339 = (known after apply)
      + second  = (known after apply)
      + unix    = (known after apply)
      + year    = (known after apply)
    }

attribute id is exported by the time_static resource.

To refer to this attribute we'd use ${time_static.time_update.id} which is the timestamp in form e.g. 2022-04-28T06:13:18Z.

To see resource attribute values that are known after apply, first execute terraform apply and then terraform show.

If value we want to specify is only a reference to the attribute of another resource, we don't need to use interpolation expression but just name of that attribute:

content = random_pet.my_random_text.id


If we use interpolation:

content = "${random_pet.my_random_text.id}"

...then terraform plan will issue the following warning:

Warning: Interpolation-only expressions are deprecated

  on key.tf line 8, in resource "local_file" "key_details":
   8:     content="${tls_private_key.pvtkey.private_key_pem}"

Terraform 0.11 and earlier required all non-constant expressions to be
provided via interpolation syntax, but this pattern is now deprecated. To
silence this warning, remove the "${ sequence from the start and the }"
sequence from the end of this expression, leaving just the inner expression.

Template interpolation syntax is still used to construct strings from
expressions when the template includes multiple interpolation sequences or a mixture of literal strings and interpolations. This deprecation applies only
to templates that consist entirely of a single interpolation sequence.



Implicit vs Explicit Resource Dependencies


When Terraform creates resources from the example above, it knows about the dependency so it will first create dependee (random_pet) and then dependent (local_file) and in case of destruction, the order will be first dependent and then dependee. This is a form of implicit dependency as we don't explicitly specify which resources is dependent on which but Terraform figures this out by looking reference expressions used. 
 
If we want to make sure one resource is created after another, we can specify this explicit dependency via depends_on meta-argument: 

resource "random_pet" "my_random_text" {
    length = 2
    prefix = "my_rnd_txt_"
    separator = "."
}

resource "local_file" "foo" {
    filename = "${path.cwd}/temp/foo.txt"
    content = "This resource will be created after the random_pet" 
    depends_on = [
        random_pet.my_random_text
    ]
}

In this case we don't use resource attribute reference. Resource R1 is explicitly dependent on Resource R2 it it relies on Resource R2 but doesn't access any of its attributes in its own arguments.

terraform graph command creates a visual presentation of dependencies in TF configuration or execution plan. It can be run as soon as there is a configuration file and configuration directory has been initialized with terraform init. If the configuration directory is not initialized, we'll get the following error:

$ terraform graph                     
Error: Inconsistent dependency lock file
│ 
│ The following dependency selections recorded in the lock file are inconsistent with the current configuration:
│   - provider registry.terraform.io/hashicorp/local: required by this configuration but no version is selected
│   - provider registry.terraform.io/hashicorp/random: required by this configuration but no version is selected
│ 
│ To make the initial dependency selections that will initialize the dependency lock file, run:
│   terraform init

So we first need to initialize the directory:


$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/local...
- Finding latest version of hashicorp/random...
- Installing hashicorp/local v2.2.2...
- Installed hashicorp/local v2.2.2 (signed by HashiCorp)
- Installing hashicorp/random v3.1.3...
- Installed hashicorp/random v3.1.3 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ terraform graph
digraph {
        compound = "true"
        newrank = "true"
        subgraph "root" {
                "[root] local_file.foo (expand)" [label = "local_file.foo", shape = "box"]
                "[root] provider[\"registry.terraform.io/hashicorp/local\"]" [label = "provider[\"registry.terraform.io/hashicorp/local\"]", shape = "diamond"]
                "[root] provider[\"registry.terraform.io/hashicorp/random\"]" [label = "provider[\"registry.terraform.io/hashicorp/random\"]", shape = "diamond"]
                "[root] random_pet.my_random_text (expand)" [label = "random_pet.my_random_text", shape = "box"]
                "[root] local_file.foo (expand)" -> "[root] provider[\"registry.terraform.io/hashicorp/local\"]"
                "[root] local_file.foo (expand)" -> "[root] random_pet.my_random_text (expand)"
                "[root] provider[\"registry.terraform.io/hashicorp/local\"] (close)" -> "[root] local_file.foo (expand)"
                "[root] provider[\"registry.terraform.io/hashicorp/random\"] (close)" -> "[root] random_pet.my_random_text (expand)"
                "[root] random_pet.my_random_text (expand)" -> "[root] provider[\"registry.terraform.io/hashicorp/random\"]"
                "[root] root" -> "[root] provider[\"registry.terraform.io/hashicorp/local\"] (close)"
                "[root] root" -> "[root] provider[\"registry.terraform.io/hashicorp/random\"] (close)"
        }
}


The output is a graph in DOT format (which is a text and hard to follow). Using graph visualization software can help in making a graph image. We can use e.g. Graphviz

To install Graphviz on Linux:

$ apt update && apt install graphviz -y

To install Graphviz on Mac:

% brew install graphviz

Command to create a graph image is same on Linux and Mac:

% terraform graph | dot -Tsvg > graph.svg


graph.svg



---