AWS EFS with Terraform

The minimal aws_efs_file_system | Resources | hashicorp/aws | Terraform Registry resource example is:

resource "aws_efs_file_system" "my-app-data-efs" {
  tags = {
    Name = "my-app-data-efs"
  }
}

In AWS Console, we can go to Amazon EFS >> File systems and verify that it's created. Its attributes are:

Name: my-app-data-efs

File system ID: fs-1d130ce4a92769f59

Encrypted: Unencrypted

Total size: 6.00 KiB

Size in Standard / One Zone: 6.00 KiB    

Size in Standard-IA / One Zone-IA: 0 Bytes

Provisioned Throughput (MiB/s):    -     

File system state: Available

Creation time: Thu, 09 Mar 2023 10:41:55 GMT

Availability Zone: Standard

 

Performance mode: General Purpose

Throughput mode: Bursting
Lifecycle management:

Transition into IA: None

Transition out of IA: None

Availability zone: Standard

Automatic backups: Disabled
Encrypted: No
File system state: Available
DNS name: No mount targets available

 

It will have no Access points and no Mount targets defined:

 

To provide mount target, we need to use aws_efs_mount_target | Resources | hashicorp/aws | Terraform Registry. Required attributes are EFS (for which we want to create mount target) and subnet (in which we want this mount target to be):

resource "aws_efs_mount_target" "my-app-data-efs-mt" {
  file_system_id = aws_efs_file_system.my-app-data-efs.id
  subnet_id = "subnet-14321c874d6d35c6a"
}

terraform plan output:

Terraform will perform the following actions:

  # aws_efs_mount_target.my-app-data-efs-mt will be created
  + resource "aws_efs_mount_target" "my-app-data-efs-mt" {
      + availability_zone_id   = (known after apply)
      + availability_zone_name = (known after apply)
      + dns_name               = (known after apply)
      + file_system_arn        = (known after apply)
      + file_system_id         = "fs-1d130ce4a92769f59"
      + id                     = (known after apply)
      + ip_address             = (known after apply)
      + mount_target_dns_name  = (known after apply)
      + network_interface_id   = (known after apply)
      + owner_id               = (known after apply)
      + security_groups        = (known after apply)
      + subnet_id              = "subnet-14321c874d6d35c6a"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

After applying this change, we can check again Network settings for EFS where we'll see that mount target is now available:

 

 

The next step will be mounting EFS onto EC2 instance.

 

Resources:

 

aws_efs_file_system | Resources | hashicorp/aws | Terraform Registry  

aws_efs_mount_target | Resources | hashicorp/aws | Terraform Registry

File System Performance Metrics

image source: https://www.dnsstuff.com/latency-throughput-bandwidth

 

 

File system performance is measured by:

• Latency
• delay between request and response
• a measure of the length of time it takes for a single I/O request to be completed from the application's point of view
  
• measured separately for read (usually in microseconds) and write (usually in milliseconds) operations
• If the I/O is a data read, latency is the time it takes for the data to come back. If the I/O is a write, latency is the time for the write acknowledgement to return.
• affects application's acceleration
  
• Throughput / Bandwidth
  
• measures how many units of information a system can process in a period of time
• describes the amount of data able to flow through a point in the data path over a given time
• throughput and latency are often competing goals - the lower the latency, the higher the throughput
  
• measured separately for file system read (usually in GiBps) and file system write (usually in MiBps) operations 
• typically the best storage metric when measuring data that needs to be streamed rapidly, such as images and video files.
  
• Input/Output operations per second (IOPS)
• number of I/O operations per second
• measured separately for read and write operations
  
• as the number of IOPS requested from the device increases the latency will increase
• affects application's scalability
  

Amazon Elastic File System (EFS)

 

 

 

From How to Mount an NFS Share in Linux | Linuxize:

Network File System (NFS) is a distributed file system protocol that allows you to share remote directories over a network. With NFS, you can mount remote directories on your system and work with the remote files as if they were local files.

On Linux and UNIX operating systems, you can use the mount command (see 

Ubuntu Manpage: mount - mount a filesystem) to mount a shared NFS directory on a particular mount point in the local directory tree.

How to Mount a File System on Linux | Linode Docs

 

Amazon Elastic File System (EFS) is:

• one such shared, remote, network directory that can be mounted to Unix/Linux OS
• cloud-native data store
• shared file storage - can be accessed by multiple computers at the same time
• can be made available to VPC
• EC2 instances can then securely mount EFS to store and access data
• applications running on multiple EC2 instances can access the EFS at the same time
  
• EFS can also be mounted on on-premises data center servers when connected to Amazon VPC with AWS Direct Connect or VPN making it easy to:
• migrate data to EFS
• enable cloud bursting
• back up on-premises data to EFS
• supports low latency applications and also highly-parallelized scale out jobs requiring high throughput (read here what's the difference between latency and throughput: File System Performance Metrics | My Public Notepad)
  
• high throughput
• throughput for a file system scales automatically as capacity grows
• for workloads with high throughput and low capacity requirements, throughput can be provisioned independent of capacity 
• there are 2 storage classes: 
• Standard
• EFS IA (Infrequent Access) - for less frequently accessed data we can configure EFS to store data in a cost-optimized IA storage class
• LifeCycle Management automatically and transparently moves files access less frequently to EFS IA
  
• has 2 performance modes so we can tailor EFS to our application needs
• General Purpose
• Max I/O
  

 

Benefits of using EFS

• file storage system which is:
  
• simple - supports Network File System (NFS) versions 4.0 and 4.1 (NFSv4) protocol. This means that computers can access files on EFS by using standard file system tools and interfaces provided by OS. This is the reason why nfs is specified as the filesystem type supported by kernel when using mount command to mount EFS device on the EC2 instance (mount -t nfs ...).
  
• serverless - no need to provision infrastructure
  
• scalable performance - lifecycle management
  
• elastic - automatically grow or shrink as we add/remove files
• can grow to petabytes (PB)
• fully managed - no need to manage it
  
• easy to set up via AWS Management Console, API or CLI
• "set and forget"
• cost-effective data store: you pay for the storage you use
• access data securely, via existing AWS security infrastructure (IAM)
  

EFS symbol

Drawbacks of EFS

• supports Linux only (it doesn't support Windows)

 

When to use EFS?

• when thousands of EC2 instances from multiple availability zones or on-premises servers need concurrently to access data
• EFS provides concurrent access for tens of thousands of connections for EC2 instances, containers and lambda functions
  
• designed for high availability and durability, for storing data redundantly across multiple (3) availability zones
• ideal for machine learning, analytics, web serving, content management, media storage, DB backups
  

How to create EFS?

In AWS Console, go to EFS and click on Create file system.

 

 We can then set:

• Name of our file system
• VPC where we want EC2 instances to connect to our file system
• Storage class [EFS storage classes - Amazon Elastic File System]
• Standard (AWS used to name this Regional) - Stores data redundantly across multiple AZs (recommended)
• One Zone - Stores data redundantly within a single AZ
• we need to select desired availability zone
  

 

 

We can customize File system settings:

 

 

Note that by default Lifecycle management sets that files that haven't been access for 30 days will automatically be transferred from Standard to Standard-Infrequent Access storage (which is cheaper and so this is cost-effective measure).

We can then customize Network access:

Note that EFS is an entity connected to a network. EFS has assigned IP address in each availability zone. It is a mount target that provides an IP address for an NFSv4 endpoint at which we can mount an Amazon EFS file system.

 

So mount target provides a network interface (in the selected subnet in the AZ) for EFS mounted at it. 

 

When mount target state is available, our EFS system is mounted onto mount target and can be referred to via its url (or IP address). 

This does not mean it can still be accessible from EC2 instances. We need to mount EFS onto EC2. For that we need to specify a mount point (the local directory on the client where the EFS file system is mounted & accessible). This is one of settings that can be set when launching EC2 from AWS Console (the right-hand value in File system setting).

We can create a security group for EFS and use it everywhere - for each subnet/AZ. This security group can allow e.g. TCP traffic from anywhere.

Finally, we can customize File system policy:

 

Once EFS is created, it will take some more time for network interfaces to be created.

 

 

How to mount EFS on EC2 instance?

 

When creating EC2, we can select our EFS when setting File systems:

 

 

We also need to add EFS security group to the list of security groups used by this EC2 instance.

Once our EC2 instance is up and running, we can SSH to it and check mounted file systems with df tool:

 

$ df -T -h

 

-T - display file system types (Type column)

-h - display information about disk drives in human-readable format (kilobytes, megabytes, gigabytes and so on)

 

Resources:

Amazon EFS

 

Lecture 10 - File System Performance

Amazon Elastic File System Documentation
 

Amazon Elastic File System - Wikipedia

 

Amazon EFS performance - Amazon Elastic File System

 

Creating and managing mount targets and security groups - Amazon Elastic File System
 

EFS mount point vs access point | AWS re:Post

 

Mounting on Amazon EC2 Linux instances using the EFS mount helper - Amazon Elastic File System

 

AWS EFS Tutorial | Amazon Elastic File System | AWS Storage Services | AWS Training | Edureka - YouTube

Amazon EFS: How it works - Amazon Elastic File System

 

Walkthrough: Mount a File System from a Different VPC - Amazon Elastic File System

AWS EC2 Auto Scaling with Terraform

 

aws_autoscaling_group | Resources | hashicorp/aws | Terraform Registry

The minimum implementation that will pass terraform plan checks is:

resource "aws_autoscaling_group" "my_app" {
  min_size = 1
  max_size = 1
}

terraform plan output:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_autoscaling_group.my_app will be created
  + resource "aws_autoscaling_group" "my_app" {
      + arn                       = (known after apply)
      + availability_zones        = (known after apply)
      + default_cooldown          = (known after apply)
      + desired_capacity          = (known after apply)
      + force_delete              = false
      + force_delete_warm_pool    = false
      + health_check_grace_period = 300
      + health_check_type         = (known after apply)
      + id                        = (known after apply)
      + max_size                  = 1
      + metrics_granularity       = "1Minute"
      + min_size                  = 1
      + name                      = (known after apply)
      + protect_from_scale_in     = false
      + service_linked_role_arn   = (known after apply)
      + vpc_zone_identifier       = (known after apply)
      + wait_for_capacity_timeout = "10m"
    }

 

If we try to run terraform apply, we'll get the following error:

Error: One of `launch_configuration`, `launch_template`, or `mixed_instances_policy` must be set for an Auto Scaling Group 

 

Using Launch Configuration for defining EC2

Let's use a launch configuration (despite AWS discouraging the use of launch configurations in favour of launch templates; example with launch template is further down in this article).

We need to know the ID of the AMI we want to use. We'll choose the latest Amazon Linux 2 image, of t2.micro type which allows free tier.

If we select it, the next page will show its ID:

Terraform resource we'll use is aws_launch_configuration | Resources | hashicorp/aws | Terraform Registry.

 

# EC2 >> Launch configurations
resource "aws_launch_configuration" "my-app" {
  name          = "my-app"
  image_id      = "ami-006dcf34c09e50022"
  instance_type = "t2.micro"
}

We can now update our auto scaling group:

resource "aws_autoscaling_group" "my-app" {

  min_size = 1

  max_size = 1

  name = "my-app"

  launch_configuration = aws_launch_configuration.my-app.name

}

 

terraform apply still complains:

Error: Error creating Auto Scaling Group: ValidationError: At least one Availability Zone or VPC Subnet is required.
        status code: 400, request id: ad34ea76-a6d5-419a-bc48-0ffb15b4e76f

 

Let's define the subnet which we want our instances to be launched into:

resource "aws_autoscaling_group" "my-app" {
  min_size = 1
  max_size = 1
  name = "my-app"
  launch_configuration = aws_launch_configuration.my-app.name
  vpc_zone_identifier = [ "subnet-14321c874d6d35c6a" ]
}

terraform apply will now create the autoscaling group together with launch configuration. This can be verified by looking at EC2 >> Auto Scaling groups and EC2 >> Launch configurations. And most importantly, auto scaling group will launch the new EC2 instance, in subnet we denoted in the configuration. This instance can be found in EC2 >> Instances.

 

Using Launch Template for defining EC2

AWS discourages the use of launch configurations in favour of launch templates.

Terraform resource is aws_launch_template | Resources | hashicorp/aws | Terraform Registry. Its description says:

Provides an EC2 launch template resource. Can be used to create instances or auto scaling groups.

Here are the key differences between launch templates (LT) and launch configuration (LC):

• LT have more EC2 options than LC
  
• LT are getting latest features from Amazon EC2
• LC are still supported but are not getting the latest EC2 features
• LC is immutable (resource can't be edited; if we want to change it, we need to destroy it first and then re-create it)
• LT can be edited and updated
  
• LT can have multiple versions which allows creation of parameter subsets (With versioning, you can create a subset of the full set of parameters and then reuse it to create other templates or template versions. - Partial configuration for reuse and inheritance)
• LT allows using T2 unlimited burst credit option
• LT allows provisioning using both On-demand and Spot Instances.
• LT can be used to launch a standalone instance using AWS Console, SDK and CLI.
  

...

 

---