AWS NAT Gateway

 

What is NAT?

From AWS documentation:

A Network Address Translation (NAT) gateway is a device that forwards traffic from private subnets to other networks.

There are two types of NAT gateways:

• Public: Instances in private subnets can connect to the internet but cannot receive unsolicited inbound connections from the internet.
• Private: Instances in private subnets can connect to other VPCs or your on-premises network.

Each private or public NAT gateway must have a private IPv4 address assigned to it. Each public NAT gateway must also have an elastic IP (EIP) address (which is static public address associated with your AWS account) associated with it. Choosing a private IPv4 address is optional. If you don't choose a private IPv4 address, one will be automatically assigned to your NAT gateway at random from the subnet that your NAT gateway is in. You can configure a custom private IPv4 address in Additional settings.

After you create the NAT gateway, you must update the route table that’s associated with the subnet you chose for the NAT gateway. If you create a public NAT gateway, you must add a route to the route table that directs traffic destined for the internet to the NAT gateway. If you create a private NAT gateway, you must add a route to the route table that directs traffic destined for another VPC or your on-premises network to the NAT gateway.

 

When to use NAT?

From AWS documentation:

The instances in the public subnet can send outbound traffic directly to the internet, whereas the instances in the private subnet can't. Instead, the instances in the private subnet can access the internet by using a network address translation (NAT) gateway that resides in the public subnet. The database servers can connect to the internet for software updates using the NAT gateway, but the internet cannot establish connections to the database servers.

 

Note that NAT is required if instances in private subnet need to send a request (initiate a new connection) to the host in Internet. If request has reached private instance (via Application Load Balancer for example), then NAT is not required. See: amazon web services - Can a EC2 in the private subnet sends traffic to the internet through ELB without using NAT gateway/instance? - Server Fault

 

How to create NAT?

 

Private NAT gateway traffic can't reach the internet.

 

 

From AWS documentation about Additional settings:

 

When assigning private IPv4 addresses to a NAT gateway, choose how you want to assign them:

• Auto-assign: AWS automatically chooses a primary private IPv4 address and you choose if you want AWS to assign up to 7 secondary private IPv4 addresses to assign to the NAT gateway. AWS automatically chooses and assigns them for you at random from the subnet that your NAT gateway is in.
• Custom: Choose the primary private IPv4 address and up to 7 secondary private IPv4 addresses to assign to the NAT gateway.

You can assign up to 8 private IPv4 addresses to your private NAT gateway. The first IPv4 address that you assign will be the primary IPv4 address, and any additional addresses will be considered secondary IPv4 addresses. Choosing private IPv4 addresses is optional. If you don't choose a private IPv4 address, one will be automatically assigned to your NAT gateway. You can configure custom private IPv4 addresses in Additional settings.

Secondary IPv4 addresses are optional and should be assigned or allocated when your workloads that use a NAT gateway exceed 55,000 concurrent connections to a single destination (the same destination IP, destination port, and protocol). Secondary IPv4 addresses increase the number of available ports, and therefore they increase the limit on the number of concurrent connections that your workloads can establish using a NAT gateway.

You can use the NAT gateway CloudWatch metrics ErrorPortAllocation and PacketsDropCount to determine if your NAT gateway is generating port allocation errors or dropping packets. To resolve this issue, add secondary IPv4 addresses to your NAT gateway.You can assign up to 8 private IPv4 addresses to your private NAT gateway. The first IPv4 address that you assign will be the primary IPv4 address, and any additional addresses will be considered secondary IPv4 addresses. Choosing private IPv4 addresses is optional. If you don't choose a private IPv4 address, one will be automatically assigned to your NAT gateway. You can configure custom private IPv4 addresses in Additional settings.

Secondary IPv4 addresses are optional and should be assigned or allocated when your workloads that use a NAT gateway exceed 55,000 concurrent connections to a single destination (the same destination IP, destination port, and protocol). Secondary IPv4 addresses increase the number of available ports, and therefore they increase the limit on the number of concurrent connections that your workloads can establish using a NAT gateway.

 

Here are some typical architectures that include NAT:

 

Source: https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-scenarios.html

 

 

How to associate instances in private subnets with NATs?

 

The following diagrams show how routing tables are used to associate instances running in private subnets with NAT gateway created in public subnets thus allowing outbound traffic to Internet.

 

Source: https://www.packetswitch.co.uk/content/images/2020/06/Ghost-3-x-NAT-Gateway.png

 

 

Source: https://serverfault.com/questions/854475/aws-nat-gateway-in-public-subnet-why

 

Source: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Scenario2.html

 

 

References:

 

NAT gateways - Amazon Virtual Private Cloud

NAT gateway use cases - Amazon Virtual Private Cloud 

VPC with public and private subnets (NAT) - Amazon Virtual Private Cloud

Example routing options - Amazon Virtual Private Cloud

 
amazon web services - AWS: NAT Gateway in public subnet. Why? - Server Fault

DNS (Domain Name System)

 

 

 

 

 

DNS (Domain Name System)

• a protocol, part of the Internet Protocol (IP) Suite
  
• hierarchical and distributed naming system for computers, services, and other resources in the Internet
• naming database in which internet domain names are located and translated into Internet Protocol (IP) addresses 
• translates domain names to IP addresses so browsers can load Internet resources
• helps Internet users and network devices discover websites using human-readable host names, instead of numeric IP addresses; For humans, domain names are a lot easier to remember than a sequence of numbers.
  
• DNS configuration settings of some website are what allows visitors to still access that website even after it gets moved to a new hosting provider (its IP address will change but domain name will not)
  

 

For a hosted web site we need to specify (usually 2) DNS servers. These could be provided by hosting provider but we can specify custom ones e.g. Cloudflare DNS servers. These DNS servers will be nodes in DNS distributed database system which will be providing DNS records about our domains for whoever queries about them. Let's see which DNS records we can set.

Common DNS Records

• A (A record, Address record,  IPv4 address record)
• maps from an IPv4 address to a domain name
• used to point the domain name at one or multiple IP addresses
  
• also referred to as a host or hostname
• AAAA (IPv6 address record) maps domain name to IPv6 address
  
• CNAME (Canonical Name record)
  
• used to create an alias from one hostname to another
• maps one domain name (an alias) to another (the canonical name)
  
• Example: example.com has an A record which points to the IP address. If we say "www.example.com is a CNAME to example.com" and "ftp.example.com is a CNAME to example.com" that means that someone accessing www.example.com or ftp.example.com will be pointed to the same IP address that example.com points to. This is useful so that when your IP address changes, you only have to update example.com’s entry (DNS A record for example.com), and www.example.com and ftp.example.com automatically point to the right place.
• If you already have an A record, you will not use a CNAME
• CNAME record tells anyone visiting a subdomain to also use the same DNS records as another domain or subdomain. 
• This sort of thing is convenient when running multiple services from a single IP address (e.g. FTP server and web server share the same IP address but different port)
  
• CNAME records only work for subdomains and must always point to another domain or subdomain and never directly to an IP address.
• When a DNS resolver encounters a CNAME record while looking for a regular resource record, it will restart the query using the canonical name instead of the original name.
  
• MX (Mail eXchanger)
• allows you to control the delivery of mail for a given domain or subdomain. In our context, MX records can be set on a host-by-host basis to point to other hosts on the Internet (usually with permanent connections) that are set up to accept and/or route mail for your hostname(s). Setting a backup MX makes the entry you specify a secondary mail exchanger. This means that delivery will be attempted to your host first, and then to the backup host you specify if that fails.
• TXT (TXT records) 
• used to store information. Common uses include SPF, DKIM, etc.

 

It can take up to 72 hours for setting new DNS records to take effect - while change is replicated across all DNS servers on the internet. (see DNS Propagation)

 

The network of DNS servers is hierarchical. Types of DNS servers are:

• Recursive resolvers (DNS recursors)
• clients first send to them DNS queries
• they are assigned by ISP but can be set manually:
• Cloudflare 1.1.1.1
• Google (8.8.8.8 and 8.8.4.4)
  
• they respond either with cached data or send the request to root, TDL and finally to Authoritative nameserver from which they receive IP address
• every recursive resolver knows about 13 (types of) DNS root nameservers
  
• Root nameservers
• when receive query about some domain name e.g. example.com they return the address of the TLD nameserver which contains information about the domain extension e.g. .com 
• there are over 600 root nameservers which sync among themselves (anycast routing) and all contain the same data
  
• TLD (Top-Level Domain) nameservers
• they are domain extension-specific - each of them contains the list of authoritative servers for only a single domain e.g. .com or .ai. 
• they return the address of authoritative servers
• Larger TLDs and registrars (like GoDaddy, Namcheap etc...) use an API call to notify the TLD operator of any new registrations and changes
  
• Authoritative nameserver
• resolver’s last step in the journey for an IP address
• they are domain-specific - each of them contains the list of IP addresses for a particular doman e.g. ftp.example.com or www.example.com
• they return the IP address for a given hostname or, if domain has a CNAME (alias domain name), resolver needs to repeat the whole process in order to get the IP address for that alias host name.
• when you register your web site, name servers you set for it are authoritative nameservers
  

 

DNS server types | Cloudflare

To manage DNS records of domain e.g. example.com means setting DNS records for its root and subdomains. For each record we set:

• Type: A, CNAME, MX, ...
• Name: e.g. ftp (for ftp.example.com)
• Content: this is the value which depends on the type e.g. IPv4 address if A record, alias if CNAME etc....
• Proxy status: DNS only (proxy disabled) or Proxied (proxy enabled)
  
• TTL (Time to Live) - in minutes
  

Resources:

Domain Name System - Wikipedia

CNAME record - Wikipedia 

DNS server types | Cloudflare

DNS root server | Cloudflare

What is Anycast? | How does Anycast work? | Cloudflare 

How are DNS name-servers for a TLD updated when a new domain is registered? - Server Fault
DNS Lookup: How a Domain Name is Translated to an IP Address

DNS Propagation

AWS Security Groups

AWS Security Groups control the inbound and outbound traffic for various AWS resources: 

• EC2 instance
• running applications e.g. web server
• running as DNS server
  
• RDS - Database server
• EFS file system
• Elastic Load Balancer
• VPC peering rules
  

 

Security groups are VPC-specific (and therefore region-specific). They can only be used within the VPC they are created. The exception is where there is a peering connection to another VPC, in which case they can be referred to in the peered VPC. 

For Security Group we can set:

• Name
• Description
• VPC. VPC is region-specific so is security group.
  
• Inbound rules
• Outbound rules
  

For Security Group Rule (Inbound or Outbound) we can set:

• Type. The protocol to open to network traffic. You can choose a common protocol, such as SSH (for a Linux instance), RDP (for a Windows instance), and HTTP and HTTPS to allow Internet traffic to reach your instance. You can also manually enter a custom port or port ranges.
  
• Protocol. The type of protocol, for example TCP or UDP. Provides an additional selection for ICMP.
  
• Port range. For custom rules and protocols, you can manually enter a port number or a port range.
  
• Source. Determines the traffic that can reach your instance. Specify a single IP address, or an IP address range in CIDR notation (for example, 203.0.113.5/32). If connecting from behind a firewall, you'll need the IP address range used by the client computers. You can specify the name or ID of another security group in the same region. To specify a security group in another AWS account (EC2-Classic only), prefix it with the account ID and a forward slash, for example: 111122223333/OtherSecurityGroup.
  
• Description. A description for a security group rule.
  A description can be up to 255 characters in length.
  Allowed characters are a-z, A-Z, 0-9, spaces, and ._-:/()#,@[]+=;{}!$*.
  

Example: we have a Node.js application that is receiving traffic on port 8080, only from a Load Balancer that is on the same VPC. This means we need to create an Inbound rule:

• Type: Custom TCP
• Protocol: TCP
• Port range: 8080
• Source: Custom; CIDR block: 172.0.0.0/16 (in our example we're using a default VPC so we'll put here its private IP address block thus allowing only access from the private network)

 

Terraform Security Group resource

aws_security_group | Resources | hashicorp/aws | Terraform Registry

 

Terraform Security Group Rule resource

It represents a single ingress or egress group rule, which can be added to external Security Groups: 

aws_security_group_rule | Resources | hashicorp/aws | Terraform Registry

Required arguments: 

• from_port: start port
• to_port: end port
  
• protocol
• icmp
• icmpv6
• tcp
• udp
• all
  
• security_group_id: Security group to apply this rule to.
• type
• ingress (inbound)
• egress (outbound) 
  

Optional arguments:

• self. Whether the security group itself will be added as a source to this ingress rule.
  
• source_security_group_id.  Security group id to allow access to/from, depending on the type.
• ...
  

Because security group rule gets attached to the security group, we need to instruct Terraform to provision security group rule after the security group. We do this by using depends_on meta argument:

resource "aws_security_group_rule" "my_ec2_ssh" {
  type            = "ingress"
  from_port       = 22
  to_port         = 22
  protocol        = "tcp"
  cidr_blocks = var.ssh_ip_range
  security_group_id = aws_security_group.my_ec2_sg.id
  depends_on = [aws_security_group.my_ec2_sg]
}

Resources:

Security group rules for different use cases - Amazon Elastic Compute Cloud

AWS EC2: Auto Scaling

If our application is getting more traffic, we need to scale it: we need to create more virtual machines to run it so can handle the load, we need a load balancer to distribute the network traffic to these instances. Instead of doing this manually (creating new instances and e.g. using NGINX as load balancer) we can use AWS Auto-scaling Group and AWS Application Load Balancer.

• Automatically maintains application performance based on the user requirement at the lowest possible price
• Service which helps user to monitor applications and automatically adjusts capacity to maintain steady, predictable performance at the lowest possible cost
• Benefits:
• better fault tolerance
• better cost management
• reliability of your service
• scalability
• flexibility - changes can be made on the fly
• Snapshot vs AMI
• Snapshot
• used as a backup of a single EBS volume attached to the EC2 instance
• opt for it when the instance contains multiple static EBS volumes
• pay only for the storage of the modified data
• a non-bootable image on EBS volume
• AMI
• used as a backup of an EC2 instance
• widely used to replace a failed EC2 instance
• pay only for the storage that you use
• bootable image on EC2 instance
• creating an AMI image will also create EBS snapshots

How does AWS auto scaling work?

• Configure single unified scaling policy per application source
• explore the application
• choose the service you want to scale 
• select what to optimize e.g. cost or performance
• keep track of scaling

Different scaling plans

• scaling plan helps user to configure a set of instructions for scaling based on software requirement

Launch Template

Let's first explore a tool that can save time when scaling (creating multiple EC2 instances) manually.

How to set up a Launch Template?

EC2 >> Instances >> Launch Template:

We can set:

• Name
• Version description
• AMI. We can create AMI for each version of our application an name them e.g. amy-my-app-v1, ami-my-app-v2 etc...In the same way we can create new version of launch template and bind desired AMI (application) version to it.
  
• Instance type e.g. t2.micro 
• Key pair (for secure connection to the instance)
• Network settings
• Launch into:
• VPC
• Shared network
• Security group
  
• Storage (volumes e.g. Volume 1(AMI Root, 8GB, EBS, General Purpose SSD))
• Resource tags
• Network interfaces
• User data
  

User data example for setting an environment variable in our application's env config file:

#!/bin/bash -ex

sudo -u ec2-user bash -c "echo \"

MY_ENV_VAR="My env var value, set from the template" 

\" > /home/ec2-user/my-app.env

systemctl restart --now --no-block version-my-app.service

 

Shebang arguments explained:

-e Exit immediately if a command exits with a non-zero status.
-x Print commands and their arguments as they are executed.

 

Launch templates are versioned but we can't manually set the version number. AWS does it automatically by incrementing numbers from version 1.

Launch templates can be used outside Auto-scaling or Load Balancing: whenever we want to launch the instance, we don't need to manually fill details about the new instance, we can just use Launch instance from template.

In our scenario though, we want auto-scaling group to use launch template in order to launch EC2 instances. 

 

Auto scaling group

 

Auto scaling group manages how many EC2 instances will be running in parallel.

How to set up an Auto scaling group?

EC2 >> Auto scaling groups >> Create Auto scaling group

 

• Choose launch template or configuration
• Auto scaling group name
• Launch template. We can select it and also select:
  
• Launch template version (we can use this to select the version of our application that we want to be running on these new instances)
  
• Choose instance launch options
  
• Network
• VPC
• Availability Zones and subnets (in the form az_name|subnet_name); We want to list here all of them so we have a large pool of AZs in case something happens to EC2 instances running in some of them. 
• Instance type requirements - we can override launch template here
  
• Configure advanced options
• Load balancing (optional)
• No Load Balancer - traffic to auto scaling group will not be fronted by a load balancer
  
• Attach to an existing load balancer
• We'll specify here target groups actually (not load balancer directly)
  
• Attach to a new load balancer - quickly create a basic load balancer
  
• Health checks (optional)
• Health Check type
• EC2 - always enabled
• ELB - if load balancing is enabled
• Health check grace period - amount of time until EC2 auto scaling performs the first health check on new instances after they are put into service e.g. 300 seconds
  
• Additional settings (optional)
• Monitoring - enable group metrics collection withing CloudWatch
  
• Configure group size and scaling policies
• Group size
  
• Desired capacity. How many EC2 instances do we want running simultaneously? Must be between minimum and maximum. E.g. 2
  
• Minimum capacity e.g. 2
  
• Maximum capacity e.g. 8
  
• Scaling policies - choose whether to use a scaling policy to dynamically resize auto scaling group to meet changes in demand
  
• Target tracking scaling policy - choose a desired outcome and leave it to the scaling policy to add and remove capacity as needed to achieve that outcome
• Name - we can chose an arbitrary name
• Metric type e.g. Average CPU utilization (average over all EC2 instances running)
  
• Target value e.g. 50 (%)
  
• Instances need ____ seconds warm up before including in metric
• Disable scale in to create only a scale-out policy
  
• None
• Instance scale-in protection (optional); if enabled, newly created instances will be protected from scale-in by default
  
• Add notifications - send notifications to SNS topics whenever Amazon EC2 auto-scaling launches or terminates the EC2 instances in your auto scaling group
  
• Add tags
• Review

As soon as auto scaling group is created it becomes active. If we set to have 2 as a desired number of instances, auto scaling group will immediately create them. If we try to delete them (e.g. manually stop them), they will get to Terminated state and auto scaling group will immediately re-launch 2 new instances. If we want permanently to stop these instances we need first to delete auto scaling group and then delete those EC2 instances.

As seen above, Load balancing is optional so auto scaling (using auto scaling groups) can all happen with NO Load Balancer involved. But typically, we want to have load balancer so the load is equally distributed across all running EC2 instances. We can set up NGINX server as load balancer or use AWS (Application) Load Balancer in which case, in Load balancing options above, we'll choose to attach auto scaling group to an existing or newly created Load Balancer. It's worth mentioning that auto-scaling group is associated to load balancer indirectly, via target groups.

 

Resources:

App Scaling - AWS Application Auto Scaling - AWS

amazon web services - AWS EC2 Auto Scaling Groups: I get Min and Max, but what's Desired instances limit for? - Stack Overflow

Set capacity limits on your Auto Scaling group - Amazon EC2 Auto Scaling  

 amazon web services - When stopping EC2 instance, it starts again automatically listed separately, previous one changes to terminated - Server Fault

AWS EC2: Elastic Load Balancing

AWS Elastic Load Balancing:

• has the task of distributing traffic throughout the cluster of servers to ensure higher responsiveness and availability of applications, websites, or databases
• automatically distributes the incoming traffic across multiple targets, such as:
• EC2 instances
• containers
• IP addresses  

             ...in one or more Availability Zones.

• monitors the health of its registered targets, and routes traffic only to the healthy targets
• targets can be added and removed from load balancer as the needs change, without disrupting the overall flow of requests to the application 
• scales your load balancer as your incoming traffic changes over time
• supports the following load balancers:  
• Application Load Balancers
• Network Load Balancers
• Gateway Load Balancers
• Classic Load Balancers 

 

 

 

 

Load balancer:

• serves as the single point of contact for clients
• distributes incoming application traffic across multiple targets which increases the availability of your application
• has one or more listeners added to it

Listener:

• checks for connection requests from clients, using the protocol and port that you configure
• has rules defined for it
• a default rule must be defined for each listener, additional rules can be defined optionally 
• Rules determine how the load balancer routes requests to its registered targets
• Each rule consists of:
• priority
• one or more actions
• one or more conditions; when conditions for a rule are met, then its actions are performed
• specified target groups
• When load balancer receives a request, it evaluates the listener rules in priority order to determine which rule to apply, and then selects a target from the target group for the rule action.
   

Target groups:

• Each target group routes requests to one or more registered targets, such as EC2 instances, using the protocol and port number that you specify.  
• target can be registered with multiple target groups
• Routing is performed independently for each target group, even when a target is registered with multiple target groups
  
• health checks are configured per target group 
• used to monitor the health of the registered targets
• they are performed on all targets registered to a target group
• they can be configured so that the load balancer can send requests only to the healthy targets
  
• routing algorithm is configured at the target group level
• default: round robin
• least outstanding requests routing algorithm can alternatively be specified
  

References:

What is an Application Load Balancer? - Elastic Load Balancing 

How Elastic Load Balancing works - Elastic Load Balancing 

What is Round Robin Load Balancing? Definition & FAQs | Avi Networks

How does AWS Elastic Load Balancer Work? - Whizlabs Blog
Load balancer subnets and routing - AWS Prescriptive Guidance