How to access a private EC2 instance using a bastion host

 

A bastion host is a server whose purpose is to provide access to a private network from an external network, such as the Internet. [source: How to Record SSH Sessions Established Through a Bastion Host]

 

The idea is to have a public EC2 instance which would be a bastion host, SSH to it first and then from that session to SSH to a private EC2.

 

The process follows the security rule that SSH private keys should never be copied on shared systems or servers - we never copy SSH private key that belongs to (default) user on private EC2 instance onto bastion host. But SSH connection from bastion host to private EC2 instance is still possible thanks to SSH feature called agent forwarding.

 

Prerequisites

 

Here is the list of resources we need to create, step by step:

• Custom VPC in which we want to run EC2 instances
• Internet Gateway which connects VPC with Internet; depends on VPC (ID)
  
• (Internet) Route Table (for this VPC) which allows connection to Internet (routing to Internet Gateway); depends on VPC (ID) and CIDR chosen for public subnet
• (Local) Route Table (for this VPC) which allows only local traffic (within VPC); depends on VPC (ID) and CIDR chosen for private subnet
  
• A (public) subnet in chosen AZ in this VPC
• connected to route table which allows Internet connection => that makes this subnet public
  
• A (private) subnet in the same AZ in this VPC 
• connected to route table which allows local traffic only => that makes this subnet private
• Security group in this VPC that allows SSH; depends on VPC (ID)
  
• Running (public) EC2 instance (bastion host)
• running in the same VPC
• running in the same AZ
• running in the created public subnet
• having attached security group mentioned above
  
• Running (private) EC2 instance
• running in the same VPC
• running in the same AZ
• running in the created private subnet
• having attached security group mentioned above
  
• SSH client available on a dev machine

 

Resource Details

 

VPC:

• Name: test-vpc
• CIDR: 10.10.0.0/16
   

We need to have a security group which is created in the same VPC as in which we want to run EC2 instances and which allows inbound and outbound SSH traffic. 

• Name: Allow public SSH
• VPC: test-vpc
• Inbound rules: 
• IP version: IPv4
• Type: SSH
• Protocol: TCP
• Port range: 22
• Source: 0.0.0.0/0 
• Description: SSH access
  
• Outbound rules: 
• IP version: IPv4
• Type: All traffic
  
• Protocol: All
• Port range: All
• Destination: 0.0.0.0/0
• Description: -
  

EC2 is public if it runs in a public subnet and has Auto-assign public IP enabled. Subnet is public if it is associated with routing table which has at least one Target set to Internet Gateway. 

 

Public EC2 instance needs to have attached to it Allow public SSH security group.

 

EC2 is private if it runs in a private subnet. Subnet is private if it is associated with routing table which has all Targets set to local. The following settings don't make subnet a private by default:

• Subnet has Enable auto-assign public IPv4 address disabled
• When launching EC2 instance, this can be overridden with EC2 setting Auto-assign public IP
  
• Subnet uses a private IP range as its CIDR block

A private EC2 has a network interface which does not have a public IPv4 address but has a private IPv4 address e.g. 10.10.3.226 if it's in 10.10.3.0/24 subnet.

Private EC2 instance needs to have attached to it Allow public SSH security group (SSH Access Security Group in the diagram below).

Architecture diagram:

 

SSH Connection

We'll use SSH feature called agent forwarding. If we load a key into a local SSH agent, we can use ssh -A command to allow any remote ssh server to read this key as if it was added to the local SSH agent. We need to use -A only when establishing the first SSH connection.

From man ssh:

     -A      Enables forwarding of connections from an authentication agent such as ssh-agent(1).  This can also be specified on a per-host basis in a configuration file.
Agent forwarding should be enabled with caution.  Users with the ability to bypass file permissions on the remote host (for the agent's UNIX-domain socket) can access the local agent through the forwarded connection.  An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent. A safer alternative may be to use a jump host (see -J).

To add a private key to the local ssh agent:

$ ssh-add MyTestMachine.pem
Identity added: MyTestMachine.pem (MyTestMachine.pem)
 

Note that in this setup we are using the same SSH key (MyTestMachine.pem) for secure connection to both bastion and private host.

Now we can connect to the bastion host (ec2-user is the default user for Amazon Linux 2 AMIs):

$ ssh -A ec2-user@34.255.30.127
Last login: Wed Jun 15 16:38:34 2022 from r-123.45.67.195.ptr.example.com

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
16 package(s) needed for security, out of 26 available
Run "sudo yum update" to apply all updates.

If each host has its own SSH key we can load to SSH agent the one of the target private EC2 host and then later use -i option to specify the key for connecting to bastion host:

$ ssh-add private_ec2_host.pem

$ ssh -A -i bastion_host.pem user@bastion-host.com

(These .pem files store private keys but in general, they can contain also public keys, or both, or certificates...security - Does .pem file contain both private and public keys? - Stack Overflow)

From inside the bastion host we can connect to the target private EC2:

[ec2-user@ip-10-10-0-85 ~]$ ssh ec2-user@10.10.3.226

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
[ec2-user@ip-10-10-3-226 ~]$

This is possible because Agent Forwarding added MyTestMachine.pem to bastion host's SSH agent:

$ ssh-add -L
ssh-rsa AAAA...E2XN MyTestMachine.pem  

On bastion host, authorized_keys for the custom user contains public key from the key-pair provided by that user (private key bastion_host.pem is kept by the user and is used to connect to bastion host):

$ cat ~/.ssh/authorized_keys 
ssh-rsa AAAA...VeXQ== john.smith@example.com

---

Internet Protocol (IP) Addresses and Ranges

Internet Protocol (IP) is one of protocols from Network Layer.

Each node in IP network has IP address assigned so IP packets can be routed between them. 

IPv4 Addresses

image source: juniper.net

• IPv4 - Internet Protocol v4 (1981)
• each node has its identifier - IP address  
• 32-bit number divided in octets
• example: 56.122.17.4
• dot-decimal notation
• four octets, four sections of 8 bits
• each octet (each section) is 8-bit number so in total 32-bits required to represent the full addres; there are 2^32 combinations in total
• 0.0.0.0 to 255.255.255.255 
• some addresses and ranges are reserved for e.g. private networks 

IPv6 Addresses

 

image source: juniper.net

• Introduced in order to expand the range of IP addresses.
• 128-bit address scheme 
• 2^128 (undecillion) address combinations 
• IPv6 as hexadecimal: 8 segments of 16 bits separated by colons
• Example: 1e03:b32f:042d:0000:0000:0000:0436:4aef
• How to shorten IPv6 address:
• Leading zeros in each segment can be omitted.
• Segments with all zeros can be replaced with single zero:
• 1e03:b32f:42d:0:0:0:436:4aef
• Furthermore, a single subset of consecutive zeros can be replaced with two colons: 
• 1e03:b32f:42d::436:4aef
• This can be done only once as if we had two occurrences of double-colons we wouldn't know how many zeros each represent. 

Classless Inter-Domain Routing (CIDR) Notation

• A way of specifying a range of IP addresses, including the case of a single IP address. 
• Example: 192.10.0.0/16
• Number after slash denotes leading bits in a 32-bit number that get frozen. In this example, 192.10 (first two octets) stays the same and the rest two octets change so we get the range from 192.10.0.0 to 192.10.255.255. Two last octets are free to change which gives us 2^16 addresses in this range.
• Examples:
  
• 192.10.0.0/24 gives the range 192.10.0.0 to 192.10.0.255
• 192.10.0.0/32 gives the single IP address: 192.10.0.0  
• 192.10.0.0/31 gives two IP addresses: 192.10.0.0 and 192.10.0.1
• 192.10.0.0/30 gives four IP addresses: 192.10.0.0 to 192.10.0.4 
• 192.10.0.0/17 freezes first 17 bits which is first 2 octets and first bit of 3rd octet so we get the range 192.10.0.0 to 192.10.127.255 (0111111 = 127)
• 192.10.0.0/28 freezes first 3 octets and first half (first 4 bits) of the 4th octet giving the range of 16 IP addresses: 192.10.0.0 to 192.10.0.15 (00001111 = 15)
• AWS VPC allows leading bits between 16 (/16) and 28 (/28).

 

Private Network Ranges

• Public IP range is routable to the Internet. These IP addresses can directly communicate to Internet. 
• Private network ranges solve the problem of not having enough of IPv4 addresses for all devices connected to Internet.
• Private Network Ranges (IETF specification RFC1918):
• 10.0.0.0/8 => 10.0.0.0 to 10.255.255.255 (~16 million addresses)
• 172.16.0.0/12 => 172.16.0.0 to 172.31.255.255 (~1 million addresses)
• 192.168.0.0/16 => 192.168.0.0 to 192.168.255.255 (~65k addresses)
• Private IP range is not routable to the Internet, they are not publicly available, can be used only in private networks. Devices with private network IPs can reach Internet via Network Address Translation or proxy service - something that translates private IP address to public IP address. 

 

ipcalc Tool

 

 

ipcalc is a useful tool which visualizes subnet calculations.

 

To install it on Ubuntu Linux:

 

$ sudo apt install ipcalc
 

To visualize CIDR:

$ ipcalc 192.168.0.0/24

Address:   192.168.0.0          11000000.10101000.00000000. 00000000

Netmask:   255.255.255.0 = 24   11111111.11111111.11111111. 00000000

Wildcard:  0.0.0.255            00000000.00000000.00000000. 11111111

=>

Network:   192.168.0.0/24       11000000.10101000.00000000. 00000000

HostMin:   192.168.0.1          11000000.10101000.00000000. 00000001

HostMax:   192.168.0.254        11000000.10101000.00000000. 11111110

Broadcast: 192.168.0.255        11000000.10101000.00000000. 11111111

Hosts/Net: 254                   Class C, Private Internet

---

Terraform Workspaces

 

By default, there is one state file (terraform.tfstate) per configuration directory. Sometimes, we want to reuse the same configuration files for multiple projects. Instead of creating a directory for each project and copy-pasting files, we can use Terraform's feature called workspaces. Each workspace has its own, isolate state. When in particular workspace, terraform plan can see only its state.

When we create a configuration file, before explicitly creating any workspaces, Terraform implicitly creates a workspace named default:

$ terraform console
> terraform.workspace
default

To create a workspace:

$ terraform workspace new ProjectA

To list all workspaces:

$ terraform workspace list
  default

* ProjectA

  ProjectB

Asterisk indicates the currently active workspace.

To switch to another workspace we need to use select command:

$ terraform workspace select default

Switched to workspace "default".
$ terraform workspace select ProjectB 

Switched to workspace "ProjectB".

terraform.workspace variable contains the name of the current workspace and it can be used in configuration files:

variables.tf:

variable region {

    default = "eu-west-1"
}

variable instance_type {

    default = "t2.micro"

}

variable ami {

    type = map

    default = {

        "ProjectA" = "ami-0123456789"

        "ProjectB" = "ami-9876543210"

    }

}

main.tf:

resource "aws_instance" "my-server" {

    ami = lookup(var.ami, terraform.workspace)

    instance_type = var.instance_type

    tags = {

        Name = "terraform.workspace"

    }

}

Terraform creates one state file for each workspace. They are stored in a directory named terraform.tfstate.d:

$ tree terraform.tfstate.d/

terraform.tfstate.d/

|--ProjectA

|       `-- terraform.tfstate

`--ProjectB

        `-- terraform.tfstate

---