When an EC2 instance (acting as a CI/CD runner, bastion, or deployment node) runs Terraform or Helm to interact with a private EKS cluster, it needs two things:
- network visibility (which it gets by being in the same VPC or connected network) and
- proper IAM-to-Kubernetes mappings
AWS modern access management feature uses EKS Access Entries. This native API feature entirely replaces the messy, deprecated aws-auth ConfigMap.
Here is how to configure the IAM Role and wire it into the cluster's auth configuration using Terraform.
1. The EC2 IAM Role (AWS Side)
The EC2 instance does not need any specific EKS admin permissions attached directly to its IAM role policies. It just needs a standard IAM Role that it can assume via an EC2 Instance Profile. The actual Kubernetes cluster access is granted inside EKS by referencing this role's Amazon Resource Name (ARN).
# 1. Create the IAM Role for the EC2 Instance
resource "aws_iam_role" "cicd_runner" {
name = "eks-cicd-runner-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = { Service = "ec2.amazonaws.com" }
}
]
})
})
# 2. Create the Instance Profile so the EC2 can use the role
resource "aws_iam_instance_profile" "cicd_runner_profile" {
name = "eks-cicd-runner-profile"
role = aws_iam_role.cicd_runner.name
}
2. Wiring it into the Cluster (EKS Side)
To authorize this IAM role to run Helm deployments or manage Kubernetes resources via Terraform, you use EKS Access Entries (aws_eks_access_entry) combined with Access Policy Associations (aws_eks_access_policy_association).
⚠️ Prerequisite: Ensure your aws_eks_cluster resource has authentication_mode = "API_AND_CONFIG_MAP" or "API" enabled so it accepts Access Entries.
The Terraform Configuration
# 1. Define the Access Entry mapping the IAM Role ARN to EKS
resource "aws_eks_access_entry" "cicd_runner_entry" {
cluster_name = "my-cluster-name"
principal_arn = aws_iam_role.cicd_runner.arn
type = "STANDARD" # Standard workflow for users/roles
}
# 2. Attach an AWS-managed access policy to the Access Entry
resource "aws_eks_access_policy_association" "cicd_admin_mapping" {
cluster_name = "my-cluster-name"
principal_arn = aws_iam_role.cicd_runner.arn
policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
access_scope {
type = "cluster"
}
}
Another example (in the cluster's Terraform source code):
# read-only access entry for the in-VPC runner's instance role, so operators can kubectl the
# private cluster via SSM-on-the-runner for diagnostics without an out-of-band `aws eks create-access-entry` grant each time. AmazonEKSAdminViewPolicy = read all resources incl.
# CRDs, excluding Secrets — safe for diagnostics. The runner role is defined in another repo
# (applications/test/github-runner); it also gets eks:DescribeCluster there so `aws eks update-kubeconfig`
# works. Writes still require github-actions-role (the CD/harness OIDC principal above).
resource "aws_eks_access_entry" "runner_readonly" {
cluster_name = local.cluster_name
principal_arn = "arn:aws:iam::1234567890123:role/github-runner-role"
type = "STANDARD"
depends_on = [module.eks]
}
resource "aws_eks_access_policy_association" "runner_readonly" {
cluster_name = local.cluster_name
policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSAdminViewPolicy"
principal_arn = aws_eks_access_entry.runner_readonly.principal_arn
access_scope {
type = "cluster"
}
}
Alternative: Custom Kubernetes Groups
If you don't want to grant full AmazonEKSClusterAdminPolicy and prefer mapping the EC2 instance to custom Kubernetes RBAC roles, you can pass target groups directly via the access entry:
resource "aws_eks_access_entry" "cicd_runner_entry" {
cluster_name = "my-cluster-name"
principal_arn = aws_iam_role.cicd_runner.arn
kubernetes_groups = ["my-custom-helm-deployers-group"]
type = "STANDARD"
}
(You would then manage a native Kubernetes ClusterRoleBinding pointing to my-custom-helm-deployers-group using the kubernetes provider.)
3. How the EC2 Instance Connects
Once the IAM and EKS access configurations are applied, you don't need to pass raw AWS access keys to the EC2 instance. The tools leverage the Instance Profile automatically.
For Helm & kubectl:Log onto the EC2 instance (or configure your shell script/user-data) to update the local kubeconfig using the AWS CLI:
aws eks update-kubeconfig --region us-east-1 --name my-cluster-name
When Helm or kubectl is invoked, it calls the aws eks get-token command under the hood, uses the EC2's IAM profile to generate a signed token, routes over the private VPC endpoint network, and authenticates flawlessly.
For the Terraform Kubernetes/Helm Providers:
If Terraform itself is running on that EC2 instance and deploying helm charts into EKS, configure your Terraform provider block to pull tokens dynamically using the instance profile credentials:
data "aws_eks_cluster_auth" "cluster" {
name = "my-cluster-name"
}
provider "helm" {
kubernetes {
host = "https://ABC123XYZ.gr7.us-east-1.eks.amazonaws.com"
cluster_ca_certificate = base64decode("CLUSTER_CA_DATA")
token = data.aws_eks_cluster_auth.cluster.token
}
}
Just-In-Time (JIT) Access (Least Privilege Workflow)
Another approach is a Just-In-Time (JIT) access or Least Privilege workflow. It is highly secure, but it can quickly become an operational headache for a DevOps engineer if it isn't completely automated.
Instead of giving your GitHub Actions runner continuous, permanent admin access to your private EKS cluster, you only grant access for the exact duration of an ad-hoc task, and then immediately rip it away.
Here is exactly how that breakdown works mechanically step-by-step, why someone would build it, and the hidden risks you should look out for.
The Workflow Cycle Breakdown
When you trigger a workflow or step that needs to run an ad-hoc kubectl command, an automation tool (like a pipeline script, a step function, or a privileged security wrapper) executes this lifecycle:
[Pipeline Starts]
│
▼
1. CREATE ACCESS ENTRY ──► (Allows 'github-runner-role' network identity into EKS)
│
▼
2. ASSOCIATE POLICY ──► (Binds ClusterAdmin or specific RBAC permissions)
│
▼
3. RUN KUBECTL COMMAND ──► (The runner executes its ad-hoc tasks safely)
│
▼
4. DISASSOCIATE POLICY ──► (Strip away the RBAC permissions)
│
▼
5. DELETE ACCESS ENTRY ──► (Completely remove the IAM role mapping from EKS)
│
▼
[Pipeline Finishes]
1. The Gate: create-access-entry
By default, your runner's IAM role (github-runner-role) is completely blocked at the EKS front door. Even if it can physically route to the API endpoint, EKS will return a 401 Unauthorized.
The setup script calls the AWS API to create an EKS Access Entry. This tells EKS: "Be ready to recognize this specific IAM role ARN."
2. The Permission: associate-access-policy
Creating the entry just gets the runner past the front door; it still has zero privileges inside the cluster. The next API call binds an access policy (like AmazonEKSClusterAdminPolicy or a custom namespace policy) to that entry. Now kubectl commands will actually work.
3. The Execution: Ad-hoc Commands
The runner runs aws eks update-kubeconfig, grabs its temporary token, and runs the necessary kubectl or helm actions.
4. The Clean-up: Revoking Access
Once the kubectl step finishes, the pipeline runs a cleanup block (usually wrapped in a always() or finally clause to ensure it runs even if the deployment fails). It reverses the process by disassociating the policy and deleting the access entry entirely. The runner is now unauthorized again.
Why would someone architecture it this way?
- Blast Radius Reduction: If that specific GitHub runner instance or the AWS role credentials are ever compromised, the attacker cannot access your Kubernetes cluster because the role has no standing permissions.
- Audit Trails: Every single time access is granted, used, and revoked, a permanent trail is logged in AWS CloudTrail. It makes passing compliance checks exceptionally easy because you can prove no system has persistent, unchallenged access.
The Catch: Why this pattern can trip you up
While it sounds bulletproof on paper, this setup has a few major operational trade-offs that you have to manage closely:
- Pipeline Failures Leave Backdoors: If the runner gets abruptly killed (e.g., the GitHub Actions job is manually canceled mid-run, or the runner machine loses power), the cleanup step might never run. The role will remain an admin of your cluster until someone goes in and manually deletes the entry.
- Race Conditions on Concurrent Runs: If you have multiple workflows trying to use that same runner role at the same time, Run A might delete the access entry while Run B is right in the middle of executing a kubectl apply.
- AWS API Throttling: If you scale up your CI/CD pipelines and are constantly creating and destroying access entries every few minutes, you can easily hit AWS API rate limits (throttled EKS control plane requests).
For the same EC2 instance (github-runner-role) to grant itself access via the JIT loop, it must possess two entirely separate sets of permissions:
- Control Plane Permissions (AWS IAM): It needs permission to call the AWS EKS APIs (CreateAccessEntry, AssociateAccessPolicy) to tell AWS, "Hey, let me in."
- Data Plane Permissions (Kubernetes RBAC): This is what it actually gives itself inside the cluster to view pods, run Helm charts, or apply manifests.
Here is how this works in practice, how to configure the IAM policy so the runner can self-authorize, and why you must be extremely cautious with this pattern.
1. The Setup: The Instance's Base IAM Policy
To make this happen, the IAM role attached to the EC2 instance must be pre-provisioned with standard AWS permissions allowing it to modify the cluster's access configuration.
You must attach an IAM policy like this to the github-runner-role:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowEKSAccessEntrySelfManagement",
"Effect": "Allow",
"Action": [
"eks:CreateAccessEntry",
"eks:DeleteAccessEntry",
"eks:AssociateAccessPolicy",
"eks:DisassociateAccessPolicy",
"eks:DescribeCluster"
],
"Resource": "arn:aws:eks:us-east-1:123456789012:cluster/my-cluster"
},
{
"Sid": "AllowSTSAssumeRoleSelf",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::123456789012:role/github-runner-role"
}
]
}
💡 Note on the STS block: This allows the runner role to assume itself to generate those unique, concurrent session ARNs (.../github-runner-role/gh-run-123) we discussed earlier.
2. The Execution Flow
When your automation script starts on the EC2 instance, it uses its own local AWS credentials (provided by the EC2 Instance Profile) to run the exact commands that grant it access:
# 1. The EC2 instance talks to AWS IAM & STS to get a unique session ARN
# (This isolates concurrent runs from each other)
ASSUMED_ROLE_ARN="arn:aws:sts::123456789012:assumed-role/github-runner-role/gh-run-${GITHUB_RUN_ID}"
# 2. The EC2 instance calls the AWS EKS API using its base IAM permissions
aws eks create-access-entry \
--cluster-name my-cluster \
--principal-arn "$ASSUMED_ROLE_ARN" \
--type STANDARD
# 3. The EC2 instance elevates its own session's privileges inside Kubernetes
aws eks associate-access-policy \
--cluster-name my-cluster \
--principal-arn "$ASSUMED_ROLE_ARN" \
--policy-arn "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy" \
--access-scope type=cluster
At this point, the instance has successfully leveraged its permanent AWS-level permissions to dynamically grant itself Kubernetes-level permissions.
3. The Security Catch: "The Illusion of Security"
While this satisfies the requirement of keeping the Kubernetes cluster clean of permanent admin entries, you need to look closely at the security trade-off here.
If an attacker compromises this EC2 instance, they inherit the base IAM role. Because that base role has permission to call CreateAccessEntry and AssociateAccessPolicy, the attacker can simply run the exact same script to grant themselves cluster admin privileges whenever they want.
How to make it actually secure:
To make this pattern truly robust, the EC2 instance should not have direct permission to grant itself access. Instead, a external "Gatekeeper" should handle it:
- Lambda Gatekeeper: The runner sends a request to an internal AWS Lambda function (or an internal API gateway). The Lambda verifies the job context (e.g., checks if there is a valid, active GitHub Action job running right now), and if verified, the Lambda creates the EKS Access Entry on behalf of the runner.
- OIDC/IRSA Approach: If this runner is executing jobs from GitHub, skip the EC2 instance profile entirely. Use GitHub's OIDC token to exchange directly for a short-lived AWS session that EKS recognizes natively. This removes the "self-granting" risk entirely.
No comments:
Post a Comment