Amazon GuardDuty is a threat detection service which continuously monitors, profiles and analyses events across AWS accounts and resources. In Introduction to Amazon GuardDuty I wrote about its general features.
Its support for Amazon Elastic Kubernetes Service (Amazon EKS) comes in two ways:
- GuardDuty EKS Protection (EKS Audit Log Monitoring)
- GuardDuty feature that monitors cluster control plane activity by analyzing EKS audit logs
- Helps detect potentially suspicious activities in EKS clusters
- Enabled by default when GuardDuty is enabled
- GuardDuty EKS Runtime Monitoring
- mechanism to detect runtime threats from over 30 security findings to protect our EKS clusters
- can identify specific containers within your EKS clusters that are potentially compromised and detect attempts to escalate privileges from an individual container to the underlying Amazon EC2 host and the broader AWS environment
- fully managed EKS add-on
- managed as a part of GuardDuty Runtime Monitoring
- visibility into individual container runtime activities (on-host operating system-level behavior) such as:
- file access
- process execution
- network connections
- lightweight security agent is deployed as a Kubernetes Daemonset which has a pod running on every node
- Detection of a potential threat triggers creation of a security finding that pinpoints the specific container, and includes details such as:
- pod ID
- image ID
- EKS cluster tags
- executable path
- process lineage
- can be enabled both on existing and new EKS clusters
- available from March 2023 [Amazon GuardDuty now monitors runtime activity from containers running on Amazon EKS]
- Needs to be enabled explicitly, after GuardDuty is enabled
In Introduction to Amazon GuardDuty we saw that to enable EKS Audit Log Monitoring we only need to provision and enable GuardDuty detector (aws_guardduty_detector | Resources | hashicorp/aws | Terraform | Terraform Registry):
resource "aws_guardduty_detector" "this" {
enable = true
}
To enable GuardDuty EKS Runtime Monitoring (which is a prerequisite for activating GuardDuty EKS add-on) we need to use resource aws_guardduty_detector_feature.
To add and activate GuardDuty EKS add-on we have two choices:
- use resource aws_eks_addon where addon_name is set to aws-guardduty-agent. See examples:
- Amazon GuardDuty- deploy EKS Runtime Monitoring using Terraform | by Marcin Cuber | Medium
- Introducing GuardDuty EKS Runtime Monitoring
- Amazon GuardDuty for EKS - wikieduonline
- use resource aws_guardduty_detector_feature, see examples:
We can enable EKS Runtime Monitoring AND automatic GuardDuty add-on management (automatic deployment and updates of the security agent in accounts where Runtime Monitoring is enabled) with the following:
resource "aws_guardduty_detector_feature" "eks_runtime_monitoring" {
detector_id = aws_guardduty_detector.this.id
name = "EKS_RUNTIME_MONITORING"
status = "ENABLED"
additional_configuration {
name = "EKS_ADDON_MANAGEMENT"
status = "ENABLED"
}
}
Note that if current account does not have required privileges, provisioning this resource will throw an error:
Error: updating GuardDuty Detector (92bXXXXXXXd1) Feature (EKS_RUNTIME_MONITORING): BadRequestException: The request is rejected because member accounts cannot manage specified resources or properties.
In AWS Console we can now see that EKS Runtime monitoring is now also enabled:
EKS monitoring is now covered in full (EKS Audit Logs monitoring + EKS Runtime monitoring + automatic GuardDuty EKS addon management).
After provisioning the EKS cluster we can see that GuardDuty picked it up:
We can check out the addons section in the EKS cluster:
Amazon GuardDuty EKS runtime monitoringAmazon GuardDuty is a security monitoring service that analyzes and processes multiple supported data sources to identify any unexpected and potentially unauthorized suspicious or malicious activity within your AWS environment.EKS Runtime Monitoring in Amazon GuardDuty provides runtime threat detection to protect EKS clusters in your AWS environment. EKS Runtime Monitoring uses a fully-managed EKS add-on (GuardDuty security agent) that adds visibility into individual Kubernetes container runtime activities, such as file access, process execution, and network connections.Once you enable EKS Runtime Monitoring within Amazon GuardDuty and install the EKS add-on within your cluster, GuardDuty starts to consume runtime activity events from all EC2 hosts and containers in the cluster. These events are analyzed by GuardDuty for potential threats. As GuardDuty detects a potential threat, it generates a security finding. Navigate to the GuardDuty console to view the generated findings in your account.
Disabling EKS_ADDON_MANAGEMENT will keep the GurdDuty addon active but management will switch from Automatic to Manual:
resource "aws_guardduty_detector_feature" "eks_runtime_monitoring" {
detector_id = aws_guardduty_detector.this.id
name = "EKS_RUNTIME_MONITORING"
status = "ENABLED"
additional_configuration {
name = "EKS_ADDON_MANAGEMENT"
status = "DISABLED"
}
}
terraform plan output:
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# aws_guardduty_detector_feature.eks_runtime_monitoring will be updated in-place
~ resource "aws_guardduty_detector_feature" "eks_runtime_monitoring" {
id = "e6c7f038a9682cf6ff6bb514c110a66f/EKS_RUNTIME_MONITORING"
name = "EKS_RUNTIME_MONITORING"
# (2 unchanged attributes hidden)
~ additional_configuration {
name = "EKS_ADDON_MANAGEMENT"
~ status = "ENABLED" -> "DISABLED"
}
}
Plan: 0 to add, 1 to change, 0 to destroy.
resource "aws_guardduty_detector_feature" "eks_runtime_monitoring" {
detector_id = aws_guardduty_detector.this.id
name = "EKS_RUNTIME_MONITORING"
status = "ENABLED"
}
terraform plan output:
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
# aws_guardduty_detector_feature.eks_runtime_monitoring must be replaced
-/+ resource "aws_guardduty_detector_feature" "eks_runtime_monitoring" {
~ id = "e6c7f038a9682cf6ff6bb514c110a66f/EKS_RUNTIME_MONITORING" -> (known after apply)
name = "EKS_RUNTIME_MONITORING"
# (2 unchanged attributes hidden)
- additional_configuration { # forces replacement
- name = "EKS_ADDON_MANAGEMENT" -> null # forces replacement
- status = "DISABLED" -> null
}
}
Plan: 1 to add, 0 to change, 1 to destroy.
Nothing changed in the AWS Console - addon is active but manually managed.
Let's now disable EKS_RUNTIME_MONITORING feature completely:
resource "aws_guardduty_detector_feature" "eks_runtime_monitoring" {
detector_id = aws_guardduty_detector.this.id
name = "EKS_RUNTIME_MONITORING"
status = "DISABLED"
}
terraform plan output:
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
# aws_guardduty_detector_feature.eks_runtime_monitoring must be replaced
-/+ resource "aws_guardduty_detector_feature" "eks_runtime_monitoring" {
~ id = "e6c7f038a9682cf6ff6bb514c110a66f/EKS_RUNTIME_MONITORING" -> (known after apply)
name = "EKS_RUNTIME_MONITORING"
~ status = "ENABLED" -> "DISABLED"
# (1 unchanged attribute hidden)
- additional_configuration { # forces replacement
- name = "EKS_ADDON_MANAGEMENT" -> null # forces replacement
- status = "DISABLED" -> null
}
}
Plan: 1 to add, 0 to change, 1 to destroy.
Addon is not present anymore:
Let's see what happens if we add GuardDuty addon to EKS cluster via resource aws_eks_addon.
For it we need to find out the correct name of the addon and also its version. We can use aws eks describe-addon-versions for that:
$ aws eks describe-addon-versions --output table --query 'addons[].{Name: addonName, Publisher: publisher}'
------------------------------------------------------------------------
| DescribeAddonVersions |
+---------------------------------------------+------------------------+
| Name | Publisher |
+---------------------------------------------+------------------------+
| vpc-cni | eks |
| upwind-security_upwind-operator | Upwind Security |
| upbound_universal-crossplane | upbound |
| tetrate-io_istio-distro | tetrate-io |
| teleport_teleport | teleport |
| stormforge_optimize-live | StormForge |
| splunk_splunk-otel-collector-chart | Splunk |
| solo-io_istio-distro | Solo.io |
| solo-io_gloo-mesh-starter-pack | Solo.io |
| solarwinds_swo-k8s-collector-addon | SolarWinds |
| snapshot-controller | eks |
| rafay-systems_rafay-operator | rafay-systems |
| new-relic_kubernetes-operator | New Relic |
| netapp_trident-operator | NetApp Inc. |
| leaksignal_leakagent | leaksignal |
| kubecost_kubecost | kubecost |
| kube-proxy | eks |
| kong_konnect-ri | kong |
| kasten_k10 | Kasten by Veeam |
| haproxy-technologies_kubernetes-ingress-ee | HAProxy Technologies |
| groundcover_agent | groundcover |
| grafana-labs_kubernetes-monitoring | Grafana Labs |
| factorhouse_kpow | factorhouse |
| eks-pod-identity-agent | eks |
| dynatrace_dynatrace-operator | dynatrace |
| datree_engine-pro | datree |
| datadog_operator | Datadog |
| cribl_cribledge | Cribl |
| coredns | eks |
| cisco_cisco-cloud-observability-operators | Cisco Systems, Inc. |
| cisco_cisco-cloud-observability-collectors | Cisco Systems, Inc. |
| calyptia_fluent-bit | Calyptia Inc |
| aws-mountpoint-s3-csi-driver | s3 |
| aws-guardduty-agent | eks |
| aws-efs-csi-driver | eks |
| aws-ebs-csi-driver | eks |
| amazon-cloudwatch-observability | eks |
| akuity_agent | akuity |
| adot | eks |
| accuknox_kubearmor | AccuKnox |
+---------------------------------------------+------------------------+
Now when we know the name of the addon we're interesting in (aws-guardduty-agent) we can check its latest version:
$ aws eks describe-addon-versions --output table --addon-name=aws-guardduty-agent --query 'addons[].{Name: addonName, Publisher: publisher, Version: addonVersions[0].addonVersion}'
-----------------------------------------------------------
| DescribeAddonVersions |
+----------------------+------------+---------------------+
| Name | Publisher | Version |
+----------------------+------------+---------------------+
| aws-guardduty-agent | eks | v1.6.1-eksbuild.1 |
+----------------------+------------+---------------------+
Let's now add this resource to our eks cluster (Terraform module) and provision it:
resource "aws_eks_addon" "guardduty" {
cluster_name = aws_eks_cluster.this.name
addon_name = "aws-guardduty-agent"
addon_version = "v1.6.1-eksbuild.1"
resolve_conflicts_on_update = "OVERWRITE"
}
terraform plan output:
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
~ update in-place
Terraform will perform the following actions:
# module.voting-app-eks-cluster.aws_eks_addon.guardduty will be created
+ resource "aws_eks_addon" "guardduty" {
+ addon_name = "aws-guardduty-agent"
+ addon_version = "v1.6.1-eksbuild.1"
+ arn = (known after apply)
+ cluster_name = "example-voting-app"
+ configuration_values = (known after apply)
+ created_at = (known after apply)
+ id = (known after apply)
+ modified_at = (known after apply)
+ resolve_conflicts_on_update = "OVERWRITE"
+ tags_all = (known after apply)
}
During the provisioning, AWS Console shows the accurate status:
Provisioning the addon was not successful after 20 minutes timeout:
module.voting-app-eks-cluster.aws_eks_addon.guardduty: Still creating... [19m30s elapsed]
module.voting-app-eks-cluster.aws_eks_addon.guardduty: Still creating... [19m40s elapsed]
module.voting-app-eks-cluster.aws_eks_addon.guardduty: Still creating... [19m50s elapsed]
module.voting-app-eks-cluster.aws_eks_addon.guardduty: Still creating... [20m0s elapsed]
╷
│ Warning: Running terraform apply again will remove the kubernetes add-on and attempt to create it again effectively purging previous add-on configuration
│
│ with module.voting-app-eks-cluster.aws_eks_addon.guardduty,
│ on ../../modules/eks-cluster/main.tf line 174, in resource "aws_eks_addon" "guardduty":
│ 174: resource "aws_eks_addon" "guardduty" {
│
╵
╷
│ Error: waiting for EKS Add-On (example-voting-app:aws-guardduty-agent) create: timeout while waiting for state to become 'ACTIVE' (last state: 'CREATING', timeout: 20m0s)
│
│ with module.voting-app-eks-cluster.aws_eks_addon.guardduty,
│ on ../../modules/eks-cluster/main.tf line 174, in resource "aws_eks_addon" "guardduty":
│ 174: resource "aws_eks_addon" "guardduty" {
After this I commented out resource "aws_eks_addon" "guardduty" {...} and executed terraform apply so this resource was removed.
The reason for failing to install this add-on might be the fact that GuardDuty Runtime monitoring is not enabled for EKS.
Let's try to manually add this add-on, without enabling the EKS Runtime monitoring:
I was not able to stop creation of this add-on in AWS Console but it worked via AWS CLI:
$ aws eks delete-addon --cluster-name example-voting-app --addon-name aws-guardduty-agent --profile terraform --region eu-west-2
{
"addon": {
"addonName": "aws-guardduty-agent",
"clusterName": "example-voting-app",
"status": "DELETING",
"addonVersion": "v1.6.1-eksbuild.1",
"health": {
"issues": []
},
"addonArn": "arn:aws:eks:eu-west-2:471112786618:addon/example-voting-app/aws-guardduty-agent/cec7f182-1052-421b-76db-9fa8e353af80",
"createdAt": "2024-06-04T12:30:49.852000+01:00",
"modifiedAt": "2024-06-04T12:42:29.170000+01:00",
"tags": {}
}
}
When I was adding this add-on I didn't enable EKS Runtime Monitoring although its activation is required for an optimal operating experience:
Ensure to enable EKS Runtime Monitoring within Amazon GuardDuty.
Let's enable EKS Runtime Monitoring via:
resource "aws_guardduty_detector_feature" "eks_runtime_monitoring" {
detector_id = aws_guardduty_detector.this.id
name = "EKS_RUNTIME_MONITORING"
status = "ENABLED"
}
After enabling EKS Runtime Monitoring I tried to provision resource "aws_eks_addon" "guardduty" {...} and again it started creating it only to error out after 20 minutes.
Let's try to enable RUNTIME_MONITORING feature (which includes EKS_RUNTIME_MONITORING):
resource "aws_guardduty_detector_feature" "eks_runtime_monitoring" {
detector_id = aws_guardduty_detector.this.id
name = "RUNTIME_MONITORING"
status = "ENABLED"
}
We can now see in AWS Console that runtime monitoring is enabled for EKS, ECS and EC2:
Provisioning resource "aws_eks_addon" "guardduty" {...} again error out after 20 minutes.
I will try to find out what prevents successful provisioning of the aws_eks_addon resource.
After Disabling GuardDuty Detector
Error: updating GuardDuty Detector Feature (RUNTIME_MONITORING): BadRequestException: The request failed because you cannot enable a data source while the detector is disabled.
No comments:
Post a Comment