Introduction to Elastic Agents

Elastic Agents are unified, lightweight software components developed by Elastic to collect, ship, and (optionally) protect data—including logs, metrics, traces, and security events—from your infrastructure to the Elastic Stack (Elasticsearch, Kibana, etc.)

Elastic Agents are not strictly required components in every Elastic Stack deployment, but they play a crucial role in certain scenarios. Here's an explanation based on use cases:

Key Functions of Elastic Agents (When Elastic Agents Are Required?)

• Unified Data Collection:

• They provide a single, centralized solution to collect  various types of observability and security data from hosts, containers, and Kubernetes clusters (logs, metrics, traces, and security data)
• They replace individual Beats (e.g., Filebeat, Metricbeat) for streamlined data ingestion.

• Kubernetes Monitoring: 
• When deployed on Kubernetes (often as a DaemonSet), Elastic Agent runs on every node, collecting:
• System metrics (CPU, memory, disk, etc.)
• Kubernetes resource metrics (pods, nodes, deployments)
• Logs from nodes and containers
• Security posture and events

• Fleet Management:

  • Elastic Agents can be centrally managed using Elastic Fleet, allowing you to configure, update, and monitor all agents and their integrations from a single Kibana interface
  • Elastic Agents are required when using Fleet, the centralized management interface in Kibana.
  • Fleet allows you to:
  • Manage agent configurations from a single UI.
  • Deploy updates and policies at scale.
  • Monitor agent health and performance.
  
  
• Endpoint Security:
• Elastic Agents are necessary for using endpoint Security features, like malware detection, endpoint protection, and threat monitoring, host intrusion detection, and Kubernetes Security Posture Management (KSPM)

When Elastic Agents Are Not Required:

1. Traditional Beats Usage:

   • If you are already using specific Beats (e.g., Filebeat, Metricbeat, Heartbeat) for data collection and do not need unified management, Elastic Agents are optional.
   • Beats can ship data directly to Elasticsearch or Logstash without requiring Fleet or Elastic Agents.

2. Direct Data Ingestion:

   • If you are ingesting data directly into Elasticsearch via APIs, custom applications, or third-party tools, Elastic Agents are not needed.

3. Standalone Elastic Stack:

   • For use cases focused purely on search, analytics, or visualization where data is ingested manually or through custom integrations, Elastic Agents are unnecessary.

Key Considerations:

• Unified Management: Elastic Agents with Fleet simplify large-scale deployments and are recommended for environments with many data sources.
• Compatibility: Elastic is gradually consolidating data collection around Elastic Agents, so they are the future-proof choice for managing observability and security data.
• Flexibility: You can still mix and match Elastic Agents and Beats, depending on your requirements.

How Elastic Agents Work in Kubernetes

Deployment

Typically deployed as a DaemonSet so that each Kubernetes node runs an agent instance, ensuring complete coverage for data collection.

Leader Election

One agent may be elected as a leader to handle cluster-wide metrics (like Kubernetes events), while others focus on node-specific data.

Data Flow

Data collected by agents is shipped to Elasticsearch for storage and analysis, and visualized in Kibana

---

In summary, Elastic Agents are not mandatory for all Elastic Stack setups, but they are highly beneficial for unified data collection, centralized management, and security monitoring.