Monitoring and Observability

 

Monitoring vs Observability

In the world of IT and DevOps, monitoring and observability are two related but distinct concepts used to manage system health and performance. 

Core Difference

The simplest way to distinguish them is:

• Monitoring tells you what is happening (and when). It is reactive and focuses on known problems using predefined metrics.
• Observability tells you why it is happening. It is proactive and uses the system's outputs to understand its internal state, especially for "unknown unknowns". 

Key Comparison Table

Feature         Monitoring                 Observability

----------           ---------------                    -----------------    

Purpose         Detect known issues Diagnose root causes

Perspective External (symptoms) Internal (system state)

Question         "Is the system healthy?" "Why is it behaving this way?"

Approach Reactive                         Proactive

Focus         "Known knowns"         "Unknown unknowns"

Data Types Metrics, logs                 Metrics, logs, and traces

Analogy: The Car

• Monitoring is your dashboard. It has dials for speed and fuel, and a "check engine" light. It tells you if you are speeding or if something is broken.
• Observability is the mechanic’s diagnostic tool. When the "check engine" light comes on, the mechanic plugs in a tool to see exactly which sensor failed and why, without having to take the entire engine apart. 

Common Tools

• Monitoring Tools: Nagios, Zabbix, Prometheus.
• Observability Platforms: Datadog, New Relic, Honeycomb, Dynatrace.

Three Pillars of Observability

The three pillars of observability—metrics, logs, and traces—are essential telemetry data types used to understand the internal state of complex, distributed systems. They enable teams to detect, investigate, and resolve performance issues by providing high-level trends, granular event details, and full request-flow paths. 

Metrics

Quantitative measurements over time (e.g., CPU usage, error rates).

Numerical measurements that describe the health, performance, and behavior of a system over time (e.g., CPU usage, error rates, throughput). They are ideal for alerting, capacity planning, and spotting trends or symptoms.

Logs

Granular, timestamped records of discrete events.

Timestamped, granular records of discrete events. They provide the detailed context (text or structured data) necessary to understand exactly what happened within an application or service.

Traces

End-to-end journeys of a single request through a distributed system, showing how different 

Records showing the journey of a single request as it travels through a distributed system, encompassing multiple services. They are critical for pinpointing bottlenecks, latency, or failures in microservices architectures. 

Why They Are Used Together

While metrics indicate that a problem exists, logs provide the context of why it happened, and traces show where it is occurring. Correlating these three data types provides actionable insights rather than just raw data.

Are Logs concern of Monitoring or Observabilty?

Both monitoring and observability deal with logs, but they do so in fundamentally different ways, representing a shift from simply knowing something is broken to understanding why. 

Monitoring is generally used to detect known issues using logs. It is reactive and focuses on pre-defined metrics or alert thresholds, such as alerting when error logs spike or when a specific error code appears.

Observability is used to investigate and understand the "why" behind issues by exploring logs, metrics, and traces together. It is proactive, allowing you to debug complex, distributed systems without needing to know every question ahead of time. 

Comparison: Logs in Monitoring vs. Observability

Feature                 Log Monitoring                                   Log Observability

----------                  ---------------------                                      -------------------------

Primary Question   What went wrong?                                   Why did it go wrong?

Approach        Reactive: Alerts when logs meet criteria   Proactive: Explores data to find root causes

Log Handling        Searchable, indexed logs for active alerts  Contextualized, correlated logs (with traces)

Data Usage       Simple monitoring and basic dashboards   Deep, ad-hoc, and exploratory analysis

Typical Usage       "Error rate > 5%"                                   "Why did this transaction fail?"

How They Work Together

Logs are one of the "three pillars" of observability—alongside metrics and traces—that provide the detailed, granular context necessary for troubleshooting, notes Grafana. 

Monitoring tells you the system is unhealthy (e.g., an alert fires because of high error rates in log files).

Observability allows you to use tools like Splunk or Datadog to dive into the logs and traces to find the specific line of code or database failure causing the issue. 

In short, monitoring is a component of observability—you cannot have true observability without comprehensive logging. 

Monitoring vs Observability on the example of AWS Lambda

For AWS Lambda, monitoring identifies what is wrong (e.g., an execution failed), while observability reveals why it happened by connecting logs, metrics, and traces across your entire serverless architecture. 

Monitoring AWS Lambda: Detecting the Known

Monitoring focuses on pre-defined health indicators. You use it to track "known-knowns" and trigger reactive alerts when thresholds are breached. 

• Key Tool: Amazon CloudWatch collects standard metrics automatically.
• Monitored Metrics:
• Invocations: The total number of times your function runs.
• Errors: The count of failed executions.
• Duration: How long your function takes to run.
• Throttles: Occurrences where invocations are blocked due to concurrency limits.
• Example Scenario: You set a CloudWatch Alarm to notify you if your Lambda's error rate exceeds 5%. Monitoring tells you there is a problem, but not the specific line of code that caused it. 

Observability in AWS Lambda: Investigating the Unknown

Observability is a property of the system that allows you to understand its internal state from external outputs. It uses high-cardinality telemetry to investigate complex, distributed issues. 

• Key Tool: AWS X-Ray provides distributed tracing to visualize the request path across multiple services.
• Key Elements:
• Distributed Traces: Seeing a "waterfall" view of a request as it moves from API Gateway to Lambda, then to DynamoDB.
• Log Insights: Using CloudWatch Logs Insights to run ad-hoc queries across massive log volumes to find specific patterns.
• Enhanced Instrumentation: Using libraries like AWS Lambda Powertools to add structured logging and context to your telemetry.
• Example Scenario: You notice high latency in a specific user's request. Using X-Ray, you see that the Lambda function itself is fast, but it is waiting 2 seconds for a downstream third-party API call. Observability provided the "why". 

Comparison Summary for Lambda

Feature   Monitoring (CloudWatch)           Observability (X-Ray + Logs + Metrics)

----------     ---------------------------------            ---------------------------------------------------

Goal   Track health against thresholds.   Understand root causes and behavior.

Questions  "Is my function failing?"           "Why is this specific request slow?"

Visibility   Isolated metrics for one function.   Request paths across multiple services.

Action   Reactive (Alarms/Notifications).   Proactive (Debugging/Optimisation).

---