Prometheus Metrics

Container Metrics

container_memory_rss

container_memory_rss is a Prometheus metric (exposed by cAdvisor) that measures a container's Resident Set Size—specifically, the amount of physical RAM allocated to non-reclaimable, non-file-backed memory.

Key Technical Breakdown

Under the hood in Linux cgroups, container_memory_rss tracks:

• Anonymous Memory: Process heap allocations, execution stack, and memory allocated via malloc or mmap(MAP_ANONYMOUS).  
• Swap Cache: Memory swapped out to disk that is being brought back into RAM.

What it EXCLUDES:

Unlike standard Linux host RSS metrics, container_memory_rss in cAdvisor excludes file-backed page caches (memory used to cache files read from disk). Because of this, it only measures memory that cannot be automatically freed by the Linux kernel under memory pressure.  

container_memory_rss vs. Other Container Metrics

To understand its role, it helps to see how it fits into cAdvisor's other core memory metrics:

Metric

• Includes
• Purpose / Key Characteristic

container_memory_rss

• Heap + Stack (Anonymous memory)
• Stable indicator of process memory footprint. Does not fluctuate with disk I/O.

container_memory_working_set_bytes

• Heap + Stack + Active Page Cache
• What Kubernetes actually monitors. Fluctuates with file reads/writes.

container_memory_usage_bytes

• Heap + Stack + Active Cache + Inactive Cache
• Raw total RAM usage. Can be misleading because inactive cache is easily reclaimed by the OS.

Why is container_memory_rss Important?

• Memory Leak Detection: Because it excludes cached file reads, container_memory_rss provides a much cleaner signal for application-level memory leaks. If this metric steadily climbs over time without dropping, your application (e.g., Go heap, JVM heap, Node process) is holding onto unmanaged memory. 
• Debugging OOM Kills: While Kubernetes triggers OOMKills based on container_memory_working_set_bytes hitting resource limits, container_memory_rss helps you determine why it happened:  
• High RSS + High Working Set --> Application memory leak or underestimated heap limit.
• Low RSS + High Working Set --> Heavy disk I/O / file caching (e.g., reading massive log files or database indexes into memory).  

When container_memory_rss is NOT a relevant metric?

Example:

WiredTiger is the default storage engine that MongoDB (here, the Percona Server for MongoDB / PSMDB cluster) uses to actually read and write data to disk. In <ticketID> it matters specifically because of how it uses memory, which is what breaks those four Grafana alert rules.

The relevant behavior:

WiredTiger keeps a large in-memory cache. It maintains its own cache of frequently-accessed data and indexes, and it deliberately sizes that cache to roughly half the container's memory limit (the issue cites ~10 GiB against a 21Gi limit on rs0/rs2, and it's tunable per <ticketID>). WiredTiger runs its own eviction, targeting about 80% cache fill under normal operation and triggering aggressive eviction around 95%.

That cached data shows up as page cache, not RSS. This is the crux of the ticket. container_memory_rss counts anonymous/resident process memory but excludes the OS page cache — and for a WiredTiger workload the file-backed pages (the cache) are the bulk of the real footprint. So rss / limit sits structurally pinned around 50-58% no matter how much memory pressure the pod is actually under. An alert keyed on container_memory_rss > 0.90 can therefore never fire for a WiredTiger container. It's dead. (alert is defined as: container_memory_rss{container="mongod"} / limit > 0.90)

But you can't just switch to working-set either. container_memory_working_set_bytes does include those active file pages, so on the busy replicas it reads 95-96%. That looks alarming but is expected and healthy: it's the WiredTiger cache sitting at its designed ~half-of-limit size and WT's own eviction watermark. The kernel reclaims those pages under real pressure, and the evidence is that nothing in the mongodb namespace has ever been OOMKilled. Flipping the metric would just move the rule from never-firing to always-firing.

The signal that actually matters is WiredTiger cache fill. Because WT stalls user operations when its cache can't evict fast enough, the meaningful early-warning metric is cache utilization (the existing MongoDB WiredTiger cache fill — above 95% for 30m rule from <ticketID>), not container RSS or working set. That's why the ticket argues the RSS rules are redundant, not just broken, and leans toward deleting them.

So in this issue's context, "WiredTiger" is essentially shorthand for "a workload whose memory lives mostly in a large, self-managed, page-cache-backed database cache" — which is exactly the profile that makes RSS-based memory alerting meaningless and makes cache-fill the correct signal instead. (The same logic applies to the Elasticsearch rules, where the JVM heap plays the analogous role.)

Example PromQL Query

To track application heap growth without the noise of filesystem caching, query RSS like this:

# Rate of container RSS growth over 5-minute intervals

rate(container_memory_rss{namespace="production", container!=""}[5m])

container_memory_working_set_bytes

...

container_memory_usage_bytes

...