It's monitoring inverted: instead of alerting when you observe something bad, you alert when you stop observing something good.
Example: ping on a successful run. If run is unsuccessful, monitoring catches missing ping and triggers alert.
The name comes from industrial safety — the lever on a train's throttle or a chainsaw that has to be actively held down. If the operator dies or lets go, the machine stops. Safety is the default state; it takes continuous positive action to keep running.
Normal alerting is presence-based. Something goes wrong, it emits a signal, you alert on the signal: error rate spikes, latency crosses a threshold, a pod enters CrashLoopBackOff. It works well when failures are noisy.
A dead-man's switch is absence-based. The healthy system periodically says "still fine." You alert when that message doesn't arrive on time.
No comments:
Post a Comment