Late and down
How LastSeen decides a heartbeat monitor has gone quiet, and how to tune it so it tells you soon without crying wolf.
A heartbeat monitor has three settings for timing:
| Setting | Meaning | Default |
|---|---|---|
| Heartbeat every | How often it pings. At least 30s. | 60s |
| Missed before down | How many heartbeats in a row may be missed before it's down (1 to 10). | 2 |
| Extra grace | Time added to the deadline, for slow networks and flaky Wi-Fi. | 0s |
The states #
| State | When | Alert? |
|---|---|---|
| new | Added, but no heartbeat yet. It can't be late until it has started. | No |
| up | Heartbeats are arriving on time. | A recovery, if it was down |
| late | One interval has passed with no heartbeat. | No: it's a warning on the page |
| down | The missed heartbeats and the grace have passed, or it sent /fail. | Yes |
Late is deliberately quiet. Most late heartbeats are a slow network or a device retrying, and turn up seconds later. It's there so you can see trouble brewing on the page, not to wake you.
When exactly #
Counting from the last heartbeat:
- late after one interval;
- down after the missed heartbeats times the interval, plus the grace.
With the defaults (every 60s, 2 missed, no grace), a monitor whose last heartbeat was at 12:00:00 is late from 12:01:00 and down from 12:02:00. The form shows these times as you type.
The next heartbeat brings it straight back to up, and the recovery alert says how long it was gone.
Choosing the numbers #
- Set the interval to how often it really pings. If a device pings every 5 minutes, say 5m. Too short and it's always late; too long and you hear late.
- Missed before down is your patience. 2 is a good start for devices on Wi-Fi: one lost ping won't alert, two will. For a server on a wired network, 1 is fine.
- Grace absorbs slowness that isn't a missed heartbeat: a device that takes 20 seconds to reconnect after a Wi-Fi blip, or a job that pings at the end of a run that varies in length.
- For a job that runs at set times, rather than every few minutes, use a cron schedule instead.
The monitor's page shows lost heartbeats (from seq gaps, if the device sends them) over the last day, which tells you whether to loosen or tighten.
Reboots and restarts #
If a device sends /start, or its seq or up counter goes back to the start, LastSeen notes a restart in the monitor's history. When a monitor comes back up, the recovery says what the outage looked like: a reboot, lost heartbeats, or plain silence.
When LastSeen can't hear anyone #
If every monitor goes quiet at the same moment, it's much more likely that LastSeen's own receiving end has a problem than that all your devices died together. LastSeen watches for that and holds down alerts until it's sure. See Staying quiet when you should.