There is a class of failure that dashboards handle badly: a background process exits, nothing takes its place, and the system carries on looking healthy because the web tier is still answering. The queue backs up quietly. Somebody notices hours later.
The instinct is to reach for better alerting. Often the smaller and more effective fix is supervision — something whose only job is to notice the process is gone and start it again.
Alerting and supervision are different problems
Alerting tells a human that something is wrong. Supervision makes it right without waiting for one. They are complementary, but they are not substitutes, and teams frequently build the first while assuming they have bought the second.
If a process can be safely restarted, restarting it automatically is nearly always correct. The alert should still fire — but as a record of what happened, not as a summons.
Make restart safe first
Automatic restart is only safe if the work is idempotent. A job that half-completed and left a partial write will happily corrupt things a second time, faster.
So the ordering matters: make the work resumable, make the side effects idempotent, and only then automate the restart. Skipping to the automation is how a reliability feature becomes a data-integrity incident.
Back off, and give up loudly
A supervisor that restarts instantly and forever will hammer a downstream dependency that is already struggling. Exponential backoff with a ceiling handles the common case, where the dependency simply needs a moment.
And there must be a point at which it stops trying and escalates. A crash loop that never surfaces is the original problem wearing a different hat.