Writing

Does this alert need a response?

Do you get alarms and notifications that you find not actionable?

Every alert should answer one question: do I need to do something? If not, then why are you telling me this.

Control rooms settled this decades ago. In an incident at the Milford Haven refinery in 1994 there was a huge fire and they found that operators faced 275 alarms in the last eleven minutes before the explosion and dismissed all of them. Most of them were marked urgent. But when everything is urgent, nothing is. The industry wrote the fix into a standard in 1999. Their target for a plant running normally: no more than one alarm every ten minutes. Now scroll back a day in your team's alert channel.

If 85% or more of the alarms are not actionable, the machine loses the trust. An alert that is usually wrong teaches everyone near it to turn it off or ignore it.

The fix is really dull. Go through your alerts one at a time. Ask whether each one needs a response. Keep those. Which ones can you automatically recover? Implement the mechanism for that. Separate out if some alerts are just status reading, tracking or informative and route them elsewhere.

If you are designing systems, products that need human input and intervention, this is critical work, even if it is boring.