Go to All Forums

How to stop alert storm

We are monitoring a number of services in each server which are related to antivirus, web protection and security. There are altogether 10 + services per server. The issue is whenever there is an update on AV signature or threat database etc, services stop and starts automatically and we do receive 20+ email alert per server - one for each service down and the other for each service up. We are monitoring 400+ servers. You can calculate the number of emails this is generating!!

Is there a way to group these services into one and I only receive one alert instead of 10+ per server when service goes down?

 

Thanks,

Ash

 

Like (3) Reply
Replies (4)

Re: How to stop alert storm

Hi Ash, 

      I can understand the operational problem you have to face. We don't have the ability to consolidate alerts at present.

However, would you be fine if the server in which the services are running could be Marked as Maintenance..? Since updating AV is a maintenance activity, you can mark the servers as maintenance which will avoid alerts. You can schedule the maintenance if you know that the update happens in a time bound manner.

-Jasper 

Product Manager, Site24x7

Like (0) Reply

Re: Re: How to stop alert storm

Hi Jasper, 

These updates are done automatically by the software and no way, we can schedule when the updates will be applied. thus, cannot add a maintenance window.

Though, a question, is it possible crate a schedule maintenance window specifically for those services as a group and not monitors as a whole? We have noticed that most of these updates are normally applied between 7 PM and 11:00 Pm and its a very long window to put all my server monitor in a maintenance window.

 

Thanks,

Ash

 

Like (0) Reply

Re: How to stop alert storm

1. Create a notification profile that you want to be applied to your systems.

2. Set the "Downtime Notification Delay" to "Notify after two continuous failures"

 

We do this for the default notification profile and have found it to significantly improve the quality of life for our admins. This means that you do not get alerts for restarting services, but still get alerted if the service fails to restart, which is awesome.
We also have a "Critical Devices" profile, which notifies immediately. 

Like (1) Reply

Re: Re: How to stop alert storm

Hi Brose,

Thank you for sharing your thoughts.

Yes, we are already using it that way and still getting lots of alerts. I don't want to go more than two as  critical/downtime alerts may potentially be delayed otherwise.

 

Thanks,

Ash

 

 

 

Like (0) Reply

Was this post helpful?