In the notification profile set the "Downtime notification delay" to "Notify after five continuous failures". With this setting you will get notification only after 5 continuous alerts (assuming your polling interval is 5 min you will get the alert after 25 min). "Escalation Settings" is meant for escalating an alert to an Escalation group if a downtime is not resolved for a certain period of time. Here the 20 min value for "Duration of downtime before escalation" will send an escalation alert if the status of the server is down for more than 20 min.
However, keep in mind that setting "downtime notification delay" will send alert only after 25 min (in case 5 min polling). If you are dealing with production servers and need alerts instantly its not a good practice to set it, the reason being you will get a genuine alert only after 25 min which will be too late. Your SLA for that server may be affected.
If you know you are restarting a server or doing some maintenance work, you can mark the server as maintenance. By doing this we won't send you any alerts during the maintenance period and your SLA will not be affected.
Product Manager, Site24x7