Operations: Update on Service Disruption
UPDATE: 1825 pacific time, Nov 12
Services are normal. Read more about it in this post.
UPDATE: 2319 pacific~time, Nov 11
Services have been near normal in the last 2 days. We are rolling in infrastructure changes to make it better to handle attacks.
An update on our service.~We were~posting updates over the last many days on our forums.~The good news is services have been near normal for the past couple of days.
We are posting this~update on our blog too as~the forums sub-domain had to be brought down for~technical reasons by the Zoho Discussions Product Team.~I am reproducing the updates here for your reference.
Frequently asked questions on the attacks answered:
1. I am unable to reliably access Site24x7 / Zoho~services this week. What is happening?~
We’re facing a criminal cyber-attack called a DDoS (Distributed Denial of Service) aimed at denying~access~to~genuine users of our services. This started on Nov 4, 2015. Updates are available on our ~@Site24x7Cares and~@Zoho~Twitter handle. You can also follow this blog post.
2. How are you planning to mitigate this attack?
Our network and infrastructure teams have been working~– non-stop~–~on mitigating these~attacks~since~they~started.~We have received and handled many of them already.~We are also working with other experts to erect additional~defenses, using multiple approaches.~
3. When will services be fully restored?
Unfortunately, this kind of attack makes this unknowable. These are criminals using the internet to extort companies and deny service to their paying customers. If past experience of~other companies is any guide, such attacks can last from hours to days.~Rest assured that we are doing our best and working with all the right people to mitigate these attacks and will keep you posted.
4. Is my data safe?
Yes, all your data will remain safe and sound. You may not be able to access your data reliably, given the unpredictability of the multiple attack sequences, but you will not lose any data.
5. Do I need to change my password?
No, you do not need to change your existing password. However, you may experience disruption in accessing services.
~6. Why is forums.site24x7.com not accessible?
The forums domain had to be brought down due to~technical reasons by the Zoho Discussions Product Team.
UPDATE: 1815 pacific time, Nov 9
The attacks continue and we are managing to stay afloat. We are not assuming that the worst is over yet and customers should not either. Stay with us.
UPDATE: 1034 pacific time, Nov 9
Things look quite positive now.
UPDATE: 0847 pacific time, Nov 9
Here is what is happening with our infrastructure. We had several pending updates that were designed to mitigate DDoS attacks. Those updates were scheduled to go live in the next 2-3 weeks. We have had to squeeze them in over the last weekend. That has created a bit of instability in our systems, given the emergency nature of the upgrades themselves.
So while we have taken successful counter-measures against DDoS, we are now working on strengthening our systems. We have traffic rerouted through our secondary data center and that introduces more hops that adds latency (delay) as well. We are working on all these issues right now. Thank you for your patience.
Update: 0650 pacific time, Nov 9
Due to relentless attacks we have taken emergency counter measures. Part of this is to reroute traffic and data, through additional network hops to filter out the attack. This added complexity is making the web client access unstable and slow for customers. We are working on it as we speak.
Services update: 2015 pacific time, Nov 8
Services are up and we are mitigating the attack. We fully expect them to keep coming though and disruptions may yet occur.
New attack update: 1705 pacific time, Nov 8
Zoho services were targeted again on November 8 at 1705 hours pacific time (0115 GMT). Services were down for about 33 minutes and came back up. Many are not reliably up yet. Our teams have been working nonstop over the last 72 hours with various counter measures. We are still at it and are prepared for tough days ahead. We expect more attacks and more service disruption as we get into the work week. Please stand with us at this time.
Zoho under distributed denial-of-service-attack: ~Nov 6
Zoho was~subject to a criminal attack to our networks called a distributed denial-of-service attack (DDoS). This started at 8:15am pacific time on November 4, 2015 and has continued intermittently. The obvious intent is to make Zoho and all of our services unavailable to customers by flooding our servers with bogus requests from multiple sites. The attack was accompanied by threats and a blackmail attempt at extortion to prevent ongoing attacks.
This attack is focused on denying access since it targets the network connections to our servers. All your data are sound and secure, but unfortunately you cannot access it. This is like a crowd of people standing around the entrance to your bank, preventing your entry. Your money is in there and safe, you just can’t get at it until we have them all moved.
Companies like us that have offered services accessed through the internet for many years carefully prepare and expect these attacks. But the attacks are~getting worse and more sophisticated.~In fact, there have been other attacks this very week. Secure email providers like ProtonMail and Runbox have been hit. Many~major banks and businesses with online customers have seen these attacks in the last year.
We’re working round the clock, with service providers, experts, and others to continuously improve our defenses. We are also contacting law enforcement. There is no single silver bullet to fix these issues and we have to work at it diligently and with the right experts behind us. We expect that these attacks will continue, but we also expect to prevail.
In the meantime, things will be rough for you, and you have our deepest apologies. The most unfortunate part is that we cannot tell you exactly when everything will be back to normal. We regretfully ask for your patience in advance, knowing fully well that you rely on us to run your business and to serve your own customers. We are deeply sorry for your trouble.
Please stand by us as we fight this attack. We cannot give in to criminals and embolden them to perpetuate other attacks.~Thank you again for placing your trust in~us.