Good evening everyone,
I’m happy to announce that Anime Herald is back online after a long downtime today. However, due to the circumstances and time, I would like to give you a candid explanation of the events that led to our downfall and their overall impact.
what happened
At 5:45 AM ET, we received a report from the monitoring host informing us that Anime Herald resource usage was high. Two minutes later we received notification that the site was completely shut down. After logging into the instance, we found it completely unresponsive, preventing us from performing typical diagnostics. We initiated the restart at 6:30 AM ET.
After restarting, we were unable to get the site back online. According to logs, a large-scale sustained attack from a generative artificial intelligence crawler bot disrupted critical system updates and forced the server into a functionally inoperable state. Attempting to restore from backup proved ineffective as the bots immediately locked the IP address and caused an almost immediate shutdown.
I should point out that we actively block GenAI bots because we believe the system Actively steal From independent creators, anything they create is Built on massive theft. We have codified this in our contributor guidelines and made it public on our social media accounts.
So, we have all the protections enabled in the Cloudflare instance, and we’re taking every step possible to make sure they don’t steal our information. However, the number of robots increases Spoof user agent stringand rotate agents to avoid such measures. These rogue robots actively ignore instructions such as robots.txt, thus breaking the social contract that has existed since the birth of the World Wide Web.
Our course of action
At 7:42 AM ET we decided to deploy a new VPS with a new IP address and restore our existing backups. However, doing this means taking the time to restore more than fifteen years of images, articles, and other resources that we have accumulated over the years. As a result, our recovery ended up taking a few hours, resulting in us being offline until 12:37 PM ET.
Since our old VPS format was deprecated and no longer suitable, we needed to upgrade. We are now running a system with twice the RAM and storage, and multiple dedicated CPU cores. This will have a more expensive knock-on effect, but hopefully it will also prevent us from being overwhelmed for the foreseeable future.
Anime Herald, as a publication, is important to me – I want to do everything I can to make sure it’s still around five, ten, twenty years from now. I personally apologize for our downtime and thank you for the trust you have placed in us over the past fifteen years.
greeting,
Samantha Ferreira