Brightspot supports several big names in the media, all over the world, and the elections in the U.S. are a big deal for a lot of those customers 鈥� so they鈥檙e a big deal for us, too, and not just for the politics. Last night some of us were spending more time looking at graphs than at the news, and here鈥檚 a little drama we saw unfold.
Starting at about 7 p.m. (U.S. Eastern), we detected a distributed denial of service, or 鈥淒DoS,鈥� against one of our customers. A DDoS is a lot of different computers 鈥� and sometimes other devices 鈥� sending connection requests to a website all at the same time, trying to overwhelm the website鈥檚 servers and break the website.
While our customer鈥檚 site was serving record-breaking pageviews, we saw a new attack happen 鈥� instead of Edgar鈥檚 300,000 requests in 4 minutes, this one was 37 million requests in one minute, which is a lot. (On a typical day, this site may get about 180,000 requests per minute.)
The 7 p.m. attack lasted about 4 minutes and sent about 300,000 requests at us, then it went away. But then it came back a few minutes later, did the same thing, and went away again. It did this six times, each time about the same duration and about the same volume. It was an ineffective attack, deflected by the security infrastructure. We named it Edgar. I made a picture of him. Or ChatGPT did, anyway.
This story is not about Edgar; it鈥檚 about Edgar鈥檚 much bigger brother, who we didn鈥檛 name.
At a little before 9:30 p.m., while our customer鈥檚 site was serving record-breaking pageviews, we saw a new attack happen 鈥� instead of Edgar鈥檚 300,000 requests in 4 minutes, this one was 37 million requests in one minute, which is a lot. (On a typical day, this site may get about 180,000 requests per minute.)
There were other attacks throughout the night and into today, but none of them were nearly as big as the 9:30 attempt and, I hasten to say, the website itself was fine the whole time.
It takes a lot of devices to generate that many requests in a minute; for an attack like Edgar it can just be a handful of servers or a dozen personal computers, but 37 million requires either a sizable botnet (a botnet is a collection of hacked devices) or several small ones, working together. Or both.
After failing to find anything in common among all the requests that came in at 9:30 (IP Address, ASN, user agent, geographic info, etc.), we took a look at something called a JA4 fingerprint. The JA4 fingerprint is a sophisticated way to identify any software that makes secure connections, no matter where the software is or what it鈥檚 doing. We found that almost all of the 9:30 attack came from a single fingerprint, across hundreds of devices, all over the world. A single, very large, botnet.
Depending on what tools you have, you can actually block a particular JA4 fingerprint. We didn't feel it necessary to do so at the time, because our automated security services identified and mitigated the requests before any impact was seen on the site. Because we didn鈥檛 block it, Edgar鈥檚 big brother managed to attack two more times, once around 2 a.m. and once around 6 a.m. But those attacks were much smaller; only about 3 million requests. Still a good amount of traffic, but nothing like the first one. What happened?
It looks to me like a lot of the individual devices in the botnet disappeared sometime between 9:30 p.m. and 2 a.m. The fingerprint remains the same, and some of the devices are still there in all three attacks, but the drop in volume (and some other evidence) suggests that the botnet took a pretty big hit. I鈥檓 guessing that big chunks of the botnet were detected and neutralized by security teams at network and cloud providers all over the world. I鈥檒l never know, and will probably never get to meet the folks who did it. But it worked, the whole thing worked, the way it鈥檚 supposed to. And you probably never hear about that.
So to those people: hey, thanks.
