Doug MadoryJul 15
Doug Madory

Just to clear up some misinfo circulating, a BGP hijack was not the cause of
Cloudflare DNS going down today.

At 21:51 UTC, Cloudflare (AS13335) withdrew both 1.1.1.0/24 and 1.0.0.0/24 for an unknown reason.

I suspect AS4755 was always announcing 1.1.1.0/24, when CF went away, it leaked a bit (i.e. "%2").

https://infosec.exchange/@GossiTheDog@cyberplace.social/114854023690856642

Infosec Exchange

Jul 15 at 12:26amWeb
Show threadDoug MadoryJul 15
Also, the IPv6 route for Cloudflare's public DNS resolver (2606:4700:4700::/48) also went down at the same time as 1.1.1.0/24 and 1.0.0.0/24. It wasn't hijacked.
Show threadDoug MadoryJul 16

Here’s the @Cloudflare write-up with a description of what caused the outage. It was caused by an internal error not a BGP hijack, but we already knew that.

https://blog.cloudflare.com/cloudflare-1-1-1-1-incident-on-july-14-2025/

Cloudflare 1.1.1.1 Incident on July 14, 2025

On July 14th, 2025, Cloudflare made a change to our service topologies that caused an outage for 1.1.1.1 on the edge, resulting in downtime for 62 minutes for customers using the 1.1.1.1 public DNS Resolver as well as intermittent degradation of service for Gateway DNS. We’re deeply sorry for this outage. This outage was the result of an internal configuration error and not the result of an attack or a BGP hijack. In this blog post, we’re going to talk about what the failure was, why it occurred, and what we’re doing to make sure this doesn’t happen again.

The Cloudflare Blog
Show threadMartin Boller     Jul 16
@dougmadory Besides privacy, another reason to not use these centralized resolvers.
Show threadJohn KristoffJul 15
@dougmadory 1.1.1.0/24 and 1.0.0.0/24 have had valid ROAs. If networks accepted announcements from another origin, that's kind of on them. They would have been rejected by anyone doing #RPKI ROV.
Show threadDoug MadoryJul 15

Yeah, there are ROAs for 1.1.1.0/24 and 1.0.0.0/24, however, 6453 accepted 1.1.1.0/24 from 4755 and passed it on so they aren't validating routes from 4755.

It didn't go too far. It was only the absence of 13335 that allowed it to propagate at all.

Show threadMax ResingJul 15

@dougmadory - Just wondering, what could cause Tata to actually announce the 1.1.1.0/24?

BGP.tools reveals that there are actually paths where 4755 or 6453 is the second-last hop to 13335.

For instance: 3320, 6453, 13335

Is it that Tata hosts instances for Quad 1, and failure response mechanisms ensure the prefix is still announced? Somewhat as a resiliency effort?

Show threadDoug MadoryJul 15
@resingm my guess is that 4755 always announces 1.1.1.0/24 for some internal routing and it never propagates anywhere because of 1) 13335’s huge peering base, and 2) RPKI ROV enforcement.

1.1.1.0/24 is used internally in a lot of places. There are local hijacks happening all the time for that range.

Since there is a ROA (and the fact that 13335 originates it) the hijacks don't go very far.
The main problem was AS13335 withdrawing 1.1.1.0/24 and 1.0.0.0/24.

Only 13335 could take down these routes.
Show threadrk: it’s hyphen-minus actuallyJul 15
@dougmadory
Show threaddrop(_paddi)Jul 15
@dougmadory I'm not that experienced in BGP, so please excuse the potentially stupid question, but why did TATA announcing 1.1.1.0/24 not cause issues even when Cloudflare was still announcing it?
Show threadDoug MadoryJul 15

@paddi my guess is that 4755 always announces 1.1.1.0/24 for some internal routing and it never propagates anywhere because of 1) 13335’s huge peering base, and 2) RPKI ROV enforcement.

1.1.1.0/24 is used internally in a lot of places. There are local hijacks happening all the time for that range.

Since there is a ROA (and the fact that 13335 originates it) the hijacks don't go very far.

The main problem was AS13335 withdrawing 1.1.1.0/24 and 1.0.0.0/24.

Only 13335 could take down these routes.

Show threaddrop(_paddi)Jul 15
@dougmadory Thanks!
Show threadJohn KristoffJul 15

@dougmadory If it always was, I don't think it was being seen anywhere. The first time rrc00 appears to record an announcement originating from 4755 is in updates.20250714.2150.gz.

Note, I spot checked a few updates to narrow it down, not every one of them.

Show threadDoug MadoryJul 15
@jtk yeah it might not have been seen. Does RIPE have collectors downstream of 4755? If not, then they wouldn’t see it as 6453 would probably accept the route from 13335.
Show threadDoug MadoryJul 15
@jtk or is there a scenario where 4755 prioritized 1.1.1.0/24 from 13335 over its own static version? Then when 13335 withdrew it, it reverted to the internal route and advertised it externally.
Show threadJohn KristoffJul 15

@dougmadory I really can't afford to spend a lot of time on this. :-)

But I'll also note that community tags associated with the 1.1.1.0/24 prefix from 4755 seemed to suggest it was a customer route at London Telehouse North, a location Cloudflare and Tata are both at.

I was wondering if Cloudflare deploys anycast instances to certain nets and Tata had one, but leaked their own internal announcement. Seems like a stretch, but yet another possibility?

Show threadDoug MadoryJul 15
@jtk we’ll have to read their blog post, which should be out any minute now. 😁
Show threadRootWyrm 🇺🇦Jul 15

@dougmadory but but NaziFlare and it's employees are smarter than everyone on the internet! Look at their lava lamps and how quirky they are! They would never do something so grossly incompetent or sloppy!

(Yes, 4755 is announcing a ton of space it doesn't own and bogons. As usual.)