So, three nameservers out of seven for the pubmed.ncbi.nlm.nih.gov are broken. If you try to look at the web site, you stand a 3/7 chance of encountering something that is broken.

The nameservers have the following addresses:

$ host ns.nih.gov
host ns.nih.gov has address 128.231.128.251
$ host ns2.nih.gov
host ns3.nins2.nih.gov has address 128.231.64.1
$ host ns3.nih.gov
ns3.nih.gov has address 165.112.4.230

We can see that the addresses for the first two both start with 128.231 and might guess that they are relatively nearby to each other. This can be confirmed using traceroute. Go ahead and open a terminal and try it out!

traceroute 128.231.128.251
traceroute 128.231.64.1

for me, the paths, the sequence of routers between me and those addresses look broadly the same.

The other one is different,

traceroute 165.112.4.230

it goes a totally different way. So whatever problem is happening is not limited to a single site or datacentre.

Now we can turn to RIPE for some help.

Let us inspect the last address, https://stat.ripe.net/widget/bgplay#resource=165.112.4.230

This is very peculiar. Notice how that network is announced from two different places. And there seems to be a partition, they are not (visibly) connected to each other! This is not normal. I attach a screenshot. There is also some volatility, shown as path changes, the yellow bars at the bottom.

Looking at ns.nih.gov, https://stat.ripe.net/widget/bgplay#resource=128.231.128.251

this appears more coherent, but volatile. The network was withdrawn about 45 minutes before I took the second screenshot, about an hour ago. And then re-announced.

RIPE's BGPPlay is very nice, you can time travel and replay this incident as observed from the Internet. It takes a bit of background knowledge to decode what is going on though.

Someone is doing networking... Badly...

@researchfairy

As of this morning, maybe late yesterday depending on your timezone, PubMed should work again, and cancer.gov.

$ host -t spa ncbi.nlm.nih.gov
ncbi.nlm.nih.gov has SOA record dns1-ncbi.ncbi.nlm.nih.gov. systems.ncbi.nlm.nih.gov. 2025030201 32400 5400 2419200 82800

The domain ncbi.nlm.nih.gov is now handled only by the NLM nameservers without the top level NIH ones in the mix:

$ host -t ns ncbi.nlm.nih.gov
ncbi.nlm.nih.gov name server lhcns1.nlm.nih.gov.
ncbi.nlm.nih.gov name server lhcns2.nlm.nih.gov.
ncbi.nlm.nih.gov name server dns1-ncbi.ncbi.nlm.nih.gov.
ncbi.nlm.nih.gov name server dns2-ncbi.ncbi.nlm.nih.gov.

Interestingly, cancer.gov nameservers are the NIH ones,

$ host -t ns cancer.gov
cancer.gov name server ns.nih.gov.
cancer.gov name server ns2.nih.gov.
cancer.gov name server ns3.nih.gov.

and it seems to work just fine now.

So, on Saturday, someone broke part of the NIH network, which took down the NIH nameservers, apparently unintentionally given that they fixed it by late Sunday.

This just looks like a fairly high profile configuration error that took a while to fix. What they were actually trying to do we do not know...

Big Balls' Fat Fingers? Who knows?

@researchfairy @briankrebs