An interesting day in #Barcelona it was...

I expected it to be one as I was scheduled to do a presentation with the "Autoritat Catalana de Protecció de Dades" on the risk of S3 buckets and other data leaks. For me it was a special presentation because the first time in my life I would do a presentation in Spanish. Put a lot of preparation into it but was nervous anyway. But it went fine and we wrapped it up at about 12:00 local time.

As we (Directora Meritxell Borràs i Solé and me) were debriefing, suddenly the lights went out. I expected a small localized hiccup, so we didn't think much about it and started our journey to the scheduled lunch. A taxi was waiting for us.

1/7

On the way to the taxi we got the news, the outage was city wide. That was giving me bit of a worry for our lunch as I don't expect restaurants to keep working without electricity. This turned out to be right and wrong at the same time.

Traffic in Barcelona is a mess at best times and these weren't. But a total collapse was avoided as most traffic lights kept working, a thing a reliability enthusiast like me noted positively. But I also saw a fraying of the mobile network that frankly shocked me. I would have expected problems, but not that soon. During our 15min drive to the restaurant I had everything from excellent connectivity to no reception at all.

By the time we arrived at the restaurant, it was clear that something big was happening. We had reports of outages from Portugal, all over Spain and from Southern France at well.

2/7

As it turned out (after four floors upwards on ramps), the restaurant was completely out of commission. While they still had gass to cook, no light made the kitchen inoperable and the beverage dispenser were out of commission as well. So we went four floors down again.

At that point I presented my opinion that we would be dealing with hours of outage. Small outages can usually be fixed quickly, but nation wide is a totally different game. I said that I would be very happy, if we had electricity before sundown. I recommended to my hosts to make their way home as traffic would not get better and that I needed not to be taken care of. My hotel was about a mile away from the restaurant so we split up. This was about 13:30.

At that point I started some actions of my own:

  • Phone was set to power saving mode and brightness reduced to "barely legible"
  • At the next open store I bought non-alcoholic drinks and food to get me through two days
  • I asked my colleagues in the SOC to go to an higher alert level

I am kind of a pessimist in such things...

3/7

At that point I had only terrible mobile phone coverage. My evaluation showed:

  • I could do outgoing calls in terrible quality, but not receive incoming calls
  • Text messages could neither be sent nor received
  • Internet was completely down

I would love to read the post mortem from the mobile phone companies. That should not have happened within the first hour of the outage and is a big "no no" from my PoV.

As I was already three weeks in Barcelona, I noticed an unusual pedestrian traffic pattern. There were a lot more people on the streets than there were usually at this time of day. All the large stores and restaurants had closed and the patrons were pushed to the streets. People were no longer using their phones in the usual way. Usage in unsual ways was mostly shouting or staring frustrated at it.

4/7

Due to my ruined knee, it took me quite some time to get to the hotel. I arrived at about 14:30 there. At the hotel rumors were flying: Marocco, Italy and Belgium were said to be down as well and someone even said that Ireland was affected.

Connectivity was practically zero at that time. Nobody could get any web site.

I climbed 8 floors up to the rooftop bar (and explored the emergency stairs in the process), expecting the best chances for any reception there. To my surprise, the rooftop bar was operating. There were no warm dishes, but most drinks were available and cold snacks. My kudos to the staff and I will fill the tip jar very generously.

I managed to get a voice call through to @isotopp (in about 7 attempts) to get a rough briefing. That didn't look good. Expected time to recovery was 6-10 hours. Especially 10 hours would have been very bad because that meant no power till past midnight.

Over the next two hours, I scanned the WIFI spectrum. And it was awing. My hotel is located about 50m from "Las Ramblas" and usually I could get more APs than my phone could show. Now there was NOTHING AT ALL.

During that time a hotel about 400-500m from mine seems to have brought up their emergency power generator and their free WIFI went up. I managed to connect to it twice for about 5min during the next 2 hours. Whoever operates the SSID "Liceu_Opera" in #Barcelona. You have my thanks and respect.

5/7

At about 16:30 I noticed a significantly improved mobile network. I could get Internet through 5G again. This was a great relief and I tried to use the service as much "text only" as possible.

I went to stroll through the streets and check the connectivity down there. On the streets it was still pretty bad. Also I noticed a significant change in the composition. There were far less women on the streets and there were a lot more young men in small groups looking bored. If the outage persisted after sundown, looting would not surprise me and I discussed my observations with the hotel staff. It turned out they were way ahead of me concerning that curve and they had quietly started preparing for it.

Another observation was: if I am ever short on hardy men and women, I would recruit them among the small store operators and tapas bar staff in Barcelona. They kept the city operating, come what may. They had no light, no cash register, no credit card terminals, but they kept going. Food was improvised and payments were handled cash only.

6/7

At 17:30 I was back at the hotel and in my room. Around 18:50 the lights came back on. One could hear cheering in the streets through the closed window. The mood on the streets shifted immediately. My thoughts are with the people in Madrid and Lisbon who reportedly still are in the dark.

Whoever worked on repairing the damage, you have my thanks.

In order to be able to sleep, I have to get the thoughts out of my head, therefor this post.

Good night, an interesting day is ending

7/7

@masek
Thanks for the impressions. Keep my fingers crosses that the power will stay on and that the whole grid is back tomorrow. The Portuguese energy company REN point to induced atmospheric vibrations as the cause.

https://www.bbc.com/news/live/c9wpq8xrvd9t?post=asset%3Addda9592-0346-4fe8-a17a-2261efc1ba5b#post

There will be much to learn from the postmortem analysis.

Spain rules out cyber-attack as cause of power outage as travel chaos continues - live updates

Portugal's prime minister also says there is no indication of a cyber-attack, as officials investigate what caused the power outages.

BBC News

@0815 @masek

So...solar flare? Or something else?

@MissGayle @0815 We will have to wait. Still would not rule out anything yet.