Mastodawn

walfischbucht 3d ago

An interesting day in #Barcelona it was...

I expected it to be one as I was scheduled to do a presentation with the "Autoritat Catalana de Protecció de Dades" on the risk of S3 buckets and other data leaks. For me it was a special presentation because the first time in my life I would do a presentation in Spanish. Put a lot of preparation into it but was nervous anyway. But it went fine and we wrapped it up at about 12:00 local time.

As we (Directora Meritxell Borràs i Solé and me) were debriefing, suddenly the lights went out. I expected a small localized hiccup, so we didn't think much about it and started our journey to the scheduled lunch. A taxi was waiting for us.

1/7

Martin Seeger 3d ago

On the way to the taxi we got the news, the outage was city wide. That was giving me bit of a worry for our lunch as I don't expect restaurants to keep working without electricity. This turned out to be right and wrong at the same time.

Traffic in Barcelona is a mess at best times and these weren't. But a total collapse was avoided as most traffic lights kept working, a thing a reliability enthusiast like me noted positively. But I also saw a fraying of the mobile network that frankly shocked me. I would have expected problems, but not that soon. During our 15min drive to the restaurant I had everything from excellent connectivity to no reception at all.

By the time we arrived at the restaurant, it was clear that something big was happening. We had reports of outages from Portugal, all over Spain and from Southern France at well.

2/7

Martin Seeger 3d ago

As it turned out (after four floors upwards on ramps), the restaurant was completely out of commission. While they still had gass to cook, no light made the kitchen inoperable and the beverage dispenser were out of commission as well. So we went four floors down again.

At that point I presented my opinion that we would be dealing with hours of outage. Small outages can usually be fixed quickly, but nation wide is a totally different game. I said that I would be very happy, if we had electricity before sundown. I recommended to my hosts to make their way home as traffic would not get better and that I needed not to be taken care of. My hotel was about a mile away from the restaurant so we split up. This was about 13:30.

At that point I started some actions of my own:

Phone was set to power saving mode and brightness reduced to "barely legible"
At the next open store I bought non-alcoholic drinks and food to get me through two days
I asked my colleagues in the SOC to go to an higher alert level

I am kind of a pessimist in such things...

3/7

Martin Seeger 3d ago

At that point I had only terrible mobile phone coverage. My evaluation showed:

I could do outgoing calls in terrible quality, but not receive incoming calls
Text messages could neither be sent nor received
Internet was completely down

I would love to read the post mortem from the mobile phone companies. That should not have happened within the first hour of the outage and is a big "no no" from my PoV.

As I was already three weeks in Barcelona, I noticed an unusual pedestrian traffic pattern. There were a lot more people on the streets than there were usually at this time of day. All the large stores and restaurants had closed and the patrons were pushed to the streets. People were no longer using their phones in the usual way. Usage in unsual ways was mostly shouting or staring frustrated at it.

4/7

Martin Seeger 3d ago

Due to my ruined knee, it took me quite some time to get to the hotel. I arrived at about 14:30 there. At the hotel rumors were flying: Marocco, Italy and Belgium were said to be down as well and someone even said that Ireland was affected.

Connectivity was practically zero at that time. Nobody could get any web site.

I climbed 8 floors up to the rooftop bar (and explored the emergency stairs in the process), expecting the best chances for any reception there. To my surprise, the rooftop bar was operating. There were no warm dishes, but most drinks were available and cold snacks. My kudos to the staff and I will fill the tip jar very generously.

I managed to get a voice call through to @isotopp (in about 7 attempts) to get a rough briefing. That didn't look good. Expected time to recovery was 6-10 hours. Especially 10 hours would have been very bad because that meant no power till past midnight.

Over the next two hours, I scanned the WIFI spectrum. And it was awing. My hotel is located about 50m from "Las Ramblas" and usually I could get more APs than my phone could show. Now there was NOTHING AT ALL.

During that time a hotel about 400-500m from mine seems to have brought up their emergency power generator and their free WIFI went up. I managed to connect to it twice for about 5min during the next 2 hours. Whoever operates the SSID "Liceu_Opera" in #Barcelona. You have my thanks and respect.

5/7

Martin Seeger 3d ago

At about 16:30 I noticed a significantly improved mobile network. I could get Internet through 5G again. This was a great relief and I tried to use the service as much "text only" as possible.

I went to stroll through the streets and check the connectivity down there. On the streets it was still pretty bad. Also I noticed a significant change in the composition. There were far less women on the streets and there were a lot more young men in small groups looking bored. If the outage persisted after sundown, looting would not surprise me and I discussed my observations with the hotel staff. It turned out they were way ahead of me concerning that curve and they had quietly started preparing for it.

Another observation was: if I am ever short on hardy men and women, I would recruit them among the small store operators and tapas bar staff in Barcelona. They kept the city operating, come what may. They had no light, no cash register, no credit card terminals, but they kept going. Food was improvised and payments were handled cash only.

6/7

Martin Seeger 3d ago

At 17:30 I was back at the hotel and in my room. Around 18:50 the lights came back on. One could hear cheering in the streets through the closed window. The mood on the streets shifted immediately. My thoughts are with the people in Madrid and Lisbon who reportedly still are in the dark.

Whoever worked on repairing the damage, you have my thanks.

In order to be able to sleep, I have to get the thoughts out of my head, therefor this post.

Good night, an interesting day is ending

7/7

Martin Seeger 3d ago

Please don’t take my post as „It’s all fine“.

Millions are still without electricity and significant parts of the country will most likely have 24h or more of outage.

This was released 15 hours after the start of the outage:

Red Eléctrica, Spain’s national power grid operator, said in a statement the company had restored more than 20% of the Iberian Peninsula’s demand for electricity, with power “progressively restored” across 45% of Spain’s power grid.

@masek
Glad things ended well and congratulations on your talk. Thanks for sharing, always interesting to hear first person accounts from a knowledgeable spectator.

Patrick Dersjant RCX 3d ago

@masek Thanks for your report!

SpaceLifeForm 3d ago

Thank you for the excellent report.

Kudos for thinking "text only". It really helps.

@masek thank you, very interesting and insightful... (a bit chilling the looting part, so quickly!) good luck!

Martin Seeger 3d ago

@miracorvino My trust in humanity is very limited, a side effect from my job.

@masek
Thanks for the impressions. Keep my fingers crosses that the power will stay on and that the whole grid is back tomorrow. The Portuguese energy company REN point to induced atmospheric vibrations as the cause.

https://www.bbc.com/news/live/c9wpq8xrvd9t?post=asset%3Addda9592-0346-4fe8-a17a-2261efc1ba5b#post

There will be much to learn from the postmortem analysis.

Spain rules out cyber-attack as cause of power outage as travel chaos continues - live updates

Portugal's prime minister also says there is no indication of a cyber-attack, as officials investigate what caused the power outages.

BBC News

@0815 @masek what exactly are "atmospheric vibrations" in this context?

It sounds like this was radio activity (think CMEs), not air "vibrations."

I suspect a mistranslation?

Martin Seeger 3d ago

@draeath @0815 I am unsure and would wait a bit. There are some things that still have me wondering. Wouldn't rule out an attack yet.

Miss Gayle 3d ago

So...solar flare? Or something else?

Martin Seeger 3d ago

@MissGayle @0815 We will have to wait. Still would not rule out anything yet.

ubrummack 3d ago

@masek Good night and thank you for telling us. Hope everything is ok.

@masek in my experience in barcelona the subway was the only reliable way of transportation. Since this probably also wen‘t out of service a lot of people mist have gotten a lot of sunlight on their way through the city. Mobile coverage outage concerns me, bcn has due to MWC one of the best city wide mobile networks. I could do a videoconference from the fair to the plane passing the subway and airport security (just put your phone on the security machine) without any problem.

mirabilos 3d ago

@masek @nido009 so have you seen the L4S demo at MWC? (Vay/DTAG/Ericsson, teledriving a car)

@masek TY for sharing your detailed experience. I suspect you handled it much better than many. Cheers!

Nils Hammer 3d ago

@masek Good night an thanks for the report. ¡Que duermes bien!

Martin Seeger 3d ago

@nshr Sleep didn't come easy. Sensed much disquiet everywhere.

KielKontrovers Blog 3d ago

@masek do they not yet know the cause or are they not telling us. my idea of a power grid would be that you have information where a failure happens and when? so like why did it happen in three countries? I have heard about Texas that their grid was very weak, but thought Europe was better prepared. Thngs like this on this scale should not happen, or at least I expect the officials should know why it happened.

hackbyte 3d ago

@kielkontrovers I can imagine pretty good that in such a huge emergency, research about the fault source is secondary to getting services backup A.S.A.P..

Besides that, some sources point to a rare phenomenon called induced atmospheric vibrations as a cause.

Btw, portugal has _ONLY_ connections towards spain. So they had to be completely independent ..... or - as it happened - will go down with the spanish power grid..

Which third country went dark too? I didn't read about any.

Christoph Petrausch 3d ago

@hackbyte
AFAIK parts of portugal and southern france where affected as well. The european grid is very interconnected and in cases like this local grid operators have to act fast to stabilize their local grids. It's a bit easier for those "far away".

My local PV inverter logs the network frequency observed and it went as low as 49,85Hz. If the frequency had dropped below 49,80 Hz, load shedding would also have occurred here in Germany.

@kielkontrovers

Martin Seeger 3d ago

@hackbyte @kielkontrovers I do not rule out anything yet. Some things are very unclear and strange. I consider an attack still a possible cause.

Markus LaPalma 3d ago

@masek THANKS A LOT! Here on the Canary Islands we had a few internet hick ups. Stay safe!
¡Y buenas noches - bona nit!

Wokebloke for Democracy 3d ago

@masek
Martin, thanks for your account of events. I doubt we'll get this kind of detail in any news story.

Cyril Brulebois 2d ago

@masek Wow, thanks for sharing, that was an impressive read.

@masek I was in the same area and around that time notably more police presence was in the streets. The situation seemed relaxed. At the same time, people from the city in neon vests were also checking in with store front owners.

Martin Seeger 3d ago

@dahie Yes, the urban police was very present.

Christoph Petrausch 3d ago

But that is a common practice in that case. First of all to show presence to prevent looting etc, but also at least in Germany, the police has radios powered by an independent grid, where they can call in the fire brigade or medical services if needed.

Martin Seeger 3d ago

@hikhvar @dahie Barcelona is more difficult than most German cities. Extremely narrow road in the old town. Police can only get through on motorcycles, I have no idea how the fire brigade can work there.

Christoph Petrausch 3d ago

I'm sure they have their plans and tactics how to excute the job their. But with every emergency, the quicker you are on site, the better the help. Therefore it's crucial that police is patrolling the street offering points of contacts to get emergency services in the first place.

@hikhvar @masek yeah and as I said, it wasn't only the police. Civil servants by the city were also checking around. Must have been stressful, but I met nobody aggressive, neither official nor regular people. Most were relaxed, a bit confused. Shop owners unsure what do to, especially later when they were supposed to close shops, but couldn't die to electronic shutter gates and lock mechanisms.

Dr Sorcha O’Brien 3d ago

@masek What was the situation with cash machines? I’d be so stuck if they were out, too, I only tend to have about €5 in cash in my wallet these days.

Martin Seeger 3d ago

@_sorcha I didn't try, but I would expect that none of them worked at all. AFAIK they don't have emergency power.

I think, this caused a lot of people a lot if headaches yesterday. My host asked me, if I needed cash but I was well equipped.

There is a reason, why I always have some cash at least in the hotel safe and about 50€ with me.

Björn 🇪🇺 Starkimarm 3d ago

My parents taught me to always have 50 with me for gasoline. Nowadays it wouldn't fill a tank, but it's a good tip anyway. A folded note fits nicely between phone and case.

@masek Interesting to read, that your mobile Network improved over time. I was in Faro (PT) at outage and my network worked mostly fine until roughly 16:00.

After that, reception was still full 5G, but without any Data throughput.

I kept my phone on airplane mode most of the time (whilst checking every once in a while). My Wife’s 4G worked (while slow) all of the time. (In Germany I’m on T-Mobile, she’s with O2)

Power came back at 22:18 local time with cheering on the streets.

@masek Also interesting was, that my wife received a text message from portugals emergency services to stay calm and not call 112 for information and only in emergencies, that one didn’t reach me.

After power came back, I could immediately use my network again (might have been working before, but I settled with being in Airplane Mode).

I experienced the mood on the streets way more relaxed than you described. But: I was in a tourist spot. The biggest problem there was atms and cash 🫣

Truth Matters 1d ago

@sico93 @masek i was less fortunate in Portimão as mobile Internet stopped working already at noon. I managed to call someone in Germany but the line dropped after some seconds.

Energy was finally back only at 23:40. What a day 😠

2 super markets (Continente, Auchan) worked perfectly, including card payments 😊People seemed to be chilled, despite 300m queue organized by security staff and police at the entrance of the super market. 2 ATMs inside super market also worked.

clubchonky 3d ago

@masek @isotopp
for those gnarly no uplink situations I built atacama for any android. instagram like p4p sharing without uplink over wlan,phone ap share.
9mb no google no bs, try out.

https://wormhole.app/ZMjzPY#zjhjKOMa5m--c41yFmL6KA

Wormhole - Simple, private file sharing

Wormhole lets you share files with end-to-end encryption and a link that automatically expires.

Wormhole

clubchonky 3d ago

@masek @isotopp
this is something like secure scuttlebutt (fully decentralized) but million times faster and better. shareware, pay what you can with XMR

Alecia Batson (she/her)3d ago

@masek @isotopp Ah! #opera ftw. The Liceu Opera House (@liceu) must be nearby. Rest well. Thanks for sharing your experience.

@masek I used to work in civil resilience in part of the UK government and we ran an exercise in total grid outage.

1 to 2 hours maximum is all you can expect from cell sites. They have a small battery designed to cover 15 minute blips to the power grid, nothing more.

Remember it has to run both the radios and the backhaul.

Martin Seeger 3d ago

@skrrp Thank you for that input. I had something in my mind about federal regulation being 4 hours for wireless services in Germany and 24 hours for landline. But that may have changed.

Korenwolf 3d ago

@skrrp @masek and even if the cells remain up the core mobile network may have failed if the generator backups aren't as good as everyone was expecting, or if the kit keeping the inter-data center links gets flaky that can cause cascading failures within the core as well.

Christoph Petrausch 3d ago

@masek Also in germany Internet is gone really really quickly if the power went down. Especially for discount carrieres like O2. We have some regular small power outages here in the village (most 30m-2h), as the local network is strained by Tiefbau (Deutsche Glasfaser) and quick adoption of PV.

The thing going away the first is the Internet for low cost carriers. For all others, the bandwidth will go down to pre-LTE levels.

I suspect this is some optimization for the batteries.

Martin Seeger 3d ago

@hikhvar At home, I have three carriers (Fiber, DSL, Mobile) with an automatic fail-over. I hope at least one will work.

With the PV, I can use my battery as big backup power battery.

Christoph Petrausch 3d ago

@masek From my experience, non of them work if there is a complete power outtage in the village. Both Fiber and vDSL are immediatly dead. Mobile becomes shitty very soon.

I have a PV battery with backup capabilities. Often I use that to get me some coffee and enjoy the time for some local-non-internet programming work. Everything else is impossible.

Christoph Petrausch 3d ago

@masek
As both residential VOIP and mobile is gone so often, we as voluntary fire fighters man some POI with our cars, so the people have at least a point where they can go and find help in case of emergencies.

Björn 🇪🇺 Starkimarm 3d ago

And that's why something like StarLink would be a great backup line. Too bad there are (moral/ecological) constraints.

Heleen Kuiper 3d ago

@hikhvar @masek what is PV?

Christoph Petrausch 3d ago

German abbreviation for photovoltaics.

Heleen Kuiper 2d ago

@hikhvar @masek and that is a solar panel?

Christoph Petrausch 2d ago

@heleenkuiper
Yes, I have a home solar panel installation and a battery which is managed by my solar inverter. The inverter can draw power from the battery to feed my house, if there is no sun. What is more, this even works, if the grid is offline. Many older and cheap inverter/battery combinations will not generate any power if the grid is offline as it was in spain yesterday.

Heleen Kuiper 2d ago

@hikhvar @masek thanks for the info!