Today we had a fire alarm in the office. A colleague wrote to a Slack channel 'Fire alarm in the office building', to start a thread if somebody knows any details. We have AI assistant Glean integrated into the Slack, and it answered privately to her: "today's siren is just a scheduled test and you do not need to leave your workplace". It was not a test or a drill, it was a real fire alarm. Someday, AI will kill us.
@tagir_valeev "Dear Munich emergency services, there's a strong smell of smoke in our open office, and the flames are really messing up the colors of my IDE's dark mode. Is this an actual emergency? Looking forward to hearing from you…"
The IT Crowd - Series 1 - Episode 2: Fire!

YouTube
@mhd @tagir_valeev I knew without the preview that this would be a link to the IT crowd episode xD Nice to know that I'm not the only one who still thinks of that! Fire! Exclamation mark!
@ditsch42 @mhd @tagir_valeev It warms my heart to see people referencing that show 😀
@mhd @tagir_valeev
„I hope this e-mail finds you well and unscorched …“
@tagir_valeev *in GLaDOS voice* The Enrichment Center apologizes for this clearly broken test chamber.
@tagir_valeev From an AI point of view it’s the most probable answer, as there are much more „drills“ than „fires“ in the database. Maybe you should connect the assistant to the alarmsystem, so it can check it out. Or maybe… not.
@Matt_999 @tagir_valeev it is still probably a good idea to leave your workplace and go to the emergency meeting point, so you get used to the process, and don't forget to take e.g. shoes, jacket, wallet, keys … with you if the alarm turns out to be a real alarm. Or even have to find out first where the next fire extinguisher or the emergency meeting point is…

Omg so much yes to what @daniel_bohrer wrote. You should even if you know it's a drill actually still leave the building because that's the point of a drill.

The only situation where that's clearly not necessary is: they are inspecting the fire alarm system itself. But that would be communicated very clearly in advance.

@Matt_999 @tagir_valeev

@betalars @daniel_bohrer @Matt_999 the 'siren test' AI referring to is exactly that. It's not a drill. The planned siren test may take several hours of occasional alarms, and it doesn't mean that you should leave the building every time you hear it. We had one a few months ago. Probably AI found that announcement in the chat history and assumed that it is related to today's alarm.
@tagir_valeev @betalars @Matt_999 yes, if it is really only a siren test, all office workers should know about it already in advance. If it's not, then usual emergency protocol applies.

@daniel_bohrer @tagir_valeev @betalars @Matt_999

I do siren tests as part of my job (as well as setting the bells off for the 6 monthly fire drill), if its a bell test I always warn everyone in person (its a small office so not a big problem), any other time they should follow emergency evacuation procedure if the alarm sounds..

@betalars @daniel_bohrer @Matt_999 @tagir_valeev Exactly. And the other rule is: evacuations are atomic. I.e. once started, you evacuate until full completion. No arguments like "but Simon says it's a false alarm, so we can abort the evacuation".

@daniel_bohrer @Matt_999 @tagir_valeev I mean, it's a free break that you can't be penalised for. Might suck if it's bad weather out, but still, it's a break.

Don't dawdle while leaving or you'll be trampled by the smokers. Extra smoke break!!!

@daniel_bohrer @Matt_999 @tagir_valeev Given an alarm can happen at any point, either spares should be available near the emergency exit or one should also ensure never to be too far from such resources.

Yes, this especially includes going to the restroom. Being stuck in -20℃ windy weather without a coat because one didn't bring it with oneself when going is a safety issue.
@tagir_valeev sure the AI bit is bad, but the employee also needs another fire drill since "writing to slack to find out if somebody knows anything" is definitely not what you should be doing. Leaving the building immediately is.
@nd of course, the employee wrote this while already staying at the meeting point outside the building. I did not say that she wrote this before the evacuation.
@tagir_valeev @nd But you did write the original post saying AI will kill us. There is no other reasonable inference with that context.
@TerryBTwo @nd I'm not saying it's killing already. It will be worse.
@tagir_valeev AI might. Being so stupid that they hear an alarm and start a thread about it instead of getting out certainly will.
@TerryBTwo there's no 'instead of' in my post. Please read carefully.
@tagir_valeev As noted-the implication.of "AI will kill us " is precisely that.

@TerryBTwo @tagir_valeev No one would ask this question while shuffling down a crowded stairwell and turn back? Or be in the bathroom and hoping not to have to rush out?

It's weird to assume that people won't be a little lazy when LLMs are so successful BECAUSE people are eager to be lazy. Also weird to focus only on a very specific event as how "AI will kill us" when we already have machines driving cars into people, deleting production databases, encouraging people to kill themselves, etc.

@tagir_valeev how would a conversational « agent » know about a real fire unless it’s somehow hooked up to sensors (in which case, you actually wouldn’t need any AI at all, anyway)…
People are way too ready to give up their thinking ability to chat bots that are fairly good at pretending to be human…
@metacosm nobody asked the AI input at all. It just was configured in the particular channel to answer automatically if it thinks it can help faster than fellow humans (sometimes people actually ask something which was asked before, so AI could be helpful). The configuration will be adjusted after this incident.
@tagir_valeev @metacosm > The configuration will be adjusted after this incident.

On a probabilistic bullshit generator that does its own determination of topics (if it even has that much semantic analysis machinery, which it very well might not). Such that even if it was strictly allow-list based, it would still frequently fail to stay silent.
@tagir_valeev @metacosm You need to stop anthropomorphising LLMs. LLMs do not think! They do not even hallucinate! They just spit out the most probable next tokens from their training set, and the training set is all of the human knowledge plagiarized + all of the human bullshit their crawlers could find on the web! If everyone turns them off, there will be fewer fires in the future (in both senses)!

@tagir_valeev LLM-cosplaying-as-AI has already killed a bunch of people.

It will keep killing.

@tagir_valeev Has Glean been a well evidenced data driven positive impact on the business?

If not, this would be a great opportunity to campaign to remove such tooling that which may not be providing the positive impact the business was intending

I’ve found AI chatbots via Slack has not worked out worked out well for our teams

@rail sometimes, it's helpful when asked explicitly. Automatic answers are indeed annoying. But it looks like it's getting better, so we are still evaluating.

@tagir_valeev

I don't know, man. If I had just received "advice" that might have caused physical harm or even death, I wouldn't just suppress this specific error and happily continue evaluating, because the bot looks like getting better, "otherwise".

I hope accountability for such decisions is well documented. That won't prevent harm from happening, but at least give employees or their surviving relatives a chance with respect to liability issues and compensations.

@rail

@katzenberger @rail the incident was reported to the vendor, and they are looking at it. Of course, such things should be taken seriously.

@tagir_valeev @mathowie Decades ago, I read a story (I want to say on Slashdot?) about a baffling email sent to thousands of employees at a large corporation, that said, simply, “DO NOT LEAVE IT IS NOT REAL.” Ultimately, it turned out that one of their buildings at one location had a false fire alarm, and some well-meaning person emailed everybody to inform them of that fact.

In 2026, we have automated DO NOT LEAVE IT IS NOT REAL, and it's wrong.

@tagir_valeev I'm thoroughly convinced that the AI will kill us with bad directions on the maps and things like what you described!
@tagir_valeev Fake mushroom gathering e-books on Amazon quite probably already have.
@tagir_valeev Just me, but I would not start a thread when my building's fire alarm goes off. I'd go stand in the parking lot.🚒

@tagir_valeev München Calling...

Engines stop runnin', but I have no fear
'Cause London is drownin' and I live by the river

@tagir_valeev
> Someday, AI will kill us.
Hopefully
@mossyfoot @tagir_valeev
I was sure Moss would turn up in this thread :D

@tagir_valeev

Jebus. The fire alarm goes off and their reaction is to ... start a Slack thread?

Not, "get up from desk and leave expeditiously" like you practice in the fire drills?

Some people have no common sense.

@cazabon The Slack thread was started after the person evacuated:

https://mastodon.online/@tagir_valeev/116059238703673745

Tagir Valeev (@[email protected])

As it gets much more attention, than I expected, here are two clarifications: 1. The Slack message was written after the person evacuated properly. It was written via phone while staying at the designated area outside the building. 2. Nobody asked AI advice explicitly. It was configured to answer automatically if it thinks it can help you. The configuration was updated after this incident.

Mastodon

@tagir_valeev

If it was a test then folks should do what they are supposed to do in a fire alarm to test the complete system.

@tagir_valeev A deadly case of the general principle that the situationally useful information lies not in the statistical pattern but in where and how deviations from the pattern occur.
@dalias @tagir_valeev At least when the environment is properly secure.

An old undermaintained chemical factory is always dangerous, deviations just make it even more so.

@tagir_valeev More galling still, a scheduled test of a fire alarm system typically *still includes evacuation.* Leaving the building *is* the drill. I have never worked in an office where there was any condition under which occupants are told to ignore the alarm.

Ignoring alarms leads to alarm fatigue which then leads to failure. Alarms either exist for a reason or they don't. A device that says otherwise is a broken device. You're right, devices like that will kill.

@majick well, in our case there were definitely cases of 'scheduled alarm maintenance'. In that time, random alarms occurred many times during two or three hours. Evacuating after every single of them would mean nobody is doing the actual work during the good part of the day.
@tagir_valeev Exceptions like that are reasonable, I think, with the caveat that 'being prepared to not die' is the actual work on any good day.
@majick @tagir_valeev There are two kinds of alarm testing. One is as you described, where they are testing the alarm structure and functionality. You should get advance notice to ignore the alarms, preferably with a reminder to listen to announcements just in case there's a real emergency in the middle of their test. The other kind is testing the human element, so yeah, you have to leave when they tell you to because you never know.

@sbourne @tagir_valeev I don't agree with this because no alarm should go ignored, but I do understand why it's done that way in the real world. And why it's the default method.

Nobody[1] calls a reliability engineer before putting together their building maintenance punchlist and sending dudes with ladders.

1. note: nobody except my kid's boyfriend who is a chief facilities engineer, or my kid who is a marine engineer and grew up around rigid high-reliability high-risk operations. They're exceptions.

@sbourne @majick @tagir_valeev I definitely remember at least one fire alarm test (scheduled ahead of time, no evacuation required) where in the middle of the test The Powers That Be got on the intercom and said “actually this isn’t part of the test, gtfo of the building”

@majick @tagir_valeev There could be some cases where the test checks sound etc, where evacuation is not the drill, but those indeed would be exceptions.

In Latvia, in 2013 fire alarms were repeatedly set off in a supermarket. Security just reset them, and employees & shoppers returned, then ignored the alarms.
The building collapsed and 54 people died.

Whenever I hear a fire alarm, I first get the fuck out, then I figure out what's happening.

https://lv.wikipedia.org/wiki/Lielveikala_%22Maxima%22_sagr%C5%AB%C5%A1ana_R%C4%ABg%C4%81

@richlv @tagir_valeev Operators resetting/muting the alarm without understanding why it fired is a perfect example of alarm fatigue. A tragedy like that underscores why it's a Big Fuckin' Deal to avoid it.

The root cause of a failure like that is almost never the dude who did that. It's the circumstances that led to that dude thinking it was the correct thing to do.

Then people die.

My own opinion that evacuation is always the drill. Working on the alarm device, be it wiring, programming, or the noise that comes out of it, is part of working on an end-to-end system that includes people going away from the alarm.

@majick @tagir_valeev As much as I love to talk about alert noise/fatigue, in that case other factors were contributing.
A bit of pressure from the supermarket chain, a bit of leftover mentality from the russian occupation times. Similar to the "I'm not afraid of no virus, I'm not gonna mask!" etc.

@richlv @tagir_valeev Yeah, I completely agree those factors can very much influence how bad something like that turns out to become. Rarely does someone think a thing they do routinely is actually high risk.

I'd say they fall into the category of "factors" I mentioned as root causes: cultural, management-based (I've lived that life), and so on. Fatigue's one thing, for sure, not the only thing, and sometimes it's stuff like that deprioritizing the seriousness of an alarm. A different cause of fatigue/blasé attitude/misinterpretation.

Like I said, it's almost never the dude and almost always what influenced the dude to be the last link in the chain up to a horrible tragedy.

@majick @tagir_valeev

Fwiw I’ve worked in buildings with a regularly-scheduled alarm test (same time and day every week) which you were expected to ignore, reporting if there was a fault. It was preceded by a recorded announcement saying it was a rest, and followed by one saying the test was over and any further alarms should be responded to normally by evacuating.

(The drills where you do leave are more common, of course.)

@tagir_valeev As 90% of fire alarms are drills, it makes perfect sense to respond with the most likely scenario.

( /s in case it wasn’t obvious. Rather startling that people are arguing with you.)

@tagir_valeev idk Dude, when theres a fire alarm I leave the building and make sure everyone made it out, and dont write shit in Slack