Pennydavies90Sep 10, 2024
Stefano Marinelli

This morning, I took my wife to the hospital for routine blood tests that had been scheduled for some time. Everything was going smoothly: check-in, number, waiting room. Suddenly, everything came to a halt and shut down. I was connected to the hospital’s public Wi-Fi and noticed that my connection also went down.

Having managed a couple of similar facilities, I immediately understood what had happened. I saw the staff panicking and calling the technicians, but they quickly reorganized within 10 minutes. They managed to process everyone who already had a number and then proceeded with the others in the order of their arrival. Despite the ten-minute delay (even though people started complaining right away), they were extremely efficient.

I later confirmed that the entire booking, check-in, and queue system is “in the cloud.” The hospital experienced a connectivity interruption, and all related services stopped. The staff no longer had access to anything, so a technician sent the lists to a manager via another channel, and everything resumed manually.

For years, I’ve insisted that certain things MUST be local. The healthcare facilities I manage have all the necessary systems for the operation of the facility internally, including patient records. External services like websites, emails, etc., are secondary.

Everything essential must always be accessible locally and, in special cases, it should be possible to physically access the servers and connect directly to them, bypassing any network/switch failures.

There has been only one interruption in the past, due to human error. Today, we have redundant servers (not HA on virtualizers, but two machines running the same software with replicated databases - on separate power lines) so such an issue shouldn’t happen anymore.

Not everything can be anticipated, but history is a great teacher. The Internet connection will eventually be interrupted :-)

When it comes to the health and survival of people, there are no compromises.

#IT #Internet #Networking #Outage #Health #HA #Cloud #CloudComputing #OwnYourData

Sep 10, 2024 at 3:51pmWeb
Show threadEnigmaRotorSep 10, 2024
@stefano #amen
Show threadTomAokiSep 10, 2024
@stefano
Exactly.
All critical data should be held locally as primary, and clouds should be for backups and shares. If making cloud as primary, multiple physical connections with different routes should be always kept, if the data there is "critical".
Show threadKevin RussellSep 10, 2024

@TomAoki @stefano

There are rules that can be followed, but medical records for 51 million people do not need to be stored at every hospital.

Your point is a start.

Show threadTomAokiSep 10, 2024
@kevinrns @stefano
Of course, single hospital doesn't need to host and manage records of other hospitals. But keeping their own patients' records locally as primary AND backing up in real time to (shared governmental) cloud would be wanted.
With this, they can access to their records even when external connections are lost and safe keep their records for hazards.
Show threadKevin RussellSep 10, 2024

@TomAoki @stefano

Yep. Good rules can be made, they have not been made.

Seeing military organisations running Windows 95 is terrifying.

Show threadAndy FletcherSep 11, 2024

@kevinrns @TomAoki @stefano A lot of embedded systems use elderly operating systems. The essential thing is to control their access to networks to only what is absolutely essential for them to operate correctly.

The issue is that the manufacturer certifies operation of all parts of the software stack when systems are delivered. Random upgrades are impractical for some systems - consider a building lighting system control system which can operate for 25 years - if it is working why upgrade it?

Show threadTomAokiSep 11, 2024
@X31Andy @kevinrns @stefano
Old enough embedded systems doesn't have any ability for Internet access, or even network access (except for their specially-crafted control lines).
This made them "secure".
But recent IoTs like webcams has too much functionality (as needed to function) to be able to insecure. Thus, if any of its components turned out to be vulnerable, there SHALL not be any option NOT to update.
Once the vendor stops providing security fixes for them, it should be considered as (equivalent as) reaching physical EoL of it and need quick replacing, if still needed. If not, should be disconnected from network.
Show threadKevin RussellSep 11, 2024

@TomAoki @X31Andy @stefano

This is the seed of solutions. This is insight.

Show threadKevin RussellSep 11, 2024

@X31Andy @TomAoki @stefano

Upgrade sure, bring into compliance with channels of subversion, takeover and exploited networks? No .

Thanks Andy

Show threadEvannakitaSep 10, 2024
@TomAoki @kevinrns @stefano A shared governmental cloud does worry me considering the recent efforts by some Republican governors to obtain the medical records of trans people, though.
Show threadTomAokiSep 10, 2024
@evannakita @kevinrns @stefano
Well, there would be multiple aspects.
With shared records, especially if you have some chronic health problem and got ill on the trip, shared records would help you a lot.
OTOH, it surely has risks on privacy.
Show threadSLCW💥Sep 10, 2024
@kevinrns @TomAoki @stefano I think a cloud-based system makes sense for networks with multiple hospitals. It's the most effective way to have real-time, highly-available data sharing across the network. But redundancy has to be carefully designed and implemented so that no single point of failure can bring the whole thing to its knees.
Show threadRalph058 (S/he/it) AF4EZSep 10, 2024
@TomAoki @stefano In life critical systems, NASA mandated triple redundancy. It was mainly to handle the Byzantine General problem but at least one system I worked on (DoD, not NASA), two processors went down (redundancy didn't go up the path enough in the initial design, handshake between processors failed)
Show threadBrian StaffordSep 10, 2024
@TomAoki @stefano Cloud == someone else's computer.
Show threadTomAokiSep 11, 2024
@briancs @stefano
Not limited as such.😉
There can be "local cloud" built using the organization's (or personal) own computers co-located on some datacenter. (Would be costly, though, but put them as remote backup site helps BCP. No need to be public cloud, but it would be trade-offs.)
Show threadEpistatacadamSep 10, 2024
@stefano To support you, when cloud services first arrived, we frontline workers received explicit instructions: no clinical data was to be put on the cloud.
The director of IT changed, the new one had no experience of running the system having gone on an IT course after attaining his management qualifications.
Within weeks a new directive came out, all vital data was to be stored on the cloud as it was more secure.
It was but at peak times it just wasn't available!=> Copies on USB drives!
Show threadUkeleleEricSep 10, 2024
@stefano So many people are so reliant on things that they ultimately have no control over. I know how to do things the 'old-fashioned' way. I also eschew devices or systems that are needlessly complicated or reliant on technology. Many years ago, I was a programmer - I am no technophile, just realistic. Nothing wrong with following a Satnav, as long as there is a map in the car for when something goes wrong...
Show threadkikebenllochSep 10, 2024
@UkeleleEric @stefano Exactly, previous technologies can be reliable backup options in adverse circumstances, that's the same reason why I find it absurd to completely depend on wireless solutions, for instance landlines can save your day if everything goes south.
Show threadUkeleleEricSep 11, 2024
@kikebenlloch @stefano Also, even if you're still connecting to the same computer, Wired connection is more secure, faster and less prone to dropouts. wifi is a second-best for those mobile devices that don't have space or size for a proper ethernet port.
Show threadMeidasRomiSep 10, 2024
@stefano thei should have redundancy for the internet access (from two different providers) like most good offices have now.
Show threadZeki Çatav 🤔 ☕ 🕯️🎶Sep 10, 2024
@stefano
Show threadAngela ScholderSep 10, 2024
@stefano I absolutely fully agree!
Show threadBig Kumera EnergySep 10, 2024
@stefano hospitals I've worked with usually have back-to-paper drills semi regularly so that a network or software failure won't dramatically impact patient care.
Show threadosfa_2030Sep 10, 2024
@depereo @stefano The digital record at the public hospital I go to is rather bad and messy. No doctor knows well how to use it. Patients are supposed to have access to their own records but that is rather a joke. Always incomplete, recent info not showing up... It's the typical thing that in theory should be efficient, but in reality never works well enough, costly to maintain, to keep updated with external changes... It's the false utopia of digital administration.
Show threadDavid Palk#RejoinEU 💖🇪🇺🕊🌈Sep 10, 2024
@stefano Absolutely. Not as life critical as your experience but I was the IT manager of a broking house in the City of London. Absolutely everything was backed up, both on site behind fire walls, literal & programmed. We also had off-site secure back ups. We were bombed out by the IRA twice in the eighties, up and running in two hours. Just goes to show & privately, I wouldn't put a damn thing up to the cloud. An 'accident' waiting to happen IMO
Show threadMark vWSep 10, 2024
@stefano Storing HIPAA stuff on somebody else's computer...who has explicitly disclaimed any liability for the loss or misuse of that information.... What could possibly go wrong?
Show threadKris Gainsforth - KB0STGSep 10, 2024

@stefano Local also means that you need to have more FTE IT staff in order to deploy, secure, update and monitor. I understand why SaaS vendors like Epic are being used more and more. Budgets vs. redundancy.

Do I like it? No.
Am I justifying it? No.

Just looking at it from a patient that works in security and sees budget problems all the time.

Show threadPhilip KaludercicSep 10, 2024
@stefano Perhaps to put it this way: If you have a (local) generator, fearing the power might go out, you should also manafe your data locally, anticipating the cloud could "go out".
Show threadMAMA TSep 10, 2024
@stefano about 6 years ago a doctor asked me the best way to set up his new office and patients files. I recommended a server that was not connected to the internet to keep his patients records. The program on the web he wanted to use we could basically restyle to suit his needs. He found that idea unacceptable and went with a cloud based company. I often wonder how many times he goes down or needs to tell patients their files have been compromised.
Show threadkikebenllochSep 10, 2024
@stefano I couldn't agree more. Choose whichever architecture you wish (IDK -even periodical local copies of cloud repositories if nothing else is available or supported by management) but ask yourself if critical services are really autonomous or else can be brought down by not-so-extreme circumstances and render basic vital information inaccessible. If I depend on a single wire (literally or figuratively speaking) I'm likely to run into trouble.
Show threadJérôme CarreteroSep 10, 2024

@stefano

One example to add to
@bert_hubert 's https://berthub.eu/articles/posts/cyber-security-pre-war-reality-check/

Cyber Security: A Pre-War Reality Check - Bert Hubert's writings

This is a lightly edited transcript of my presentation today at the ACCSS/NCSC/Surf seminar ‘Cyber Security and Society’. I want to thank the organizers for inviting me to their conference & giving me a great opportunity to talk about something I worry about a lot. Here are the original slides with notes, which may be useful to view together with the text below. In the notes there are also additional URLs that back up the claims I make in what follows.

Bert Hubert's writings
Show threadSLCW💥Sep 10, 2024
@stefano Cloud-based critical infrastructure has to have redundant connectivity using multiple providers to prevent things like this from happening. And ideally, one cloud should be mirrored to another with a second provider so that N+1 redundancy is maintained at every level. Redundant clouds, and redundant connectivity, otherwise you're at risk of failure at multiple single points.
Show threadAskPippa🇨🇦Sep 10, 2024
@stefano Of interest to @Pathealth ?
Show threadRickSep 10, 2024
@stefano @eschaton I won’t install anything in my home automation that requires an internet connection to operate.
Show threadCody Boone FergusonSep 10, 2024

@stefano as the joke goes (but it’s true) ‘the cloud is just another way of saying "another person’s computer" ‘.

Yes above all else medical facilities need to be local networks.

And yet as time changes it unfortunately becomes harder for that to happen. I remember years back a medical device was connected to the Internet and not only was it TELNET enabled but it required NO authentication. That’s never acceptable.

Show threadCody Boone FergusonSep 10, 2024
@stefano and I hope your wife’s blood work and everything else is fine.
Show threadGalactic Jew 🇮🇱Sep 10, 2024
@stefano Building on your idea, hospitals should have their own wells for water supply as a contingency, since municipal infrastructure may fail.
Show threadCykonotSep 10, 2024
@stefano markets in the information age? Maybe not lol
Show threadJohn FrancisSep 10, 2024
@stefano they call it a "Code Grey" in Ontario....
Show threadnigelSep 10, 2024
Yah my work moved their accounting/job/invoicing system to the cloud. It is annoyingly slow and I wish they just done hybrid but apparently they were never given that option.

But yeah, people aren't generally aware of how fragile infrastructure is.
Show threadderptronSep 10, 2024
@stefano Just wait until you can't order your appendectomy from the autosurgeon until it is able to connect to homeland and make sure the subscription the hospital is using is up to date and your insurance is valid. The onboard assistant AI is mostly braindead because 99% of its brain power is severed and it just keeps asking you to rephrase the question. The head nurse is just staring at the wall with her jaw jerking aimlessly trying to sing a soothing lullaby she can't remember.
Show threadfind you on   fediverseSep 10, 2024

@stefano i once had a gaping wound that needed to be treated and i possibly needed a tetanus shot. i showed the wound to a secretary and asked that my name not go in the computer. when they said they had to put it in the computer i asked again for them to just write it down, they caouldn't so i left.

never got treated it could have gone really bad but i decided then and there i will probably never go to a doctor ever again. terrible, but i'm sick to death of the panopticon.

Show threadkaramazov1879Sep 11, 2024
@stefano Don't much trust the cloud.
Show threadulutohSep 11, 2024

@stefano Regardless of your experiences, a hospital's records and processes *mus* still be hosted in a secure cloud.

Especially in countries where large medical corps operate multiple hospitals in different parts of the country under their name.

Such hospitals *must* be able to access a patient's records from their other branches at all times, and schedule appointments for patients in different branches where required.

Show threadQoleSep 11, 2024
@stefano I work in healthcare IT, and this would be great, an on-site replication of recent data. Even if the data isn't in some distant cloud, it's often off site in a regional data centre, humorously referred to as "on prem".
Show threadJan EdenSep 11, 2024
@stefano A case can be made for both robust local infrastructure and for cloud-based services, depending on the requirements. Our organization recently experienced a severe local outage for ~ 2 hours, and our colleagues working at home were (mostly) unaffected.
Show threadPrabirSep 11, 2024
@stefano Should be on #MedMastodon
Show threadToken Sane PersonSep 12, 2024
@stefano Speaking as someone who has worked on safety-related systems in the past, I'm always amazed that hospital IT doesn't require a safety case.
Show threadsquelch41Sep 12, 2024
@stefano in the uk, it is mandated that we use cloud systems for primary care notes and telephone.
Fine until the connection fails (rare, but it happens)
Show threadGeorgeSep 21, 2024
@stefano same thing happened here in Missouri plus most of the rest of the USA about 6 months ago. It had to do something with Microsoft and a update. The cloud can be a good thing but there should be local data back-up too.
Show threadJens ElbækDec 21, 2024
@stefano Well put
Show threadSzescstopniDec 21, 2024
@stefano Had similar issues recently in a small clinic that has our rural Internet link. The BTS crashed, and it turned out that their backup, which is not from us, was misconfigured. They later fixed their backup link, but a few days later, after a strong storm, both their primary and backup links went down for a while. No chance of moving their system to their premises or getting a more reliable backup – it would be too expensive. Rural Internet in relatively poor Eastern Poland really sucks.
Show threadXenotarApr 23, 2025
@stefano Agree
Show threadCleverson has movedApr 23, 2025
@jackf723 @stefano Totally agree. Not in this area only, but in many others, localism is the key.
Show threadthedæmon (Clay Ayers) [he / him]Apr 23, 2025
I love when businesses run on a day to day basis where missing a day can kill their business... and they switch to cloud applications! Very smart people XD