Stefano MarinelliSep 10, 2024

This morning, I took my wife to the hospital for routine blood tests that had been scheduled for some time. Everything was going smoothly: check-in, number, waiting room. Suddenly, everything came to a halt and shut down. I was connected to the hospital’s public Wi-Fi and noticed that my connection also went down.

Having managed a couple of similar facilities, I immediately understood what had happened. I saw the staff panicking and calling the technicians, but they quickly reorganized within 10 minutes. They managed to process everyone who already had a number and then proceeded with the others in the order of their arrival. Despite the ten-minute delay (even though people started complaining right away), they were extremely efficient.

I later confirmed that the entire booking, check-in, and queue system is “in the cloud.” The hospital experienced a connectivity interruption, and all related services stopped. The staff no longer had access to anything, so a technician sent the lists to a manager via another channel, and everything resumed manually.

For years, I’ve insisted that certain things MUST be local. The healthcare facilities I manage have all the necessary systems for the operation of the facility internally, including patient records. External services like websites, emails, etc., are secondary.

Everything essential must always be accessible locally and, in special cases, it should be possible to physically access the servers and connect directly to them, bypassing any network/switch failures.

There has been only one interruption in the past, due to human error. Today, we have redundant servers (not HA on virtualizers, but two machines running the same software with replicated databases - on separate power lines) so such an issue shouldn’t happen anymore.

Not everything can be anticipated, but history is a great teacher. The Internet connection will eventually be interrupted :-)

When it comes to the health and survival of people, there are no compromises.

#IT #Internet #Networking #Outage #Health #HA #Cloud #CloudComputing #OwnYourData

Show threadTomAokiSep 10, 2024
@stefano
Exactly.
All critical data should be held locally as primary, and clouds should be for backups and shares. If making cloud as primary, multiple physical connections with different routes should be always kept, if the data there is "critical".
Show threadKevin RussellSep 10, 2024

@TomAoki @stefano

There are rules that can be followed, but medical records for 51 million people do not need to be stored at every hospital.

Your point is a start.

Show threadTomAokiSep 10, 2024
@kevinrns @stefano
Of course, single hospital doesn't need to host and manage records of other hospitals. But keeping their own patients' records locally as primary AND backing up in real time to (shared governmental) cloud would be wanted.
With this, they can access to their records even when external connections are lost and safe keep their records for hazards.
Show threadKevin RussellSep 10, 2024

@TomAoki @stefano

Yep. Good rules can be made, they have not been made.

Seeing military organisations running Windows 95 is terrifying.

Show threadAndy Fletcher

@kevinrns @TomAoki @stefano A lot of embedded systems use elderly operating systems. The essential thing is to control their access to networks to only what is absolutely essential for them to operate correctly.

The issue is that the manufacturer certifies operation of all parts of the software stack when systems are delivered. Random upgrades are impractical for some systems - consider a building lighting system control system which can operate for 25 years - if it is working why upgrade it?

Sep 11, 2024 at 8:43amWeb
Show threadTomAokiSep 11, 2024
@X31Andy @kevinrns @stefano
Old enough embedded systems doesn't have any ability for Internet access, or even network access (except for their specially-crafted control lines).
This made them "secure".
But recent IoTs like webcams has too much functionality (as needed to function) to be able to insecure. Thus, if any of its components turned out to be vulnerable, there SHALL not be any option NOT to update.
Once the vendor stops providing security fixes for them, it should be considered as (equivalent as) reaching physical EoL of it and need quick replacing, if still needed. If not, should be disconnected from network.
Show threadKevin RussellSep 11, 2024

@TomAoki @X31Andy @stefano

This is the seed of solutions. This is insight.

Show threadKevin RussellSep 11, 2024

@X31Andy @TomAoki @stefano

Upgrade sure, bring into compliance with channels of subversion, takeover and exploited networks? No .

Thanks Andy