Ever run a failover test that worked perfectly… and still felt like everything was falling apart?

In Episode 12, we take you into a disaster recovery test during a busy release weekend — where the tech held up, but communication didn’t.

Subcontractors weren’t aligned, assumptions didn’t match reality, and suddenly a ‘simple test’ turned into a full coordination puzzle.

No production impact — but plenty of lessons.

Because resilience isn’t just about systems… it’s about people, timing, and actually talking to each other.

Listen now to IT Horror Stories with Jack Smith
You can find us on Spotify, Apple Music, Youtube, Deezer and of course at ITHorrorStories.eu

You are one of us.

https://youtube.com/shorts/k_SyFbQ71TU

#podcast #technology #failover #failure #techlife

For those who run #ProsodyIM as #xmpp server, I did something simple but effective in my failover architecture:

• 2 Prosody instances in two different regions in a datacenter
• lsyncd syncing from primary to stand by instance all data
• an entrypoint script supervising Prosody execution
• a lock file controlling if entrypoint script can up Prosody
• a daemon checking if floating ip is linked to hosts and controlling the lock file and the lsyncd execution and configuration to primary/standby modes

Perfect solution? Of course not.
Effective solution? Hell yeah.

#xmpp #failover #container #vrrp #prosodyim #prosodyim

During a busy release weekend, a planned failover exposed not technical flaws, but something more familiar: misaligned teams, unclear responsibilities, and communication that didn’t quite arrive when it should have. Production stayed safe — but confidence took a hit.

This episode dives into how a “simple test” turned into a coordination challenge, and why resilience is just as much about people and processes as it is about systems.

Find all links to listen on our website : https://ithorrorstories.eu/#ep12

#podcast #datarecovery #failover #test #technology

You can find our podcast on :

Spotify : https://open.spotify.com/show/7LqbtykS0IQctSCucvQVHW
Apple Music : https://podcasts.apple.com/us/podcast/it-horror-stories-with-jack-smith/id1812612272
YouTube : https://music.youtube.com/playlist?list=PL9A9yzpnkOdVQvmFjgTsZRrE-zDCuIVcX
Deezer : https://link.deezer.com/s/30dyH3RoKvN8N24zgsbhj

[Перевод] Осваиваем replication slots в Postgres: как предотвратить разрастание WAL и другие проблемы в продакшене

Логическая репликация в Postgres редко ломает прод внезапно — чаще она долго и методично копит проблему, пока replication slot удерживает всё больше WAL, потребитель отстаёт, а свободное место на диске начинает таять. В этой статье разбирается именно такая зона риска: как устроена работа replication slots, почему одних базовых настроек здесь недостаточно и какие практики реально помогают держать под контролем WAL, публикации, heartbeats, failover и мониторинг. Материал особенно полезен тем, кто работает с CDC, Debezium и production-инстансами Postgres, где цена ошибки измеряется уже не теорией, а стабильностью системы. Разбор PostgreSQL

https://habr.com/ru/companies/otus/articles/1018444/

#PostgreSQL #replication_slots #логическая_репликация #WAL #CDC #Debezium #pgoutput #failover #мониторинг_Postgres

Мониторинг SQL Server Always On в Zabbix

Если у вас стоит Always On Availability Groups, вы наверняка бывали в такой ситуации: в SSMS всё зелёное, дашборд показывает «Synchronized», а пользователи звонят с жалобами на тормоза. Смотришь на secondary — а там redo_queue_size 600 МБ, реплика отстаёт на полчаса. Ни одного алерта. У нас это случилось на продуктивном кластере с 1С: secondary молча отвалился в SYNCHRONIZING, а мы узнали только при плановом переключении. Полтора часа redo queue. Стало понятно, что встроенный дашборд SSMS — это не мониторинг. Дальше — как мы это закрыли Zabbix'ом за вечер.

https://habr.com/ru/companies/cloud4y/articles/1017578/

#SQL_Server #Always_On #Zabbix #мониторинг #DMV #WSFC #кворум #failover #DBA

ssh service failed to start #ssh #services #reboot #failover #bsd

https://askubuntu.com/q/1565146/612

Starlink Mini as a Failover

https://www.jackpearce.co.uk/posts/starlink-failover/

#HackerNews #Starlink #Mini #Failover #SpaceTech #Connectivity #InternetSolutions

#Multi-WAN setup for #failover and #loadbalancing with #opnsense ready! I enjoyed the implementation and also the concepts are quite nice. Until now I just used the big #COTS players like #cisco #juniper #paloalto for this kind of network stuff, but I liked the opnsense approach quite a bit. If you want to have look here [1] you will find an overview.

[1] https://docs.opnsense.org/manual/how-tos/multiwan.html#example-configuration

Cross Region Failover

Auto-switch regions when latency spikes.

#php #python #failover #multiregion #latency #reliability #backendsafety #performance #sre #viralcoding

https://www.youtube.com/watch?v=022LKi89rNY

Python vs JS Scraper Failover: Outsmart IP Bans

Cache, retry, and switch sources before sales tank.

#python #javascript #scraping #cache #failover #pricing #viralcoding #codecomparison #growthhacks #reliability

https://www.youtube.com/watch?v=t13VD57xHiQ