| Website | kmcd.dev |
| Languages | English, Danish (beginner) |
The Incident Log: January 31, 2017 [23:00] *** Topic: DB Replication Lag | Status: 🔴 Critical [23:05] *** tired_sysadmin has joined [23:10] <tired_sysadmin> Replication is stuck again. The secondary node (db2) is refusing to sync. [23:11] <tired_sysadmin> I’m going to wipe the data directory on db2 and let it pull a fresh copy from master. [23:12] <tired_sysadmin> rm -rf /var/opt/gitlab/postgresql/data [23:12] <tired_sysadmin> Weird. It’s taking a while. Usually empty directories delete instantly. [23:13] <helper_dev> Hey, why did the website just go 500? [23:13] <tired_sysadmin> … [23:13] <tired_sysadmin> I’m looking at my terminal prompt. [23:14] <tired_sysadmin> It says root@db1. [23:14] <helper_dev> db1 is Prod. You are deleting Prod. [23:15] <tired_sysadmin> CTRL+C CTRL+C CTRL+C [23:15] <tired_sysadmin> Okay, I stopped it. How much is left? [23:16] <helper_dev> Checking… The directory is 4.5KB. [23:16] <tired_sysadmin> We had 300GB of data. [23:17] <helper_dev> Okay, don’t panic. We have 5 different backup mechanisms. Let’s check S3. [23:20] <helper_dev> S3 bucket is empty. The backup script has been failing silently since version 8.1. [23:21] <tired_sysadmin> Check the Azure disk snapshots. [23:22] <helper_dev> Not enabled. [23:23] <tired_sysadmin> …LVM snapshots? [23:24] <helper_dev> We take them every 24 hours. We just lost 6 hours of data. [23:25] <tired_sysadmin> I am going to live stream the restoration on YouTube so people don’t kill us. Postmortem of database outage of January 31
An archived log from the #dev-help channel on the Freenode (RIP) afterlife network. [03:12] *** Topic: reckless_dev is now known as broke_dev | Don’t paste API keys [03:14] *** vibe_coder_99 has joined [03:15] <vibe_coder_99> yo! anyone good with terraform? i’m trying to launch my new crypto-based to-do list app and the tutorial is too slow. [03:15] <sysadmin_dave> What tutorial? Also, crypto to-do list? Why? [03:16] <vibe_coder_99> it’s strictly vibes based. i just need it to scale. like, infinite scale. i found this script on a forum that says it “maximizes throughput”. [03:17] <cloud_guru> Paste the plan. Don’t just run random scripts. [03:18] <vibe_coder_99> nah its fine, i just want to know what “p4d.24xlarge” means. sounds powerful. i changed the count to 50 just to be safe for launch day. [03:18] <sysadmin_dave> STOP. [03:18] <sysadmin_dave> DO NOT RUN THAT. [03:19] <cloud_guru> That instance is $32 an hour. PER INSTANCE. [03:19] <vibe_coder_99> wait really? lol whatever i have $100 in credits. running terraform apply --auto-approve now. [03:19] * sysadmin_dave screams internally [03:22] <vibe_coder_99> man aws is slow today. it’s been provisioning for like 2 minutes. [03:22] <cloud_guru> You are requesting 50 top-tier GPU instances. You are trying to provision a supercomputer to host a to-do list. [03:23] <vibe_coder_99> gotta go fast right? 🚀 [03:24] <vibe_coder_99> uh guys [03:25] <vibe_coder_99> my bank called. fraud alert. something about “unusual spending pattern”. [03:25] <sysadmin_dave> You just burned your credits in 4 minutes. You are now spending ~$1600/hour. [03:26] <vibe_coder_99> how do i undo??? ctrl-z doesn’t work in terminal!! [03:26] <sysadmin_dave> terraform destroy. PRAY that the API isn’t rate limiting you. [03:27] <vibe_coder_99> it says “state lock”. i think i closed the window too fast. [03:28] *** vibe_coder_99 has left [03:28] *** vibe_coder_99 has quit (Connection reset by peer: fleeing the country)
