Boring reminder: If an alert isnât actionable, it shouldnât exist. Alerts are for action, not noise.
Captain Obvious strikes again: when setting up blackbox_exporter to monitor your endpoints, host it on a separate server/cloud. Not on the same cluster you're going to monitor. A downed blackbox_exporter doesnât export much đ
#Terraform lets you keep modules in the same repo as your code, which is great for getting started. But as your deployment logic grows, consider moving modules to a separate repo.
This brings major benefits: better version control and easier QA/testing for module changes.
When planning a new service that handles customer data, and if maintenance procedures are required, always ask: how can this operation be safeguarded? How will we observe progress? How will we handle failures?
Design maintenance operations to be safe by default.