As part of the constant tinkering on #keyboardvagabond, today I did a few fixes:
1) the talos registry mirror fallbacks were wrong, so #harborregistry went down and so did #cloudflared and #tailscale. Scary. I got back in to the system and separated the mirror and fallback configs (for disaster only) and then scaled harbor to 2 replicas, hopefully to prevent this failure in the future

2) added a #piefed watchdog script that restarts if the queue is stuck (cont'd) #kubernetes #selfhosting

#keyboardvagabond is now migrated from using #ingress-nginx to #gatewayApi. It took most of the day, but almost the entire networking stack has been swapped out for what I should have been using anyway.

I'll try to get a write-up or something on it later to go into more detail. Overall, it wasn't too bad.

#kubernetes #selfhosting #programming #softwareEngineering

@meta
@programming

#keyboardvagabond just upgraded #writefreely from v0.15.1 to v0.16.0. I was previously using someone else's image for the docker container, but just switched to using official images as part of my own build for my #kubernetes #cluster. Everything is up and running with the latest features! Woo!

#selfhosting #softwareEngineering

I finally got to add #DeepL #translations to this #mastodon instance! It took a while because they flagged me as suspicious, but this week it finally went through and configuration was super easy!

#fediverse #keyboardvagabond #selfhosting

Moar adventures in #selfhosting for #keyboardvagabond! I think that I finally got the longhorn too-many-s3-calls networking issue resolved https://blog.keyboardvagabond.com/michael-dileos-software-and-tech-blog/longhorns-backup-cost-savings-or-not-getting-surprise-bills

I also found that the #bitnami #helmchart for #harborRegistry didn't get updated with the latest migration they did, so I had to find out that they moved their repositories to `bitnamilegacy` and update my helm chart. That took a while to figure out why the image pulls failed!

@selfhosted
@selfhosting
#kubernetes

#keyboardvagabond successfully updated #piefed to v1.3.0! I'm looking forward to the new features! @rimu thanks for all of the good work!

https://codeberg.org/rimu/pyfedi/releases/tag/v1.3.0

[Blog Post] - Longhorn's Backup Cost Savings – or not getting surprise bills

https://piefed.keyboardvagabond.com/post/137210

Longhorn's Backup Cost Savings - or not getting surprise bills

I'm experimenting with where to put these types of blog posts. I have a copy over at the Keyboard Vagabond blog

Longhorn, the system that is used to manage storage for Keyboard Vagabond, performs regular backups and disaster recovery management. I noticed that on the last few billing cycles, the costs for S3 cloud storage with BackBlaze was about $25 higher than expected, and given that the last two bills were like this, it's not a fluke.

The costs are from s3_list_objects, over 5M calls last month. It turns out this is a common thing that has been mentioned in github, reddit, Stack Overflow, etc. The solution seems to be just to turn it off. It doesn't seem to be required for backups and disaster recovery to work and Longhorn seems to be doing something very incorrectly to be making all of these calls.

My expectation is that future billing cycles should be well under $10/month for storage. The current daily average storage size is 563GB, or $3.38 per month.

#kubernetes #longhorn #s3 #programming #selfhosting #cloudnative #keyboardvagabond