René Moser (resmo) レネToday is my #Debian13 upgrade day.
Many nodes in my #hashicorpNomad environments. Smooth so far without any app downtimes.
I love #hetzner #hcloud, not only a good (or best) price/performance ratio but the UI and user experience are top.
mxabEver hesitated to switch to HashiCorp #Nomad because NACP lacked OpenTelemetry support?
Great news! The latest NACP release now includes OpenTelemetry instrumentation! Observe with ease while admission controlling your Nomad deployments.
You'll get not just logs, but also fancy metrics and traces! 🥳 🤓
#nomad #hashicorp #opentelemetry #o11y #admissioncontroller #observability #devops #HashiCorpNomad
fredix 🐧Et boom, un wiki pour centraliser mes docs dispersées sur mon blog ... On commence par #HashicorpNomad https://wiki.fredix.xyz/nomad
MichaelAnd that's it, the end of a Homelabbing era. The Nomad cluster is devoid of jobs.
The issue is seemingly a change in how the fingerprinting of Consul/Vault is done during the Nomad client startup. If Consul/Vault aren't up yet, fingerprinting fails, and then Nomad seems to think Vault/Consul is not available...?
Anyway, the result is that Nomad thinks the nodes can't run anything.
A simple restart of the clients after Vault/Consul are up seems to fix the issue. A server restart was not necessary.
If you're running Nomad with Consul/Vault, skip the 1.9.6 update for now, or be prepared to restart the Nomad clients after your node reboot: https://github.com/hashicorp/nomad/issues/25097
All nodes are seen as unavailable for any job, meaning nothing gets started. This only becomes visible after too many nodes have been updated for the remainder to run all of your services.
1/2
Consul/Vault fingerprints never updated if missed during startup (1.9.6 update) · Issue #25097 · hashicorp/nomad
Nomad version 1.9.6 from your PPA. With Consul 1.20.1 also from your PPA. Operating system and Environment details Ubuntu Jammy on metal Issue Updating the cluster from 1.9.5. to 1.9.6. using by up...
fredix 🐧Une infra avec Nomad, Consul, Vault, Tailscale et Tricot : S02E01
https://www.fredix.xyz/2025/02/une-infra-avec-nomad-consul-vault-tailscale-tricot/
#HashiCorpNomad
https://www.fredix.xyz/2025/02/une-infra-avec-nomad-consul-vault-tailscale-tricot/
#HashiCorpNomad
Tous mes services sont up et fonctionnels sur #Hetzner. Le plus long c'était pour le service #LND (#bitcoin) qui va chercher son pass dans #Vault (et ma conf n'était pas bonne).
#HashiCorpNomad
#HashiCorpNomad
j'ai enfin configuré mon HCL #HashiCorpNomad de #LND pour qu'il utilise #Vault afin de récuper le pass du wallet après un restart du conteneur Docker.
#Bitcoin
#Bitcoin
I think it's just not possible to map the same port onto a port on host and have a consul proxy listen on it. Admittedly, it's a somewhat weird config, but I could have sworn I had it working at one point. But looking through all my Nomad jobs, I found that I only ever map ports other than the Consul Connect one to other Consul services.
I finally ended up resurrecting an old Consul ingress gateway config to allow entry into the Connect Mesh.