poddingueShipped BuildKit for RISC-V64. Green CI, packages built. Users tried it: crash loop.
"No worker found" - missing runc, wrong ENTRYPOINT/CMD, hanging tests.
Lessons:
✓ Test integration, not just compilation
✓ Add timeouts everywhere
✓ Document WHY, not just WHAT
Visible success hiding more profound problems. Post-deployment debugging is the real work.
#BuildKit #RISCV64 #Docker #DevOps #Debugging #PostDeployment #LessonsLearned #OpenSource #devEco
BuildKit for RISC-V64: When Your Package Works But Your Container Doesn’t | Bruno Verachten
I shipped BuildKit for RISC-V64 two days ago. The workflows were green, packages built successfully, documentation looked thorough. Then users tried to actually use it. The container crash-looped immediately. "No worker found." Turns out "successful compilation" and "functional deployment" are very different things. The problems: - Missing runc binary meant OCI worker couldn't initialize (BuildKit needs at least one worker to run) - ENTRYPOINT/CMD split was wrong, so Docker Buildx was unable to pass runtime arguments - CI tests hung indefinitely when buildkitd tried to initialize workers without proper privileges - Cached GHCR credentials caused false "access denied" errors Three PRs later, the container works. But here's what I learned: - Testing individual components isn't enough. You need to test how components integrate with the systems that actually use them. It's not enough to verify the car starts; you need to verify it drives. - Always add timeouts. Even commands that "should never hang" will eventually hang. - Use your own packages consistently. Mixing Debian's runc 1.1.12 with our runc 1.3.0 builds created unnecessary version complexity. - Document the WHY, not just the WHAT. Explain when to use container images vs. APT packages. Users need context, not just instructions. The iceberg metaphor fits perfectly: visible success hiding more profound integration problems. Post-deployment debugging is where the real work happens. #BuildKit #RISCV64 #Docker #DevOps #Debugging #ContainerTechnology #LessonsLearned #PostDeployment #devEco
Sanjay Mohindroo