Release Engineering Is Exhausting So Here's cargo-dist!

Helping your Rust projects ship prebuilt binaries that others can actually use without you having to become a Github CI Expert (or Rust Toolchain Expert) ((or OS integration expert)) (((or...)))

https://blog.axo.dev/2023/02/cargo-dist

axo blog - Release Engineering Is Exhausting So Here's cargo-dist

Release Announcement for cargo-dist, a tool to help you package up binaries for a Rust application.

protip if you ask me to write an announcement blogpost for a tool i WILL just send you back an extended rant about random toolchain problems and then make the tool we're announcing hit 3 bugs in the demo for good measure
@Gankra If it doesn't uncover any new bugs, is it even a demo?

@Gankra congratulations on the release!!

This looks really cool and definitely a thing I'll want to use with my tools! Thank you!

@Gankra this is exactly what I need!

@Gankra
> Clippy complained that you weren't!!

Having spent a morning fixing the code for the latest clippy changes this week, this one hits a bit too close to home 😅

@Gankra @brainblasted that’s actually super cool and really helpful! Thank you!
@Gankra I moved a bunch of my projects to cargo dist yesterday and all of them worked perfectly, thanks for this!

@msfjarvis haha yeah i saw your toot yesterday and was like "uhhh hold up the blog post is tomorrow"

glad it works!

@Gankra I have not only been a release engineer, I’ve built entire CI systems from scratch. So I love everything about this.
@Gankra FYI, the ASCII art on the axo.dev home page is very annoying for screen reader users.
@matt will fix, thanks for the heads up!
@matt sorry for the delay, should be fixed! (went with aria-hidden since it's not really informative content)
@Gankra Sounds good to me. Looks like there's also ASCII art on the first navigation link and the signup form's submit button. For those I'd use aria-label to override the accessible text.
@matt will do (when we're back on monday), thanks for the tips!
@Gankra
That looks awesome, despite the glitches. I had so many headaches getting my simple app into binaries (and I've never worked out how to sign them - I just tell users how to fix the unsigned code warning manually)
@calanais I know you're a fan of release engineering! :)
@Gankra But anyway, cargo-dist looks really cool. I almost want to add some kind of tool component to AccessKit (which is, so far, purely a library project) just so I can use it.

@matt oh also i've been meaning to dig deeper into AccessKit/egui stuff more. has anyone written docs/tutorials for that integration yet?

(would you like me to...? would probably require chatting you or someone else up a bit to understand it more.)

@Gankra Sorry, no tutorials yet, and docs are pretty thin. Happy to chat any time.
@matt would you prefer a "look at integrating this into an existing app" (https://github.com/rust-minidump/minidump-debugger) or more of a toy tailored example?
GitHub - rust-minidump/minidump-debugger: An experimental GUI for rust-minidump

An experimental GUI for rust-minidump. Contribute to rust-minidump/minidump-debugger development by creating an account on GitHub.

GitHub
@Gankra I will look at your app and see if there are any gaps in the egui AccessKit integration, or AccessKit itself, that need to be filled before the app can be made usefully accessible.

@matt oh! uhhh ok..!

you can use this as a test input if you plan to actually run it:

https://github.com/rust-minidump/rust-minidump/blob/main/testdata/test.dmp

rust-minidump/test.dmp at main · rust-minidump/rust-minidump

Type definitions, parsing, and analysis for the minidump file format. - rust-minidump/test.dmp at main · rust-minidump/rust-minidump

GitHub
@Gankra This will, in fact, be my first opportunity to put AccessKit to use in a real-world application.

@matt to be clear I'm not sure if anyone uses minidump-debugger For Real -- I made it last year to mess with egui and also demo the functionality of rust-minidump (whose primary users use it as a library or cli in their infra)

but it definitely *works* and is useful (although some of the log viewer stuff is dubious absent a good Text Editor widget in egui that can handle enormous amounts of text)

@Gankra AccessKit itself probably isn't very good at handling enormous amounts of text yet, and the immediate-mode nature of egui makes it worse.
@Gankra This application looks like exactly the kind of niche tool that I hope AccessKit will help make accessible. The kind of thing that's far enough off the beaten path that it probably won't be made accessible through typical corporate accessibility efforts, thus limiting job prospects for disabled people. Looking forward to exploring this app.
@Gankra not sure whether to be more excited about this tool or about the idea of the Axo company/team that builds tooling like this.
@Gankra Ooooh, I'll absolutely have to try this out! I currently have a pet project with a horrendous set of CI jobs for publishing releases that do many of the things mentioned here. Having a unified tool to manage all of this junk would be wonderful, especially because it means I wouldn't have to copy-paste config snippets from 6 other places whenever I set up a new project, only for the first run to *still* fail because I forgot a line somewhere 🙃. More CI tooling is always a good thing!
@Gankra Congrats on the release! Btw did you see my treatise on DWP? https://github.com/rust-lang/rust/issues/105991
Are dwp files the right choice for split-debuginfo=packed on Linux? · Issue #105991 · rust-lang/rust

@davidtwco @bjorn3 @philipc @Gankra @khuey Hi all, I was reading up on split dwarf and DWP files and came away rather confused. What is a situation in which using split dwarf + DWP is preferable ov...

GitHub

@mstange yes! i agree dwp isn't ready for primetime, but am mostly waiting for

https://github.com/rust-lang/cargo/pull/11572

before re-evaluating ...which in grabbing that link I just saw landed like, yesterday!

Make cargo aware of dwp files. by khuey · Pull Request #11572 · rust-lang/cargo

When using -Csplit-debuginfo=packed on Linux, rustc will produce a dwp file. Unlike the dwo files, whose paths are embedded into the binary, there's no information in the binary to help a debugger ...

GitHub
@Gankra Ah nice, I'm curious to see your evaluation once you get a chance.

@mstange my main concern is getting this stuff uploaded at all to a level where, even if it's not useful *today*, someone can come back later and process the uploaded artifacts into a more useful form

i'm pretty confident I'm doing this for pdbs (I don't grab any ids but they should be embedded in the binaries and the naming scheme indicates they pair)

i'm less clear on dsym/dwp, if I need to make sure to grab anything extra.

@Gankra Ah I see, that makes sense.

Ok so if you just grab the dSYM bundle and the dwp file and upload those as extra artifacts, that should be all that’s needed. And don’t strip the Linux binary; it contains debug info which is not in the dwp.

A dSYM is quite similar to a pdb. It’s a self contained unit with all debug info. It is paired with its corresponding binary by virtue of having the same mach-O UUID. This UUID is written down in the files and is usually not spelled out in the filename.

@mstange perfect, thanks for confirming my assumptions!

@mstange would absolutely love to hear whatever opinions you have on split debuginfo support :)

https://github.com/axodotdev/cargo-dist/issues/30

Splitting Debuginfo · Issue #30 · axodotdev/cargo-dist

Another future killer app: properly setting the flags and running the tools to produce proper symbol/debuginfo files (similar to source maps, but for native code). This comment covers a lot of usef...

GitHub
@Gankra I'm excited to try this! In place of the scripts I copied and modified from ripgrep that, er, don't work at the moment

@Gankra Niiiice.

You mentioned thinking about SDKs etc and controlling deployment version, what is the plan for your platforms (especially building somewhere new-ish and deploying to older versions)? MacOS seems to at least have sdk controls in its toolchain (maybe not for Rust?), windows iirc has symbols you define, and linux usually requires a sysroot, or is the situation a lot nicer in Rust-land than C++ land?

Also, on the topic of "old glibc", passenger's holy-build-box should be a possibility, but I'm not sure it's optimal when you need to install so much new-ish toolchain.

@lenary unfortunately I only know about the Problems and know nothing about the solutions, help definitely wanted, because I want cargo-dist to help with this for sure!

https://github.com/axodotdev/cargo-dist/issues/78

target sdk version support · Issue #78 · axodotdev/cargo-dist

When making prebuilt binaries you "need" to decide how old of a system you're willing to support. i.e. if you want to rely on APIs introduced in windows 10, your binary won't nece...

GitHub
@Gankra I am happy to drop some thoughts/pointers in that issue for you!
@lenary please do! 🙇
@Gankra well, of course i hit submit 30% of the way into the comment, but it's updated now. I couldn't find one link I wanted to, I'll have to ask someone else what to do there.
@lenary this definitely helps -- also it's nice and reassuring to know that there wasn't an obvious "it's easy" approach I was missing x3

@Gankra I've had to firefight now a few different upgrades at work: a surprise "oh shit one of our testing deps needs a better glibc so someone updated a docker image, and now we can't ship to customers on older linuxes", and another surprise of "this source file no longer compiles with very old gcc, we need to update our whole build tree, including static library dependencies" and tbh, I opted out of thinking deeply about this a third time for Windows as I am not actually a build expert, just someone who tries to pay attention and knows this stuff is harder than one would hope.

My best advice is never to undervalue the ability to rebuild your whole dependency tree from source, which is sometimes an easier way to get yourself out of an ABI difficulty than any other way. Rust is young enough this should still be doable. It isn't with C/C++.

One thing I specifically didn't cover was lots of the random bullshit to do with packaging C++ apps, where you have to learn that it is not the case that gcc and clang C++ objects can always be linked together, or even objects from two different versions of the same compiler without greater care than you want to know.

Actually, now I think about it, there's even C11 objects that can easily not be compatible between gcc and clang, but in that little corner of the compiler (atomics) I can also find issues linking Rust code to C/C++ code.

@lenary yeah the ABI issues between compilers fucking suck, that's why i made https://github.com/Gankra/abi-cafe

(still need to get back to v2 on that...)

GitHub - Gankra/abi-cafe: Pair your compilers up at The ABI Café!

Pair your compilers up at The ABI Café! Contribute to Gankra/abi-cafe development by creating an account on GitHub.

GitHub
@Gankra this is all ABI! We are cursed to think about the hardest problems in toolchains forever.
@Gankra Tool looks super useful in its own right, but this release announcement is next-level. Well done!
@Gankra fyi, links in a <li></li> don't display as clickable unless you hover over them / tab to them

@Gankra

I AM ASKING YOU AGAIN TO ENABLE FRAME POINTERS IN YOUR BUILDS, YOUR LOCAL PERFORMANCE ENGINEER WILL THANK YOU FOR THE GIFT.

is there somewhere i can read more about this? i'd come to understand that the frame pointer wasn't needed but clearly i'm missing something

@cxberger the css is a bit messed up but the line you're quoting was actually a link to the explanation x3
@Gankra this post was a joy to read!
@Gankra thank you. I do not want to become an expert of any kind 🙏
@Gankra have you considered supporting some sort of automatic signing mechanism (like sigstore) for releases?

@ariadne on the roadmap as "this is important but I don't know anything about it yet"

so if you're an expert on this then i'd love suggestions https://github.com/axodotdev/cargo-dist/issues/21

code signing? · Issue #21 · axodotdev/cargo-dist

Some platforms complain if you run unsigned code :( Not sure if this is in scope for dist or a different tool altogether, but it should probably be out of scope for the MVP.

GitHub
@Gankra i will have some of my people reach out!
@Gankra Very cool! 🦀 We've been running into similar obstacles and created the Ploutos project to generate Debian and RPM packages, as well as Docker images. #rustlang https://github.com/NLnetLabs/ploutos
GitHub - NLnetLabs/ploutos: Reusable packaging workflow for Rust projects

Reusable packaging workflow for Rust projects. Contribute to NLnetLabs/ploutos development by creating an account on GitHub.

GitHub
@Gankra Worked like a charm on my little project! (I can't compile cargo-dist on Apple silicon, so I'm glad I also learned about cargo-binstall from your blog. The binary version just works)