Mastodawn

Zack Weinberg Apr 2, 2024

It's late enough to be hacker hours, if you're as old as I am. Gonna write down a bunch of rambly thoughts about #xz and #autoconf and capital-F Free Software sustainability and all that jazz. Plan is to edit it into a Proper Blog Post™ tomorrow. Rest of the thread will be unlisted but boosts and responses are encouraged.

Zack Weinberg Apr 2, 2024

Starting with the very specific: I do not think it was an accident that the xz backdoor's exploit chain started with a modified version of a third party .m4 file to be compiled into xz's configure script.

It's possible to write incomprehensible, underhanded code in any programming language. There's competitions for it, even. But when you have a programming language, or perhaps a mashup of two languages, that everyone *expects* not to be able to understand — no matter how careful the author is — well, then you have what we might call an attractive nuisance. And when blobs of code in that language are passed around in copy-and-paste fashion without much review or testing or version control, that makes it an even easier target.

So, in my capacity as one of the last few people still keeping autoconf limping along, I'm thinking pretty hard about what could be done to replace its implementation language, and concurrently what could be done to improve development practice for both autoconf and its extensions (the macro archive, gnulib, etc.)

Zack Weinberg Apr 2, 2024

Side bar, I know a lot of people are saying "time to scrap autotools for good, everyone should just use cmake/meson/GN/Basel/..." I have a couple different responses to that depending on my mood, but the important one right now is: Do you honestly believe that your replacement of choice is *enough* better, readability wise, that folks will actually review patches to build machinery carefully enough to catch this kind of insider attack?

On the subject of implementation language, I have one half-baked idea and one castle in the air.

The half-baked idea is: Suppose ./configure continues to be a shell script, but it ceases to be a *generated* shell script. No more M4. Similarly, the Makefile continues to be a Makefile but it ceases to be generated from Makefile.am. Instead, there is a large library of shell functions and a somewhat smaller library of Make rules that you include and then use.

For ./configure I'm fairly confident it would be possible to do this and remain compatible with POSIX.1-2001 "shell and utilities". (Little known fact: for a long time now, autoconf scripts *do* use shell functions! Internally, wrapped in multiple layers of M4 goo, but still — we haven't insisted on backcompat all the way to System V sh in a long, long time.) For Makefiles I believe it would be necessary to insist on GNU Make.

This would definitely be an improvement on the status quo, but would it be *enough* of one? And would it be less work than migration to something else? (It would be a compatibility break and it would *not* be possible to automate the conversion. Lots of work for everyone no matter what.)

Zack Weinberg Apr 2, 2024

Suppose that's not good enough. Bourne shell is still a shitty programming language, and in particular it is really dang hard to read, especially if you're worried about malicious insiders. Which we are.

Now we have another problem. The #1 selling point for autotools vs all other build orchestrators is "no build dependencies if you're working from tarballs," and the only reason that works is you can count on /bin/sh to exist on anything that purports to be Unix. If we want to stop using /bin/sh, we're going to have to make people install something else first, and that something else needs to be a small and stable Twinkie. Python need not apply (sorry, Meson).

What's small and stable enough? Lua is already too large, and at the same time, too limited.

There's one language that's famous for being tiny, flexible, and pleasantly readable once you wrap your head around it: Forth.

If I had investments to live off, I would be sorely tempted to take the next year or so and write my own Forth that was also a shell language and a build orchestrator, and then have a look at rewriting Autoconf in *that.* This is the castle in the air.

Zack Weinberg Apr 2, 2024

Side bar 2: Let's table the whole "shouldn't everyone build from git nowadays?" discussion. I'm quite sure the xz insider could've found a way to hide the stage 0 exploit in a checked-in file. If you care about ways to make the output of "make dist" verifiable and reproducible, and to facilitate building from VCS checkout for those who want that, we're actually having a productive discussion about that on one of the autotools mailing lists right now.

(Not sure which list — I sort them all into one mailbox — and I have to warn you that several other less helpful conversations are happening under the same subject line.)

Zack Weinberg Apr 2, 2024

Moving to the more general.

I said this over on the autoconf lists earlier today: just as I think it is a mistake to focus on the stage 0 exploit having been concealed by not checking it into the VCS, I also think it is a mistake to focus on the next few stages having been concealed in a binary file. There are binary files that are naturally editable and auditable as themselves (raster images, for instance) and there are text files that nobody wants to look at at all (ever tried to fix a merge conflict in an SVG image?)

A more interesting line to draw, IMO, is between code and tests. I feel quite confident in saying that the files written to $prefix by "make install" should never need to have any sort of dependence on the project's test suite, and that is something that ought to be possible to detect mechanically (the biggest challenge is determining what files of the source repo are exclusively part of the test suite).

Zack Weinberg Apr 2, 2024

tea break

Zack Weinberg Apr 2, 2024

Last bit. Community, sustainability, and trust.

The early free software movement (1983–1994 give or take) was, as I've heard the tales, consciously revolutionary, and, as revolutions often do, it ran on the spare time of relatively young people with time and energy to spare.

I came on the scene in 1997, right about the time it became reasonably possible to run Linux as your only desktop OS if you knew what you were doing — or, to put it another way, right about the time the original goal of the GNU Project had been achieved.

Like many other revolutions, GNU had no answer, and still doesn't, to the question: now what?

This is not the only reason the young, energetic revolutionaries of 1997 are now the exhausted maintainers of an archipelago of individual "projects" that sort of add up to a computing environment that one might fairly describe as "the worst (except for all the others)". But I think it's an important reason.

Zack Weinberg Apr 2, 2024

Side bar 3: In the middle 1990s someone — either Eric Raymond or Guy Steele — wrote as part of the "Portrait of J. Random Hacker" appendix to the Jargon File

> [Among hackers] racial and ethnic prejudice is notably uncommon and tends to be met with freezing contempt.

This was not true even at the time, and twenty years later ESR was cheerfully making common cause with Vox Day and the Sad Puppies.

I'm a white guy (albeit some of my grandparents weren't). I already knew how to program when I got to college. If I'd made different choices in the early 2000s, I could very well now be sitting on enough investment income to take a sabbatical and invent a new shell language.

When we look around and say "where do we find the helping hands we so desperately need?" we must recognize that part of the problem is that hacking was never as inclusive a club as we claimed.

(This sidebar is not *only* a response to the commenters who saw a name like "Jia Tan" and immediately started hating on China as a whole.)

Zack Weinberg Apr 2, 2024

Last thought for tonight: Riding on the time and energy of revolutionaries no longer works. Giving away stuff for free and then asking corporations to pay us *never* worked. Grants from governments and NGOs works only for the stuff you can successfully write grants for, which is almost never "five years' salary for invisible maintenance tasks." What's left?

Scott M. Stolz Apr 2, 2024

https://authorship.studio/item/64e6ba64-84f6-4605-ac00-bcd602cb2aa1

-

AdeptVeritatis Apr 2, 2024

Who needs the maintenance? The devs?

Do we need better platforms for us to collect money for developers or even forks? The culture on Twitch with the subs shows, people are willing to pay, if they are used to.
But then only some may get most of the money. And the hacker scene already had enough problem with its stars.

We don't need more competition. It leads to > an archipelago of individual "projects" <.

Maybe the question is:
How can we federate open source software?

AdeptVeritatis Apr 2, 2024

Federate like in: "I am able to do this. You are able to do that. This one has that thing. If we bring everything together and work on it, we will all have what we wanted. To work together successfully, we need to agree upon a standard."

No competition. No price. No market.

But we still need people, who know and can do stuff, and people, who have stuff. But without competition.

(This doesn't sound very convincing or conclusive, but I am still sending this post.)

Zack Weinberg Apr 2, 2024

@AdeptVeritatis Thank you for clarifying, that is essentially what I meant by "mutual aid network" ;-)

Zack Weinberg Apr 2, 2024

@AdeptVeritatis Good questions. It seems to me that answers may come from the space of "trade unions" and "mutual aid networks" and " community organization" but I don't think I'm the right person to spearhead that part of the solution

Zack Weinberg Apr 2, 2024

@AdeptVeritatis also I think that a discussion *solely* in terms of "people need to get paid" is going to leave out some important bits.

@zwol
I'm not smart enough to hang here so just gonna add

> toasting in an epic bread

and if you start a patreon to fund your forth project I'm in for ten bucks

Rep. Eric Gallager (no "h"!)Apr 3, 2024

@zwol I've been trying to get governments to fund this kind of stuff without requiring people to file grant applications first, but it's tough going... I think if we could lobby for more people who understand FOSS to be appointed as state IT directors, that could help, though. It would require political organizing and lobbying, but it could be worth it.

Rep. Eric Gallager (no "h"!)Apr 3, 2024

@zwol Specifically, bird-dogging gubernatorial candidates would be one strategy. If candidates for governor in your state hold publicly-open town halls, go and ask, "What sort of qualities would you look for when nominating a state IT director?" and then grade them based on how FOSS-maintenance-friendly their answers are.

Nire Bryce Apr 7, 2024

@zwol I'm hoping that a general push towards realizing that less complex code is easier for new people to contribute to helps this, but we've got a long way to go.

along with curriculums and corporate hiring practices making people think they need to know a lot more than they do to get involved with open source...

Debabrata Bhattacharya || SE Apr 7, 2024

@NireBryce @zwol

>making people think they need to know a lot more than they do

I had this exact experience the first time I made a contribution: it was so easy, very much against my expectations

Zack Weinberg Apr 8, 2024

@LikesCalendars @NireBryce Several times while I was teaching at CMU, people who'd taken my class (sophomore level "introduction to computer systems") told me that they went in expecting to hate it and find it incomprehensible, but then they really enjoyed the experience and now they were planning to take more systems courses

I don't know if I'll ever teach again, but if I could find a less exhausting way to convey that one piece of enlightenment—that the machine is not magic and you *can* understand it—to the general public...

Nire Bryce Apr 9, 2024

@zwol
💯
@LikesCalendars

Graydon Apr 13, 2024

@zwol What's left is taxes; computers are to a large extent the public communications infrastructure, and ought to be public in the full sense.

(And even if not in the full sense, substantially. Because having it work at all depends on dreary stuff you have to pay people to do, just like clearing clogged culverts.)

The Doctor Apr 2, 2024

@zwol I'm pretty sure that was esr.

maco Apr 2, 2024

@zwol (semi-related: earlier today, I saw a reddit thread asking "why aren't there more women in STEM?" and told my friends I wanted to comment "men in STEM.")

Zack Weinberg Apr 2, 2024

@maco Oof, yeah. I didn't quote it but that same page of the Jargon File has a bunch of unconscious gender essentialism as well (the usual "the wimminz just aren't *interested*" horseshit).

Kevin Lyda Apr 2, 2024

@zwol in CI/CD build systems the test build is separate (at least it should be). It should depend on the actual build, but the only output that's kept from the test build are the test results.

Zack Weinberg Apr 2, 2024

@lyda That's not enough of a firewall; the xz backdoor pulled data out of the testsuite during the normal build.

Wendy (prev Jason) A Apr 2, 2024

@zwol it’s 2024. Assuming Python exists would not be out of line. That would cover all the hobbyists leaving behind organizations that should probably be coughing up money to solve their own problems.

Zack Weinberg Apr 2, 2024

@norgralin the problem with assuming python exists is it has an enormous dependency list and a big hairy configure script itself

Rep. Eric Gallager (no "h"!)Apr 3, 2024

@norgralin @zwol IMO, a statement to the effect of "it's safe to assume the existence of Python" ought to get added to POSIX first

Ludovic Courtès Apr 2, 2024

@zwol Do check out this post by @jas4711 on the question of reproducible tarballs:
https://blog.josefsson.org/2024/04/01/towards-reproducible-minimal-source-code-tarballs-please-welcome-src-tar-gz/

Towards reproducible minimal source code tarballs? On *-src.tar.gz – Simon Josefsson's blog

Zack Weinberg Apr 2, 2024

@civodul @jas4711 Yep, saw that yesterday, right after I posted about that on the lists

ArneBab Apr 7, 2024

@civodul for the tarballs the main missing piece I see is standardized documentation of dependencies.

But that’s a huge can of worms, because every distribution and most language-specific package-managers already include something for that.

mid_kid Apr 8, 2024

@zwol I'd also argue that it's very easy to tag a commit that's disconnected from the master branch, and people would be just as wise.

Josh Grams Apr 2, 2024

@zwol Lua is larger than the bash? Or is that not the point?

Zack Weinberg Apr 2, 2024

@JoshGrams See my reply to Ludovic; Lua is much smaller than bash, but is still too large IMO, and not *enough* more readable to be worth the switching costs.

Haelwenn /элвэн/

@zwol Didn't the "no build dependencies if you're working from tarballs" kind of died with pkg-config or are you referring to the `make dist` thing? (which I'd absolutely want gone as a packager).

And I think outside of effectively switching to meson/muon (muon being meson-compatible without pulling in python btw), would be a shell library which could entirely error out with a nice message when it fails to be sourced.
But you could very quickly end up repeating the errors of CMake here.

Kevin Lyda Apr 2, 2024

@zwol sure, but tools like shellcheck exist to at least encourage better standards.

Jamey Sharp Apr 2, 2024

@zwol I'm not sure I'm quite prepared to accept Forth as the standard build language, but I certainly wouldn't be sad about it if that turned out to be the answer everyone went with.

That makes me curious, though, how well WebAssembly might do as a compromise. Its text representation can be reasonably clear, at least as far as stack languages go, and it's nicely explicit about what non-computational (I/O) capabilities you're asking for. I can think of various ways that could work that seem nice to me, but this is already a thought experiment on top of a thought experiment so I'll stop there.

Zack Weinberg Apr 2, 2024

@jamey For possibly irrational reasons having to do with my past employment history I don't trust WebAssembly to continue being a going concern. I might change my mind in another 15 years or so.

Jamey Sharp Apr 2, 2024

@zwol Fair enough! It was pretty much idle speculation anyway.

Josh Triplett Apr 2, 2024

Leaving aside *which* specific sandbox we use, there's something to be said for build systems being a sandbox that has well-defined capabilities and non-capabilities. Earlier in the thread you mentioned having test suites not write to the source directory, for instance; a sandbox would allow enforcing that. Similarly, we could set rules for what's unreasonable for a build system to read and to write in which phases, and such constraints could make it easier to review and catch weirdness.

Zack Weinberg Apr 2, 2024

@josh @jamey Yeah, sandboxes in general I like, except for sandboxes that are themselves a security disaster (have you ever looked at the code for firejail? )

For clarity, I agree with "tests shouldn't write to the source directory" (indeed, *nothing* should write to the source directory) but that's not what I was saying. What I was saying was "the regular build should never need to *read* bits of the testsuite."

Thomas Depierre Apr 2, 2024

@zwol na, Forth too big too

But we can cheat. We can compile the build system to a binary. Or bring its own interpreter. Embrace the reality. Build systems are a programming language. So let's properly build one.

Ludovic Courtès Apr 2, 2024

@zwol In #bootstrapping circles, we have GNU Mes and Gash (the combination of which is good enough to run ./configure scripts).

The Racket folks have switched to Zuo as their build system, also based on a minimal Scheme implementation.

Maybe not a universal option, but I can imagine a build system based on Mes/Zuo, at least in the circles I care about.

Simon Tournier Apr 2, 2024

@civodul Hum the attack would be a bit more sophisticated for GNU Mes and Gash as implemented in Guix. But still…

Instead of targeting plain Bash, one needs to target the Guix package ’guile-bootstrap’. This package depends on tar, bash, mkdir and xz; it adds some surface.

Else, it would also be possible to exploit the non-deterministic Gash compilation to hide stuff.

https://simon.tournier.info/posts/2023-10-01-bootstrapping.html

The attack would be much more complicated, I guess.

Is Guix full-source bootstrap a lie?

Zack Weinberg Apr 2, 2024

@zimoun @civodul I thought about it some more and absolute size is not the most important issue here; the most important issues are (1) how difficult is it to install the thing, and (2) how much more readable than sh(+m4)+make do you get for the effort. That said, size does matter in that someone might want to audit the language they're being asked to install, on top of everything else. And the big popular interpreted languages tend to have large dependency graphs, which makes their true size even bigger, makes them harder to install, and makes problems for bootstrapping.

Python and Perl are very large (current releases are ~1.2M lines of code each according to SLOCCount), nontrivial to install from source, and problematic at the lowest levels of the bootstrap chain.

A mostly complete implementation of POSIX shell and utils, namely busybox, can be fit into 200,000 lines. bash+coreutils has important missing pieces (grep, sed, awk, find, diff are the ones I know about) and is about twice as big.

mes+gash+gash-utils is ~70,000 lines. Lua is ~20,000. Neither Scheme nor Lua feels like *enough* of a readability improvement over sh to be worth the switching costs.

I would say that 20,000 lines of C is about the upper limit for what I'd feel comfortable demanding people install before they can build the thing they actually wanted to build.

Furthermore, any such component cannot require a complex configure+build process itself lest we have a circular dependency.

Ludovic Courtès Apr 2, 2024

@zwol @zimoun To be fair, Mes includes a C library, a C compiler with 4 backends, etc. The parts that would matter here are the interpreter, which is ~6K lines of C under src/.

Zuo has an interpreter with ~8K lines of C and ~5K lines of Zuo (Scheme).

This should be compared with the line counts of Perl + Auto{conf,make} + Make or CMake + Make/Ninja.

Ludovic Courtès Apr 2, 2024

@zimoun @zwol Speaking of build systems: in 2008, Tom Tromey wrote Quagmire, a proof-of-concept replacement of Autoconf + Automake, mostly compatible with the latter, implemented in GNU Make (~1K lines).

https://tromey.com/blog/?cat=16
https://github.com/tromey/quagmire

It’s appealing because GNU Make is ubiquitous and ‘Quagmire’ files looked very much like ‘Makefile.am’.

The downside is that it’s hard to debug and work with (lots of ‘eval’ tricks…). Less appealing than Zuo or similar to me.

quagmire — The Cliffs of Inanity

Zack Weinberg Apr 3, 2024

@civodul @zimoun Oh, I remember when Tom announced Quagmire. Hard to debug is exactly what we don't want, though — I bet it would be easy to hide a back door in 1000 lines of cryptic Makefile tricks.

@zwol @zimoun @civodul I’ve seen Fossil use Tcl for their configure script and shipping a micro-Tcl called jim that’s a single .c file for when you don’t have Tcl

ArneBab Apr 7, 2024

@zimoun regarding gash in another scheme: that would be solved most beautifully if gash could be run by the Scheme from @janneke ’s MES. @civodul @zwol

Simon Tournier Apr 12, 2024

@ArneBab Hum, not really from my understanding.

There is a chicken-or-the-egg problem for the "driver" (currently guile-bootstrap). From my understanding, the only option for removing tar, bash, mkdir and xz is to have an implementation directly in binary (hex).

Well, that’s what I detail in the sections:

« Analysing guile-bootstrap derivation »
« Opinionated next steps »

from: https://simon.tournier.info/posts/2023-10-01-bootstrapping.html
@zwol @civodul @janneke

Is Guix full-source bootstrap a lie?

ArneBab Apr 13, 2024

@zimoun I think I understand what you mean: this still needs something to run it.

Maybe avoiding compressions for tarballs could make xz unnecessary. Would it then suffice to have the scheme from mes capable of running as driver?

mes bootstraps from binary, IIRC, so this could alleviate more problems? https://www.gnu.org/software/mes/manual/mes.html#Full-Source-Bootstrap

@zwol @civodul @janneke

GNU Mes Reference Manual

GNU Mes Reference Manual

ArneBab Apr 13, 2024

@zimoun (or is the driver needed for mes itself? ⇒ we’d need a driver built on m2-planet? Or only on parts of mes?) @zwol @civodul @janneke

Simon Tournier Apr 15, 2024

@ArneBab « Would it then suffice to have the scheme from mes capable of running as driver? »

See Fig.1 https://simon.tournier.info/posts/2023-10-01-bootstrapping.html

The question is how to run ’bootstrap-seeds’, which is stage0 and M2-Planet. We need guile-bootstrap (driver) which relies on helpers (bash, tar, etc.)

Chicken-or-the-egg problem. We need a binary driver to get MES. IMHO, the only option’s binary driver written by hand, somehow.

@janneke @civodul @zwol

Is Guix full-source bootstrap a lie?

Janneke Apr 13, 2024

@zimoun @ArneBab @zwol @civodul that's a pretty nice writeup.with some valid concerns.

Philip McGrath Apr 2, 2024

@civodul @zwol Upstream Chez Scheme now also uses Zuo. And by “minimal scheme implementation” there is a single file, zuo.c, that builds with `cc -o zuo zuo.c`—no configure required, and even very old or limited C compilers supported.

Philip McGrath Apr 7, 2024

@civodul @zwol In this context, maybe it’s worth mentioning that Racket and Chez Scheme use Zuo to replace make (keeping a stub makefile): they specifically don’t try to replace configure. Racket uses Autoconf; Chez Scheme uses a handwritten shell script.

The Zuo language certainly could be used to write a configure script. I mention this just to reaffirm that implementing ./configure does have specific challenges!

Philip McGrath Apr 7, 2024

@civodul @zwol In fact, I nudged Zuo to use Autoconf for its optional configure script to get the standard installation flags and such to work.

Andy Wingo Apr 7, 2024

@LiberalArtist @civodul @zwol probably the biggest obstacle for guile to switch away from autoconf would be gnulib (include-only lib to polyfill modern posix+gnu extensions); it integrates tightly with autoconf/m4/automake