Mastodawn

✧✦Catherine✦✧

people: ask their dependencies to follow semver, for fuck's sake already
also people: make a surprised pikachu face when the major version is incremented with every release

(i have been both, at times. this is about me. this is also about others who i've seen be a lot more militant about this issue)

the thing is, if you have a sufficiently complicated application it is not feasible to determine what is a "breaking change" or not. this complexity limit kicks in long before you get to a "browser" or a "JIT compiler" but it is definitely well applicable by that point

i think what people mean when they do both of those things are a mix of "please stop adding features entirely. only fix bugs" and "please only make changes i like, but not the changes i dislike" depending on maturity level. that's not really how open source software works though

✧✦Catherine✦✧2d ago

anyway that's how you end up with https://codeberg.org/whitequark/track-pypi-dependency-version

track-pypi-dependency-version

A script for use with GitHub Actions that updates the upper bound in requirements.txt when a package is released on PyPI

Codeberg.org

datenwolf 2d ago

Yep.

I recently switched to the following scheme

${n-th iteration of "major cleanups that got rid off legacy cruft"*2 + is a release ? 1 : 0}.${year%100}${week of year/2 + 'a'}

@whitequark It doesn't hurt to try, but yes semver is being a bit delusional about the complexity of reality. Law of leaky abstractions type stuff

✧✦Catherine✦✧2d ago

@bubbline yeah basically

✧✦Catherine✦✧2d ago

@bubbline i think you can define a scope for what a breaking change is and then tell people to deal with it if their specific change is out of scope, but this leaves people exactly as mad (or madder) than if you never followed semver in first place

@whitequark this is exactly why I am a bit of a CalVer zealot. abandon the illusion. only by freeing yourself from desire can you achieve enlightenment

✧✦Catherine✦✧2d ago

@glyph i think that's pretty rude to your downstream code in the Python ecosystem in particular (you know, the one that doesn't let two different versions of a package coexist on the same system)

if i did this for Amaranth i'd have split the ecosystem into tribes based on which year they started using it more or less. that would fucking suck

@whitequark I don’t know if amaranth has any special requirements but we have been doing it for 15 years on twisted and no such bifurcation has occurred

✧✦Catherine✦✧2d ago

@glyph amaranth routinely deprecates and removes broken features in a way that requires manual upgrades of significant amounts of hard-to-verify code

@whitequark how many major-version branches do you maintain simultaneously?

✧✦Catherine✦✧2d ago

@glyph two (main and last release); if main ever breaks something that worked in the last release instead of emitting a diagnostic it's considered a critical bug

✧✦Catherine✦✧2d ago

@glyph this still results in some stratification but considering that some other projects in this domain are quite literally frozen forever because the downstream consumers object to any changes i think this is a major win

@whitequark in python (and many other languages) you can’t even get your tools to produce something as crude as an soversion. If we had a robust tool that could do something like “make a DAG of all publicly defined names and tell me if anything has been changed or removed between these two git tags” then we could maybe TRY to do semver but as it is I cannot imagine anyone doing it correctly

✧✦Catherine✦✧2d ago

@glyph I think that's a really silly absolutism. who cares if all the names are there if the types have changed? who cares if the types are the same if the behavior changed? this exact line of thought implies you can never have any notion of compatibility at all

i think of version numbers as a communication tool, or sort of a filter: they tell you if shit's definitely broken or maybe fine. this is a useful signal

Janne Moren 2d ago

@whitequark @glyph
Yep, versioning is at least as much a social signal as a technical one. With enough users *any* change can be a breaking change. Fix a misspelling of an error message and someone's been grep-ing for that in their wrapper script.

Semver speaks of you intentions, not just the results. You bop the minor number and it breaks someone's code? You're likely to take the bug report seriously. You increased the major number? You send them a link to the release announcement.

@whitequark I may be making some "live internet-facing service using libraries" assumptions about how updates need to be managed, which is to say, you have some compliance and possibly legal obligations to upgrade to new security supported versions in a timely way, every application is in its own venv (and probably its own container or even its own VM), there's a staff of people doing regular maintenance. for dependencies of such environments semver doesn't make sense

@whitequark the social function semver serves in that context is "I want to use version 3 forever because I don't give a shit about the health or development direction of the projects I depend on, I just want free security support where I can do no-look upgrades for eternity". every time you do a "breaking change" (which is to say: remove any deprecated name) you bump the major version which means all your dependents stop getting updates and then they get mad

@whitequark pinning to a major version in this context is just allowing downstreams to accumulate pain to turn every "major version" upgrade into a looming catastrophe when the last major version goes out of support. meanwhile you end up having to support 9 major version branches, and if you don't, Red Hat comes along and starts pretending you have security support for those versions too

✧✦Catherine✦✧2d ago

@glyph yeah so what I've seen with my downstream is that if I break too many things in a way that's annoying they just pick a version they like and never upgrade at all. which effectively results in a fork

Garth Kidd 1d ago

@glyph @whitequark we’ll be maintaining Python 3.6 code until 2029 because EL8 /usr/libexec/platform-python

✧✦Catherine✦✧2d ago

@glyph yes, you are, which is my point. Amaranth is (in the vast majority of contexts) not a security-relevant application and if Python was more like Rust in terms of how it manages dependencies, it would have been actively desirable to have some stub library that lets you link together modules built with different versions of the language; typically if you make a HDL module, you verify it and then you just don't touch it ever. (this is not the only possible workflow, but this is the de-facto standard workflow in the industry, and a lot of the time it's close to the best you can do because the interface of the module, including the bugs, gets implicitly embedded in projects like "Linux" where if you change anything your actual end users will see an update maybe in five years)

✧✦Catherine✦✧2d ago

@glyph the other thing is that I guess this makes me not particularly interested in using Twisted (I have no other context here) because I don't want to be on call for this any more than humanly feasible, and I am not a startup that can afford to burn infinite VC cash chasing upgrades

@whitequark to be clear, we don't just do "calver, but every micro-release breaks everything", we have a rolling window where every release (potentially) removes deprecated stuff, and deprecates new stuff. if you have tests and they run without warnings, you should be able to upgrade without any work. the workflow for downstreams (who, again, must be upgrading periodically anyway) is "run & fix tests until no warnings, upgrade one version, repeat"

@whitequark in practice, among our cohort of comparable dependencies we have a reputation for extreme stability because we don't have these big major-version jumps where huge arbitrary clumps of stuff all break at once, every individual upgrade is usually between "zero" and "trivial" in terms of the amount of work it takes

✧✦Catherine✦✧2d ago

@glyph yes, but your test workflow is of a kind that doesn't involve "you probably have to run it on a real hardware, and maybe plug in an oscilloscope and spend a day chasing a single-cycle problem resulting in updating an expression somewhere in the guts of an application"

@whitequark indeed not. I am pretty impressed that you get people to upgrade at all in the face of that. do your downstreams just pick a major version and ride it out until the end of support or can you convince them to upgrade major versions? and what kind of stuff do you change in minor/micro versions that don't risk breakage?

✧✦Catherine✦✧2d ago

@glyph I am impressed too, hah. I think this works because I'm really zealous about either not breaking stuff at all, or breaking it in a way that's legible: where you get a diagnostic pointing you to the place that's broken and, usually, telling you exactly how to fix it and what the rationale is

it's a mix of several things: some downstreams pick a major version and ride it out, some run against main, some test against both main and last major

(keep in mind that Amaranth has a 0.x.y version, but x is de-facto major and y is de-facto micro; there is no minor because there are neither resources nor justification to do minor releases)

micro versions exist for two things:

fixing regressions that should've never gotten into the release in first place
exceptionally, making backports that are necessary for ecosystem health, like introducing an interface that an accepted RFC places in the next release so that people can upgrade in a more gradual way. this doesn't happen a lot.

one thing that not using calver allows is having a predictable deprecation train. there are currently 0 people working on Amaranth full-time. the RFC can say "we deprecate this over two x increments" but it can't say "we deprecate this over the course of the next year" because truthfully who knows how much stuff gets implemented over the next year?

✧✦Catherine✦✧2d ago

@glyph also, the high-level plan is to get things in a good enough shape for a conceptual "1.0" and then freeze the core language Rust-style; I think this is both viable and desirable and I have funding for this, just have to take care of immigration and stuff first

@whitequark this is surprisingly close to where twisted landed despite the very different requirements and ecosystem pressures. our version of a "legible break" is that we try to never ever touch the presence or the signature of a public name; changes are supposed to show up in the form of new name introduced / old name deprecated / (mandatory minimum delay) / old name removed. the minimum delay is 2 versions or a year, whichever is longer

@whitequark we have a lot more flexibility on "behavior" and sometimes security fixes require behavioral changes, but in practice we really rarely make intentional behavioral changes that actually break stuff. if there's a regression we'll do a micro-version that fixes the behavior but those are extremely rare nowadays, since there are a few downstreams that test regularly against trunk and report bugs (so when stuff like https://github.com/twisted/twisted/pull/12599 happens it's rarely in a release)

fix optionsForClientTLS to negotiate ALPN again by glyph · Pull Request #12599 · twisted/twisted

Scope and purpose Fixes #12598 Add a few words about why this PR is needed and what is its scope. If the associate ticket(s) fully explain the need you can just refer to it/them. Add any comments a...

GitHub

✧✦Catherine✦✧2d ago

@glyph yes, I agree! I think the versioning scheme is a bit of a red herring in that we use it to communicate more or less the same thing to very different audiences. (I somewhat expected this to be the case which is also why I objected to the, in your words, CalVer zealotry.)

Amaranth doesn't really have the liberty of doing something like "replacing the If keyword with an If2"—technically it could work but it's not how people expect programming languages to work—but we do similar things where possible. in general we only ever remove public names, or add new keyword names or add keyword-only arguments to existing functions

Alyssa Coghlan 2d ago

@glyph @whitequark Many hardware downstreams *don't* upgrade in place, they upgrade on new device builds (which is then covered by factory acceptance testing). Physical installations can then be ring fenced with less critical network devices that receive routine updates to mitigate the security risks (complete air gaps are also nice, but the lack of routine telemetry then becomes a risk in its own right). Major software upgrades get lumped in with hardware revisions (and the associated testing).

✧✦Catherine✦✧2d ago

@ancoghlan @glyph yep, that's definitely one way in which this is commonly done! I would describe this on a higher level as "there is an upgrade workflow that is already in place because of some other, more powerful forces, and so fitting your software into this existing workflow can be a good match"

@whitequark @ancoghlan I am definitely going to encode the calver zealotry in a longer blog post about upgrade workflows, and this is a good contextual distinction to keep in mind. I still think that most of my religious beliefs about the way upgrades need to work apply here (the big one being: tools mostly don't exist to enforce the ability to upgrade without testing so you cannot upgrade to ANY version without testing) but there are definitely cultural distinctions about expectations

@whitequark @ancoghlan it's interesting that this superficially resembles what, in my mind, was the Bad Old Days of the internet-service context, where downstreams annoyed by some small breakage would just refuse to upgrade forever or would create internal, broken, unsupported and unsupportable forks, which they had to stop doing in ~2014 because PCI and SOC2 auditors started getting _real_ mad, which was extremely healthy leverage at the time. it's very different, though.

Alyssa Coghlan 2d ago

@glyph @whitequark Yeah, it's untenable as a universal practice for networked devices, as they need to evolve in parallel with the networks they're connected to. Ring fencing also leaves you horribly exposed if the perimeter ever gets compromised. Testing software updates against every hardware revision ever published is its own special flavour of awful though, so there are no good answers, just differently bad choices to weigh against the specifics of a deployment model.

@whitequark @glyph "major version changes when existing tests are updated"?

✧✦Catherine✦✧2d ago

@gabe @glyph that addresses intentional breakage, but that is the easy bit: the reason we're having this discussion is that in a complex enough system, any change—including changes that aren't tested because nobody thought they'd be relevant—could and eventually will be a "breaking" one (in the sense that it breaks something important)

eviloatmeal, resident Death Stranding 2 apologist (arch, btw)2d ago

@whitequark I, for one, am in the camp of the former ("please stop adding features").

Tim van der Lippe 2d ago

@whitequark sounds like an instance of https://www.hyrumslaw.com/

Hyrum's Law

James Healy 2d ago

@whitequark labels aside, if actual breaking changes are regularly made then it pushes workload to package consumers and builds resentment. Sane deprecation policies that avoid intentional breaking changes for a reasonable time build trust

David Chisnall (*Now with 50% more sarcasm!*)2d ago

I absolutely detest SemVer because it has no way of doing graceful deprecation, which arises because it conflates implementation and interface.

A library that wants to do graceful deprecation will have three releases, A, B, and C, where B introduces a new API and C removes the old one. C is a breaking change for anything coming from A. C is a breaking change for anything using B and not moving to the new interfaces.

You really want a set of SemVers, for each interface you support. So you actually have something like:

A: {1.0}
B: {1.1, 2.0}
C: {2.1}

I have never seen a system using SemVer that supports this. You can kind-of do it with UNIX SONAME and symlinks, where A has symlinks names libfoo.so.1, libfoo.so.1.1, and then B has symlinks named libfoo.so.2, libfoo.so.2.0, libfoo.so.1, libfoo.so.1.1 and libfoo.so.1.0, but I’ve never seen it done.

✧✦Catherine✦✧2d ago

@david_chisnall yeah you're on point

David Chisnall (*Now with 50% more sarcasm!*)2d ago

One of the few things I like about Windows is that COM doesn’t let you change interfaces. You can create a new interface, with a different name or version, but once you’ve shipped an interface it’s stable forever. Windows DLLs can export many different versions of interfaces. And COM has an IDL so you can surface these in many different languages without needing to go via C.

EDIT: COM is far from perfect, but it’s also over 30 years old at this point. There’s no excuse for having things that are worse than COM.

✧✦Catherine✦✧1d ago

@david_chisnall yeah, COM is an improvement on many of its successors.

what I do in Amaranth is I split it into pieces small enough that each, individually, can be maybe reasonably approximated with SemVer, but it's far from perfect

I think the closest approximation in mainstream PLs for what you're proposing is Rust, where you could have e.g. rand 0.9 depend on rand 1.0 (and reexport some types from the latter) so you can have both of them used in the same project and you don't end up with horrible diamond dependency issues. but it's far from intuitive or convenient.

@david_chisnall @whitequark

tangentically related, theres an interesting set of papers nd research on encoding versioning at the type level, from ppl at the tokyo institute of technology

Some links:
https://prg.is.titech.ac.jp/projects/modularity/version-programming/

https://prg.is.titech.ac.jp/wp-content/uploads/2023/03/tanabe-phd-dissertation.pdf (https://github.com/yudaitnb/vl)

https://prg.is.titech.ac.jp/papers/pdf/pro2024-4-8.pdf (https://github.com/prg-titech/vython)

https://prg.is.titech.ac.jp/papers/pdf/sle2022.pdf (https://github.com/ansharlubis/batakjava)

i also did a bunch of design work on simmilar ideas but lost steam on my project

Programming with Versions – Programming Research Group

David Chisnall (*Now with 50% more sarcasm!*)2d ago

@mini_ninja_64 @whitequark

Oh, interesting! Do they have a good solution to the diamond dependency problem? If my program depends on libraries X and Y, but X depends on Z version A, and Y depends on Z version B, but both X and Y expose types from Z in their interfaces.

I have some tentative ideas for how to solve this in Verona, but I’d love to see if someone else has a better solution already!

David Chisnall (*Now with 50% more sarcasm!*)4h ago

@mini_ninja_64 @whitequark

I’ve skimmed a few papers and they seem to be not really handling that case. In particular, there are a bunch of issues that come up in structural (or late-bound dynamic OO) type systems.

For example, consider version A of some concrete type exposed from Z lacks a method, but version B adds it. This changes the behaviour of pattern matches on that type (and on interface conformance with structural typing). Library X using the old version must depend on the method not existing, whereas library Y expects to call that method.

I believe the solution to this requires viewpoint adaptation in pattern matching, which can be done by versioning interfaces and a little bit of trickery in the runtime. But I’d really love to see a type system that actually does this.

@david_chisnall @whitequark soz was distracted yesteqrday, I remember some vague mentions of the indirect dependency use problem, when i read the VL paper, if memory serves, my understanding was that the 2 re-exports would be distinct types due to the version constraint I would imagine, the 2 types to be incompatible in the X & Y interfaces?

yehh makes sense, my design at the time was v simple typing so avoided lots of problems but the thought deffo crossed my mind.

@mini_ninja_64 @david_chisnall @whitequark Oooh, seeing a paper by Hidehiko on the fediverse is wild :D
this world is small.

@krono @david_chisnall @whitequark oooo u know them?

@mini_ninja_64 @david_chisnall @whitequark Hidehiko Masuhara has been my PhD thesis' third reviewer.
I just met him last week. He's a very kind, clever, and all around nice person.

verified_dragon

@david_chisnall @whitequark I feel like pride version makes it a nicer experience despite not really fixing the things you mention lol

Martin Uecker 2d ago

@david_chisnall @whitequark On Linux, libraries would provide a mix of 1.1, 2.0 versions on the symbol level in the same binary.

@david_chisnall @whitequark So much this... definitely one of the main reasons I find claims of "Semantic versioning solves all your versioning problems" frustrating... the other being that interface contracts are very rarely tightly specified enough to be certain that any given change is "non-breaking" (if only in an https://xkcd.com/1172/ sense!)

Workflow

xkcd

@whitequark It's in effect very similar to using version 0 forever, but a bit more information about larger changes.

David Foster 1d ago

@whitequark It’s funny: I work on mostly applications with GUI/web interfaces (with no expected dependent packages), where it’s frequently not clear what a “breaking change” would mean. So in practice I use romantic versioning a lot more than SemVer. https://dafoster.net/articles/2015/03/14/semantic-versioning-vs-romantic-versioning/

Semantic Versioning vs. Romantic Versioning | DaFoster