Part of the problem in the software industry (and one that open source partially addresses) is that the business models fundamentally do not reflect economic realities.

Software is, to within a rounding error, a zero marginal cost good. This means that, once you’ve made one of a thing, the additional cost of another (the marginal cost) is zero. Most things benefit from economies of scale. If you make one bespoke widget, you have a bunch of costs in fabrication on top of the materials costs. If you build a factory to create widgets, the cost of the factory is big, but it’s amortised across all of the widgets you make. If you spend $1,000,000 building a factory and aim to sell 1,000,000 widgets, the factory cost contributes $1 to the cost of each widget. For simplicity, I’m going to lump all of the R&D effort involved in designing the widget and figuring out how to build it in the factory costs. They are slightly different classes of costs (they don’t scale with the rate of production, whereas if you want to make widgets twice as fast you will need a bigger and more expensive factory) but they’re broadly similar.

If you somehow increase your market and can now sell 10,000,000 widgets, the factory cost adds only $0.10 to each widget. The more you sell, the lower your fixed up-front costs are as a proportion of your total costs. As these go down (or are paid off by early profits, so the factory is now effectively free), other costs matter more. Factories aren’t magical replicators. You need to feed them with raw materials (or components). You need to pay people to operate them. You need to pay for maintenance and repairs on the factory itself.

Even after the cost of the factory and R&D are all paid for, there’s still a cost. If you’re selling widgets for $10, these costs may add up to $8/widget and, as the other costs are paid off your profit goes from $1/widget to $2/widget. The model where you build the factory as an investment and aim to recoup the money selling widgets makes sense to you, but also to your customers. They can view it as spending $8 on materials and operating costs and $2 renting a factory (which is far cheaper than the $1,000,000 to build their own).

Software does not work this way. The cost of writing software is high. It takes time and skill, and more time and skill to test it. If I wanted to write a word processor, for example, I’d need to hire a moderately large team (tens to hundreds of people, depending on my exact feature requirements) and it would take them a few years to ship the product. Once they have built it, creating a second copy costs (to a first approximation) nothing. If you have a bunch of users who want a word processor, I can sell it to you, and the cost to me is no different if you have one user or 10,000 users.

Business models that make sense for zero-marginal-cost goods are tricky. If it cost me $10,000,000 to make and I think I can sell it to 100,000 people, I can pick a price of $100/user. If you are buying it as an individual, that seems like a great deal: far lower than the cost of building it yourself. But if you are a company or government with 1,000,000 users, you could build it in house for a tenth the amount. Except you aren’t a software company, so maybe you don’t want to. And I really want that sale, so I can give you a discount.

And that’s where it starts to get weird. If I can guarantee 100,000 sales at $100 each, any additional sale I make at any (positive) price is profit. If you want 10,000,000 licenses, I can five you a great deal of only $500,000 and that’s still pure profit for me. Wonderful.

But, at some point, I’ve sold my word processor to everyone who wants a word processor, and I’m employing a team of people who are experts in word processors. Because I’m selling software as an immutable artefact, the obvious thing for me to do is figure out some new features people want and do a new version.

New versions are great because they cost you a lot less to make than the original (they are incremental changes) but would cost a new competitor more because they would have to start from scratch. Now, my total development cost is $15,000,000, but I’ve already paid for two thirds of that. I had to fix some technical debt, but a new competitor would have to spend $13,000,000 to reach the same feature set, so I can sell this as if it cost me as much to make as the first version and customers still think they have a good deal. And maybe you give existing customers a discount on an upgrade version, so they are even more likely to stick with you than go elsewhere.

Eventually though, I hit diminishing returns. You, the customer, look at version 47 and say ‘actually, I can’t think of a single feature introduced since version 42 that I use, why should I pay for the new version?’. And a big customer may think ‘I’ve been buying this for ages and version 23 was pretty much feature complete from my perspective, maybe I should pay someone else to build an equivalent of version 23 that I own and can add new features to if I need them’.

My incentive is to make the program big and complicated so that you don’t really believe you could create an equivalent. The more features, the better. Because someone probably needs them, and you need to be able to interoperate with them and so you can’t replace my product with something that isn’t exactly feature compatible.

To address these risks, I you need to keep adding features and convincing users that they want them. Spell checking, grammar checking, cloud syncing, AI, and so on. Some of them are actually useful, some I put marketing effort into so people are convinced they need them even if they don’t know why.

And, hey, if that fails, I can always make some changes to the file format so that old versions can’t open the new documents. Oh, sorry, your customers are using version 50 to send you purchase orders? I guess you’ll need to upgrade to be able to read them. Please talk to my sales team, I’m sure we can come to some arrangement.

And all the while, there’s an elephant in the room: bugs. Version 38 had a nasty parser bug that let someone who sent you a maliciously crafted file compromise your computer. Version 39 had one where if you made your company name bold, it would crash. And you do want these fixed.

And now we get to a really tricky problem. If I’d done the level of formal verification required to absolutely guarantee that none of these bugs existed, I’d have had to charge $10,000 for v1 and no one would have bought it. At the same time, you (quite reasonably) think you paid for your copy and so should expect bug fixes (and, especially, fixes for security issues): it was sold as a working product, even if the small print did have some disclaimers of liability.

So I propose a compromise. You buy the current version and I’ll give you free bug fixes for three years. But what bugs will I fix? That bug with crashes for bold on your company name is triggered on your corporate letterhead and nothing else. Engineer time to fix that will cost more than I made on sales to you. So maybe I don’t fix that one especially, but it turns out that v40 had refactored that code while adding a new feature and now the bug is gone. So now you need to buy the new version. Unfortunately, the new features also introduced some new bugs.

Okay, this clearly isn’t working. I have a new idea. I’ll sell you a subscription instead of a product. For $50/year, you are always running the latest version. This is good for me in two ways. First, I don’t have to maintain back-ports of bug fixes to old versions, I just have one supported version. Second, I have guaranteed revenue. It’s good for you because you can predict your spending well in advance.

But, again, at some point you are going to have spent more on the subscription than it would cost to have built the subset of the product that you need for yourself.

So my incentives are the same. I must make the app as big as possible so you don’t think you can replace it. I must tightly couple it to all of my other products, so you can’t replace just one bit. Sure, you could replace my word processor, but it talks a proprietary protocol to my cloud back end for collaborative editing, and that uses the same namespace as my file storage, so you lose that if you switch. And the collaboration is built around my proprietary account management system which you use to restrict access to people in your company (and specific people or groups).

I build plugin interfaces that make sure the plugin is subservient to my program, so other people can add value to my ecosystem but can’t become competitors, rather than allowing interoperability with peers where you can select the components you want and that might not include any of mine.

To maintain my business model of writing software for free and then charging you to use it, I must build things that are bad software engineering and user hostile.

A F/OSS system has none of those incentives, which is why it makes me so sad to see them copying these structures. A F/OSS platform could focus on lightweight interoperable components that users can compose in arbitrary ways. A system in which the cost of developing a new component for a specific use case is small enough that individual programmers can do it and other people can afford to pay a programmer to do it is great for a F/OSS ecosystem. Paying someone to add a feature to a massive monolith like LibreOffice is hard (not least because they must be able to upstream it, which requires time and effort from upstream maintainers, if you want to use it long term). Paying someone to write a component for a modular component system is much easier and doesn’t require anyone else to like it.

Component systems were popular in the early to mid ‘90s, but died out because they are hostile to the COTS business model. Microsoft realised that replacing Office with a set of COM components that composed to build arbitrary document classes was completely feasible, but paved the way for competitors to incrementally replace them and displace MS Office’s dominance. The trend has been in the opposite direction, towards walled-garden operating systems with siloed apps, each striving to become the legendary ‘everything app’.

F/OSS doesn’t need to copy this. It can build structures that proprietary software simply cannot replicate because they are directly incompatible with a rent-extraction model for funding development. And, in this world, licenses like the GPL are both unnecessary (someone wants to incorporate my component in a proprietary system? Go ahead, it won’t be competitive in my ecosystem) and harmful (in a system with a hundred individual components tightly collaborating, where my document includes a custom plugin that is a derived work of ten other plugins, what are my legal obligations if I send this document to my lawyer under attorney-client privilege?).

Wow, 15 characters short of the limit! I’ve never come that close to the character limit here before!

Congratulations to anyone who actually made it to the end!

@david_chisnall Made it to the end!? This is the most brilliant description of what went wrong with tech I've read so far 🙌 🙌 🙌
@david_chisnall

Epic!

I just posted a video about photo editors and subscription pricing, but it's shorter than your toot today :)

The incentives for software companies is a huge deal, and you've explained in great detail where the problem lies.
@david_chisnall Awesome! It was a pleasure reading it.

@david_chisnall I stuck on this sentence:

someone wants to incorporate my component in a proprietary system? Go ahead, it won’t be competitive in my ecosystem

and was thinking, if someone is incorporating my component, it definitely is adding value to their product. If the product does whatever my ecosystem does and that company has enough resources to operate at a loss until my stuff are irrelevant to the world, then I'd see incentive in avoiding that.

Am I missing something here?

@pft Yes. First you’re missing that the flexibility of an open system is the benefit. If you build systems where poking the internals of a component and distributing the result is something normal users do routinely, someone who comes along and says ‘we have a thing like this, but without the ability to do the thing you do all the time’ will be laughed at. Their proprietary system with your component will be better than their proprietary component without it. But that’s not the right comparison, it’s whether the open system will be better with their proprietary product, and it won’t be because, by definition, you can do less with a proprietary component.

Second, you’re ignoring that the ideal component is small. Something a single person can easily maintain. So if they don’t want to take your component due to license or other reasons, they can always rewrite it in house. But that’s where an open model is inverted from a proprietary one. In the proprietary world, you want large systems so that the cost of copying them is high. In an open system, you want small components so that they can individually be ‘done’ and require almost no maintenance. To compete with such a system, you don’t have to reproduce a component, you have to reproduce all components. And this brings me to a maxim from (I think) Scott McNeally:

Always remember that a lot more smart people don’t work for you than do.

If you do it well, a company trying to build a proprietary alternative to your system needs to produce as much software as everyone else who doesn’t work for them does.

@david_chisnall I do not give all toots the "bookmark, star & boost" treatment, but this one even deserved the "reply" cherry on top.

Nicely put. Anyone with some experience in software has probably noticed this, but it was very well written, and topping it with the example on the finances around the problem (hopefully) make the idea approachable to the larger public.

@david_chisnall wow. excellent piece, sir. and you just tooted it out for free haha. bur seriously: well written. keep it up

@david_chisnall

This is why I'm building Uni's Easy. The current system's incentives are completely broken for education.

@david_chisnall I liked this, especially your call for composable FOSS, and how much sense that makes.

Current software, imho, is completely held back by walled garden and siloed App mentality, and this isn’t something we need to keep doing.

I wondered if you’d address how FOSS developers might be funded, though, because this remains a mystery to me 🙂

My ongoing side project is trying to build common data storage and collaboration infrastructure to support the kind of composable software you describe. Think: local SQLite like storage with Git-like (but conflict free) merging).

It’d be a lot easier to build this if it came with an income stream! Then I could do it with others, and do it instead of a main job.

@benjohn

I wondered if you’d address how FOSS developers might be funded, though, because this remains a mystery to me 🙂

I think funding becomes a much more tractable problem if components are small, but there is still a problem of user visibility. I've seen a few F/OSS apps put donation things in their GUI, and that's fine, except that 95%+ of the code that they ship is libraries that are not directly visible to the user. With a component model, that becomes a bit better, because each of those might be exposed on a tool palette or similar and you can dig in and see that ones you need.

But I think the key thing that Free Software (as an ideology, versus Open Source as an economic model) needs to do is ship things where end users want to and are able to make changes. Hiring a developer to add a new feature to LibreOffice is well out of scope for most small businesses. But going to a consultancy like Igalia and asking them to write a simple bespoke component for a modular system would be much cheaper. Done right, I think you can build an ecosystem where all users are able to make some changes on their own, which shows them the value of having user-modifiable code, and then an easy ramp for paying someone else to do the things that you can't do. Most people can put up a few shelves, but they know when to call in a professional. I'd like to see software more like that.

It's much easier for small projects to be 'done'. A few things I maintain are basically feature compete. I spend no more than a few hours a year on them. Occasionally the requirements evolve a bit and either I do the work (if I care about it), or I find someone to pay for it (if I don't personally care about the new requirements). Even big new features cost at most a few thousand dollars worth of work, which is a bit much for most individuals to pay but easily affordable by a small company that needs the feature.

My ongoing side project is trying to build common data storage and collaboration infrastructure to support the kind of composable software you describe.

You might be interested in some inspiration from the one we were building 20 years ago.

CoreObject

@david_chisnall

> Most people can put up a few shelves, but they know when to call in a professional. I'd like to see software more like that.

Yes! This is what I'd like to see too, and I think it is unlocked by much more modular software.

There's some nice recent writing about "Bare foot developers" that I liked. A position between "user" and "developer" that's filled by "power users" at the moment… probably can't build a whole application, but can heavily customise.

https://maggieappleton.com/home-cooked-software/

> You might be interested in some inspiration from the one we were building 20 years ago.

… I'll take a look, thank you. A _fairly_ recent idea I'm leaning are CRDTs which allow a great deal of decentralisation.

Home-Cooked Software and Barefoot Developers

The emerging golden age of home-cooked software, barefoot developers, and why the local-first community should help build it

@david_chisnall :-) Not sure how you got those lovely inline quotes, but that's very cool!

@benjohn as someone who is working on general purpose data storage for asynchronous collaboration (think nodes with cbor/json/adts with references to other nodes, similar to a language-agnostic CoreObject afaict) - i am curious about your design.

i admit that i am broadly skeptical of fully automatic merging - to avoid problems i think you need to match the human consensus process on that merge - but that doesn’t stop me being interested in folks’ ideas!

@tryst Fantastic!

I anticipate using CRDTs for merging.

I hope "fully automatic" merge will work in many cases. Some CRDT semantics are to choose an arbitrary (but consistent) "winner". I agree there are situations where this won't be appropriate!

However, CRDTs don't have to do this.

Semantics can be defined to merge consistently in to a _conflict_ state. It's then possible for any user to repair that state. Two users doing this concurrently will again cause a conflict, so this could continue. Hopefully they talk to each other at this point!

… While CRDTs define a consistent automatic merging, you don't have to use it automatically where that doesn't serve you.

You can think of them as versioned files that know how to reconcile. I expect this to also be useful, perhaps in audited environments / for source code.

The core of my approach is strongly typed, immutable, content addressable Merkle DAG for storage and exchange – https://docs.ipfs.tech/concepts/merkle-dag/

I want to use something like Typical for data schema (algebraic data type schema) – https://lib.rs/crates/typical

I'm currently working on BTrees on top of the DAG. These are very useful for databases, but they also allow for edits while reusing almost all (immutable) data, so I think will support efficient and simple state exchange.

Merkle Directed Acyclic Graphs (DAG) | IPFS Docs

Learn about Merkle Directed Acyclic Graphs (DAGs) and why they're important to IPFS.

@tryst … Conflict states would want tool support to help users detect and resolve these where wanted. That’s fine though.

I want the user’s editing environment to come with lots of understanding of the types it works with (and support evolution of these). I envisage the UI here coming mostly automatically (and extensibly) from the data schema.

@tryst I want all of this (underlying system and UI) to be broadly general, although parts of it could be independently taken and used in other tools.

I don't think stand alone applications are at all a good idea, so I want it mostly to be an extensible environment to which new visualisation, data and editing support can be brought incrementally.

I think a fairly good initial choice of application design would be something somewhat like a message / chat app + notes. This is a familiar metaphor that provides a reasonable setting for the collaborative element and would let users discover the possibilities here with both their own notes and project, shared parts of those, and fully collaborative group owned documents.

Some initial built in types I want to support are structured text, relational entities, and spreadsheet like cells.

… it's quite a big project!

@benjohn i agree that this kind of graph-linked data is a natural fit for lots of discrete components. have you seen BTRON? i sometime characterize my broader goal as a distributed collaborative version of BTRON.

my first “application” is a naming system, exposed as a file system. it is somewhat challenging to expose the full capabilities over FUSE, but hopefully it’ll let me move data into the system and have some interop with my existing tools while i work on other components.

are the structured text, relational entities, and tables going to be directly supported in your data model or be built from primitives?

as i’ve been picking away at this i’ve often mused that i’d like to be able to just directly bring an object into a chat. i think there’s some interesting UI work to be done there around what sort of access is being granted to that object - like whether its read-only (but receiving updates), read-write, or read-only snapshot, or a read-write clone (that you could then merge back into the original)…

BTRON - Wikipedia

@tryst wow! That’s a whole world I’ve never heard of. The aspirations of the project sound great. It’d be nice to see a video of someone using that in practice.

@benjohn Nina Kalinina tried it out last year and did a lovely set of screenshots! https://tech.lgbt/@nina_kali_nina/111885033842209483

I haven’t tried it as yet - my japanese is both extremely minimal and out of date. I also haven’t seen video of it.

Nina Kalinina (@[email protected])

Attached: 2 images These days, finding the installation media for the B-right/V OS itself can be hard. But the operating system is still sold on Amazon as "Chokanji V, TRON runs on Windows" input system. 超漢字V means "Super Kanji 5", by the way. There is MacOS X version, too. Either version is just installing BTRON in VMWare, and runs the input method somewhat seamlessly. Archive.org has installation media for B-right/V 4.5 from 2006, but it will not run in Qemu properly. https://archive.org/details/chokanji The iso contains a file called bright00 which is a boot floppy for B-right/V installer. It should work on most i386-based computers, but it requires supported CD, IDE and mouse. After a bit of tinkering with PCem, I managed to make it work.

LGBTQIA+ and Tech
@benjohn yeah, for mine i imagine having a default editing and conflict resolution tool that by default presents a nice view into the data tree and where conflicting changes happen. but each node also has a format field, which points to a node holding some description of the format. so i imagine that tool asking the system for an editor-conflict resolver for that format first. but if there isn’t one and description has type information legible to the tool, it could create a more tailored interface from that type information.

@tryst I think this sort of thing is really powerful. I’d probably work on the UI instead, but there is so much research in to great UI and editing approaches.

I wondered “why don’t we see this in the wild?”, and concluded that a big part of this is that to use any of it, you are locked in to that niche tool, which is ultimately very limiting. These tools need a shared underlying collaborative data mechanism to flourish, imho.

(I think this is why plain text is so loved. It is currently the best collaborative data mechanism for interoperable tools - but it’s not good enough)

@benjohn

The core of my approach is strongly typed, immutable, content addressable Merkle DAG

i started this all out with storing dependently typed terms and ended up backing that off to raw byte vectors before building back to where i am now, which is more dynamically typed, but keeping an affordance for serializing data an a type/format specific way.

are you planning to bring static typing all the way between nodes of the data? like… each BTree node could be (haskellish) data BTree k v = Interior [(k, Node (BTree k v))] | Leaf [k, v]?

It is possible to merge consistently in to a conflict state.

i am having a bit of trouble visualizing a join-semilattice with conflict states that can be repaired? something like each conflict gets a unique identifier, but then you have to contend with merging conflict resolutions that you haven’t seen the conflict for yet?

i guess in my case, the “conflict state” is that there are two versions of a node, neither of which is an ancestor of the other. and the set of versions of nodes forms a join-semilattice. and user agents resolve that conflict state by adding a version with both of those versions as ancestors. but it seems a bit more natural to model that at the version level than the set of versions level.

i guess one advantage to what you’re discussing would be that non-conflicting changes made after the conflict change could be merged while it is still marked conflicting… but that would also break my ideas for locking…

@tryst on the BTree typing, yes.

It’s fairly general and is not key value specific. It is generic on the leaf, which mostly just needs to be a Monoid of some kind.

A leaf could be an array, or a list of kv pairs. Or fairly specific things as needed, so, perhaps fragments of an attributed text document.

The BTree has a generic `join` function (That I got passing tests last night since working from start of the summer!) that can be told how the underlying leaves should be joined. It’s hopefully efficient and minimises work where trees are similar.

So, the tree doesn’t inherently know how to join data, the content leaves need to know this.

Eg, I anticipate a generic leaf for a CRDT list, which I’m hoping to work on roughly next.

@david_chisnall I think the bigger issues MS/others ran in to were
1. Performance (eating memory)
2. Initial quality transition cost
3. Relatively decentralized teams being forced to rewrite.
@david_chisnall is there a pair of F/OSS apps you have in mind for "dudes, build on the same platform"?
Conversely, if you were starting an app, what would you build it on?
@david_chisnall I was noodling on cloud platforms a couple years ago.
http://webseitz.fluxent.com/wiki/PersonalCloud
Personal Cloud

Personal Cloud

WebSeitz
@david_chisnall interesting fresh thread about composability in the context of emacs vim, lisp machines....
https://zotum.net/item/17018846-251c-4b0f-be3a-6cba11547ac1
Zotum