Mastodawn

ineemio 20h ago

if the goal of "free software" is to give the user control over how the software behaves then is minecraft[1] more free than, say, gcc[2]? thread locked by moderators after reaching 100 pages of intense debate

[1]: where an end-user can usually install >100 mods with a lot changing the game in nontrivial ways reasonably trivially without major conflicts
[2]: where combining multiple features from different forks requires lengthy conflict resolution usually dependent on deep knowledge of the systems being modified

what im trying to say is github.com/FabricMC/Mixin & github.com/LlamaLad7/MixinExtras rock and we need clones of them for more languages and environments

GitHub - FabricMC/Mixin: Mixin is a trait/mixin framework for Java using ASM

Mixin is a trait/mixin framework for Java using ASM - FabricMC/Mixin

GitHub

https://j0̷.lol 3d ago

@kopper i feel like jvm being jvm makes this stuff a lot easier than like c

@j0 oh absolutely yeah. this is more philosophical than anything, in the real world of course a bytecode jit is more malleable than raw machine code

https://j0̷.lol 3d ago

@kopper what we want first is better runtime reflection in a language before we can even hope for smth like mixin. and the ability to do things like access widening

https://j0̷.lol 3d ago

@kopper i think access widening is a consequence of runtime reflection tho

@j0 @kopper yea, the jvm has an idea of what a local variable is, and only inlined stuff ever if constants (depends on compiler / obfuscater settings too).

something similar for native is, while not impossible, going to be much harder and much less powerful (again, jvm generally has only one form for certain things, native has as many as codegens * their versions

even just inject at head is going to be difficult because where does function prepare end and body begin ?

@j0 @kopper i think there is much more value in making our software's features more decoupled so instead of rewriting parts, you swap them in and out for ones that suit your needs better

@SRAZKVT @j0 this bit is interesting to me. how granular is the swapping parts bit supposed to be? executables? libraries? what's the real difference between the two?

@kopper @j0 it's imo better to standardise executables because a library's api shape how it is implemented, while executables gives a lot more freedom (heh) on how to do things

but there's cases where we need libraries

there's also the possibility to standardise a daemon's protocol for certain operations, though this is in this case mostly useful for things which would need some form of privilege (like ping)

d@nny disc@ mc² 3d ago

@SRAZKVT @kopper @j0 spack has done the most work on swapping libraries that i'm familiar with both through the concretization that reconciles compatibility directives and then through the installation process which e.g. rewrites RPATH. there's not really a distinction there between executables and libraries but it's not magic and as you say there are library decisions that affect this

d@nny disc@ mc² 3d ago

@SRAZKVT @kopper @j0 i also agree that a cli api is standardizeable like a daemon protocol but this is not done for some reason and in fact google engineers will say outright that a cli is not an api when they whine on twitter about linus banning them from the kernel for breaking the perf cli

@hipsterelectron @kopper @j0 a cli is an interface that can be constructed to do needed operations by a program. it's an API.

a restricted one, sure, but it's still one

princess pancake

@SRAZKVT @hipsterelectron @kopper @j0 it's a terrible boundary to cross though

@natty @kopper @hipsterelectron @j0 how so ? your build system does it every day by calling your compiler, i don't see why we shouldn't generalise it

API implementation freedom is inversely proportionnal to how specific to the internals the API is, executables in that regard just give a very generic thing. that's why we can have wildly different C compilers and yet all of them can work together to build something that works

princess pancake

@SRAZKVT @kopper @hipsterelectron @j0 I have no idea what you picked up from our reply, are working on one such project ourselves (it works well in shells)

Generalizing CLI might improve the interface and UX/DX by avoiding repeated parsing and serializing, however it's unlikely to open any new opportunities given how sloooow executing a program is. Microservices already show roughly how small you can go

It's kinda like RPC in that sense, without the R. Replacing argparse with something like a Protobuf definition won't really fix the inherent limitations. It would be an improvement to the status quo tho

You could have the processes running and talk via IPC- hold on that's what Mojo is

@natty @kopper @hipsterelectron @j0 oh i think i misunderstood you because you misunderstood me

i meant generalising the idea of delegating work to another process with the cli itself acting as interface, i didn't said anything about standardising the cli itself, that's pretty much a lost battle already anyway

@natty @kopper @hipsterelectron @j0 and for your last point, yes, a daepon is an api. but when something is done once only, spawning a daemon is a bit overkill, unless you merge in more features and capabilities into it, but then you go back to the original problem of having a single monolith you can't meaningfully modify the inner workings of

princess pancake

@SRAZKVT @kopper @hipsterelectron @j0 I think I misunderstood your misunderstanding because you misunderstood whether we misunderstood, because I am getting lost

We do argue that work-splitting via CLI API can work, but also that the use cases are limited

Although it could be interesting to generalize busybox-style commands for "CLI libraries"

d@nny disc@ mc² 3d ago

@natty @kopper @SRAZKVT @j0 i provided in a separate subthread a characterization of the cases where work-splitting is particularly effective https://circumstances.run/@hipsterelectron/116261376121926115 this ended up being the key insight necessary to parallelize scala compiles where the outline compiler ran in a persistent process in topological order while the longer-compiles could be fully parallelized via this exact approach (although in that case we actually did RPC process executions which pay an additional cost to upload input files)

d@nny disc@ mc² (@[email protected])

@[email protected] @[email protected] @[email protected] no fucking way they added a retry flag but only for process execution. i am always proven right. anyway if you have a coroutine framework available then you can leverage asynchrony in ways the individual tool cannot because it cannot control the execution environment https://docs.google.com/document/d/1iWINqg30nXaCYmBxma5h5RgWzx7sgFqWGJcQJ-RYpRE/edit?usp=drivesdk outline for a conference talk i gave at CGO 2019

GSV Sleeper Service

d@nny disc@ mc² 3d ago

@natty @kopper @SRAZKVT @j0 one notable insight of that document is that it's extremely difficult for the persistent process to generalize across scales of input whereas the ephemeral one-shot invocation can do that without harming either end of the spectrum.

i find your mention of a busybox-esque cli api tantalizing but i am not terribly certain i understand it yet. i have been accidentally producing a shell script compiler for bootstrap scripts after spending a lot of time with autoconf and specifically found incorporating process executions (which becomes a form of constant folding) to be very difficult as it conflicts with the layered output framework of m4

princess pancake

@hipsterelectron @kopper @SRAZKVT @j0 To elaborate, busybox could be generalized to something like a shared library for shell scripts

We're in our free time working on a semi-statically-typed shell that would have a similar property by design, making functions definitions double as a DSL for CLI definitions

It's still heavily WIP though

princess pancake

@hipsterelectron @kopper @SRAZKVT @j0 Essentially it would be first-class reflection kinda like D-Bus

d@nny disc@ mc² 3d ago

@natty @kopper @SRAZKVT @j0 that's super interesting!! i was working on a framework that exposed argv parsing and json deserialization from a shared specification for the zip crate cli which also incorporated named input/output streams and also felt this would be functionality useful to a shell. positional argv parsing by specification also enables some degree of analysis/type checking/optimization but most certainly argv splitting

d@nny disc@ mc² 3d ago

@natty @kopper @SRAZKVT @j0 spack uses json schema for its yaml config which does not change often and idk about the applicability of that more generally but the build system i'm working on has structured argument registration with precedence between hardcoded < env < config file < cli arg, where list and dict argument types support extend operations distinct from completely overwriting the prior value

d@nny disc@ mc² 3d ago

@natty @kopper @SRAZKVT @j0 RPC as IPC and using shared memory for immutable shared data is what i started working on many years ago https://codeberg.org/cosmicexplorer/upc i don't know that i would call executing a process slow but rather the interfaces exposed for local execution (particularly linear path traversal) are not only inefficient but also nonreproducible. serializing the result of that process could well be done through the filesystem

upc

Ultra-high-performance local IPC framework with Zipkin tracing to conduct a beautiful symphony of (brotherhood) build tooling.

Codeberg.org

d@nny disc@ mc² 3d ago

@SRAZKVT @kopper @j0 if the daemon is the os and the protocol is exec then the cli for an exe is a subset of that daemon protocol

@hipsterelectron @j0 @SRAZKVT i may be getting too "architecture astronaut" here but if you replace the plain text a process (usually) receives as input and output with something structured then how different is it from importing a library and calling a function in it? can't these things be the same? or similar enough to make processes less unwieldy compared to libraries?

@hipsterelectron @j0 @SRAZKVT the original posts were a bit philosophical in nature, currently the differences really seem to boil down to historical accidents more than purposeful design

@hipsterelectron @j0 @SRAZKVT sure it makes sense for, say, a hashmap to be a library (unless i'm not seeing something radical [cool]), but what's stopping, say, an image decoder from being a separate "process" you pipe image data into and get rgb out of?

@hipsterelectron @j0 @SRAZKVT in fact, for the image decoding use case, aren't a lot of them working in a similar fashion already? for sandboxing? what's stopping us from generalizing this approach

@kopper @hipsterelectron @j0 a project of mine (eventual, not yet started, but when i have time i WILL do it) is a format, somewhat similar to pandoc internal, which allows readers, filters, and writers to be separate executables, all of which being replaceable by any part you want. don't want to be able to import latex ? sure, don't install the latex reader. want to make a pipeline with your own filter added into it ? go right ahead

there is no reason the same idea wouldn't work for images, though images are a lot more data than documents

d@nny disc@ mc² 3d ago

@kopper @j0 @SRAZKVT this is why build systems are generally process executors because the process execution interface is the most general API format which makes it easy for a sandbox to implement

d@nny disc@ mc² 3d ago

@kopper @j0 @SRAZKVT libraries are specifically useful for operations within your own process memory space. they also execute synchronously on the processor and use the same stack. this is useful in some scenarios. process execution can be synchronous but it's nicer if the language has coroutines because in fact the process execution interface decouples the execution from process-local state. this makes it reproducible and limits the vulnerability surface, but is also exactly what you would need to parallelize it in-process anyway.

d@nny disc@ mc² 3d ago

@kopper @j0 @SRAZKVT i apologize for the docs sucking they didn't accept my PR to fix it and they recently broke all docs urls https://www.pantsbuild.org/2.30/docs/writing-plugins/the-rules-api/processes

Processes | Pantsbuild

How to safely run subprocesses in your plugin.

d@nny disc@ mc² 3d ago

@kopper @j0 @SRAZKVT no fucking way they added a retry flag but only for process execution. i am always proven right. anyway if you have a coroutine framework available then you can leverage asynchrony in ways the individual tool cannot because it cannot control the execution environment https://docs.google.com/document/d/1iWINqg30nXaCYmBxma5h5RgWzx7sgFqWGJcQJ-RYpRE/edit?usp=drivesdk outline for a conference talk i gave at CGO 2019

Swapping in Graal native-image to multiply Pants JVM build performance

Swapping in Graal native-image to multiply Pants JVM build performance Abstract: Pants is an open source build tool which emphasizes extensibility and composability of build tasks, including integrating tools and build products from different languages and frameworks in a structured and performan...

Google Docs

d@nny disc@ mc² 3d ago

@kopper @j0 @SRAZKVT i actually don't really like this abstraction for file and directory content (which is what is provided as input and collected from output of a process execution) https://www.pantsbuild.org/2.30/docs/writing-plugins/the-rules-api/file-system partially because it is a completely unstructured byte stream and requires context to make sense of. it is a google api we copied

File system | Pantsbuild

How to safely interact with the file system in your plugin.

d@nny disc@ mc² 3d ago

@kopper @j0 @SRAZKVT the CAS part is precisely what makes it lose context. instead i have a terrible scheme to succeed the zip file format https://codeberg.org/cosmicexplorer/dice/src/branch/main/TODO.org which maintains the tabular structure of a zip while making it somewhat easier to transfer subsections than a zip file where the format allows this but implementations often get confused

dice/TODO.org at main

dice - faster than light archives

Codeberg.org

Wolf480pl 3d ago

@SRAZKVT
you could compile with LTO, which leaves the compiler's IR inside the object files

and then do the hooking at link time, before LTO and codegen

you'd still be bound to one compiler tho
@j0 @kopper

d@nny disc@ mc² 3d ago

@kopper i have said this before and the build system i have built before and the one i am working on are intended to achieve this

@kopper i've been working on a library for analysis and patching of a certain other bytecode language and, while there's obviously a lot of things that don't really transfer over to other languages, i aspire to be at least as usable and powerful as mixins. it's been a really good learning experience too, though i'm stuck procrastinating and being burned out at the moment.

princess pancake

@kopper https://github.com/microsoft/detours

princess pancake

@kopper The problem is not that it would be impossible in native compiled stuff

But rather that compilers demolish functions on purpose

princess pancake

@kopper The claim that this is hard out of Java is not entirely true

We have call conventions and symbols

It's that Java inlines in HotSpot (at runtime)

d@nny disc@ mc² 3d ago

@kopper this fails to consider the software's importance to user control over other systems. minecraft is not a dependency of anything. gcc is a dependency of everything. the functionality from a fork is often useful in one context and detrimental in another without lengthy conflict resolution. this is kind of a classic concern epitomized by "move fast and break things". llvm chooses to do exactly this and in fact has several forks who will never be made fully compatible.

@hipsterelectron yeah gcc was perhaps not the best example here but i was trying to think of something that was both forked frequently and substantially large. picking something GNU seemed to be funny considering this is talking about "free software" and hence, gcc

tusharhero 3d ago

@kopper GCC is not a videogame.

@Luanti is much freer than Minecraft.

@tusharhero look into reading comprehension someday

tusharhero 3d ago

@kopper writing a compiler is a different task when compared to writing mods for game.

I am sure you are aware GCC has support for many languages as well.

@tusharhero please look into the meaning of words in context instead of simply pattern matching on the words you recognize

tusharhero 3d ago

@kopper You don't have to be so rude. If you don't like my replies block me.

Wolf480pl 3d ago

@kopper
I think what @tusharhero is tryng to say is that gcc is solving a problem that is inherently more complex than the problem Minecraft is solving. And when such difference in inherent complexity exists, difficulty of modification is not a good measure of user freedom.

I.e. apples and oranges.

I don't entirely agree with that point, but I don't think it's fair to accuse tusharhero of lack of reading comprehension

@wolf480pl gcc is simply an example and is irrelevant to the overall point of the post (and i do agree it's not a great example, i have another reply somewhere else for that)

🇺🇦 haxadecimal 🚫👑3d ago

@kopper
I'm not sure about "more free", but gcc is definitely more challenging and more fun, even though the scoring is quite subjective.

@kopper

hmm. does that mean that having founded a second, incompatible mod loader makes me an enemy of software freedom 🤔 /s