Mastodawn

if the goal of "free software" is to give the user control over how the software behaves then is minecraft[1] more free than, say, gcc[2]? thread locked by moderators after reaching 100 pages of intense debate

[1]: where an end-user can usually install >100 mods with a lot changing the game in nontrivial ways reasonably trivially without major conflicts
[2]: where combining multiple features from different forks requires lengthy conflict resolution usually dependent on deep knowledge of the systems being modified

what im trying to say is github.com/FabricMC/Mixin & github.com/LlamaLad7/MixinExtras rock and we need clones of them for more languages and environments

GitHub - FabricMC/Mixin: Mixin is a trait/mixin framework for Java using ASM

Mixin is a trait/mixin framework for Java using ASM - FabricMC/Mixin

GitHub

https://j0̷.lol 4d ago

@kopper i feel like jvm being jvm makes this stuff a lot easier than like c

@j0 @kopper yea, the jvm has an idea of what a local variable is, and only inlined stuff ever if constants (depends on compiler / obfuscater settings too).

something similar for native is, while not impossible, going to be much harder and much less powerful (again, jvm generally has only one form for certain things, native has as many as codegens * their versions

even just inject at head is going to be difficult because where does function prepare end and body begin ?

@j0 @kopper i think there is much more value in making our software's features more decoupled so instead of rewriting parts, you swap them in and out for ones that suit your needs better

@SRAZKVT @j0 this bit is interesting to me. how granular is the swapping parts bit supposed to be? executables? libraries? what's the real difference between the two?

@kopper @j0 it's imo better to standardise executables because a library's api shape how it is implemented, while executables gives a lot more freedom (heh) on how to do things

but there's cases where we need libraries

there's also the possibility to standardise a daemon's protocol for certain operations, though this is in this case mostly useful for things which would need some form of privilege (like ping)

d@nny disc@ mc²

@SRAZKVT @kopper @j0 spack has done the most work on swapping libraries that i'm familiar with both through the concretization that reconciles compatibility directives and then through the installation process which e.g. rewrites RPATH. there's not really a distinction there between executables and libraries but it's not magic and as you say there are library decisions that affect this

d@nny disc@ mc² 4d ago

@SRAZKVT @kopper @j0 i also agree that a cli api is standardizeable like a daemon protocol but this is not done for some reason and in fact google engineers will say outright that a cli is not an api when they whine on twitter about linus banning them from the kernel for breaking the perf cli

@hipsterelectron @kopper @j0 a cli is an interface that can be constructed to do needed operations by a program. it's an API.

a restricted one, sure, but it's still one

princess pancake

@SRAZKVT @hipsterelectron @kopper @j0 it's a terrible boundary to cross though

@natty @kopper @hipsterelectron @j0 how so ? your build system does it every day by calling your compiler, i don't see why we shouldn't generalise it

API implementation freedom is inversely proportionnal to how specific to the internals the API is, executables in that regard just give a very generic thing. that's why we can have wildly different C compilers and yet all of them can work together to build something that works

princess pancake

@SRAZKVT @kopper @hipsterelectron @j0 I have no idea what you picked up from our reply, are working on one such project ourselves (it works well in shells)

Generalizing CLI might improve the interface and UX/DX by avoiding repeated parsing and serializing, however it's unlikely to open any new opportunities given how sloooow executing a program is. Microservices already show roughly how small you can go

It's kinda like RPC in that sense, without the R. Replacing argparse with something like a Protobuf definition won't really fix the inherent limitations. It would be an improvement to the status quo tho

You could have the processes running and talk via IPC- hold on that's what Mojo is

@natty @kopper @hipsterelectron @j0 oh i think i misunderstood you because you misunderstood me

i meant generalising the idea of delegating work to another process with the cli itself acting as interface, i didn't said anything about standardising the cli itself, that's pretty much a lost battle already anyway

@natty @kopper @hipsterelectron @j0 and for your last point, yes, a daepon is an api. but when something is done once only, spawning a daemon is a bit overkill, unless you merge in more features and capabilities into it, but then you go back to the original problem of having a single monolith you can't meaningfully modify the inner workings of

princess pancake

@SRAZKVT @kopper @hipsterelectron @j0 I think I misunderstood your misunderstanding because you misunderstood whether we misunderstood, because I am getting lost

We do argue that work-splitting via CLI API can work, but also that the use cases are limited

Although it could be interesting to generalize busybox-style commands for "CLI libraries"

d@nny disc@ mc² 4d ago

@natty @kopper @SRAZKVT @j0 i provided in a separate subthread a characterization of the cases where work-splitting is particularly effective https://circumstances.run/@hipsterelectron/116261376121926115 this ended up being the key insight necessary to parallelize scala compiles where the outline compiler ran in a persistent process in topological order while the longer-compiles could be fully parallelized via this exact approach (although in that case we actually did RPC process executions which pay an additional cost to upload input files)

d@nny disc@ mc² (@[email protected])

@[email protected] @[email protected] @[email protected] no fucking way they added a retry flag but only for process execution. i am always proven right. anyway if you have a coroutine framework available then you can leverage asynchrony in ways the individual tool cannot because it cannot control the execution environment https://docs.google.com/document/d/1iWINqg30nXaCYmBxma5h5RgWzx7sgFqWGJcQJ-RYpRE/edit?usp=drivesdk outline for a conference talk i gave at CGO 2019

GSV Sleeper Service

d@nny disc@ mc² 4d ago

@natty @kopper @SRAZKVT @j0 one notable insight of that document is that it's extremely difficult for the persistent process to generalize across scales of input whereas the ephemeral one-shot invocation can do that without harming either end of the spectrum.

i find your mention of a busybox-esque cli api tantalizing but i am not terribly certain i understand it yet. i have been accidentally producing a shell script compiler for bootstrap scripts after spending a lot of time with autoconf and specifically found incorporating process executions (which becomes a form of constant folding) to be very difficult as it conflicts with the layered output framework of m4

princess pancake

@hipsterelectron @kopper @SRAZKVT @j0 To elaborate, busybox could be generalized to something like a shared library for shell scripts

We're in our free time working on a semi-statically-typed shell that would have a similar property by design, making functions definitions double as a DSL for CLI definitions

It's still heavily WIP though

princess pancake

@hipsterelectron @kopper @SRAZKVT @j0 Essentially it would be first-class reflection kinda like D-Bus

d@nny disc@ mc² 4d ago

@natty @kopper @SRAZKVT @j0 that's super interesting!! i was working on a framework that exposed argv parsing and json deserialization from a shared specification for the zip crate cli which also incorporated named input/output streams and also felt this would be functionality useful to a shell. positional argv parsing by specification also enables some degree of analysis/type checking/optimization but most certainly argv splitting

d@nny disc@ mc² 4d ago

@natty @kopper @SRAZKVT @j0 spack uses json schema for its yaml config which does not change often and idk about the applicability of that more generally but the build system i'm working on has structured argument registration with precedence between hardcoded < env < config file < cli arg, where list and dict argument types support extend operations distinct from completely overwriting the prior value

d@nny disc@ mc² 4d ago

@natty @kopper @SRAZKVT @j0 which is from v1 pants https://v1.pantsbuild.org/options.html the "task scope" mentioned here is also used to dereference named output streams and checksummed cache directories

Pants Options

d@nny disc@ mc² 4d ago

@natty @kopper @SRAZKVT @j0 RPC as IPC and using shared memory for immutable shared data is what i started working on many years ago https://codeberg.org/cosmicexplorer/upc i don't know that i would call executing a process slow but rather the interfaces exposed for local execution (particularly linear path traversal) are not only inefficient but also nonreproducible. serializing the result of that process could well be done through the filesystem

upc

Ultra-high-performance local IPC framework with Zipkin tracing to conduct a beautiful symphony of (brotherhood) build tooling.

Codeberg.org

d@nny disc@ mc² 4d ago

@SRAZKVT @kopper @j0 if the daemon is the os and the protocol is exec then the cli for an exe is a subset of that daemon protocol

@hipsterelectron @j0 @SRAZKVT i may be getting too "architecture astronaut" here but if you replace the plain text a process (usually) receives as input and output with something structured then how different is it from importing a library and calling a function in it? can't these things be the same? or similar enough to make processes less unwieldy compared to libraries?

@hipsterelectron @j0 @SRAZKVT the original posts were a bit philosophical in nature, currently the differences really seem to boil down to historical accidents more than purposeful design

@hipsterelectron @j0 @SRAZKVT sure it makes sense for, say, a hashmap to be a library (unless i'm not seeing something radical [cool]), but what's stopping, say, an image decoder from being a separate "process" you pipe image data into and get rgb out of?

@hipsterelectron @j0 @SRAZKVT in fact, for the image decoding use case, aren't a lot of them working in a similar fashion already? for sandboxing? what's stopping us from generalizing this approach

@kopper @hipsterelectron @j0 a project of mine (eventual, not yet started, but when i have time i WILL do it) is a format, somewhat similar to pandoc internal, which allows readers, filters, and writers to be separate executables, all of which being replaceable by any part you want. don't want to be able to import latex ? sure, don't install the latex reader. want to make a pipeline with your own filter added into it ? go right ahead

there is no reason the same idea wouldn't work for images, though images are a lot more data than documents

d@nny disc@ mc² 4d ago

@kopper @j0 @SRAZKVT this is why build systems are generally process executors because the process execution interface is the most general API format which makes it easy for a sandbox to implement

d@nny disc@ mc² 4d ago

@kopper @j0 @SRAZKVT libraries are specifically useful for operations within your own process memory space. they also execute synchronously on the processor and use the same stack. this is useful in some scenarios. process execution can be synchronous but it's nicer if the language has coroutines because in fact the process execution interface decouples the execution from process-local state. this makes it reproducible and limits the vulnerability surface, but is also exactly what you would need to parallelize it in-process anyway.

d@nny disc@ mc² 4d ago

@kopper @j0 @SRAZKVT i apologize for the docs sucking they didn't accept my PR to fix it and they recently broke all docs urls https://www.pantsbuild.org/2.30/docs/writing-plugins/the-rules-api/processes

Processes | Pantsbuild

How to safely run subprocesses in your plugin.

d@nny disc@ mc² 4d ago

@kopper @j0 @SRAZKVT no fucking way they added a retry flag but only for process execution. i am always proven right. anyway if you have a coroutine framework available then you can leverage asynchrony in ways the individual tool cannot because it cannot control the execution environment https://docs.google.com/document/d/1iWINqg30nXaCYmBxma5h5RgWzx7sgFqWGJcQJ-RYpRE/edit?usp=drivesdk outline for a conference talk i gave at CGO 2019

Swapping in Graal native-image to multiply Pants JVM build performance

Swapping in Graal native-image to multiply Pants JVM build performance Abstract: Pants is an open source build tool which emphasizes extensibility and composability of build tasks, including integrating tools and build products from different languages and frameworks in a structured and performan...

Google Docs

d@nny disc@ mc² 4d ago

@kopper @j0 @SRAZKVT i actually don't really like this abstraction for file and directory content (which is what is provided as input and collected from output of a process execution) https://www.pantsbuild.org/2.30/docs/writing-plugins/the-rules-api/file-system partially because it is a completely unstructured byte stream and requires context to make sense of. it is a google api we copied

File system | Pantsbuild

How to safely interact with the file system in your plugin.

d@nny disc@ mc² 4d ago

@kopper @j0 @SRAZKVT the CAS part is precisely what makes it lose context. instead i have a terrible scheme to succeed the zip file format https://codeberg.org/cosmicexplorer/dice/src/branch/main/TODO.org which maintains the tabular structure of a zip while making it somewhat easier to transfer subsections than a zip file where the format allows this but implementations often get confused

dice/TODO.org at main

dice - faster than light archives

Codeberg.org