Mastodawn

Björkus "No time_t to Die" Dorkus

A r e y o u r e a d y t o h a v e s o m e f u n ?

:3

Björkus "No time_t to Die" Dorkus Apr 6

"This is going to destroy my build system"

Nope! It's able to determine everything that will be processed by the Phase 7 compile-time-computed strings by Phase 4, and presents all of that information through already-available means, meaning CMake/build2/meson/make/ninja/etc. can all understand the dependency chain here natively!

Björkus "No time_t to Die" Dorkus Apr 6

Ultimately, this means we can process files -- recursively -- at compile-time, meaning that rather than embedded shaders with #includes that can't be touched, we can process those includes and make true single blobs without extra build steps.

compile-time python with imports is VERY possible.

Martin Uecker Apr 6

@thephd Cool, but why?

Björkus "No time_t to Die" Dorkus Apr 6

@uecker C++ has capabilities that can make great use of this. A perfect example is using (Generative) reflection to generate, at compile-time, the perfect FFI that maps to Python, or Lua, or JavaScript, with all of the utility that comes from having it mapped perfectly to C(++) interfaces and fully type-checked while always being generated directly from said Lua or Python or JavaScript source code.

This also applies to things like e.g. Rust and C++ interop, which has also been the topic of discussion and monetary investment. (Not that they're paying me; I wish they would, I could do a lot more for them than just std::embed.)

C doesn't have the systems in place to do things like this, so in most cases they'd just have to rely on the usual techniques used today: code generators, hand-written parsers, fresh data files and description files used to drive code generation (like e.g. SWIG). Certainly not bad, but not nearly as "automated luxury FFI" as C++ can make it.

Martin Uecker Apr 6

@thephd What I do not understand is why the extra build step is a problem.

Björkus "No time_t to Die" Dorkus Apr 6

@uecker I don't think the extra build step is the problem. I think the ability to e.g. parse C++ or C code and generate the proper FFI to connect to other languages, or vice versa, is a tooling investment that isn't really a fully solved problem.

Martin Uecker Apr 6

@thephd Ok, but why does it need to be solved by compile-time interpretation and not simply be a tool one runs during built? To me, this seems to solve the problem at the wrong place and using poorly suited tools (a compiler is not a good interpreter). And my only explanation so far is that people are nerd-sniped into doing it.

Björkus "No time_t to Die" Dorkus Apr 6

@uecker If that's all you took from this, okay!

mia (developer mode)Apr 6

@thephd @uecker take a look at fftw3’s build system and i think several use cases for this kind of compile-time functionality will become apparrent. for example, it currently uses some arcane OCaml code generators to turn customizable rules into SIMD code. it’s so unwieldy that the project’s README recommends users only ever build from tarballs with the sources already generated.

(after exchanging some emails i ported that whole mess to meson to make it usable but then that didn’t get reviewed at all and i stopped caring)

mia (developer mode)Apr 6

@thephd @uecker my personal opinion is that build systems should not generate code, and code should not depend on build system output. the responsibility of a build system is the build/test/deploy process alone.

Martin Uecker Apr 6

@mia @thephd Then let's agree to disagree, because my opinion is that compilers should not be misused as interpreters.

Martin Uecker Apr 6

@mia @thephd (my personal pain with build systems is typically with CMake where it throws some error and I have no idea where in the thousands of lines of CMake I am supposed to change something. Especially if it first compiles for a long time and processing constexpr as part of it just makes it worse. )

mia (developer mode)Apr 6

@uecker @thephd that’s just cmake being utter garbage (without hyperbole, it has more in common with microsoft batch than it does with an actual build system) that for no apparent reason half the software industry agreed to force on each other, the other half being google’s flavor of the week.

qt needs more than 90k lines of code—again, no hyperbole because i’ve ported some of that recently—consisting mostly of cmake workarounds to achieve what meson can express entirely declaratively in just over a hundred. and none of it would have to exist if qt didn’t require 5 or 6 different generators.

so i don’t think “this makes bad build systems even more of a pain to use” is a strong argument

Martin Uecker Apr 6

@mia @thephd I certainly agree that CMake is just bad independently of constexpr. I haven't looked at meson much, but if I needed to build and run a C program to create specialized code it would be a single line in my Makefile. Cross-compilation would be more complicated though.

mia (developer mode)Apr 6

@uecker @thephd meson can also handle that case (plus cross compilation and figuring out how to run it if it needs to compiled for a different system/architecture) in a single line as well.

but you’ve just said it: cross-compilation breaks the assumptions of many build systems. there aren’t many that properly support the canadian cross to begin with, and there’s a lot of platform-specific quirks that build systems need to figure out to make it all work as well.

that’s an incredible amount of jank and moving parts for something that could just be done by the compiler that you already use to compile the generator code.

Martin Uecker Apr 6

@mia @thephd I think "just" having an additional interpreter in the compiler is not generally a good idea. I want things to be more modular and less monolithic. Compilers are already too complex for my taste. If people use modularity to create a mess, then this is IMHO not an argument against modularity but for cleaning up the mess.

mia (developer mode)Apr 6

@uecker @thephd well i think the past 30 years of software engineering have abundantly demonstrated that people are generally bad at cleaning up messes. just look at libc.

i don’t like complexity either, but what we’re talking about here is a problem that is, much like wanting to embed binary files, nearly universal.

and good tools do not foist such things off on the user in the name of simplicity, especially when doing so results in a million ugly hacks downstream.

Martin Uecker Apr 6

@mia @thephd I do not agree that the solution needs to be ugly hacks. I simply do not see why putting the complexity into the language and the compiler is a step to overall improvement. I think it does the opposite.

mia (developer mode)Apr 6

@uecker @thephd i’m arguing in favor of putting that complexity as close to the source of the problem as possible so that it only needs to be dealt with in the one place that is in a very good position to do so, rather than an unbounded number of places that then have to wonder why it is still so hard and still requires all these crutches in the current year. like, so far your argument boils down to aesthetic preferences more than anything else and it makes me wonder who you think the languange and its tools are for

Martin Uecker Apr 6

@mia @thephd You argument is "build systems are a mess, so solve everything in the compiler close to the source of the problem". Somehow this argument is also seems weak. Similar generic arguments always are used to justify centralization of complexity. But I do think code generation is fundamentally different from code translation and should not be done by the same tool and I do not think this is purely aesthetic argument. But let's agree to disagree.

Martin Uecker Apr 6

@mia @thephd I don't know. Maybe fftw3's build system could be better, but I personally would not prefer a C++ constexpr approach to it.

@uecker @thephd https://mastodon.online/@manx/116261044523322289

@uecker @thephd This exact extra backwards cyclic dependency is probably the single most significant reason why building C and C++ code is hard.

Martin Uecker Apr 6

@manx @thephd Can you explain what you mean? I do not generally find building C code hard - not even when it runs code (which it rarely does).

Martin Uecker Apr 6

@manx @thephd You could turn this around "Do not feel proud about your clever compile-time meta programming. You should feel ashamed that you have been tricked into using your compiler as a poor interpreter."

@uecker @thephd Well, I am happy to disagree here.

No cyclic dependency is IMHO always better than any cyclic dependency.

This whole problem tends to always bite especially hard when you need to port from one build system to another (because the existing build system does not support your platform or toolchain), like trying to use an Autoconf project with MSVC.

Martin Uecker Apr 6

@manx @thephd I am not entirely sure I understand what is "cyclic" in this context. To me one reason people try to make languages into complete frameworks is that it seems an easier approach to standardize things at the language level than try to harmonize the diverse environments. And this is a valid argument, but not really a technical one.

@uecker @thephd

The build system generates input for the compiler. The compiler generates input for the build system (include dependency graph). This is cyclic, and by itself causes all sorts of complications (which are all solved in current build systems, but which IMHO makes all of them more complex than they should need to be).

This is similar to what Autoconf does (which also induces a cyclic dependency).

It is also remotely similar to what C++ Modules require (but there it is way worse).

Martin Uecker Apr 6

@manx @thephd Thanks! I guess in a reasonable approach you would only have two stages. You build a tool that creates the code you then run to create he code you ultimately compile. This is not fundamentally different to what happens with constexpr. Just there is all part of an in transparent and slow monolithic C++ compiler. The complexity is not gone, but just hidden. But this is not better IMHO.

@uecker @thephd The reasonable approach is only have two stages, I totally agree. But this is not how any build system I know models it. They all support the complexity of arbitrary graphs, which offers enormous flexibility at the cost of unbounded complexity. Unless this gets simplified in one way or another, any build system standardization is IMHO very unlikely to happen.
...

@uecker @thephd ...
I think moving complexity into the language proper is the right approach. It frees developers from learning multiple "languages", and it simplifies porting to other platforms that require different build systems.

Martin Uecker Apr 6

@manx @thephd Maybe you are right, but my perspective after observing the industry for a while is this: Complexity always grows until things suck too much and then people build alternatives and the cycle starts again. At the same time, if some part does not suck yet because it is relatively small and simple, people think that complexity needs to be moved there from the other parts nobody wants to touch. Couldn't this be part of what is happening here?

@uecker @thephd Maybe partly.
I still think that removing cyclic dependencies is something to strive for in general. I see clear practical advantages for use cases that I encounter.
However, I also see problems arising from implementing too much in constexpr, but for me, the advantages from simplifying the build outweigh them clearly here. Other people might weight things differently.
...

@uecker @thephd ...
I also still think that the C/C++ build system ecosystem only has the chance of getting addressed IFF we take away responsibility from them and simplify them first. Standardizing the complexity that currently exist across all of them would just result in an even more complex generalized "solution" that really does not help at all.

@uecker @thephd
make
ninja
cmake
msbuild
premake5
autotools
vcpkg
conan
apt
yum
rpm
...

Having a reverse dependency on the build system multiplies this nonsense, especially when cross-compiling and interacting between different build systems.

Compare this to any (really any) other language.

This is really not fixable if every build system invents its own mechanism for generating code at build time, thereby locking any project into build system vendor specific mechanisms.

Martin Uecker Apr 6

@manx I agree that the lack of standardization of build systems is a problem.

Martin Uecker Apr 6

@manx But I think it partially also reflects the size of the ecosystem. I really do not like many other languages that provide a framework that locks you into a specific way of doing things.

@uecker @thephd If you are cross-compiling, you are running the interpreter and code generator on the build host instead of on the build target, which means, if it requires any platform-specific knowledge (like the size and alignment of fundamental types), you have to manually duplicate this knowledge into your custom interpreter.

Inside the target language itself, this information is naturally trivially available.

Martin Uecker Apr 6

@manx @thephd Cross-compilation is a good argument, but it is still not clear to me that this would not be better solved at a different level.

@uecker @thephd I am also not 100% sure that doing this in constexpr context is necessarily the best approach (mainly because debugging constexpr code is a completely unsolved problem), but it is frankly the best approach I have seen offered so far, and IMHO far better than involving the build system in any way.

Martin Uecker Apr 6

@manx @thephd IMHO if people are unhappy about build systems (which seems to be the case and there are certainly good reasons), one should think hard why this is the case and fix it at that level. This is hard, because it is likely not just a technical issue, but lack of standardization and the need to support annoying environments that intentionally wanted to be different.

@uecker @thephd I agree, and I think reducing the scope and responsibility of what build systems need to do, is a good and necessary step into that direction.

Chip Collier Apr 6

@thephd whoa.gif

Alex Celeste, Princess Consort of Burnout Apr 6

@thephd just need some expression templates and you got yourself a quick'n'dirty code generator too

✧✦Catherine✦✧Apr 6

@thephd oh, very nice

mx alex tax1a - 2020 (6)Apr 6

@thephd what the fuck

@thephd Wait, is there some update on std::embed in the standard library that I missed or is it still not a thing?

Björkus "No time_t to Die" Dorkus Apr 6

@NekkoDroid just trying to finish it off, yeah.

Björkus "No time_t to Die" Dorkus Apr 6

@NekkoDroid (It's not in C++ yet, this is a personal branch.)

Avery (She/Her)Apr 6

@thephd i'm trying my best to understand what is going on here, and what i think is happening is that you have made lua into a compiled language hosted(??) in c. is that correct?

Björkus "No time_t to Die" Dorkus Apr 6

@shitpostalotl C++, not C. God forbid C having this kind of power.

Avery (She/Her)Apr 6

@thephd i definitely don't understand c++ to know which c++-exclusive features you're using to make this happen that can't be used in c. this is basically wizard techniques to me. question though: since lua is being compiled, does this mean that it could have compile-time errors for things that would have only shown up while running the program previously?

Björkus "No time_t to Die" Dorkus Apr 6

@shitpostalotl yes, you can turn all of those errors into compile-time C++ errors. (I have not done this yet, just implemented gluing the two files together at C++-compile-time.)

Avery (She/Her)Apr 6

yet👀

@thephd every day we come closer to https://web.archive.org/web/20220501154514/https://www.reddit.com/r/cpp/comments/jee26l/should_c_just_standardize_an_interpreted_step_for/ and i'm glad for it

r/cpp - Should c++ just standardize an interpreted step for compilation?

70 votes and 49 comments so far on Reddit

reddit