most SDL_gpu functions take these chonk config structs, so LRDL needs a way to initialize structs just as conveniently.
the best way to support this is through a syntax-level macro, but my C port of libsxpp still lacks the macro-based preprocessing system from the prototype. i decided to put this off until i really needed it.
but now the moment has come to put the pp in libsxpp.
... i wrote this, tried to extend libsxpp and then had to give up because gcc-14 crashed while trying to compile libsxpp's modules.
but i really really need preprocessing to work for other things so i have to finalize the port.
step 1 is to switch to gcc-15 and hope the problem is gone, otherwise i will have to de-module this C++ project ;_;
update: praise the gcc gods, it's fixed in gcc-15. work can commence!
ported the remainder of RuleMatcher yesterday, and then today the port of the last module, Preprocessor.
after which, a new C interface for it.
the rule matcher tech is really cool. instead of trying one match after another, we track all matches in parallel. every new token is looked up in a trie implemented as a flat persistent hashmap, where the key is the 128-bit hashed path of the matcher, and the u64 value is the next "instruction".
below are all "vm instructions"
patterns that start with the same tokens all deduplicate in the map, so run as a single match until they diverge.
the map is "persistent" because it itself is implemented as a HAMT; this makes it possible to add new rules on a scoped basis, something that the C preprocessor can not do, but which greatly amplifies the power and reliability of preprocessing.
matching can also understand nesting and capture complex patterns like l/r-assoc infix expressions with precedence.
the "vm machine" is slightly cursed: the matcher builds a hashed token path where each token is a list or symbol literal, or one of six possible abstract instructions (see previous listing).
to find the next path, we try all seven combinations: path + next literal, path + instruction 1, etc. all entries that have a mapping create matcher forks. if nothing fits, the match ends here.
could accelerate this with a 7 bit "roadsign", but not necessary so far.
still porting. the original code was leaning on libarcmem, so it makes ample use of efficient bidirectionally extensible arrays and auto-sharing, which i don't have in C++; so we have a few more copies and possible sources of quadratic complexity for the time being.
i decided to replace the std::vector with a std::shared_ptr<std::deque<T> > wrapped inside a std::span-like container. it's a compromise.
the lexer is documented.
(for the number parser i realized it would hit malloc hard, the way it works, so i added a sx_num_reset() function.)
documented the unit. this is the preprocessors' storage model and the high level parsing interface.
i now found the cause for the first big bug:
months ago i made a tiny change in symbol unescaping, deciding that unknown escapes would strip the backslash (e.g. \$ becomes $).
but the rules parser depends on \$ to be passed as-is, so variable names can be escaped.
because that didn't happen, all fundamental forms, which start with \$, turned into variables, and so the first of any rule would be applied and eat the entire source file this way.
this took a whole day.
pattern matching is beginning to work again. the next issue is with variadic patterns. they're not being matched. nothing crashes so this is harder to figure out...
found it through comparing with the original source.
when a matching thread is done (not a real thread, just a simulated one), the loop is supposed to skip to the next thread, but it jumps out of the loop to the end of the parent loop, skipping processing of all following threads. that was a mistranslation.
not hitting any more bugs so far, so i'm writing tests to check if rules work as intended, and also as demo.
here for example are sxpp rules which transform infix expressions into prefix expressions.
(1 * 2 + 3 * 4 + 5 * 6) -> (add (add (mul 1 2) (mul 3 4)) (mul 5 6))
(foo bar * fizz buzz) -> (mul (foo bar) (fizz buzz))
LR
