first discovery in the #nocompiletime branch: the expander will produce a `Function`. which we then compile to a module, execute, and the result of that will be another `Function`.
so in the expander i can already get a good feeling for what it is like to specialize the typechecker.

#devlog #scopes

figured out how to efficiently handle pseudoquotes. they're equivalent to inline functions constructed in the meta-module by the expander (where we are pseudoquoting anyway), but then executed in the generated module.

below an example of source code, generated meta-module, and generated module. (mocked up)

#devlog #scopes

exciting developments. since this is a rewrite of the typechecker, i'm throwing out our existing borrow checker while i'm at it. always hated to use it. the less we are like rust, the better.

instead, we'll go full libarcmem, starting with the program stack itself.

#devlog #scopes #nocompiletime

i'm now rewriting the Scopes IR API to be self-typing as you build it; similar to LLVM IR. the only part where it gets tricky is with loops and recursive calls.

for recursions, we already require such functions to either have returned once already, or specify an explicit return type.

loops need to be built twice: a dry run that merges all continue types, followed by the actual build with the correct loop argument type.

#devlog #scopes #nocompiletime

ouch. i've bitten off more than i can chew again. i'm going to restart the branch, and this time i'll instead make small transformations to the prover that move more and more of the typechecking logic into self-typing constructor functions until the prover loop is trivial and only consists of dispatched calls. this way i can keep testing as i transform the code, and keep it in production state.

#devlog #scopes #nocompiletime

step 1 complete: the special `Scope` object used during proving is gone, instead we keep all context within the nested basic block representation `Block`.

#devlog #scopes #nocompiletime

step 2: a special case for merge labels has been removed; labels no longer need a cyclical bind-early hack to be addressable. a neat syntax improvement fell out of this as well. `merge <label> ...` can now simply be written as `<label> ...`.

step 3: explicit `set_parent()` calls in prover to link up nested basic blocks have been moved to block construction functions. `owner` attribute of blocks could just be removed.

#devlog #scopes #nocompiletime

step 4: all function template information that concerns anything but the body of the function has been factored out into a TemplateHeader.

step 5: every Closure (a compile time object) carries a TemplateHeader.

step 6: the TemplateClosure subtype holds template function context as it used to be. there is now a new UserClosure subtype that carries a global (for the symbol of the plugin function) and a context, but it isn't supported yet.

#devlog #scopes #nocompiletime

step 7: all that TemplateClosure does is now implemented again, with the UserClosure interface. it compiles, but we're not actually generating UserClosures yet.

step 8: flip the switch and generate UserClosures rather than TemplateClosures. now all closure handling runs through the more generic interface that supports specialized implementations (such as our metamodule later).

all this works now and passes tests. phew.

#devlog #scopes #nocompiletime

step 9: removed TemplateClosure and renamed UserClosure to Closure. Various API functions to query information from the template pointed at by closures have been translated to query information from the closure instead (since the template behind it no longer exists as an object).

step 10: rewrote all code that depended on querying template and context from closures, such as the Capture module.

#devlog #scopes #nocompiletime

step 11: separated template-translator from reusable instruction builder flow for CondBr, Switch and Loop forms. this covers all complex flow.

#devlog #scopes #nocompiletime

step 12: painstakingly single out each interpreter case for instruction instantiation into its own constructor function, inling macros and finally get rid of all instances of `hack_change_type()` that were added when borrow checking was tacked on top of it, since we can now properly move and account for lifetime in constructor functions, and hence construct the correct type right away.

step 13: ensure all instructions of apply_from() are appended to the block.

#devlog #scopes #nocompiletime

step 14: today is the day! second attempt at changing the syntax expander to directly produce typed IR. all building blocks are in place.

with std::function, it was even possible to expose proper structured control flow constructors.

#devlog #scopes #nocompiletime

argh. nope, still too soon. i need to handle quoting and CompileStage correctly first.

step 14: quoter and CompileStage now directly construct typed IR rather than going through the indirection of generating templates, then proving them.

#devlog #scopes #nocompiletime

step 15: add native support for "function closures". we already have compile time closures, but meta-modules require runtime closures for simplified code generation.

#devlog #scopes #nocompiletime

the `based` branch is now in good enough shape for me to try once more to replace the LLVM backend with a clang C-- backend. i already have a prototype written in Scopes, but the last time i tried this it was so uncomfortable to write that I ran away screaming.

after the recent refactorings though, it should be much more straightforward to port.

#devlog #scopes #nocompiletime

day 3 of rewriting the LLVM backend to a C-- one. i'm two thirds through the file. the codegen writes only cpp macro calls, which are then translated by clang into actual C declarations through an additional include file.

that means once i'm done with gen_c.cpp, i still have to write the include header, but that's comparatively easy work.

#devlog #scopes #nocompiletime

now slowly extending the include header that makes sense of code generated by the backend, which looks like this:

https://paste.sr.ht/~duangle/07d7a52984cba8deca1e0179a3cabd1e42f5086d

#devlog #scopes #nocompiletime

using the new C-- code generator, we made it all the way to the REPL :)

now to fix tests...

108 tests executed, 66 succeeded, 42 failed.

#devlog #scopes #nocompiletime

all tests pass. LLVM backend is completely removed.

this is the entire macro translation layer, in under 500 lines. i managed to keep it header-free:

https://hg.sr.ht/~duangle/scopes/browse/lib/scopes/compiler/target/C/target_cc.inc?rev=tip

we presently still need to keep ABI API related stuff to flatten unions correctly, but after bringing the native UnionType back, this will also be taken care of.

#devlog #scopes #nocompiletime

i can't think of anything else to refactor in preparation of metamodules.

which means now it's time to do metamodules. gotta be brave.

#devlog #scopes #nocompiletime

the expander now directly builds typed IR (no metamodules), except for the AST quote part, but we only make it through two stages of core.sc before we need full AST quotes.

so i'm now adding an alternate pseudoquote path to the expander. if we then put a `spice-quote` in front of an entire module, we would automatically get a metamodule. but this way, both paths are still supported.

this next step also requires addition of many new C API functions.

#devlog #scopes #nocompiletime

these are all C API functions to construct structured control flow.

#devlog #scopes #nocompiletime

removing all template expression types and associated API functions. it's a lot. today is another good day.

#devlog #scopes #nocompiletime

generating code on the client side will become slightly easier as we're always quoting into the correct basic block anyway. so no "expression" containers necessary anymore.

right: before; left: after

#devlog #scopes #nocompiletime

these are the kinds of errors i get now for malformed IR. much easier to navigate and understand than LLVM module dumps.

#devlog #scopes #nocompiletime

we're now several stages into core. very funky bugs of all variety: calls into illegal pointers, things that don't fire when they should, ... i have to fix quite a bunch of basic extensions that build quotes out of order, and build minimal tests for analysis. but it makes the core better.

#devlog #scopes #nocompiletime

now starting to rewrite the expander to generate metamodule output.

the weirdest part is pseudoquoting. fortunately every form turns into a call on the first level (e.g. f(x) becomes apply(f,x)), so all subsequent levels just defer calls (apply(apply,f,x), apply(apply,apply,f,x), etc.)

#devlog #scopes #nocompiletime

starting in earnest to make modules as locally independent as possible so we can cache them efficiently.

I wrote down the plan for that as a new section in the existing plan: https://git.sr.ht/~duangle/texts/tree/master/item/ponder/nocompiletime.md#loose-coupling-status-quo

#devlog #scopes #nocompiletime

i'm now beginning to generate metamodules. pseudoquoting turned out quite interesting... i still don't know if it fully works like this but it looks promising.
the insight is that the metamodule only does API calls. if we record in the current block whether we are quoting or not, the API calls can translate the call either into a direct IR construction or an indirect one (generate the same API call again).

#devlog #scopes #nocompiletime

`sc_quote()` and `sc_unquote()` are used to mark these sections; they increment and decrement the quote level of the block. the functions themselves are quote-aware: when the existing quote level is already above 0, `sc_quote` also generates a `sc_quote` call. the same with `sc_unquote`. this defers the indirection to the next stage, each time removing one quote level.

#devlog #scopes #nocompiletime