NEW by
@rrix:
Checking in on the new version of
The Arcology Project
|
Updates Arcology Programming
This fall I have been spending more time with the experimental rust rewrite of my Arroyo Systems Management
libraries, the early promising progress was written about below in I am starting to experiment with a rust rewrite of the Arroyo Arcology Generator. Since then, quite a lot of progress has been made.
I spent most of this summer building a few prototypes to try out different ORMs and compare them to what exists in the current Arcology FastAPI application, which uses a library called SQLModel
which promises the best of Pydantic for validation and marshaling and SQLAlchemy for query and persistence. It's fine, but a bit overkill for what I ended up building. I spent some time prototyping the Arcology's data model with a few different Rust and Python ORMs before landing on a surprising best choice:
The next version of the Arcology will be built with Django: Meet The Arcology Project: Django Edition
I spent some time over the last year building a small application for the Data Rights Protocol in Django with a coworker who had prior experience with it.
I had used it a bit in back in maybe 2012 but never really gave it a fair shake. While there was a lot I didn't and don't like about the codebase we built for testing DRP, Django is really decent to work with once you internalize a project structure and start to work with the ORM.
While I haven't had to optimize the ORM's behavior or individual queries in anger yet, it's transparent, it's robust, and it was really easy to get the data model I wanted slotted in to place. Compared to most of the Rust ORMs which involved code generation or a large amount of boilerplate entity structures or trait implementations, this felt more akin to writing Rails or SQLModel. Only Elixir/Phoenix's Ecto came close, and while I thought hard about shipping this in Elixir, I ended up deciding not to based on the agility I have had in the Django prototype.
Ultimately, I'm coming to appreciate Django a lot more than I did a year ago. It's a good kit and I could become quite productive with it.
The latest versions have some async python3 support which I tried to use for a bit but ultimately want to rip out since the ORM has two incompatible query paths between sync/async and you can't mix them, so you end up with multiple implementations of the same getter/setter with different colored functions that don't compose. But that's python. Based on historical traffic patterns, Arcology can handle running inside a handful of gunicorn processes. Hell it could run inside of 1 most of the time!
So Django handles page routing, templating, statics, and the data model that is used to serve pages. The data model and the actual HTML generation are being handled by the Rust codebase. pyo3 and its maturin build tool are hot shit. that thing fucking slaps. it goes so hard. etc. It makes it really easy to expose Rust code to Python and the reverse, too.
In The arroyors Native Org Parser the Orgize parser turns a page in to a struct of keywords, headings, links, etc, and these are shepharded in to Python and persisted in to the DB by Django.
This allowed me to do some thing cool very quickly: The The Complete Computing Environment configurations are now generated by the Django codebase instead of Emacs Lisp, and it's faster and more robust. I can nix run a flake to generate the DB from a directory of org-mode files and run it again to tangle them in to a init.el. Soon enough you could, too. It even runs in t184256/nix-on-droid so I can make config changes to my server or emacs environment and rebuild it on the go [not that i should…. 😛]
The arroyors Native Org Parser also has a fairly basic Org to HTML exporter built in to it, extending functionality in Orgize to provide an HTML exporter that,
in a single pass, does all the work which the FastAPI process had to do in three steps. And it does so without invoking Pandoc.
The most complicated part as always is the URL rewriting from internal org-roam IDs to external URLs like the one you're reading right now.
Consider a link to this very post [[id:20231223T231032.979299]]: find the file this heading is stored in. See if that file has a page entity, this will have an ARCOLOGY_KEY keyword in it,
publishing it to an arbitrary domain/path in my system, in this case
arcology/updates. arcology maps to a domain, engine.arcology.garden. this is replaced to generate a URL to this page, and then the ID is added as an anchor to jump the viewer to this post's headline. it's complicated but it keeps me from having to maintain file system heirarchies and just lean on org-roam's linking facilities in a flat file layout. And as of today The Arcology Project: Django Edition supports rendering these URLs:
There is still a laundry-list of feature work and basic functionality to be done and this system is a long way still from being able to replace the existing system but the new codebase already reveals some very nice synergies.
Speaking of synergies, I have one last thing to share, the new system will Ingest files on-demand using Syncthing rather than inotify as I hinted at in the last post. This, along with much much faster ingestion,
means that the sites will update within seconds of a change being made on an endpoint. Syncthing basically provides a very robust HTTP long-poll wrapper around inotify and i would love to not deal with temporary files and POSIX semantics and shitty state machines like I did in the Arcology Automated Database Builder
for the FastAPI site. swimmingy.
https://engine.arcology.garden/updates#20231223T231032.979299 #Updates #Arcology #Programming
poggi
𝕿𝖍𝖊 𝕮𝖆𝖕𝖙𝖆𝖎𝖓!
amikulo
Mystech
viq
Mistigris computer arts, est. 1994
Arcology Garden