As is somewhat customary, spring time is maintenance time and so this past week was again "infrastructure week" in #ThingUmbrella land, updating various tool scripts, switching to Rclone for uploading/syncing API docs, updating/pinning Prettier version & config, reformatting thousands of source files, re-uploading hundreds of MB of documentation for all current 214 projects, updating package metadata etc.

Also lots of recent updates to several packages, including some larger ones like:

https://thi.ng/column-store

Re-indexing of dictionary-based columns now re-creates the dictionary using the histogram (usage frequency) of stored values. This can result in major file size savings (I do still care about this aspect.. 😅). For example, in my PKM system (also my main dev focus at the moment), this change alone has reduced the database file size by almost 30% (containing tens of thousands of tagged items and where tags are stored in such a dictionary-based column).

The query engine now also supports result paging and sorting by an arbitrary number of columns/criteria & custom comparators (each with ascending/descending order options).

#ThingUmbrella #OpenSource #Maintenance #TypeScript #JavaScript

Finally have been trying out https://github.com/boyter/scc for counting (unique) lines of code, comments, DRYness and complexity and getting a better and more aggregated overview of the scale of my https://thi.ng/umbrella monorepo. SCC is really much better and faster than what I've been using previously (cloc) and also provides more meaningful results, including COCOMO project stats.

So selected current data points for this mega repo (incl. 214 projects and 185 example projects):

Estimated Cost to Develop (organic) $10,032,312
Estimated Schedule Effort (organic) 33.03 months
Estimated People Required (organic) 26.98

(This is equivalent of 74 years for one person and an assumed salary of $56k/year... Considering the repo in this form is only 8+ years old, that literally makes me a 9.25x developer 🙃)

#TypeScript
Files: 5,439
Lines of code: 323,816 (unique 161,337)
Comment lines: 85,303

#Zig
Files: 46
Lines of code: 4,828
Comment lines: 2,332

Markdown (aka #Documentation)
Files: 423
Lines: 66,370 (unique 31,330)

#ThingUmbrella #OpenSource #CodeAnalysis

It's Friday, spring is here (a bit too early) — it feels like a good day to share another minute recording of a variation of my Actiniaria piece which I worked on last spring and think also captures that much needed #BloomScrolling spirit...

See #Actiniaria for more context...

(Note: Sadly Firefox still doesn't respect the Rec2020 color profile in the video, please download the video or use Chrome or Safari for full viewing pleasure...)

#GenerativeArt #AlgorithmicArt #NoAI #ThingUmbrella #GenArtAPI #Boids #Color #Video #Animation #TypeScript #WebGL

#ReleaseSunday 🎉 Quite a few https://thi.ng/column-store updates over the past month, including further performance optimizations, more tests and documentation updates...

Just also added a small section and diagram illustrating the extensive use of bit-fields and bitmap-based indexes to optimize query processing and the stepwise narrowing of result sets in multi-term queries. Depending on column configuration and query term/operator type, individual rows often do not need to be visited/checked at all (only their bitfields are combined). But even if a traversal is necessary, the use of bitfields allows the engine to only having to check rows which have already been pre-selected by previous terms in the query. Apart from some exceptions, no interim result arrays are created. Queries are also short circuiting, i.e. terminate the moment a sub-term does not produce any results...

Happy coding! :)

#ThingUmbrella #Database #Query #Optimization #Bitfield #OpenSource #TypeScript #JavaScript

RE: https://mstdn.social/@inecas/116125098722422939

PSA: All the thi.ng subdomains should be working again. I upgraded Caddy yesterday and together with the recent DNS handling changes on Hetzner, Let's Encrypt challenges stopped working. All fixed again now...

#ThingUmbrella

Light Streaks — generative photography

A small selection of long-exposures of randomly generated IFS (aka Iterated Function Systems), a family of very oldskool primitive/trivial fractal functions, but which can produce a fairly wide variety of outcomes. Each image is the result of billions of iterations of a single particle being iteratively transformed (meaning the particle's current position is used as the input for computing its next position etc.) For each iteration & position a tiny amount of light is being captured, slowly revealing an image, just like a negative does in analog film photography.

Some of these images have been "exposed" (aka computed) for up to 30 mins. The smaller the amount of light captured per iteration, the smoother (less grainy) the outcome...

(For the more technical: This is one of these projects where a floating point pixel buffer _really_ makes all the difference! My exposure rate is only 0.001 per pixel per frame, some of the images use even weaker settings... That means for a pixel to become fully white is has to be visited at least 1000 times [or more])

Made with https://thi.ng/matrices (matrix transformations) and https://thi.ng/pixel ("film" capture)...

#MonochromeMonday #Photography #Light #Simulation #LongExposure #Fractal #IFS #GenerativeArt #AlgorithmicArt #ThingUmbrella #TypeScript