Fandom-dropping progress, April 15 report

Down to 939 fandoms total. (Only 26 currently have any tags to wrangle.)

I’m keeping up the pace of “shedding about 100 per month.” Still working on the second A-to-Z sweep, just finished with the P’s.

Also, still chipping away at recruiting “wranglers who aren’t over-the-limit” to pick up unwrangled Religion/Mythology/Folklore fandoms. I’m doing a little basic research on each one first. Someone with the right cultural/research background will always be better at spotting subtle inaccuracies, but for the fandoms that don’t get a wrangler like that, at least I can request fixes for anything really glaring.

Latest win: figuring out that this Ukranian “Folk Tale” fandom needs a rename, because all the fic is actually for the adorable 2024 cartoon Pravda & Kryvda. (11-minute pilot, free on Youtube.) Ukranian folklore-inspired with angel/demon vibes (I’m 0% surprised the artist has also done Good Omens fanart), extremely f/f shippy, has a fascinating “they were created around the same time but now there’s an overt age gap” dynamic…yeah, okay, I’m subscribing.

AMT updates: With the Madoka subtags approved, I went ahead and made the new Fake News tree request last week. (Basically the draft proposal I shared in February, with some slight tweaks.) Still no response to the behind-the-scenes question I mentioned in March…so yeah, I’m going forward on the premise of “if it’s that unimportant, it won’t be a roadblock.”

tl;dr Using https://thi.ng/column-store to accelerate tag intersection queries by a factor of 880x...

Working on the static website generator/export plugin for my personal knowledge tool has been one of the main projects this past month. A key part of this setup is tagging, not just simple flat keywords/categories, but actually treating tags as sets. The system doesn't just allow browsing content by a single tag, but also supports adding (or removing) tags to narrow or widen the current topic. E.g. The combination of "3d + geometry + typescript" would select only works which have all of these three tags...

In the local version of my tool there's no limit to the number of tags (and it also supports tag negation), but for the static site generation I have to limit the set size (due to combinatoric explosion) and pre-compute all possible permutations, then create HTML documents for each these individual combinations which actually produce results.

So far I'm having ~400 unique tags in use, meaning if I want to aim for a max set size of 3, there're theoretically ~64,000,000 possibilities to check[1]! For the roughly 3500 content items used for testing, a naive JS approach to filter the result array and only retain items matching the entire current permutation is so extremely slow, that I stopped the process after 3.5 minutes just for the first 250k (aka 0.4%) of the 64 million permutations, i.e. at that rate the full process would have taken ~15 hours, pretty slow for a SSG... :)

Naive approach 🫣:

```
permutation = ["3d", "geometry", "typescript"]
results.filter(item => permutation.every(tag => item.tags.includes(tag)))
```

But since I'm using https://thi.ng/column-store as my database, such queries can be optimized by a few magnitudes, since here these intersection queries are applied only to bitfields (explained in the pkg readme). This results in all 64+ million permutations being processed in just 62 seconds (1+ million per second). Quite the difference, i.e. ~880x faster than the above approach!

Also, of these 64 million initial possibilities, there're fewer unique ones (excluding duplicates and ignoring ordering), and currently only ~24,000 are actually producing a result. Still, that's 24,000 index pages to generate & host and it's, of course, far, far too much!

So I will have to also spend more effort curating and severely reducing the tag vocabulary, at least the subset used for the website. On the other hand, I think this system will really help with browsing this large body/archive of work much more meaningfully than the boring single-tag/category approach most websites are offering. And it will do so without any backend (other than file hosting)...

[1] Permutations = 400 + 400^2 + 400^3

#ThingUmbrella #Tagging #Intersection #Query #Bitfield #WebDev #JavaScript #TypeScript #Optimization

Beets Importer

https://blog.notmyhostna.me/posts/beets-importer

#beets #music #tagging #organizing

From walking the 'net, I found this article from Alex Chan:

https://alexwlchan.net/2024/static-websites/

The idea of a small, static, local #website for my own content is beautiful, and I would like very much to do the same as Alex.

The problem is: I'm a #digital #hoarder.

https://en.wikipedia.org/wiki/Digital_hoarding

I have a barely organized collection of upwards of 10K books, 100K images, 17K songs, 3K videos, and 20K bookmarks; I have no time to #tag them all, and the collections grow steadily by the month.

While Alex can add and tag each item as she collects it, I would need hours daily, for years, to get even - and that's after many years building my current file system organization.

Does anyone know a #software for #tagging files in general (not only images or books, for instance), which matches all of the requisites below?

- Is open-source;
- Runs in both Windows and Linux;
- Does not touch or edit the files being tagged;
- Allows for exporting of #metadata to textual formats, like #JSON;
- Is easy to use;
- Has powerful search.