#AHAlodeck R&D news:
- Testdataset: 386.5 GB
- 120.000 filesystem objects (files and folders)
- 7616839 key/value informations gathered/extracted as #xattrs.
Result: A 55 MB .tar.bz2 #holotar.
Fully search-and-filter-able by any of the 7 Mio xattrs.
Extraction step required only once (6h)
Afterwards:
Indexing done in 60 SECONDS! π€© π₯
Same data (key/value pairs) viewed in different application:
`apt install eiciel`
That's a #DataCentric paradigm in action! You can literally shoot any mixed (annotated) data at that.
It really puts fun back into using data daily.
Handed in my presentation proposal for #nttw10 #notimetowait
Presenting "#DataGardens" π»οΈπ»οΈπ»οΈ
Powered by #AHAlodeck to host any arbitrary data, offload key/value to filesystem - and transform datas into annotated related object graphs.
Pure "#ScienceFaction"!
#WeAllHaveBigDataNow
in #ZFS on #Linux, #Proxmox, #Debian, #Ubuntu #FOSS #FinF #ObjectStorage
Download #wikidata dump for testing #ahalodeck and #xattrs on #zfs long-term storage... π
Found this while doing so:
https://www.wikidata.org/wiki/Wikidata:Lists/lexemes/en
It's a statistical breakdown of the #English #language. #Interesting.
#AHAlodeck R&D news:
Created holotar-copy of a real-world mixed audio collection (recording production and digitized tracks archive): ~120.000 filesystem objects (files/folders)
(donated by recording engineers)
filesystem metadata-only = 175 MB (.tar)
+exiftool metadata as xattrs = 252 MB (.tar)
...compressed (.tar.bz2): **6,3 MB** ( π€― π π€ πΎ β)
**This is AMAZING!**
I can browse, order, catalog and manifest any object in this collection using standard #GNU #Linux tools.
@beet_keeper
#AHAlodeck: My best friend wrote a mini-basic fulltext indexer for #xattrs in the shell.
Took ~1.5 seconds (!) to index ~14.000 files with "de-embedded" music tags.
π₯³ π
I love xattrs.
Imagine doing this with embedded metadata tags? π
https://git-annex.branchable.com/git-annex-metadata/
git seems to have embedded key-value #metadata annotation capabilities... π» #AHAlodeck ahoi!
https://github.com/ArkThis/AHAlodeck/blob/main/res/0byte-db_entry.png
#xattr #linux #ext4 #zfs #samba #smb
Right-click-edit-metadata happily in-use over 1 year now: Amazing!
Very helpful insights on "how much actual bytes" can be stored in #xattrs on `btrfs`:
https://github.com/kdave/btrfs-progs/issues/917
Current limit = about the nodesize (it was formatted with).
nodesize - sizeof(struct btrfs_header) - sizeof(struct btrfs_item) - sizeof(btrfs_dir_item)
Current default = 16K - 101 - 25 - 30 , which is 16228 bytes
Maximum (for now) is ~64kB, if formatted like this:
`$ mkfs.btrfs -n 64K /dev/sdX`
Peter B.