Made some good progress today on #CodableDatastore, at least on the documentation front 😅

I now have a reasonable spec document up for the rough storage format I'll be using for data stores to facilitate large data sets. Tomorrow, I'll probably start to add an implementation.

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.4

Also, shout-out to DocC and @SwiftPackageIndex for making docs so easy to publish: https://swiftpackageindex.com/mochidev/codabledatastore/main/documentation/codabledatastore/on-disk-representation

Once again, made some steady progress on #CodableDatastore. This time, it is finally writing _something_ to disk, and there are tests to verify it is writing it correctly.

I also started adding some issues, specifically for things I need to fix but don't plan to right away 😅

Tomorrow I'll focus on the index substructures so the Datastore type can start talking to the disk persistence.

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.5

A bit less progress on #CodableDatastore today as baby duties took priority, but managed to start thinking about the main snapshot type, and started dreaming of some apps that'll use this library, so so pressure, me 🤣

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.6

Just a single PR today for #CodableDatastore, this time adding the basics for creating snapshots on disk and tuning writing out the main store. I plan on having most of the loading logic live in the snapshot, so building good foundations to start!

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.7

Added a few more tests to #CodableDatastore, but otherwise spent the rest of my time spinning my wheels figuring our how to architect the index logic. Thankfully, I think I found a good solution I'm happy with for the short term, so I'll leave my changes unstaged for now and finish implementing it tomorrow!

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.8

Ran into a lot of issues with keypaths in #CodableDatastore this time around, specifically around how they are implemented on Linux vs Darwin, so had to do some backpedaling on my ideal API surface, but got something working for now…

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.9

Ok, I think I'm back on track with #CodableDatastore, no longer pulling my hair out dealing with keypaths, and ended up deleting all the branching code that they caused 😅

Next up: actually saving and loading descriptions of the data store to disk so we can detect when indexes need to be re-computed!

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.10

Quite a productive day for #CodableDatastore, cleaning up the missing bits to start loading and saving datastores to disk, and touching on the auto-migrator functionality for mismatched indexes, which I want to be easy for users of the library to deal with.

Next, I want to clean up the migration bits a tad, and jump into making write requests to the persistence — really nice seeing it all come together 🎉

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.11

Starting to really think about how transactions will interact in #CodableDatastore. For now, I’m probably going to implement top level Transactions in a serial queue, but I want the public-facing API to be open enough to making those transactions concurrent in the future.

Until then I’ll be thinking about how to construct cursors I guess, now that I have a ton of cursor accessors to fill out 😅

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.12

Got the front-half of writing instances drafted out for #CodableDatastore 🎉

This just leaves actually writing the data and indexes to disk 🤣 But first, I need to finish sketching out how Transactions are going to work as that is a really important part of how data stores can interact with other data stores.

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.13

Finally getting to hooking up Transactions and how they interact with each other in #CodableDatastore. Next will be modeling the file-based archives as actors so we can track when they change and are modified, and even have immutable copies for asynchronous reads. Getting there!

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.14

Didn’t get too much coding done with #CodableDatastore over the past few days, but spent many hours baby-in-arms thinking about how to support things like a persistent undo history for documents and libraries that use it!

Following my DDD mantra of Documentation-Driven Development, I wrote a conceptual document outlining my plan: https://github.com/mochidev/CodableDatastore/blob/0.0.15/Sources/CodableDatastore/CodableDatastore.docc/Conceptual/Persisting%20History.md

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.15

Unfortunately didn't get much done with #CodableDatastore today, only getting the chance to add some minor docs and make a few more base types to represent indexes and pages. Here’s to having time tomorrow, I guess 😅

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.16

Finally back at making progress with #CodableDatastore, this time focussing on serializing and deserializing Index manifests. Next up, pages!

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.17

Also spent a little time improving AsyncSequenceReader, now that I’ve had a chance to want more from it 😅: https://github.com/mochidev/AsyncSequenceReader/releases/tag/0.1.2

Finally getting everything into place for #CodableDatastore , with a new protocol persistences can conform to, along with serialization for the indexes and data pages!

Looking forward to finally reading pages, passing cursors back to the data store, and finally writing data, but that will have to wait for tomorrow 😅

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.18

I very excitedly started implementing the binary search algorithm in #CodableDatastore this evening, before finishing the parts that actually write anything to disk, so need to pick that back up tomorrow 😅

Other than that, finally super close to writing some unit tests for the interactivity between the high level operations and the low level lookups. Once those are in place, the rest should come together quite nicely!

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.19

Finally finished writing and documenting the file-based binary search algorithm for #CodableDatastore's indexes and primary storage!

I was initially worried that a two-tiered approach might be insufficient, but you'd be able to store terabytes of records across just a hundred thousand pages, which should need no more than 17+1 file lookups. Last I checked, this should be sufficient for most use cases, especially an iOS app 😅

Now, just need to write tests for them 🤪

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.20

It's a choose your own adventure, #CodableDatastore style! While I finish up writes to any page, what should I work on next?

https://github.com/mochidev/CodableDatastore

Finishing the day with a big set of updates to #CodableDatastore, namely the ability to write entries to disk! Not everything is hooked up yet though, namely it will only write _some_ entries to disk, not all of them 😅

I’m going to tackle the rest of the cases tomorrow. I tried getting them done tonight, but hit a separate issue of too many open files instead 😛

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.21

I had a good reminder today developing #CodableDatastore that Big O explosions can happen inconspicuously, but managed to get insertions working such that the core algorithms themselves are at least linear…

There is *one* more explosion hiding in the form of a `.filter`, so I'll be removing that tomorrow and refactoring everything to work around it. Looking forward to the large speed increase!

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.22

Finally got to the bottom of the Big O explosions in #CodableDatastore, bringing random writes to a respectable O(log n) complexity. On an M1 Max, a write takes about 3ms at 10k existing entries, including the disk roundtrip, which I think I'm happy with for now. It can certainly be improved, but there are bigger features to land first.

Next up: making sure my migration plans are possible, though I'll probably need to build out reads and secondary index lookups first 😅

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.23

Today's improvements to #CodableDatastore were all focussed on performance, improving insertion by 6×, while improving correctness of the changes. This was done by enabling child transactions to group similar work before heading to disk, allowing batches of 5000 entries to overwrite every page in a datastore, while still maintaining history of the parent transaction over time.

Next, I want to work on updating entries, which is what I originally set out to do today 😅

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.24

Major milestone today for #CodableDatastore as I completed the basic CRUD operations, which I guess makes this a legitimate ACID-compliant database, written in pure-#Swift 🎉

Although I wouldn't recommend it yet, this means CodableDatastore is actually usable, albeit missing many _minor_ features, like secondary indexes 🤪 and robust unit testing 🙄

I guess I'll work on the indexes tomorrow, as those will open the gates for migrations and ranged reads.

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.25

Another big milestone for #CodableDatastore crossed today: you now have the ability to query how many items are stored. Oh, and it supports arbitrary indexes now 🤣

Well, kinda. There is currently no ranged-read support yet, so those indexes are just sitting there, unaccessible, but they are being created, and they are being maintained!

I'll probably add ranged read support in tomorrow, though I also want to cleanup the snapshot representation 🙃

https://github.com/mochidev/CodableDatastore/releases/tag/0.0.26