Dimitri BouniolTo keep myself accountable, I’m going to start posting daily updates, no matter how small, on #CodableDatastore. The more I think about it, the more I’m realizing SwiftData is not for me, and I enjoy the challenge and thought experiment of writing something from scratch for something like this!
Made some good progress today on #CodableDatastore, at least on the documentation front 😅
I now have a reasonable spec document up for the rough storage format I'll be using for data stores to facilitate large data sets. Tomorrow, I'll probably start to add an implementation.
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.4
Also, shout-out to DocC and @SwiftPackageIndex for making docs so easy to publish: https://swiftpackageindex.com/mochidev/codabledatastore/main/documentation/codabledatastore/on-disk-representation
Once again, made some steady progress on #CodableDatastore. This time, it is finally writing _something_ to disk, and there are tests to verify it is writing it correctly.
I also started adding some issues, specifically for things I need to fix but don't plan to right away 😅
Tomorrow I'll focus on the index substructures so the Datastore type can start talking to the disk persistence.
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.5
A bit less progress on #CodableDatastore today as baby duties took priority, but managed to start thinking about the main snapshot type, and started dreaming of some apps that'll use this library, so so pressure, me 🤣
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.6
Just a single PR today for #CodableDatastore, this time adding the basics for creating snapshots on disk and tuning writing out the main store. I plan on having most of the loading logic live in the snapshot, so building good foundations to start!
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.7
Added a few more tests to #CodableDatastore, but otherwise spent the rest of my time spinning my wheels figuring our how to architect the index logic. Thankfully, I think I found a good solution I'm happy with for the short term, so I'll leave my changes unstaged for now and finish implementing it tomorrow!
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.8
Ran into a lot of issues with keypaths in #CodableDatastore this time around, specifically around how they are implemented on Linux vs Darwin, so had to do some backpedaling on my ideal API surface, but got something working for now…
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.9
Ok, I think I'm back on track with #CodableDatastore, no longer pulling my hair out dealing with keypaths, and ended up deleting all the branching code that they caused 😅
Next up: actually saving and loading descriptions of the data store to disk so we can detect when indexes need to be re-computed!
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.10
Quite a productive day for #CodableDatastore, cleaning up the missing bits to start loading and saving datastores to disk, and touching on the auto-migrator functionality for mismatched indexes, which I want to be easy for users of the library to deal with.
Next, I want to clean up the migration bits a tad, and jump into making write requests to the persistence — really nice seeing it all come together 🎉
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.11
Starting to really think about how transactions will interact in #CodableDatastore. For now, I’m probably going to implement top level Transactions in a serial queue, but I want the public-facing API to be open enough to making those transactions concurrent in the future.
Until then I’ll be thinking about how to construct cursors I guess, now that I have a ton of cursor accessors to fill out 😅
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.12
Got the front-half of writing instances drafted out for #CodableDatastore 🎉
This just leaves actually writing the data and indexes to disk 🤣 But first, I need to finish sketching out how Transactions are going to work as that is a really important part of how data stores can interact with other data stores.
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.13
Finally getting to hooking up Transactions and how they interact with each other in #CodableDatastore. Next will be modeling the file-based archives as actors so we can track when they change and are modified, and even have immutable copies for asynchronous reads. Getting there!
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.14
Didn’t get too much coding done with #CodableDatastore over the past few days, but spent many hours baby-in-arms thinking about how to support things like a persistent undo history for documents and libraries that use it!
Following my DDD mantra of Documentation-Driven Development, I wrote a conceptual document outlining my plan: https://github.com/mochidev/CodableDatastore/blob/0.0.15/Sources/CodableDatastore/CodableDatastore.docc/Conceptual/Persisting%20History.md
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.15
CodableDatastore/Sources/CodableDatastore/CodableDatastore.docc/Conceptual/Persisting History.md at 0.0.15 · mochidev/CodableDatastore
A pure-swift datastore enabling on-disk persistence for apps - CodableDatastore/Sources/CodableDatastore/CodableDatastore.docc/Conceptual/Persisting History.md at 0.0.15 · mochidev/CodableDatastore
Unfortunately didn't get much done with #CodableDatastore today, only getting the chance to add some minor docs and make a few more base types to represent indexes and pages. Here’s to having time tomorrow, I guess 😅
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.16
Finally back at making progress with #CodableDatastore, this time focussing on serializing and deserializing Index manifests. Next up, pages!
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.17
Also spent a little time improving AsyncSequenceReader, now that I’ve had a chance to want more from it 😅: https://github.com/mochidev/AsyncSequenceReader/releases/tag/0.1.2
Finally getting everything into place for #CodableDatastore , with a new protocol persistences can conform to, along with serialization for the indexes and data pages!
Looking forward to finally reading pages, passing cursors back to the data store, and finally writing data, but that will have to wait for tomorrow 😅
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.18
I very excitedly started implementing the binary search algorithm in #CodableDatastore this evening, before finishing the parts that actually write anything to disk, so need to pick that back up tomorrow 😅
Other than that, finally super close to writing some unit tests for the interactivity between the high level operations and the low level lookups. Once those are in place, the rest should come together quite nicely!
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.19
Finally finished writing and documenting the file-based binary search algorithm for #CodableDatastore's indexes and primary storage!
I was initially worried that a two-tiered approach might be insufficient, but you'd be able to store terabytes of records across just a hundred thousand pages, which should need no more than 17+1 file lookups. Last I checked, this should be sufficient for most use cases, especially an iOS app 😅
Now, just need to write tests for them 🤪
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.20
Earth, we have data being written with #CodableDatastore, I repeat, we have (the wrong) data being written (to the wrong file) with CodableDatastore 🎉
It's a choose your own adventure, #CodableDatastore style! While I finish up writes to any page, what should I work on next?
Updating Indexes
Deleting Entries
Reading Entries
Migrating Entries
Poll ended at .
Finishing the day with a big set of updates to #CodableDatastore, namely the ability to write entries to disk! Not everything is hooked up yet though, namely it will only write _some_ entries to disk, not all of them 😅
I’m going to tackle the rest of the cases tomorrow. I tried getting them done tonight, but hit a separate issue of too many open files instead 😛
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.21
I had a good reminder today developing #CodableDatastore that Big O explosions can happen inconspicuously, but managed to get insertions working such that the core algorithms themselves are at least linear…
There is *one* more explosion hiding in the form of a `.filter`, so I'll be removing that tomorrow and refactoring everything to work around it. Looking forward to the large speed increase!
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.22
Finally got to the bottom of the Big O explosions in #CodableDatastore, bringing random writes to a respectable O(log n) complexity. On an M1 Max, a write takes about 3ms at 10k existing entries, including the disk roundtrip, which I think I'm happy with for now. It can certainly be improved, but there are bigger features to land first.
Next up: making sure my migration plans are possible, though I'll probably need to build out reads and secondary index lookups first 😅
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.23
Today's improvements to #CodableDatastore were all focussed on performance, improving insertion by 6×, while improving correctness of the changes. This was done by enabling child transactions to group similar work before heading to disk, allowing batches of 5000 entries to overwrite every page in a datastore, while still maintaining history of the parent transaction over time.
Next, I want to work on updating entries, which is what I originally set out to do today 😅
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.24
Major milestone today for #CodableDatastore as I completed the basic CRUD operations, which I guess makes this a legitimate ACID-compliant database, written in pure-#Swift 🎉
Although I wouldn't recommend it yet, this means CodableDatastore is actually usable, albeit missing many _minor_ features, like secondary indexes 🤪 and robust unit testing 🙄
I guess I'll work on the indexes tomorrow, as those will open the gates for migrations and ranged reads.
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.25
Another big milestone for #CodableDatastore crossed today: you now have the ability to query how many items are stored. Oh, and it supports arbitrary indexes now 🤣
Well, kinda. There is currently no ranged-read support yet, so those indexes are just sitting there, unaccessible, but they are being created, and they are being maintained!
I'll probably add ranged read support in tomorrow, though I also want to cleanup the snapshot representation 🙃
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.26
So since Observation won the last poll for what to do next in #CodableDatastore, I went ahead and implemented ranged reads… just kidding, datastores are now observable as an AsyncSequence of events 😉
But actually, I’m only half kidding, because I also got half the necessary functionality done for ranged reads too. I shall finish that tomorrow, and finally get started on some cleanup so the persistence can (eventually…) maintain undo history across launches.
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.27
#CodableDatastore now has proper indexing support, complete with ranged reads! You still have to be a bit creative with compound indexes, but helpers for that can come in the beta stage.
Until then, time to get historical metadata in, which includes a cleanup of the snapshot format. _Or_ I can finish up the warmup and migration functionality. Either way, both need to get done before I can declare this a 0.1 and start using it in an app 😅
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.28
Finally refactored snapshots to have multiple iterations, and in the process added all the metadata needed for tracking changes over time. I probably won't wrap an undo API around this for quite some time, but it is slated as an eventual 1.0 feature.
Next, I'll work on rounding out index migrations, which is going to be the last feature before calling this a 0.1 😅 Still some ways to go before calling it a beta, but this means I can finally use #CodableDatastore in apps!
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.29
The main warm-up method for #CodableDatastore now has an implementation! This means that if you change your mind with regard to indexes, they’ll be re-computed in a graceful manner allowing you to report progress to the user.
I unfortunately can't call this a beta yet though: I added two more things to my todo list in terms of basic cleanup that would lead to incompatible persisted data… Once *that’s* done, *then* I'll call it a beta 😅
https://github.com/mochidev/CodableDatastore/releases/tag/0.0.30
Spencer Curtis🎉 I am pleased to announce the first official beta for #CodableDatastore!
CodableDatastore is a pure-Swift, ACID-compliant local database built for apps, focused on supporting value types and concurrency. It has a simple load/persist/delete model, with special support for ranged reads and observing changes.
There is still a lot to do, but all the basic functionality is… functional 😅. Most importantly, I'll officially start using it in my own apps going forward.
https://github.com/mochidev/CodableDatastore/releases/tag/0.1.0
This is probably my most ambitious open source project, and I’m excited to see how it evolves! It was also one I thought I would finish in a matter of days (hence CodableData*store* and not CodableData*base*), but here we are, months later of almost daily progress 😅
Most of all, it was fueled by my fear of #CoreData, and considering #SwiftData did nothing to ease my concerns, I figured it’ll be fun to actually design a database from scratch to tick what I thought should be supported in 2023.
I am *really* excited about the persistent history built in to the model itself — this means you’ll (eventually) be able to undo all the way to the beginning, just keep some history around, or even implement a “Time Machine” for individual objects in your store.
But my favorite feature is being able to look at the files on disk and be able to see what is stored, which is invaluable when developing any sort of software. @folivares dubbed it “the Markdown of databases”, which is pretty apt 😅
That feature may one day go away in the name of “performance” or “space efficiency”, but that’s a knob you as the user of the API can turn, since it really does just work with any Codable encoder/decoder you want, be it the built-in JSON/Plist coders, or a 3rd party BSON or other serialization scheme.
With regard to actual performance, it is quite reasonable, persisting most transactions to disk in under a millisecond, and being able to batch many more changes efficiently.
Namely, a key design goal was to not hide any performance pitfalls behind magic. Want to filter a sequence of results? Here is an AsyncSequence you can read from and filter them with `.filter()`. Want the count? Reduce the sequence and count the entries, and realize it isn’t the fastest operation.
You’ll know you are getting consistent performance loading a ranged read from any index — what you do with that sequence is up to you to reason about, letting you make the best decision for your app.
Another goal was to simplify accessing data. If you are familiar with a REST API, the async interface will be a breeze to work with. Just three basic operations to load, persist, or delete, and support for transactions to help keep changes consistent across stores belonging to the same persistence.
Since #CodableDatastore operates with value types, there is never any ambiguity whether changes got saved or not — if you persist them, they’ll be there the next time you load them 😉
There is still a whole mountain of work to get done, but that can be done little by little. Until then, happy to have shared this and hope it is either useful for your projects, or at the very least academically useful for learning how a database might be built, in a language you can actually understand 🤪
✌🏻
Bill Burgess@dimitribouniol I’ve been watching your progress on this, very exciting. I appreciate you putting this (and your progress on it) out into the world. I’m hoping to find a use for it in one of our apps. I have a need for a good persistent data store that needs more database than I would like.
@billburgess I’m glad you find it interesting! I hesitate to talk about it because at this point it is still largely a big bag of untested software with an important role to fill, while at the same time it can only get better if it is actually used…
Hopefully I can find enough time to bring it over the shipping line of stability sooner rather than later!
@dimitribouniol I look forward to using and hopefully contributing at some point.
Just released 0.1.1 of #CodableDatastore, with a much smaller maximum memory footprint! Goodbye, days of using several GB within just a few seconds, especially during migrations 😅
https://github.com/mochidev/CodableDatastore/releases/tag/0.1.1
Just released 0.2.0 (and 0.2.1 😛) of #CodableDatastore today, with a grand re-write of how indexes are declared in your code.
Gone are the wacky type gymnastics you had to jump through in favor of a single type where you describe the format of your datastore, and that's it! No macros, less boilerplate, and even new index types, what's not to love?
https://github.com/mochidev/CodableDatastore/releases/tag/0.2.0
Just released 0.2.2 of #CodableDatastore which fixed an issue migrating secondary indexes (a dumb copy paste error on my part haha).
That said, I also uncovered a potential write corruption that happened weeks ago which I'm investigating, so once again, PLEASE don't use this in production like I am, it's still unfinished and sparsely tested haha. The good news is it's all text files, so you can fix anything by hand for non-production/fun projects!
https://github.com/mochidev/CodableDatastore/releases/tag/0.2.2
Ok, good news is that I may have already fixed the #CodableDatastore corruption bug… 3 weeks ago. Bad news is the datastores I’ve been using are older than 3 weeks, so I have some surgery to do haha