New blog post:

A simple, async page cache built on top of io_uring

https://radiki.dev/posts/glowdust-page-cache-1/

#glowdust #io_uring #rust #rustlang

#Glowdust now comes with a single threaded, request based page cache. This lets me serve pages to multiple transactions, asynchronously, without spending time optimizing a thread safe version.

Which means I can start actually implementing the stuff I wrote about last week:

https://radiki.dev/posts/glowdust-tx-implementation-1/

TL;DR Transaction state is kept in the page cache and it's basically just another file. This causes a lot of interesting behavior to experiment with.

And just like that, #Glowdust now has a page cache that can actually fault pages in memory.

From some other place in memory.

And it's all it can do. Just page faults.

But hey, progress.

#database

2500 words, a princess bride meme and my surprise at finishing it.

That's about 86% of what you need to know about my new blog post.

The remaining 14% is describing a new architecture for database transaction subsystems. In #Rust , obviously.

Do let me know what you think - especially about the backpressure design.

https://radiki.dev/posts/glowdust-tx-implementation-1/

#rustlang #database #glowdust

Because transaction state is

1) stored in the page cache and so

2) can be moved freely between threads,

transactions can be served from a priority queue from a fixed number of workers.

Thus, #Glowdust has, by design, the ability to automatically balance CPU and memory use.

Memory pressure pushes back via I/O to persist overflowing tx state and serve page faults.

CPU pressure reduces time spent on each tx per worker, which makes tx state accumulate in memory, which releases CPU.

Update: Initial transaction support has landed in #Glowdust

https://codeberg.org/glowdust/glowdust/commit/37a0b94f9846611c8b5f0433711537b678c07594

Pretty large update, and I still have some work to clean up and commit locally. But it's coming together.

The independence of this code from the language syntax is deliberate. It's all in the direction of having the frontend be pluggable so you can roll your own DBMS, language and all.

Pro tip: If you want to step through a thread, you can pass it a sync_channel(1) and use write/read calls to block and unblock it.

You can do this with multiple threads and create a state machine for controlling synchronization and reproducing or checking for race conditions.

I am currently doing this with #Glowdust and it works pretty well for writing tests for transaction isolation.

#rust #rustlang

Maybe I've been thinking about #Glowdust the wrong way.

Maybe I shouldn't talk about it as a DBMS. Perhaps it makes more sense to talk about it as a database development kit, a DBDK, if you will.

I'll think about it on my way to vote #EU #Election #Europe

So #Glowdust commits stuff from different threads now, NBD.

About a month ago this seemed impossible. I still can't really believe that I got this far.

#Glowdust update: Multithreaded transaction execution works, with tx state storage in the page cache.

What remains is commit(), i.e. make per tx state visible to other transactions.

I mean, there are a million steps after that, but for now, only commit() remains.