Now on its own it isn't very interesting, its just a small AST for data structures.

The magic is that this is really an ✨intermediate representation✨

A shared interface really, between the ser and de parts of serde.

And on the other side we have modules that consume this IR and spit out other formats like JSON, s-expressions, TOML, KDL, XML, what have you.

I haven't sketched any of those yet enough to show a screenshot.

yeah, no. these meta-quotation libraries are unusable to me. I'll need tons more documentation and examples on how to do common things. I'm getting too spoiled by the amazing efforts in Rust to keep libraries well documented.

Going back to ast_builder.

so i got a deriver for serializers into the serde.ml language somewhat working, and this is already making it so much eaiser to test stuff -- viva la #metaprogramming!

#ocaml #rust #serde

fork yes! its valid #ocaml now. i think.

it also derives:

* newtypes
* tuples
* variants (unit, tuple, and record ones)

and knows how to follow modules/etc

ok, will add support for records next

friggin' brew. we're back on track. I wrote a sexpr serializer to iron out the serializer implementator interface, and it feels good!

(minus all the explicit first-class module signatures type escaping)

the important bits is that this is what you normally write as a user:

type t = { ... } [@@deriving serializer]

and then you can call

t |> serialize_t |> Serde_sexpr.to_string

wanna go to json? just change to Serde_json.to_string

maybe someone will finally implement a deriving(debug) with this!

but for authors of serializers, there's a few things that always get in the way:

1. error handling (esp. over collections)

2. mapping over the serde.ml data definition language

3. parametrizing over the kind of output you want to have (usually a string, but could be a buffer/channel that goes straight to stdout/disk/network)

all of those are provided by the Ser.Mapper interface, which is created at runtime based on the definition of the serializer you are using.

See this cursed module: https://github.com/ostera/serde.ml/blob/main/serde/ser.ml#L163-L185

IT'S ALIVE! 🧟🧙‍♂️✨

https://github.com/ostera/serde.ml

#ocaml #rust #serde #foss #buildinpublic

uuuuugh yojson why don't you have a clean clear exported ast WHY

folks, we need more _type-only_ packages. like we have for http type defs.

in any case, here we go: one line change and we got serialization to json.

sweet.

the beauty of having an abstract language for describing OCaml data, is that concrete data formats become fully pluggable.

One serializer, Many formats.

Just from yaml, to json, to protobuf in one line.

Ok, time for food. When I'm back we'll go for the Deserialize path which will be harder methinks.

Let me know what other data formats you'd like to see serialized, and go check out the code here: https://github.com/ostera/serde.ml

Okay back from food and a @anmonteiro retweeted so this is how I know I need to get back to it.

What does it take to deserialize? Let’s see how #rust #serde does it.

in short, we need:

1. a deserializer, that turns bytes into serde.ml data

2. a visitor/fold, to transform to turn serde.ml data into our custom type

3. a deriver, to generate visitors from types

so i think i'll start by making an interface for deserializers.

the idea here is that you tell serde how to deserialize these specific things from your data format, and then a visitor drives the work, only really deserializing what's needed

something like this -- you get a visitor passed in, a final value type, and you get to choose per-data-format what's the best way to consume the input

#ocaml #rust #serde

actually screw this, we'll start by manually expanding what a deserailizer would look like.

this will give me a much better idea of what the visitor/reader/deser interface needs to be.

oof, long hours #ocaml-ing here...I think I've got another ~2 hours in me before I get sleepy again. Thank you accidental nap!

The deserializer and visitor interfaces are coming along nicely! 👌🏼✨

If you were manually implementing a deserializer, this is how you'd use it to deserialize an identifier when you're coming from some data into a record with fields Name, Role, and Clearance.

Thankfully there will be a [@@deriving deserializer] too 🤩

#ocaml #serde

okay making some coffee then we're back. last night we got here:

* create a visitor for a small variant
* create a visitor for tags
* thread these into `Serde_sexpr` transparently

still a ton to figure out, but at least the structure is starting to fall into place!

now i'll brew the cofffee and thread this Variant_visitor into an implementation of the Variant_access_intf so that we actually use it.

code here: https://github.com/ostera/serde.ml

brb

#ocaml #rust #buildinpublic #devtools #serde

the sheer amount of #TypeLevel fuckery I have to do to get these modules to play along nicely and errors to surface tidily is worrying.

makes me really miss #rust traits for

In short. I’ve got a Reader interface, a Formar Deserializer, and a Visitor interface.

Together they create a Deserializer for a specific format and specific datatype over a specific source of data.

The Reader offers a basic low level next/peek/skip API over some data. So you can build a Reader for Bytes, String, In_channel, or whatever data you chunk bytes out of. Nom nom nom.

When you build a Format Deserializer (like something to parse JSON), you need a Reader. But you don’t want to read everything! That would be too inefficient. You want to only read the things you need to build up a specific datatype.

That’s where the Visitor comes in. You pass one to the Format Deserializer.

The Visitor itself decides what to do with specific pieces of data. Like what happens if we find an Int, or a Tuple, or a Variant Record.

This is specifically built for your datatype, so it knows about the fields your record needs, and the different constructors of your variant, so it can guide the deserializer forward.

It does this by receiving, in specific cases, tiny modules that offer further deserializing options. Like Variant_access (yes we do #JiraffeCase in #ocaml).

So when you visitors “visit_variant” function is called, you can peek at the variant tag name with Visitor_access.tag, and then depending on what kind of variant you know you have, you can call:

Visitor_access.unit_variant

Visitor_access.tuple_variant ~len

Visitor_access.record_variant ~fields

Each of these takes different arguments, and then drives the deserializer towards to read exactly what you’re expecting, and nothing more.

Operation wise this can be extremely efficient (as shown by #rust #serde, but I’m yet to benchmark this on #ocaml and my implementation is very naive anyway) since we only really do the minimum amount of work we need.

And yes this means that hand writing a Visitor is quite a bit of work.

Fortunately we can derive them from your types! Viva la #metaprogramming!

Anyway, I’m getting close to having a manual visitor running for a subset of S-expressions (variants).

What I’m afraid of is that there is no clear uniform interface for futures or readers (buffered or otherwise, sync or async) in the ecosystem, which makes it impossible to reuse parsers in other libraries (yojson, sexplib, tyxml, etc) out of the box.

Some say this is a feature. But in this case it looks like I’m going to have to borrow a lot of code to make these #parsers fit into #serde.ml