Irenes (many)we gave in to the urge to start writing a text editor https://code.irenes.space/ivy
(it doesn't edit anything yet)
this is our first time using the Rust
smol library, which seems quite nice. pleasingly, there isn't some big war between the authors of different rust async runtimes; rather, roughly the same group of authors wrote first tokio, then async-std, then most recently smol. this last one refactors the whole thing into a bunch of tiny, loosely-coupled libraries; smol itself is just a shorthand to import a few of those libraries at once. so that's pretty neat.thus far we have two direct dependencies and 35 transitive ones, which we're pretty pleased with, that seems nice and small to us
we have partially-implemented versions of the hjkl commands pushed, now. it's time to add an abstraction we've been really looking forward to, a helper that handles movement commands...
so in case you're keeping track of how long a project of ours can exist before we feel the need to use async closures, the answer is about three hours, 40 minutes
neat. we found and fixed a bug in our function that iterates through the file and keeps track of byte offsets to each line. it wasn't properly handling empty lines.
... see, finding a bug like that feels like progress to us because it demonstrates that the abstraction is doing the things we think it is, and when it fails it was just a minor tweak needed
like it makes us more confident of the approach than we were before
(we are way over-read on text editor implementation strategies, like in our body's early 20s our system read dozens of papers about it, so it's not like we really need more confidence, but hey)
(we're going to eventually use a buffer gap, but right this moment it's just a single consecutive buffer)
we definitely want to eventually support files larger than can fit in memory (so, like, in the hundreds of gigs)
not soon, but eventually
though it may be easier to get that into the architecture early on, rather than retrofitting it.. hm. well, we'll chew on that
technology has advanced since the last time we seriously tried writing an editor, and we clearly do not need to support files larger than 2^64 bytes, so we won't try to :)
(for you young ones: pointers used to be 32-bit!!! in fact, they used to be smaller, but by the time we were learning languages that use pointers, they were 32-bit)
(these days, unless you're on a microcontroller, you can take whatever size a pointer is on the platform and safely assume you will never need to describe a size or offset of anything that won't fit in it. that was not always true.)
not gonna lie, the continued progress seen at https://code.irenes.space/ivy/log/ feels really good
when we were young, jumping into a new project for a day or a week used to be really easy. we can do it for work just fine, but in recent years we've really struggled to channel intrinsic motivation for this sort of thing long enough to actually get anywhere
... which is fine; our habits are kind of time-oblivious, in the sense that we have a lot of dissociative memory stuff going on so we manage our tasks in ways that make forward progress regardless. there are projects we've finished in bursts of a couple hours every few months
but it's really nice to be properly deep in something
yay it can scroll through a file now
still doesn't do any actual editing, but it's starting to look quite solid as far as the viewing goes
we paid really close attention to what gets redrawn when. some of you may remember that conversation the other week about how terminal programs used to be good with screen readers because there was a natural efficiency need to only redraw things under active change, and then everyone stopped paying attention to that.
when our thing is more mature we're for sure planning to test how it feels out loud.
this code's looking and feeling a lot cleaner than the last time we tried to do byte-level terminal stuff that was in a project from a couple years ago that's still nominally ongoing but has been kinda stalled.
there was supposed to be a period in there, ah well :)
we may try to do the really fiddly thing, decoding terminal control sequences from a stream of input that is itself decoded characters having various encodings.
last time we stalled out on that, but we think smol's facility for Streams as the async equivalent to Iterators may be just what we need for it
it still feels absurd not being able to use BufReader (in any of its many versions, from many implementors)
notionally it solves a problem we have, but in point of fact it does not do that because POSIX stream semantics aren't just a list of bytes, the bytes have behavior over time and sometimes that matters
this is a pretty common nuance for language libraries to not handle well, it's just frustrating because BufReader is an abstraction that's gotten a lot of attention from a lot of people and yet it just does not let you do things like "read all the bytes that are present without blocking"
you can achieve that goal but to do it, you need to buffer those bytes on your own, externally
anyway we figured out the whole BufReader mess years ago for other stuff, and it's fine, we know not to go down that rabbit hole again and the stuff we've been doing instead has been working
Rho@ireneista either a state machine or a parser that outputs linked list entries to be consumed by something else is what we would try. curious what y'all end up doing
@rho yeah... well, you ideally want to avoid allocating all those entries explicitly because it's an inner loop, and the time spent doing that will dominate the overall cost of the entire program
@rho but de facto, yes, we're going to have some form of data structure that communicates between the layers, and we're going to pretend to ourselves that maybe Rust can do very aggressive inlining and avoid allocating it most of the time (it probably can't, but finding out for sure is months of work, so it makes a plausible avenue for willful self-deception)
@rho but it's actually the flow control between the layers that's surprisingly challenging, mostly
@rho anyway, thanks! we will for sure share more details when we do them
@rho (for the record, this is an Actual Hard Problem, we've spoken to friends who work specifically in this area about it and it really comes down to understanding exactly what the task is and optimizing relentlessly. we have no expectation ours will be high-performance in the short term, and that's fine, we care more that it's readable)
mx alex tax1a - 2020 (6)@ireneista @rho the traditional solution here is a ring buffer with separate read/write pointers iirc
SnowFox@ireneista The things I find myself frequently wanting are "un-read these bytes" (maybe that wouldn't be necessary if learned how to write async code instead of using a select()-loop) and "wait until the next byte or there's a timeout" (also maybe less necessary with an async API?).
The latter feels like a general-purpose thing to do? I have vague memories of writing double-click/long-press detection and wishing for "send an event after a synthetic timeout" instead of needing to manually schedule/unschedule a timer (and worrying about edge cases).
@snowfox no, you definitely do still need at least one byte of un-reading for a lot of things, even with async heh. you're not imagining that. we convinced ourselves of it the hard way.
@snowfox adding timeouts to stuff is one thing that most Rust async libraries are good at, so good news on that front
@ireneista I was thinking "async means I can keep the un-consumed bytes in a local variable instead of needing to use an instance variable" though I guess un-reading may be easier.
But if you can only un-read a single byte (ungetc doesn't guarantee any more than that) that also means you have to read a byte at a time, which seems inefficient?
I think what I really want is a way to read with MSG_PEEK (does that work on pipes?) and then tell the kernel how many bytes were consumed. That'd allow tricks like parsing a HTTP CONNECT, only consuming the "header" bytes, and passing the socket to another process, without necessitating reading 2-4 bytes at a time to make sure you don't overshoot the end-of-header.
@snowfox oh, yes you totally can use locals for that, we were just trying to avoid it for reasons that probably don't apply to you
don't worry about ungetc unless you're working in C, it's not a kernel facility, libc just has a byte of RAM in your address space that it manages, you can do the same thing yourself
unsure about MSG_PEEK. sounds slick if it works.