Irenes (many)we gave in to the urge to start writing a text editor https://code.irenes.space/ivy
(it doesn't edit anything yet)
this is our first time using the Rust
smol library, which seems quite nice. pleasingly, there isn't some big war between the authors of different rust async runtimes; rather, roughly the same group of authors wrote first tokio, then async-std, then most recently smol. this last one refactors the whole thing into a bunch of tiny, loosely-coupled libraries; smol itself is just a shorthand to import a few of those libraries at once. so that's pretty neat.thus far we have two direct dependencies and 35 transitive ones, which we're pretty pleased with, that seems nice and small to us
we have partially-implemented versions of the hjkl commands pushed, now. it's time to add an abstraction we've been really looking forward to, a helper that handles movement commands...
so in case you're keeping track of how long a project of ours can exist before we feel the need to use async closures, the answer is about three hours, 40 minutes
neat. we found and fixed a bug in our function that iterates through the file and keeps track of byte offsets to each line. it wasn't properly handling empty lines.
... see, finding a bug like that feels like progress to us because it demonstrates that the abstraction is doing the things we think it is, and when it fails it was just a minor tweak needed
like it makes us more confident of the approach than we were before
(we are way over-read on text editor implementation strategies, like in our body's early 20s our system read dozens of papers about it, so it's not like we really need more confidence, but hey)
(we're going to eventually use a buffer gap, but right this moment it's just a single consecutive buffer)
we definitely want to eventually support files larger than can fit in memory (so, like, in the hundreds of gigs)
not soon, but eventually
though it may be easier to get that into the architecture early on, rather than retrofitting it.. hm. well, we'll chew on that
technology has advanced since the last time we seriously tried writing an editor, and we clearly do not need to support files larger than 2^64 bytes, so we won't try to :)
(for you young ones: pointers used to be 32-bit!!! in fact, they used to be smaller, but by the time we were learning languages that use pointers, they were 32-bit)
(these days, unless you're on a microcontroller, you can take whatever size a pointer is on the platform and safely assume you will never need to describe a size or offset of anything that won't fit in it. that was not always true.)
not gonna lie, the continued progress seen at https://code.irenes.space/ivy/log/ feels really good
when we were young, jumping into a new project for a day or a week used to be really easy. we can do it for work just fine, but in recent years we've really struggled to channel intrinsic motivation for this sort of thing long enough to actually get anywhere
... which is fine; our habits are kind of time-oblivious, in the sense that we have a lot of dissociative memory stuff going on so we manage our tasks in ways that make forward progress regardless. there are projects we've finished in bursts of a couple hours every few months
but it's really nice to be properly deep in something
yay it can scroll through a file now
still doesn't do any actual editing, but it's starting to look quite solid as far as the viewing goes
we paid really close attention to what gets redrawn when. some of you may remember that conversation the other week about how terminal programs used to be good with screen readers because there was a natural efficiency need to only redraw things under active change, and then everyone stopped paying attention to that.
when our thing is more mature we're for sure planning to test how it feels out loud.
this code's looking and feeling a lot cleaner than the last time we tried to do byte-level terminal stuff that was in a project from a couple years ago that's still nominally ongoing but has been kinda stalled.
there was supposed to be a period in there, ah well :)
we may try to do the really fiddly thing, decoding terminal control sequences from a stream of input that is itself decoded characters having various encodings.
last time we stalled out on that, but we think smol's facility for Streams as the async equivalent to Iterators may be just what we need for it
it still feels absurd not being able to use BufReader (in any of its many versions, from many implementors)
notionally it solves a problem we have, but in point of fact it does not do that because POSIX stream semantics aren't just a list of bytes, the bytes have behavior over time and sometimes that matters
this is a pretty common nuance for language libraries to not handle well, it's just frustrating because BufReader is an abstraction that's gotten a lot of attention from a lot of people and yet it just does not let you do things like "read all the bytes that are present without blocking"
ferunando@ireneista i will come back to ask about how this is going later this year. i'm setting up myself a mostly-text setup-and-targets and will definitely want to hear more about screen-reader and terminals.
@gureito yeah definitely do circle back!
the esoteric programmer@ireneista the terminal programs were good with screenreaders bit is only halfway true. The only terminal apps which were really good with screenreaders were designed in such a way that the new stuff got appended to the end of the buffer. A new menu option is selected? append to end. Something about a progress bar changed? clear and write the value again, or, you guessed it, append to end. The reason why tty apps aren't very accessible with screenreaders now is that they use unicode block characters for drawing more intricate shapes, also even if they only redraw what changed, they treat the whole terminal like a screen where they can put a character anywhere, which makes the screenreader often read the whole thing. Console/tty specific screenreaders like the speakup kernel module and fenrir or tdsr, have sofisticated heuristics to somewhat deal with this, but it's quite difficult to make a complicated TTY app accessible to screenreaders
@esoteric_programmer that definitely makes sense. we specifically heard it worked okay in the context of parser-based interactive fiction, where the entire game is pretty much a transcript that only gets appended to, so that would fit with what you're saying.
@ireneista yeah, look at irssi for example, a lot of the interactions there are based on append to end, For a menu based tool, the thing that generates a kernel configuration file from menu options is accessible, but that's because of the same strategy. For something esoteric AF that doesn't work out of the box, archinstall is a good showcase, but here comes a heuristic of espeakup, speakup+ctrl+8 triggers a mode called highlight tracking, which manages to make sense of the stuff somewhat, I haven't looked into the espeakup code yet to see how that actually works, and considering that tty mode is going away soon enough...O well, sad
@esoteric_programmer hmmm, yes we see
Martin@ireneista love the description 😻
@mkhl thanks! 💜
SnowFox@ireneista That was also true for a while in the 32-bit days! Hard disks were below 2 GB, and on some platforms max file size was 2 GB, and even after that wasn't true, surely a *text* file will never have a reason to be over 2 GB, right?
I'm not sure if any text editors did "just mmap() the whole file" though (I assume classic Mac OS can't support it at all if you don't enable virtual memory; I'm not sure about Windows 98).
@snowfox we'd be shocked if anyone took the mmap()-only strategy in those days, yeah. a megabyte was a lot of RAM.
Philippa Cowderoy@ireneista My first compiler had a bunch of different memory models you could choose because the target was 8086/286...
...having 16-bit code and data pointers residing in separate segments had its uses
Graham Sutherland / Polynomial@ireneista but what if you want to edit a hundred exabytes of json? :P
@gsuberland thus far, the largest dataset we've ever had to work with was only 160 PiB, so fingers crossed that never comes up :D
amy tech (bones)@ireneista unsolicited, but you might also be interested in piece tables instead of buffer gap. Lots of interesting tricks you can do with them including baking undo directly into the buffer representation. Also allows you to mmap the initial load, and edits don't require loading anything new off the disk.
pho.spookygirl.boo/source/media-thing/browse/default/packages/text/src/PieceTable.ts;87712c3f45c29b96e6e6c0fdf15855f62a47f805?as=source&blame=off - a typescript version. I've got a paper lying around that describes them too.
pho.spookygirl.boo/source/media-thing/browse/default/packages/text/src/PieceTable.ts;87712c3f45c29b96e6e6c0fdf15855f62a47f805?as=source&blame=off - a typescript version. I've got a paper lying around that describes them too.
@amy ah yeah it's worth considering, thanks for that
rudi@ireneista how will you represent files in-memory? I think strings usually assume some valid encoding, and text files are crazy.
(In a former life I was the guy ppl came to with “we don’t know how to read this file, pls fix” and I’d find out parts of it were in some old Russian encoding and convert the lot to utf8)
@rudi oh, bytes, as far as that goes. we've been writing our own encoding-handling code because we care about not losing valid bytes during error recovery, and passing through invalid bytes unmodified, and stuff like that.
but that's not even the hard part, the hard part is that it's an editor and vectors are overly confining for that use.
@rudi wasn't it an amazing fun prank on all of us in the future how the various ISO-Latin encodings use the same codepoint for all the various currency symbols?
@ireneista it just makes me appreciate more what we have now. Unicode is messy because history but utf-8 is a marvel
Tim Ward ⭐🇪🇺🔶 #FBPE@ireneista Arrrghh.
I once had to work on a system which thought it could keep byte offsets to each line. It got into something of a mess with things like invalid character encodings. Where the line couldn't be parsed from bytes to characters, but you still needed to know where the end of line was so that you could process the *next* line, which *probably* wasn't mangled in the same way.
@TimWardCam yeah we are very much trying to do the right thing for invalid character encodings. we're a little nervous we'll inadvertently do something Unicode-specific and make things harder for ourselves, since we're starting with just UTF-8 and broken UTF-8, but we do know our way around encodings so hopefully we'll manage to avoid that.
@ireneista 😀 👍
I think the problem we had might have been to do with incompatible error handing between two of the libraries we were using - the input was supposed to be CSV which was another layer of libraries and complications (we weren't going to attempt to write our own bytes -> characters parsing).
@TimWardCam ah yep we can definitely see how that would make it significantly harder
we are doing our own bytes-to-characters stuff; we kind of feel like it wouldn't be a robust editor otherwise. that's more work, of course, but at least we get to be precise about how the weird cases are handled.
snowyfox
Will 💜@ireneista on the one hand: yay, that is small
on the other: 37 distinct dependencies.........
@Skirmisher we do think every single one of them both has a good reason to exist, and has a good reason to be in this project
R@ireneista not to be discouraging, but part of the reason there isn't some big war is because various parts of the rust async ecosystem are quite tightly coupled together
libraries used to be rather coupled to specific async runtimes (this might be finally getting better?), and low-level IO is *really* coupled to executors (this doesn't matter to most people, until it suddenly does)
@r yes, the whole point of this factored-out approach is to reduce those couplings
𐑦𐑙𐑜𐑮𐑦𐑛@ireneista what made you go for async? i’m just writing mine sync for now since terminal io is sync anyway
@ireneista oh, is this going to try to push towards "something somehow improving the world of TUIs"?
we've previously never been willing to touch projects like this because terminals are just *so* painful
@r mm, well, no, that's a separate project that we're treating this as decoupled from... because we wanted to actually get this one kicked off
@ireneista same! also in rust
@intarga looks very nice! slightly further along than ours
@ireneista yeah it (barely) edits text 😁 i spent most of my time so far on unicode width handling, since i know i will want to write japanese in it
@intarga impressive!
Badly-read literary snob@ireneista @intarga now I’ll be following both projects!