i went on IRC and asked around on the kate channel, and we browsed a bit of source code together, and --
so it turns out Kate's LSP client simply doesn't highlight strings, numbers and operators.
i filed a ticket with the project, let's see how it goes.
hm. Kate colorizes macros and comments, but not strings, numbers and keywords. idk what's wrong.
when i flip the token order, same result. when i make all tokens comments or macros, then they get colorized as such. it's weird.
the way sxpp's streaming lexer (tokenizer) works, UTF-16 and UTF-32 input streams are already supported.
since all controlling characters are well below 0x7f, and the lexer doesn't output strings, only token types and locations, you can just feed it 0xff clamped chars, and then offsets and positions are implicitly correct.
how does LSP encode text file positions: UTF-8? (what sxpp uses) UTF-32? (also sensible i guess)
neither. (well since 3.17 yes, but only if the client feels that way)
it's UTF-16. UTF-16 support is mandatory. why? because it's a damn microsoft protocol that's why¹. 😫
fortunately character offsets are line-relative so fixing that up is not too expensive.
—
¹ yes it's also because Javascript.
it's beginning to work!
numbers and strings are not being highlighted correctly yet, don't know why.
also, apparently tokens are not allowed to span multiple lines¹, so i need to fix up this part as well.
more tomorrow.
—
¹ multiline tokens can be supported by clients, but the server needs coverage for the fallback so why maintain two paths?
we have successful JSON parsing, object query, object construction and serialization.
now to provide an actual service. the most important thing is colorization, so that's first.
LR