Henri Sivonen
Tekniquelly correctWhoa. UTF-8 is older now than ASCII was when UTF-8 was invented.
avatastic 
Farce Majeure@tek and it still sucks
@vathpela Awww, I like UTF-8! I can pretend it's ASCII most of the time.
@tek I have complaints about recoverability on a mildly corrupted bitstream, but it's much too late in the evening to articulate this well.
@tek (don't get me wrong, I have to use UCS-2 often enough to know real pain...)
Elias Mårtenson@vathpela @tek Given how much worse the alternatives are, and how impossible it would have been to get people to move off of encodings, I'm glad UTF-8 exists.
Don't take me wrong, I'm quite aware of the issues with UTF-8, but I (choose to) believe that if it wasn't for UTF-8 we'd still be drowning in ASCII, and it would be impossible to tell the English-only speaking minority that supporting letters other than what was used to write inscriptions in ancient Rome might actually be useful.
Enno Rehling
mxk
Magnus Ahltorp
Glen T, heated, not stirred
Мя ��@vathpela IMHO, redundancy and/or checksums should be implemented on different layer, not in text encoding
Like, there's many, many ways to keep bits from corrupting, which are applicable in different cases
And forcing one particular inside of text encoding itself is...meh
Same for compression btw. For some texts (CJK in particular) UTF-8 is sub-optimal, but even basic deflate makes it compact enough
TL;DR: UTF-8 is not perfect, but having one encoding for every text outweighs
@mo @vathpela Also, UTF-8 is trivially easy to synchronize. If you delete a byte out of the middle of a file, at most you’ll lost the one affected character (well, code point). The ones before and after it will be fine. That’s not true of some other Unicode encodings, like double width ones where everything after would be out of sync.
root42
Mans R
David, a Bostonian in Tokyo.
Vincent Sparks 🔜 Furlingame@AVincentInSpace @vathpela Unicode currently only reserves code points 0..0x10FFFF (https://en.wikipedia.org/wiki/Unicode_block), so all existing CPs fit in 21 bits.
But @djl was saying we don't have to waste 32 bits to encode the most common code points since UTF-8 came along. This chat here only uses 8 bits per CP/character.
@tek @AVincentInSpace @vathpela
Well, I wasn't actually saying that, but at least in the Anglophile universe, since UTF-8 is just ASCII with an escape bit, it does that. Nicely.
My irritation, and rant, is that if I want to actually do some programming with strings (that use both Japanese and English), I really really really don't want to muck with a variable-width encoding.
But, as I understand it, even UTF-32 is a variable-width encoding _if I program for it correctly_.
Sigh.
@djl @AVincentInSpace @vathpela Ah, got it.
I don't think that's the case for UTF-32, though. By definition, I think every code point is encoded as exactly 4 bytes.
@tek @AVincentInSpace @vathpela
Another issue here is that one of the worst sins in programming is _premature optimization_.
As someone whose serious programming experience was all before 1990, my intuitions are way off for modern processors.
I'm dealing with 500 MB of Japanese text on disk, and reading them into Python and searching them is zippy quick on a PC.
So, IMHO, the world would work fine if the folks at Unicode defined a 32-bit fixed-with encoding, and we just used that.
@djl @AVincentInSpace @vathpela That would make all text storage 4x larger than ASCII, even for Latin text, making cache-bound operations on strings much, much slower. It also means there’s zero compatibility between “old” and “new” text (instead of being able to manipulate ASCII with UTF-8 functions). Also, big endian or little endian?
@AVincentInSpace @vathpela @tek
My understanding is that UTF-32 uses escape characters to get to rarely used characters, and thus is a variable-width encoding.
@djl @AVincentInSpace @vathpela Not quite. One UTF-32 code point == 1 32-bit int. However, you can combine multiple code points to make a single glyph. Lots of emoji work that way, for example. But the code points are still exact 32 bits.
@tek @djl @vathpela Okay, but that's still multiple codepoints. The ASCII sequence "ffi" can be displayed as a single glyph depending on your font. So what?
(Okay, technically that's a ligature, which is distinct from a glyph in that a glyph is to be treated as a single character for the purpose of selecting text and a ligature is not, but unless you are implementing a textbox where the user can select text from scratch, you do not need to care about the difference, and unless you are developing a UI toolkit you almost certainly do not need to care about anything besides codepoints.)
🇺🇦 haxadecimal 🚫👑
David Chisnall (*Now with 50% more sarcasm!*)
Sikorski Arkadiusz vel ArakuS
Fabian Köster@tek Still I am regularly confronted with IT systems that do not (properly) support it and display my name with an umlaut wrong.
Bradley M. Kühn
Michele Adduci@tek and it is still being handled wrongly in many places
deBaer
Tim Ward ⭐🇪🇺🔶 #FBPE@tek Every now and then the Cambridge CST exam papers include a question like "explain why even experienced programmers sometimes have problems with character codes".
You could write pretty well anything you liked.
Originally what was expected was an essay about things like escape sequences on Flexowriter tapes; in my day it was about conversion between EBCDIC and ASCII; these days it might be about obscure characters in URLs.
Jonathan Addleman@tek
And yet, my bank still won't let me add a contact (for etransfers) with an accent in their name.
And yet, my bank still won't let me add a contact (for etransfers) with an accent in their name.
Elena ``of Valhalla''@jaddle @tek if it's sepa (or something else with the same restrictions) it's possible that they are still using an extremely restricted set of characters with just uppercase letters, digits and a small handful of punctuation, selected back when everything had to move between obsolete mainframes, that we are still stuck with today for compatibility reasons (yay :( )
adamrice
Alper Çuğun-Gscheidel@tek MySQL will still happily mangle it.
@tek when Cleopatra invented UTF-8 she was actually closer in time to us than she was to the invention of ASCII
Arcticulate@tek The problem is that UTF-8 wasn’t universally used until the advent of smartphones.
Leonardo Fontenelle@tek Even Windows uses it!
huxley(fur) / Laelia 🔜 Furcationland DeadDog@tek if they keep adding emoji we'll have to invent UTF-9!!!
(That's a joke)