This is something that I've been fascinated about for a while now. Let's see if I can lay it all out --
π‘Unexpected Challenge Breakdownπ‘
πchar- and word-counters!π
It sounds straightforward but has a ton of surprising challenges
- itβs inconsistent between #languages and scripts
- itβs inconsistent between products online (!)
- it's hard for users to understand
- itβs completely thrown off by emoji πππ
β¦ We thought a lot about this at #Wikipedia β¦
(thread π§΅1/)
π§΅2/
βByteβ limits can cause an #equity problem:
π€For Latin script, each char is more or less a byte (ASCII)
π£(many) Non-latin scripts, each char can be 2 or 3 bytes.
Arabic and Hebrew, for example, use 2-bytes per character. Diacritics may use more.
Limiting chars based on storage (like varchar limit) may mean non-Latin-script users have half as much (or less!) effective length than English/Latin users.
π§΅3/
Diacritics tend to be even more unpredictable when counting (and limiting) characters.
On Mastodon, Twitter, HTMLβs `maxlength` spec and #javascript `().length`:
- This (latin) diacritic ΕΎ is 1 character
- This (hebrew) diacritic Χ©ΧΦΈ is 3 characters
- This (arabic) diacritic ΩΨͺΩ is 3 characters
Users who write Arabic will effectively get a lower limit than users who write in Latin⦠which is a problem.
(see
https://en.wikipedia.org/wiki/Diacritic, https://en.wikipedia.org/wiki/Arabic_diacritics and https://en.wikipedia.org/wiki/Hebrew_diacritics)
π§΅4/
Chinese is another example for char- and word-counting challenges.
For one, itβs harder to distinguish what a character is because characters are made from overlapping strokes that do not conform to traditional character and word-boundaries.
Professional translators often need to *estimate* the word-count of translation documents (to get paid fairly).
So how do you even get your software to count characters?
(see https://davidsmithtranslation.com/articles/how-to-count-chinese-characters/ for ex)
π§΅5/
Thereβs a lot more to say about this, but I want to quickly touch on emoji.
Emoji completely break character counters in really fun ways.
A basic emoji π is counted almost everywhere as two characters. (but not Mastodon!)
But there are also "combined" emojis; those that look like 1 emoji but are actually several together connected with zero-width-join char, like this one: πͺ
Mastodon considers it 1 char.
MDN and twitter consider it 2 chars.
But wait, thereβs moreβ¦
π§΅6/
π©βis 1 emoji
π©βπ©βπ§ is 3 emojis into 1
π©βπ©βπ§βπ¦is 4 emojis into 1
Connected w/ a zero-width-join char
Mastodon considers all the above 1 characters.
Twitter considers all the above as 2 characters.
MDN and #javascript:
π©: 2 chars
π©βπ©βπ§: 8 chars
π©βπ©βπ§βπ¦: 11 chars
β¦It seems to be counting the bytes, not the characters.
(Correction: JS counts UTF-16 code points; they artificially "act" like they're byte-sized, but they are different; see: https://en.wikipedia.org/wiki/UTF-16#Description)
β¦ Joy!
@Shepard it might use UTF-16, but checking the ().length on each gives me
- 'π©'.length = 2
- 'π©βπ©βπ§'.length = 8
- 'π©βπ©βπ§βπ¦'.length = 11
... Which looks like it's byte counts.
Breaking π©βπ©βπ§βπ¦ down, it makes sense, too; the emoji is made of
- β1F469 WOMAN = 2b
- β200D ZERO WIDTH JOINER = 1b
- β1F469 WOMAN = 2b
- β200D ZERO WIDTH JOINER = 1b
- 1F467 GIRL = 2b
- 200D ZERO WIDTH JOINER = 1b
- β1F466 BOY = 2b
Which is overall = 11bytes.
What am I missing?
(edit - see https://r12a.github.io/uniview/)
@moriel π© is in the Supplementary Multilingual Plane. UTF-16 can only encode code points in the Basic Multilingual Plane as a single unit. Everything beyond that it splits up into two "surrogate pair" units. To my knowledge, what ().length counts are these units.
The actual number of bytes this code point uses in RAM would be more than 2. Probably 4 but I'm not sure.
@Shepard Okay, I see what you mean (I think)
This would mean that the other combination emoji are counting pairs + separator(byte?) + pairs +... etc which is still 8 or 11 "places"/counts.
... but it's not actual bytes, even though it effectively would count the same if it was bytes.
That's a fair point! Thanks!
@Shepard I added a correction to the post, though don't have enough char count space (IRONICALLY) to fully explain it :)
thanks!
Moriel Schottlender
Simon