Mastodawn

Eugen Rochko Apr 18, 2017

Just gonna throw this out here - any Ruby gem for relatively quick & accurate language detection that you know of?

Eugen Rochko Apr 18, 2017

I didn't expect anyone to think I am asking for a General AI.

I just want semi-accurate heuristic to spare users having to tag language manually.

Proprietary APIs like Google Translate are out of the question. Already using WhatLanguage, it's not working well. If I can't find anything, welp, there's that. Not a big deal.

Show thread

葉月 Apr 18, 2017

@Gargron Wait so Mastodon is guessing our languages by random? That creates all sorts of problem for us CJK users. (See Han Unification) We should at least have an option to manually pick the correct language for the text.

Show thread

Eugen Rochko Apr 18, 2017

@hatsuki It's guessing, but there are no filters yet.

Show thread

葉月

@Gargron Oops, now I see that none of the text has lang attribute. This is a bug, CJK languages will not display correctly without lang attributes, due to Unicode Han Unification. I know this will clutter UI but I strongly recommend looking into this before Mastodon gets critical mass among Chinese-speaking communities, that will be a very hard problem to solve...