endrift 🏳️⚧️LLM translation models are going great
I think people really underestimate how fragile LLMs are for auto-translation. You can put complete garbage into it where none of the words are real words and still get out plausible-sounding "translations" just because the LLM sees it as "close" to a real sentence, and then translates what it thinks that "close" sentence is based, once again, on what seems "close".
The whole benchmarking approach really does not help with this since benchmarks rarely include testing for failures. You need to test that garbage-in is recognized as garbage, otherwise you get garbage-out too.
David Gerard@endrift this is the problem with LLMs for transcription too. They do ok, but they MAKE SHIT UP. A more specific transformer-based model does a great job! But the chatbot is more convenient.
I'm disconcerted they're putting the LLM "I know what you mean" thing into Google Translate, the original showcase for transformers. I mean, it's obvious they think that's helpful. But still, urgh.
David Chisnall (*Now with 50% more sarcasm!*)My favourite example was actually you! A post containing pivot-to-ai.com was translated (I can't remember the source language) and it decided to replace the domain name with 'pineapple.com'.
It wasn't allowed to touch the HTML, so this ended up with a link that showed 'pineapple.com' but went to 'pivot-to-ai.com'.
I strongly suspect that there are some neat ways of sneaking malicious links past existing email filters that rely on this.
@david_chisnall @endrift LLMs really are the sharp end of "convenience is king"
🆘Bill Cole 🇺🇦@david_chisnall @davidgerard @endrift No serious spam filter doesn’t already treat that sort of thing as suspect. Spammers have been trying it for >25 years. There’s a constant stream of new spammers trying it because they lack anything like technical lore. Each new idiot reinvents his own set of the same old stupid tricks.
@grumpybozo @davidgerard @endrift
Current spam filters notice when the link target doesn’t match the text. The attack I’m proposing is where they do match as they go through the filter, but then local translation makes the target the victim sees appear innocuous.