I was excited for the recent advancements in AI, but seems the area has hit another wall. Seems it is best to be used for automating very simple tasks, or at best used as a guiding tool for professionals (ie, medicine, SWE, …)

Hallucinations is common for humans as well. It’s just people who believe they know stuff they really don’t know.

We have alternative safeguards in place. It’s true however that current llm generation has its limitations

Sure, but these things exists as fancy story tellers. They understand language patterns well enough to write convincing language, but they don’t understand what they’re saying at all.

The metaphorical human equivalent would be having someone write a song in a foreign language they barely understand. You can get something that sure sounds convincing, sounds good even, but to someone who actually speaks Spanish it’s nonsense.

Calculators don’t understand maths, but they are good at it.

LLMs speak many languages correctly, they don’t know the referents, the don’t understand concepts, but they know how to correctly associate them.

What they write can be wrong sometimes, but it absolutely makes sense most of the time.

but it absolutely makes sense most of the time

I’d contest that, that shouldn’t be taken for granted. I’ve tried several questions in these things, and rarely do I find an answer entirely satisfactory (though it normally sounds convincing/is grammatically correct).