mikeceMar 19

A rogue AI led to a serious security incident at Meta

https://www.theverge.com/ai-artificial-intelligence/897528/meta-rogue-ai-agent-security-incident

A rogue AI led to a serious security incident at Meta

Last week, an AI agent similar to OpenClaw triggered a high-severity security incident at Meta by independently giving inaccurate technical advice on an employee forum.

The Verge
Show threadUhhrrr
The two errors, then, were that the LLM hallucinated something, and that a human trusted the LLM without reasoning about its answer. The fix for this common pattern is to reason about LLM outputs before making use of them.
Mar 19 at 7:17pmWeb
Show threadkrupanMar 19
It's more like, the LLM "hallucinated" (I hate that term) and automatically posted the information to the forum. It sounds like the human didn't get a chance to reason about it. At least not the original human that asked the LLM for an answer
Show threadpaxysMar 19

A big problem now both internally to a company and externally is that official support channels are being replaced by chatbots, and you really have no option but to trust their output because a human expert is simply no longer available.

If I post a question to the internal payment team's forum about a critical processing issue and some "payments bot" replies to me, should I be at fault for trusting the answer?

Show threadleptonsMar 19
If "the level of awareness that created a problem, cannot be used to fix the problem", then you're asking too much if you expect a human to reason about an LLM output when they are the ones that asked an LLM to do the thinking for them to begin with.
Show threadsomewhereoutthMar 19
However - Automation bias is a common problem (predating AI), the 'human-in-the-loop' ends up implicitly trusting the automated system.
Show threadSlinkyOnStairsMar 19

> The fix for this common pattern is to reason about LLM outputs before making use of them.

That is politics. Not engineering.

Assigning a human to "check the output every time" and blaming them for the faults in the output is just assigning a scapegoat.

If you have to check the AI output every single time, the AI is pointless. You can just check immediately.