mikece

A rogue AI led to a serious security incident at Meta

https://www.theverge.com/ai-artificial-intelligence/897528/meta-rogue-ai-agent-security-incident

A rogue AI led to a serious security incident at Meta

Last week, an AI agent similar to OpenClaw triggered a high-severity security incident at Meta by independently giving inaccurate technical advice on an employee forum.

The Verge
Mar 19 at 6:59pmWeb
Show threadwelfareMar 19
Behind paywall, is there another link to the article?
Show threadyomismoaquiMar 19
https://archive.is/A2hmz
Show threadkrupanMar 19
I hit back, clicked the link again, and it let me through
Show threadUhhrrrMar 19
The two errors, then, were that the LLM hallucinated something, and that a human trusted the LLM without reasoning about its answer. The fix for this common pattern is to reason about LLM outputs before making use of them.
Show threadkrupanMar 19
It's more like, the LLM "hallucinated" (I hate that term) and automatically posted the information to the forum. It sounds like the human didn't get a chance to reason about it. At least not the original human that asked the LLM for an answer
Show threadpaxysMar 19

A big problem now both internally to a company and externally is that official support channels are being replaced by chatbots, and you really have no option but to trust their output because a human expert is simply no longer available.

If I post a question to the internal payment team's forum about a critical processing issue and some "payments bot" replies to me, should I be at fault for trusting the answer?

Show threadleptonsMar 19
If "the level of awareness that created a problem, cannot be used to fix the problem", then you're asking too much if you expect a human to reason about an LLM output when they are the ones that asked an LLM to do the thinking for them to begin with.
Show threadsomewhereoutthMar 19
However - Automation bias is a common problem (predating AI), the 'human-in-the-loop' ends up implicitly trusting the automated system.
Show threadSlinkyOnStairsMar 19

> The fix for this common pattern is to reason about LLM outputs before making use of them.

That is politics. Not engineering.

Assigning a human to "check the output every time" and blaming them for the faults in the output is just assigning a scapegoat.

If you have to check the AI output every single time, the AI is pointless. You can just check immediately.

Show threadkrupanMar 19

"A human, however, might have done further testing and made a more complete judgment call before sharing the information"

Because a human would have been fired for posting something that incorrect and dangerous

Show threadjasonpeacockMar 19

I'm concerned that someone had the permissions to make such a change without the knowledge of how to make the change.

And there was no test environment to validate the change before it was made.

Multiple process & mechanism failures, regardless of where the bad advice came from.

Show threadFizzadarMar 19
[delayed]