Wendy Nather

Okay, could someone explain something to me please?

Why did ANYONE ever think “guardrails” would work?

We all know that blocklisting is suboptimal because you can’t possibly enumerate all the badness (see also: antivirus). And anyone who has had to write a statement of work that includes application security requirements knows how impossible THAT is without adding a whole textbook as an appendix. (Or just writing “Don’t do stupid shit with the code,” which covers it pretty broadly.)

Don’t do that. Or that. Or that, either. And not like that. Oh, we didn’t know you could do that! Don’t do that.

Seriously, why??

Jun 15 at 7:41pmWeb
Show threadEddie.Jun 15

@wendynather on this line, one would think the code of conduct for any conference could be summed up with “Be Professional.”

It isn’t for the same reason we have to be warned coffee is served hot and we shouldn’t use chainsaws on our genitals.

Show threadWendy NatherJun 15
@infoseclogger /me quietly puts down the chainsaw
Show threadgaryJun 15
@wendynather @infoseclogger conferences are not gta in real life - where is the line between reg mids professionalism and social engineering./ great places to network anyways #comp intel #recruiting #civil and persuasive #poly sci
Show threadeestileib (she/hers)Jun 16

@infoseclogger @wendynather

Ok I agree we shouldn't use the CHAIN part of the chainsaw on genitals, but I've seen more than one strapped to the front of a jumpsuit.

Maybe that's also not appropriate for professional conferences, but I'll think that's up to individual codes of conduct.

Show threadmhoyeJun 15
@wendynather I guess because "we have no idea how this works much less how to make it safe to operate" really screws up your whole financial prospectus or IPO.
Show threadChris PetrilliJun 15
@mhoye @wendynather my number one take away is that these buffoons really believe their own marketing, which prevents them from engaging with reality as it actually is, rather than the voluminous puffery they offer.
Show threadWouter HindriksJun 15
@wendynather if only the "AI" was intelligent and could understand if it someone was abusing it.
Show threadjmjJun 16

@Sikorsky78 @wendynather why? Are people any better at that? Customer service teams get training on how to detect account takeover attacks.

This effect of AI/LLM failure modes being eerily similar the human ones is the thing that makes me think something really interesting is going on. Those vectors deep in the llm are more than just words.

Show threadTindraJun 15
@wendynather similarly to watching the blockchain industry discover *why* various financial rules are in place (without actually adopting them… just getting their money stolen then wining), it’s been interesting watching people discover the difference between policy controls and technical controls (again, without actually adopting basic identity controls, etc)
Show threadMatthew ExonJun 15
@wendynather
Show threaddr_aJun 15
@wendynather I don't think anyone who actually works with LLMs believes guardrails work. Instead there is a lot of motivated reasoning around how they could theoretically work, but at the end of the day gotta make the investors happy so ship it.
Show thread🆘Bill Cole 🇺🇦Jun 15

@wendynather They think “guardrails” will work because they think they are dealing with either a normal computer program (deterministic) or a sentient being (who can understand what they mean.)

As a human emulation tech, LLMs are demonstrably good enough to trick some humans, but they’ve never emulated actually understanding a request.

Show threadjpl99Jun 16

@wendynather

A lot of people *choose* to believe this because it's convenient. And perhaps they don't have the tech skills to see the lie.

And big tech pushes guardrails hard; because they have to hide the flaws of the technology. While they must maintain the illusion of heralding the new industrial age and their own magnificence, and after convincing so much money to believe their story, they can't admit that the fundamentals of the tech has serious flaws, or that they currently lack the tools and knowledge how to fix it.