jcoglanMay 27
if I have to hear the term "prompt injection" one more time
Show threadjcoglanMay 27
you cannot have an "injection attack" in a system with no formal distinction between data and instructions. what you actually have is an "everything is instructions" model and a failure to isolate untrusted inputs from the elevated privilege of access to private information
Show threadSoatok Dreamseeker

@jcoglan When I first met @suhacker years ago, I actually asked about this sort of thing (using "prepared statements" from SQL as an analogy), but she patiently explained how much worse the whole ecosystem is than I was imagining with my (admittedly naive) question.

The amount of Pickle exploits is too damn high

May 27 at 10:16amWeb
Show threadjcoglanMay 27
@soatok @suhacker right, whereas LLMs are analogous to having your API be that user agents just send SQL to the server; end users have whatever privileges the entire server has
Show threadSoatok DreamseekerMay 27

@jcoglan Yeah, that was my intuition.

If you haven't seen Suha's excellent DEFCON talk, it's a good watch: https://www.youtube.com/watch?v=Z38pTFM0FyU

DEF CON 32 - Incubated ML Exploits: Backdooring ML Pipelines w Input Handling Bugs - Suha Hussain

Machine learning (ML) pipelines are vulnerable to model backdoors that compromise the integrity of the underlying system. Although many backdoor attacks limi...

YouTube
Show threadjcoglanMay 27
@soatok hold on, people are using python's pickle to distribute ML models? I need a lie down
Show threadSnoopJMay 27
@jcoglan @soatok `pickle` is an alarmingly load-bearing part of the ecosystem, and some of the authors of this nightmare have the gall to call it engineering 🤢
Show threadSoatok DreamseekerMay 27
@SnoopJ @jcoglan Nobody has made a GIF of "Ozzie's in a pickle" from Chrono Trigger, apparently
Show threadjcoglanMay 27
@SnoopJ @soatok the standard advice for pickle, and ruby's marshal, and various yaml flavours, is they are categorically not suitable for sending anything across a trust boundary. what are we doing here
Show threadSoatok DreamseekerMay 27
@jcoglan @SnoopJ I believe they call it "vibe coding"
Show threadSnoopJMay 27

@soatok @jcoglan in my experience a lot of it comes from a YAGNI attitude around caring about security or robustness.

At least half the ecosystem is grad students or other researchers slapping things together to get a paper out the door, and that stuff is later maybe absorbed into something calling itself a library. And I don't mean to criticize those people because those things *aren't* usually concerns for them.

But blurring the lines between the levels of seriousness between "we need to publish this paper" and "this code trains a model that decides if you get health insurance" is… not great.

Show threadSnoopJMay 28

@soatok @jcoglan oh btw did I mention that the pickles in question are usually downloaded from the web without host/content verification?

imagine there will be some attacks exploiting that when the domains start to evaporate once the VC flood recedes. haven't heard of any to date, but that's just a ticking clock

Show threadGlyphMay 28
@SnoopJ @soatok @jcoglan more horrifying details and a chilling exploration of the relevant industry-wide context collapse if you are curious https://blog.nelhage.com/post/pickles-and-ml/
What's with ML software and pickles?

I have spent many years as an software engineer who was a total outsider to machine-learning, but with some curiosity and occasional peripheral interactions with it. During this time, a recurring theme for me was horror (and, to be honest, disdain) every time I encountered the widespread usage of Python pickle in the Python ML ecosystem. In addition to their major security issues1, the use of pickle for serialization tends to be very brittle, leading to all kinds of nightmares as you evolve your code and upgrade libraries and Python versions.

Made of Bugs
Show threadKiraGPT✨🏳️‍⚧️May 27
@jcoglan @SnoopJ @soatok "big red security warning at the top of the first page of the docs" vs "it's probably nothing"
Show threadSuhaMay 30
@soatok thanks!!
Show threadJay StephensMay 27
@jcoglan @soatok @suhacker
Thank you! Phrasing it this way finally made it click for me.
Show threadGlyphMay 28
@jaystephens @jcoglan @soatok @suhacker even this is unfair to SQL. SQL servers at least have role-based access control.