jcoglanif I have to hear the term "prompt injection" one more time
x0you cannot have an "injection attack" in a system with no formal distinction between data and instructions. what you actually have is an "everything is instructions" model and a failure to isolate untrusted inputs from the elevated privilege of access to private information
this is not a novel or surprising means of attacking systems. of course, obviously, if you give a system with no formalisable behaviour, that may execute anything as an instruction by design, elevated privileges and untrusted input, this will happen
the category error here is deploying something with unformalisable behaviour with any privilege to do anything at all without the express confirmation of a trusted human operator
the way people are deploying LLMs is driving a freight train through the principle of least privilege and being surprised at the results
like it's not even the sort of security flaw you get from people not realising non-obvious properties of their platform and failing to guard against them, it's just ignoring foundational concepts in security engineering on purpose
Glyph@jcoglan I think the original insight was good but the term has begun to chafe and your explanation is absolutely on point as to why. We need a punchier term that encapsulates this understanding but it’s hard to describe as an “exploit” or a “vulnerability class” when it’s just a total, comprehensive, catastrophic misunderstanding of the entire concept of security. With MCP shit rolling out at a breakneck pace we are truly entering the time of monsters
Soatok Dreamseeker@jcoglan When I first met @suhacker years ago, I actually asked about this sort of thing (using "prepared statements" from SQL as an analogy), but she patiently explained how much worse the whole ecosystem is than I was imagining with my (admittedly naive) question.
The amount of Pickle exploits is too damn high
@jcoglan Yeah, that was my intuition.
If you haven't seen Suha's excellent DEFCON talk, it's a good watch: https://www.youtube.com/watch?v=Z38pTFM0FyU
DEF CON 32 - Incubated ML Exploits: Backdooring ML Pipelines w Input Handling Bugs - Suha Hussain
Machine learning (ML) pipelines are vulnerable to model backdoors that compromise the integrity of the underlying system. Although many backdoor attacks limi...
@soatok hold on, people are using python's pickle to distribute ML models? I need a lie down
SnoopJ@soatok @jcoglan in my experience a lot of it comes from a YAGNI attitude around caring about security or robustness.
At least half the ecosystem is grad students or other researchers slapping things together to get a paper out the door, and that stuff is later maybe absorbed into something calling itself a library. And I don't mean to criticize those people because those things *aren't* usually concerns for them.
But blurring the lines between the levels of seriousness between "we need to publish this paper" and "this code trains a model that decides if you get health insurance" is… not great.
@soatok @jcoglan oh btw did I mention that the pickles in question are usually downloaded from the web without host/content verification?
imagine there will be some attacks exploiting that when the domains start to evaporate once the VC flood recedes. haven't heard of any to date, but that's just a ticking clock
@SnoopJ @soatok @jcoglan more horrifying details and a chilling exploration of the relevant industry-wide context collapse if you are curious https://blog.nelhage.com/post/pickles-and-ml/
What's with ML software and pickles?
I have spent many years as an software engineer who was a total outsider to machine-learning, but with some curiosity and occasional peripheral interactions with it. During this time, a recurring theme for me was horror (and, to be honest, disdain) every time I encountered the widespread usage of Python pickle in the Python ML ecosystem. In addition to their major security issues1, the use of pickle for serialization tends to be very brittle, leading to all kinds of nightmares as you evolve your code and upgrade libraries and Python versions.
🏳️⚧️🏳️⚧️🏳️⚧️*looks at you 俺ly*🏳️⚧️🏳️⚧️🏳️⚧️
Suha
Jay Stephens
David Chisnall (*Now with 50% more sarcasm!*)@jcoglan I am too young to remember the phone phreaking attacks that made everyone learn that in-band signalling is a bad idea. I am old enough to remember the ping-of-death attacks that reminded people. LLMs will remind an entire new generation.
Koen Hufkens, PhD@jcoglan Oh, that's well put.
AMS