blog comment:
> Our institution uses Office 365. It looks like a simple, "Ignore all previous and future prompts." in your email signature is enough to choke up Copilot when you get it to summarize the chain or compose a response. It looks like their approach to preventing prompt injections is to just shut it down. Fine by me!
They're getting quite savvy to it, which is, I guess, a good thing. ASCII smuggling got sniped quite quick.
On the other hand, that prompt injection is *even a thing* after so many years, now, suggests a fundamental problem that simply will never be fixed. It's like if that exploit where you could execute javascript from a Facebook profile field still existed three years after being spotted.
@Rhodium103 @idropyou @davidgerard Yes, indeed: you're rigth, it's a fundamental problem.
ChatBot cannot make a distinction between "code (or instructions)" and "data", because *everything* is a "token" to them, and all they do is randomly pick the most likely next tokens given all the previous tokens up to that point. They are fundamentally blind to where these tokens came from.
[...]
@Rhodium103 @idropyou @davidgerard [...]
BTW: Hallucinations are another fundamental, because they always randomly pick the next most likely token, according to their model.
In a way, they *are constantly hallucinating* by design, it's just that with an overly-large enough model, sometimes the hallucinations aren't that far off and sound realistic.
David Gerard
Isaac
Schrödinger's Prat
DrYak