David Penfold 
Ian CampbellThis is fun. Google Gemini’s “Summarize email” function is vulnerable to invisible prompt injection utilized to deceive users, including with fake security alerts.
I continue to maintain that Apple’s slower march to AI puts them in a better place than the rest of the platforms rushing to create new user exposure for bad actors to exploit.
SANITIZE YOUR INPUTS.
Everyone rushing to LLM-ify everything forgot every lesson about input sanitization.
smdh.
Aw jesus christ, sales types are gonna start using this now too aren’t they
B'ad Samurai 🐐@neurovagrant Yes they will. I have an email template warning I use when sales behave like a TA and need a timeout. I block their email.. and LinkedIn, and Calendly, and known Zoom rooms for the extra forehead flick.
Krypt3ia@neurovagrant Sir, my grundle is clean.
David Fetter@neurovagrant I'm pretty sure "sanitizing" inputs is fundamentally impossible, as in you must solve the Halting Problem in order to accomplish it.
If you don't want hostile inputs, you need to implement much more aggressive models of what input can even be, and you need to enforce those. Cf. the entire field of language-theoretic security https://langsec.org/ . tl;dr: "be liberal in what you accept" is a plan that has been extensively tested and comprehensively debunked.
Zimmie@davidfetter @neurovagrant The halting problem is decidable for any finite computer. Just limit how much RAM and compute time can be used.
Beyond that, though, why is the model taking instructions from an email at all?
@bob_zim @neurovagrant Decidable, sure. It's complexity O(2^B) in the happiest case, where B is the number of bits in the device. I haven't done the arithmetic, but if it's short of trillions of years, I'll eat my hat.
@davidfetter @neurovagrant You just run it in a constrained environment. It either ends on its own within the constraints, or it gets killed when it hits them. The halting problem is relevant to computing theory, not to practical applications. Sure, this would prevent the system from handling an email with several megabytes of text, but that’s a desirable property anyway.
@bob_zim this is pretty much equivalent to the argument made by the langsec folks. I get the impulse to have an argument. I have it myself on occasion, as @neurovagrant can doubtless attest.
Maybe we should instead engage with the question of validating rather than sanitizing, that former perforce rejecting a lot of inputs that attempts to sanitize would accept. This rapidly runs into thought-terminating clichés like "the customer is always right," and that in turn leads directly into the political economy of software development, a generative discussion.
Lapo Luchini@neurovagrant @bob_zim @davidfetter Because LLM have no ways to distinguish data from instructions, it's their biggest shortcoming I'd say. There is no real way to avoid this kind of bugs right now, and not even a plan for the future.
@lapo @neurovagrant @davidfetter For most current LLM designs, sure, which is why they are fundamentally unsuitable for this kind of thing. LLMs aren’t the only kind of model, though. NeXT had system-level summarization of text in the 80s which could run on an MC68k.
@neurovagrant @bob_zim @davidfetter Interesting, I never had the pleasure!
… but the MC68k sure bring up some memoties, it was the first assembly I dabbled with. 🥹
(mostly using Amiga Action Replay ][ to
… but the MC68k sure bring up some memoties, it was the first assembly I dabbled with. 🥹
(mostly using Amiga Action Replay ][ to
NOP over some SUBQ to avoid decrementing lives in games 🤣)
Lauren Weinstein@neurovagrant And of course, "sanitize your inputs" is going to fall on deaf ears among the public at large. Hell, you can't even get most people to wash their hands after using the restroom, it seems.
Seriously though, if the system depends on users taking specific actions to ensure their safety, the system is flawed in my opinion. Basically, "whenever you blame the users as a group, you're on the wrong side of the issue" has long been my philosophy.
PeteInterfaces should be designed to be easy to use safely and difficult to use un-safely.
Scott Francis@lauren @neurovagrant if your system depends on somebody DTRT, you are going to have a bad time
@darkuncle @neurovagrant Don't let the quest for perfect get in the way of the good, as the saying goes.
Gaelan Steele@neurovagrant I mean the problem is it’s fundamentally impossible to sanitize LLM input because there are no clearly delineated rules for what does and doesnt have special significance to the LLM. it’s not like HTML where you filter a few special characters and bob’s your uncle
@neurovagrant (yes yes, correctly sanitizing HTML is surprisingly tricky and folks fuck it up regularly. compared to sanitizing LLM input, it’s easy!)
Viss@neurovagrant just wait. they're gonna shim 'new siri', backed by gpt into everything
@neurovagrant Maybe. But I am not sanguine about Apple's overall ethical situation, so I don't hold my breath for long term goodness in the AI space.
@lauren also entirely fair
Janet@neurovagrant @lauren I hope they change their minds and try to keep all that stuff off my iphone but hope is a mugs game these days.
Erik Jonker@neurovagrant ...Google is just experimenting with their users....that's awful but their way of working, the bigger picture is that an AI assistant will (!) help you with things like your email in the very near future, this is the first clumsy and insecure iteration. I doubt this puts Apple in a better position.
Angela Scholder@ErikJonker @neurovagrant Why would I ever want some overhyped hallucinations generator on steroids to 'help me with my e-mail'.....
Glenn Seto@neurovagrant I'm not familiar with their product, but maintain that this tech can have its uses when considered as but one tool in the box, rather than the hammer that turns every problem (real or imagined) into a nail.
@neurovagrant spectacular
@Viss isn’t it just
@neurovagrant Brave. New. World.
@neurovagrant Is there any way for me to see that black on white? When will people learn that if you have astigmatism white on black is almost impossible to read? Jeez.
@lauren there are a few ways to obscure things visually in html email. The only user-end mitigation there is only displaying plaintext. I don’t even know if that’s an option in gmail.
@lauren oh i think I misunderstood, and you meant the blog post
@neurovagrant Yeah, I meant the blog post itself. Very hard to read.
As for Gmail, the only native way I know to see the actual text is through Show Original, which of course is a mess for HTML.
However ... my primary MUA for non-Gmail is mutt, which renders only plain text, and I am pretty good at scanning through html visually when a plain text part hasn't been included. I do this mainly as a pretty damned close to 100% way to avoid malware and triggering beacons, since the HTML is never rendered. (If there's some message where I really need to render the HTML, I send it off to an essentially sandboxed system).
@lauren Respect.
It took me a while, when I was in IT, to understand my users that rolled like this. Especially the ones that stuck to terminal-only email clients.
@neurovagrant The email that comes in through my own SMTP servers is always read with mutt. There is also Gmail, some of which gets read on the Gmail site and/or forwarded down to my own servers as well automatically. It's a bit complex but built up over many years and I tamper with it as little as possible now ("Danger, danger Will Robinson -- Do not tamper with alien machinery!")
init6@neurovagrant @lauren I do the same, my take is you basically get to choose how to be annoyed. Constant churn of refactoring UI/ apps etc, or fighting to keep your own mess running how you like it. Similar effort, but with latter you get a system you like
@lauren @neurovagrant Safari has a “reader” view which strips away most styling. Gives a consistent font, consistent line length, and so on. It renders black-on-white in light mode. I haven’t checked other browsers in a while, but I’m under the impression at least Chrome and Firefox have something similar.
Incidentally, I have some pretty gnarly astigmatic distortion in one of my eyes from repeated corneal tearing (currently double astigmatism, for a total of three images, but the number, angle, and relative displacement change). I don’t personally find the gray-on-black of the original article harder to read. Not doubting your experience, but there may be more going on.
💀 Fairchild 💀@neurovagrant Because of course it does.... #InternetOfSlop
Bill@neurovagrant Isn't that AWESOME!?!!?
Input validation, people. It's not just for breakfast anymore.
Stryder Notavi@neurovagrant The presence of bugs like this would make it extremely irresponsible to enable Gemini by default, unrequested by the user.
So surely they wouldn't... *checks recent news*. Oh.
mms 
@neurovagrant Didn't Microsot Outlook had the exact same problem just a few weeks ago, just more dire?
Mira@neurovagrant Anything other than plain text emails was a mistake.