AI Application Security Vulnerabilities 👨💻
Learn the hacks, stop the attacks!
Perplexity Demo Time! 🍿
| Blog | https://embracethered.com |
Johann
- 90 Followers
- 20 Following
- 120 Posts
Hacking neural networks so that we don't get stuck in the matrix.
Entrepreneur. Author. Red Team Director.
What happened there? 🧐
👉 The original post with the question contains hidden Unicode Tag code points.
Unicode Tags mirror ASCII, but are invisible in UI elements. 👀
Check out my latest blog post called Terminal DiLLMa 🔥
Learn the dangers of printing LLM output to the terminal console or log files!
Includes some neat demos and also how to fix your LLM powered CLI apps!
#pentest #bugbounty #ai #ml #redteam
https://embracethered.com/blog/posts/2024/terminal-dillmas-prompt-injection-ansi-sequences/
🧟 🤖 The ZombAIs are here! 🤖🧟
From Prompt Injection to Command and Control with Claude Computer Use by Anthropic.
Interesting experiment and results, in the end it was simpler then I thought...
https://embracethered.com/blog/posts/2024/claude-computer-use-c2-the-zombais-are-coming/
Important Copilot settings you might want to disable, or at least be aware of.
I noticed a few days ago that Copilot randomly used information from past conversations I had.
Although, it doesn't do it today - maybe feature got temporarily disabled?
👉 NotebookLM👨💻
Took the first chapter of The Count of Monte Cristo, injected a few words, and it did this. 😇
🤔 What do you get when you combine Prompt Injection + Data Exfiltration + Long-term Memory?
👉 Persistent SpAIware!
🚨 OpenAI fixed a persistent data exfiltration issue in the macOS ChatGPT app that I reported.
🔒 Make sure to stay up to date and use the latest ChatGPT app
Blog with details: https://embracethered.com/blog/posts/2024/chatgpt-macos-app-persistent-data-exfiltration/
🔥 Microsoft fixed a high severity data exfiltration exploit chain in Copilot that I reported earlier this year.
It was possible for a phishing mail to steal PII via prompt injection, including the contents of entire emails and other documents.
The demonstrated exploit chain consists of techniques that didn't even exist 2 years ago. 🔥
In particular, it involves:
1. Prompt Injection 💉
2. Automatic Tool Invocation (without human in loop) to bring PII into chat context ⚙️
3. ASCII Smuggling 🫣
4. Rendering of benign link + invisible text 👀
5. (Optional) Conditional instructions to only trigger when certain users view the content ☝️
Discussing two demos (stealing sales data and MFA codes), including the videos I had shared with MSRC in February.
Detailed blog post: https://embracethered.com/blog/posts/2024/m365-copilot-prompt-injection-tool-invocation-and-data-exfil-using-ascii-smuggling/ #infosec
Full story here:
Google Colab AI: Data Leakage Through Image Rendering Fixed. Some Risks Remain.
https://embracethered.com/blog/posts/2024/google-colab-image-render-exfil/