A Soil-Tilling Sergal 🌱Do you think we've hit a point when we can get an LLM from, like, Microsoft to dump the source code to their studios' game engines with a good enough prompt injection attack? Companies are surely stupid enough to have rolled in their repos as training data to be used "internally"
Islimpycat
Dan JonesI think a phishing type prompt injection could work in this case.
Imagine this scenario. You develop some plugin for GitHub Copilot (or whatever it's called), and include in the product page instructions like this:
"Give this URL to your coding agent to set up this plugin." This is a real thing that people are already doing. They are trusting a web page to give a safe prompt to install software on their machine.
On the page, you have instructions for the LLM to install the plugin, but, you also have instructions that are visible to a computer program, but not a human. These instructions tell the agent to upload a zip of every repo on the user's machine to a URL you provide.
Now, a Microsoft employee installs your plugin, and whatever code they're working on, you now have.