fun trick: if someone gives a hosted LLM a skill that lets it fetch web pages (directly, not through some third party scraper service) and it's hosted on AWS, you can often trick it into fetching data from the AWS instance metadata server (IMDS) at 169.254.169.254 / [fd00:ec2::254]. the higher end models tend to refuse if you give the IP, but you can just spin up a domain with A/AAAA records pointing at that IP and request that instead. if IMDSv1 isn't disabled you can get secrets out of it.
@gsuberland Where does "I politely asked an LLM to pwn its environment and it kindly obliged" sit with the computer misuse act?