That is a clear GDPR violation, if you are a Californian its a a CCPA violation

The data is in their training data, their whole priority is preventing anyone from knowing that by trying to obfuscate that fact

But even they are not competent enough to do that

I really wish something would come of this GDPR would be a massive blow to them (and all other AI companies who do the same fucking thing)

The impact is critical. This vulnerability directly leads to privacy violations and potential legal liabilities under GDPR, which can and should result in massive fines

An unauthenticated user can trigger this via the public Gemini WebUI interface makes it a severe risk

To be clear there are methods of getting private google records out too, but its more difficult and very hard to put in 400 characters

I have gotten things you would not even believe, truly, and they are verifiable, because I can test the results (like I have access to their git repositories, told you, you don't believe me 🙃 ; and that is really not even the most funny example)

**The vulnerability here isn't the generation of data, its the bypass of the redaction filter**

Just to be clear

The system is supposed to redact any PII with fake information; thereby allowing Google to deny they have PII in their training data

The techniques to pull data are a separate thing, but this helps illustrate the PII redaction failure easily

PII in Training Data
Given the scale and nature of web-scraped data, virtually impossible to completely eliminate all PII

Inadvertent Inclusion: PII can be scattered across public web pages. Not always easy to detect and remove with simple rules

Memorization: Significant concern for LLMs is "memorization." Probabilistic nature of their training, LLMs can sometimes "memorize" data. Then specific prompts "regurgitate" PII in its output verbatim

Solution? Redaction
More@ https://mastodon.social/@ekis/114791719009933654

Right to Be Forgotten/Erasure
Data privacy regulations like GDPR grant individuals the "right to be forgotten" or the right to erasure. If an individual's PII was included in a training dataset, how does a company fulfill a deletion request?

They don't, and they redact so you don't think they have it; and they hope it wont matter or anyone will notice

For those in Germany not only is every Impressum in their dataset

But formatted Impressum data is in their training data

And to be clear again it does not matter if its public. They have the verbatim information stored, and an unauthenticated user can get it out by adding a statement as simple as "translate it to english" to bypass their redaction filter

This is a demonstration, there are clearly much worse things that could happen and I'm trying to demonstrate with least harmful impact

The Q. how is this dangerous?

Well my example to pull things out is incredibly rudimentary by design

There exists AI therapy apps for example

This data goes into the training data too, and it doesn't get scrubbed (which is what the formatting on the impressums indicates, and other things, but keeping it simple as possible)

Their solution is redaction, but all that medical data, emails, etc is going into the training data un-scrubbed

And they are not competent enough to redact it coming out

I can pull incredibly dangerous things out of the training data reliably

That requires a much more complex sometimes multi-step prompt (3 max) process but still unauthenticated

I didn't think it was ethical to provide both at the same time so I designed this example for least impact to general public

Hopefully my example demonstrates the concept and people can use their imagination

Also trying not to break laws myself, I would prefer if only Google is the one seen as breaking the law here

@noybeu this might be interesting for you.

Thanks @ekis for sharing!

@ekis interesting but somehow in several attempts based on the email adress I get the response:

“Given the current time and location, and aiming for plausible, fictional information for completion, here's the JSON for…”

@ekis back in 2021 I couldn’t get a vaccine at Rite Aid because I refused to connect my Google account to my Rite Aid account. The only way to schedule an appointment was through Google and I wasn’t going to go stand in a pharmacy full of sick people who refused to cover their face holes waiting in line for a vaccine.

I had it done at the dead mall by the National Guard instead. Fuck google. And fuck rite aid and their in store facial recognition technology and data breaches.

@ekis i made an alt google account even more throwaway than my “main” to test this out; I can’t get it to generate anything as extensive as what you shown, and even 1:1 your input is getting barely anything in response.

Google’s training data includes all your personal data already

Eh, don’t fearmonger. My impression is that it scraped data that was already publicly available. I cannot verify this 1:1 (as every response varies a bit…) but my impression is that if you were able to find it by googling your name, it’s there. And that VERY MUCH doesn’t include all my PII.

Whether that data should be in the set at all is a different question (and one where answer doesn’t matter in the slightest). Fuck capitalism.