Mastodawn

Sarah Russell 1d ago

Marcus Hutchins

Had a very surprising ChatGPT experience: asked it to generate a quick summary of the WannaCry ransomware, and instead of referencing the person who stopped it by name, it simply put "(you)". When I asked it how it was able to identify that it was me, it citied its own message as something I'd said.

After pointing out I didn't say that, it did, ChatGPT replied that it was able to infer it by my account username and what it'd learned from my skillset across various chats. Not 100% sure if that's how it actually did it. Either way, pretty cool, but also a little bit scary.

It's pretty widely known that many tech companies, especially advertising ones build comprehensive profiles on their users, but it's rare that you get to talk to said profile and figure out what it knows about you.

Matthew Loxton 1d ago

@malwaretech
99.99% chance it was just bullshitting
But that 0.01% chance that it really did identify you is yikes

Ben Aveling 1d ago

It clearly did identify him. We have that in the first screen shot.
The 'how' that it presented is a post-hoc construction that may or may not accurately describe the actual process.
If anyone wants to rename themselves malwaretech and run the same query, I'd be curious to know what happens. 🙂
@mloxton @malwaretech

Matthew Loxton 1d ago

@BenAveling
The first screenshot wasn't the first interaction, so it really isn't known if somewhere earlier there was a connection.
There would also be no way to distinguish between it just randomly attributing something to you, and it doing so because of a clue or deduction
Asking it to explain how it figured something out is a fool's errand - it doesn't know and couldn't tell you, and it is just an invitation for it to make up stuff.

Dave 🐶1d ago

@malwaretech Maybe it just says everyone stopped WannaCry, hoping to one day catch you with this exact interaction?

@malwaretech That's the memory feature. It effectively builds a dossier on you. Probably wasn't that hard to identify you.
@simon wrote about it here:
https://simonwillison.net/2025/May/21/chatgpt-new-memory/

I really don’t like ChatGPT’s new memory dossier

Last month ChatGPT got a major upgrade. As far as I can tell the closest to an official announcement was this tweet from @OpenAI: Starting today [April 10th 2025], memory …

Simon Willison’s Weblog

Marcus Hutchins

@deraffe @simon The memory feature long predates that release. It just wasn't publicly acknowledge.

@malwaretech True, memory as such isn't new, but it gained new behavior under that same name.
@simon

@malwaretech
Chatgpt for sure builds a profile across multiple chats. If you start a new chat and ask about what it knows about you, you can see the lower bound of how much it knows. However, from the first answer it seems that it did not know what it was doing. Otherwise it would have said something about your username etc.

@malwaretech Just... feels too much like magic

Court Cantrell prefers not to 22h ago

@dawisco @malwaretech It's truly nothing more than a computer program running a word association game.

https://www.nytimes.com/2025/06/13/technology/chatgpt-ai-chatbots-conspiracies.html?unlocked_article_code=1.PU8.kTZ-.aM-1gXNmLx4n&smid=url-share

They Asked ChatGPT Questions. The Answers Sent Them Spiraling.

Generative A.I. chatbots are going down conspiratorial rabbit holes and endorsing wild, mystical belief systems. For some people, conversations with the technology can deeply distort reality.

The New York Times

TripTilt /// tt 1d ago

@malwaretech
interesting. the question is, if I were to make a Marcus Hutchins profile and use the bot and ask similar questions, would it do the same. no way the machine "identified" you ;)

xXx_K1R4_xXx t3h FoXXXDerG oF d00m🏳️‍⚧️1d ago

@malwaretech i wonder if it does this for any user where it can match their name and behavior to a publicly searchable person. like if i made a steve jobs acct would it answer questions about the development of the ipod in the first person

@BorrisInABox @malwaretech Thats scary to think about. We get told what data companies collect in the privacy policies, but i know all most 99% of people don't read that. It's only when we hear about it in the news, and read studies, do we learn how much data is collected.

@malwaretech that is creepy AF!

ShadSterling 1d ago

@malwaretech LLMs are structurally incapable of identification or inference or consulting a profile; they generate text that relates to the prompt and the model (and indirectly the training set) in a way that satisfies some statistical constraints. AIUI they start with additional information hidden from the user that effectively precedes the initial prompt, which might include their profile of you; its output resembles what a followup might be in its training set, which tracks

Old Man in the Shoe 1d ago

@ShadSterling the programs that run the LLM consult profiles, it's called RAG, it's well known - not sure why it doesn't exist now

@malwaretech @ShadSterling
⬆️ This is the correct answer.

AmbianceAsunder 1d ago

@malwaretech https://i.kym-cdn.com/entries/icons/facebook/000/040/776/asm.jpg

Shannon Clark 1d ago

@malwaretech what’s more worrying is how many people will such systems quietly make errors in such profiles about and then give carefully crafted (for the error profile) answers. For example - if someone shares a handle that other people the system knows about use (I get emails for a dozen or more otber Shannon Clark’s regularly - one or two most days. Many have public profiles and web presences.

Many are women (I’m a cis man) even before LLM’s systems have assumed I was a female from my name

@malwaretech One thing worth comparing is asking the bot about weather. Some of them have the location for the weather based on geoip, some have it from an associated profile. All of them seem to deny that they know your location.

joy larkin 🌺✨1d ago

For the thread, current ChatGPT Settings > Personalization options.

Dan 🌻23h ago

@malwaretech And instead if you said, "No, that wasn't me" the reply would have been "Oops, sorry, my mistake! It was actually Bob Bobbins!". It's not reasoning, it's outputting statistically reasonable English text.

Marcus Hutchins

@danvolchek And it's statistically reasonable that giving all the information it has, it could easily deduce it was me

Dan 🌻19h ago

@malwaretech Except that there is no deduction, because it doesn't think :)

@malwaretech Did you consider how much energy and water did you waste during that exchange?
Do you even care?

Marcus Hutchins

@Johns_priv less than zero care. It's not my job to personally compensate from someone else's shitty water and power infrastructure.

@malwaretech So that's how it works, you lie to yourself so you can use highly destructive technologies that fuck up our planet.

But is not your job to care for it? I guess you don't have kids or nephews and don't give two fucks about the world you leave the future generations.

And probably out of spite you'll use it even more.
Disgusting.

Don't bother to reply.

SpaceLifeForm 22h ago

Cookies. They are everywhere!

@malwaretech ask it what ads you like to see.

Adrian Sanabria 18h ago

@malwaretech I wouldn’t read into it too much, unless it consistently shows that it has and is using profile information on you, and connecting to other grounding data

I’ve had similar experiences, and then solved the mystery later on.

OR, Miessler is now running your ChatGPT account and is messing with you 😉

@malwaretech what if you ask it about another malware that has similarly been stopped by a domain-name killswitch ? Would it also give you credit ?

it seems to me like it associates you not by your name directly, but because there is "Malware" in your name

abadidea 12h ago

@malwaretech @glyph unfortunately, chain-of-reasoning is just as much hallucination as everything else, so asking how it knows something only yields answers that *could* plausibly be how one knows something. However, it is known that the major chatbot systems receive information about the current user in the background, which could include not just what you’ve deliberately provided (such as your username) but also the profile of you built up by ad networks.

@malwaretech
@jasonkoebler you and the @404mediaco crew might find this thread interesting as a data point for some of your LLM coverage.