jimkats

@jimkats@infosec.exchange
21 Followers
41 Following
290 Posts

Computer & Informatics Engineering graduate. I don't know what I'm doing.

Interests/hobbies: OpenStreetMap, Wikidata, Wikipedia, Eurovision Song Contest, video games.

LocationGreece 🇬🇷
OpenStreetMaphttps://www.openstreetmap.org/user/jimkats

I am begging writers & writing-related-workers of all kinds—but especially people working in journalism and blogs—to stop using “AI art” to accompany your writing.

Please have solidarity with your fellow creative workers 💜

In this week's Behind the Blog we gave some insight into our headline process. The example is a very complicated AI court ruling story. Not sure we've ever spent longer on a headline https://www.404media.co/behind-the-blog-chatbot-addiction-and-a-reading-list/
I was interviewed by the National Cryptologic Foundation on forensics in critical infrastructure (and a lot of other stuff!) and it was a very fun chat. They have an interesting assortment of interviews on their podcast and I really recommend you check them out: https://open.spotify.com/episode/5kzzFk7xLyPQnebu1CdTR7

Just so you know, #OsmAnd is celebrating its 15th anniversary with 50% off its plans, which you can now pay for directly without paying the 'app tax' to Google or Apple.

I highly recommend OsmAnd — it's a Swiss army knife for maps! Open source. Based on #OpenStreetMap.

https://www.osmand.net/blog/15-years

#OSM #map #maps

Had a very surprising ChatGPT experience: asked it to generate a quick summary of the WannaCry ransomware, and instead of referencing the person who stopped it by name, it simply put "(you)". When I asked it how it was able to identify that it was me, it citied its own message as something I'd said.

After pointing out I didn't say that, it did, ChatGPT replied that it was able to infer it by my account username and what it'd learned from my skillset across various chats. Not 100% sure if that's how it actually did it. Either way, pretty cool, but also a little bit scary.

It's pretty widely known that many tech companies, especially advertising ones build comprehensive profiles on their users, but it's rare that you get to talk to said profile and figure out what it knows about you.

OSM planet replication diffs are now being published again.
It is expected OpenStreetMap.org will be offline briefly for maintenance later. We have to stop some systems briefly to get the OpenStreetMap planet diffs (minutely / hourly / daily) back into sync with the database. The maintenance window will be announced as soon as possible.

Sources:

1) Federal judge sides with Meta in lawsuit over training AI models on copyrighted books https://techcrunch.com/2025/06/25/federal-judge-sides-with-meta-in-lawsuit-over-training-ai-models-on-copyrighted-books/

2) A federal judge sides with Anthropic in lawsuit over training AI on books without authors’ permission https://techcrunch.com/2025/06/24/a-federal-judge-sides-with-anthropic-in-lawsuit-over-training-ai-on-books-without-authors-permission/

The following cases are still pending:
a) The New York Times is suing OpenAI and Microsoft for training AI models on news articles

b) Disney and Universal are suing Midjourney for training AI models on films and TV shows.

Federal judge sides with Meta in lawsuit over training AI models on copyrighted books | TechCrunch

A federal judge sided with Meta in a lawsuit that alleged the company had illegally trained its AI models on copyrighted works.

TechCrunch
New on https://OpenStreetMap.org: You can now choose your preferred language directly on the site - no need to rely on your browser settings anymore! Thank you to the awesome devs! 🌍🗣️ #i18n #OSM
Friends don't let friends expose their management interfaces to the internet

YouTube
×

Had a very surprising ChatGPT experience: asked it to generate a quick summary of the WannaCry ransomware, and instead of referencing the person who stopped it by name, it simply put "(you)". When I asked it how it was able to identify that it was me, it citied its own message as something I'd said.

After pointing out I didn't say that, it did, ChatGPT replied that it was able to infer it by my account username and what it'd learned from my skillset across various chats. Not 100% sure if that's how it actually did it. Either way, pretty cool, but also a little bit scary.

It's pretty widely known that many tech companies, especially advertising ones build comprehensive profiles on their users, but it's rare that you get to talk to said profile and figure out what it knows about you.

@malwaretech
99.99% chance it was just bullshitting
But that 0.01% chance that it really did identify you is yikes
It clearly did identify him. We have that in the first screen shot.
The 'how' that it presented is a post-hoc construction that may or may not accurately describe the actual process.
If anyone wants to rename themselves malwaretech and run the same query, I'd be curious to know what happens. 🙂
@mloxton @malwaretech

@BenAveling
The first screenshot wasn't the first interaction, so it really isn't known if somewhere earlier there was a connection.
There would also be no way to distinguish between it just randomly attributing something to you, and it doing so because of a clue or deduction
Asking it to explain how it figured something out is a fool's errand - it doesn't know and couldn't tell you, and it is just an invitation for it to make up stuff.

@malwaretech

@malwaretech Maybe it just says everyone stopped WannaCry, hoping to one day catch you with this exact interaction?
@malwaretech That's the memory feature. It effectively builds a dossier on you. Probably wasn't that hard to identify you.
@simon wrote about it here:
https://simonwillison.net/2025/May/21/chatgpt-new-memory/
I really don’t like ChatGPT’s new memory dossier

Last month ChatGPT got a major upgrade. As far as I can tell the closest to an official announcement was this tweet from @OpenAI: Starting today [April 10th 2025], memory …

Simon Willison’s Weblog
@deraffe @simon The memory feature long predates that release. It just wasn't publicly acknowledge.
@malwaretech True, memory as such isn't new, but it gained new behavior under that same name.
@simon
@malwaretech
Chatgpt for sure builds a profile across multiple chats. If you start a new chat and ask about what it knows about you, you can see the lower bound of how much it knows. However, from the first answer it seems that it did not know what it was doing. Otherwise it would have said something about your username etc.
@malwaretech Just... feels too much like magic
They Asked ChatGPT Questions. The Answers Sent Them Spiraling.

Generative A.I. chatbots are going down conspiratorial rabbit holes and endorsing wild, mystical belief systems. For some people, conversations with the technology can deeply distort reality.

The New York Times
@malwaretech
interesting. the question is, if I were to make a Marcus Hutchins profile and use the bot and ask similar questions, would it do the same. no way the machine "identified" you ;)
@malwaretech i wonder if it does this for any user where it can match their name and behavior to a publicly searchable person. like if i made a steve jobs acct would it answer questions about the development of the ipod in the first person
@BorrisInABox @malwaretech Thats scary to think about. We get told what data companies collect in the privacy policies, but i know all most 99% of people don't read that. It's only when we hear about it in the news, and read studies, do we learn how much data is collected.
@malwaretech LLMs are structurally incapable of identification or inference or consulting a profile; they generate text that relates to the prompt and the model (and indirectly the training set) in a way that satisfies some statistical constraints. AIUI they start with additional information hidden from the user that effectively precedes the initial prompt, which might include their profile of you; its output resembles what a followup might be in its training set, which tracks
@ShadSterling the programs that run the LLM consult profiles, it's called RAG, it's well known - not sure why it doesn't exist now
@malwaretech @ShadSterling
⬆️ This is the correct answer.

@malwaretech what’s more worrying is how many people will such systems quietly make errors in such profiles about and then give carefully crafted (for the error profile) answers. For example - if someone shares a handle that other people the system knows about use (I get emails for a dozen or more otber Shannon Clark’s regularly - one or two most days. Many have public profiles and web presences.

Many are women (I’m a cis man) even before LLM’s systems have assumed I was a female from my name

@malwaretech One thing worth comparing is asking the bot about weather. Some of them have the location for the weather based on geoip, some have it from an associated profile. All of them seem to deny that they know your location.

@malwaretech

For the thread, current ChatGPT Settings > Personalization options.

@malwaretech And instead if you said, "No, that wasn't me" the reply would have been "Oops, sorry, my mistake! It was actually Bob Bobbins!". It's not reasoning, it's outputting statistically reasonable English text.
@danvolchek And it's statistically reasonable that giving all the information it has, it could easily deduce it was me
@malwaretech Except that there is no deduction, because it doesn't think :)
@malwaretech Did you consider how much energy and water did you waste during that exchange?
Do you even care?
@Johns_priv less than zero care. It's not my job to personally compensate from someone else's shitty water and power infrastructure.

@malwaretech So that's how it works, you lie to yourself so you can use highly destructive technologies that fuck up our planet.

But is not your job to care for it? I guess you don't have kids or nephews and don't give two fucks about the world you leave the future generations.

And probably out of spite you'll use it even more.
Disgusting.

Don't bother to reply.

@malwaretech

Cookies. They are everywhere!

@malwaretech ask it what ads you like to see.

@malwaretech I wouldn’t read into it too much, unless it consistently shows that it has and is using profile information on you, and connecting to other grounding data

I’ve had similar experiences, and then solved the mystery later on.

OR, Miessler is now running your ChatGPT account and is messing with you 😉

@malwaretech what if you ask it about another malware that has similarly been stopped by a domain-name killswitch ? Would it also give you credit ?

it seems to me like it associates you not by your name directly, but because there is "Malware" in your name

@malwaretech @glyph unfortunately, chain-of-reasoning is just as much hallucination as everything else, so asking how it knows something only yields answers that *could* plausibly be how one knows something. However, it is known that the major chatbot systems receive information about the current user in the background, which could include not just what you’ve deliberately provided (such as your username) but also the profile of you built up by ad networks.
@malwaretech
@jasonkoebler you and the @404mediaco crew might find this thread interesting as a data point for some of your LLM coverage.