Cassandra is only carbon nowFeb 23

As mentioned before, I hate bringing this up because I have no evidence or expertise here, just a gut feeling. But I just can't help feeling like, aside from everything else aside about LLM chatbots, they're quickly becoming the leaded gasoline of our time.

Something doing real damage to human cognition, but in this diffuse and difficult to measure kind of way.

Show threadCassandra is only carbon nowFeb 23

Many, not nearly all but *many*, folks using this things seem (again, as a gut feeling) to just talk differently after contact with chatbots? I can't even quite put my finger on it, but it scares the shit out of me.

It's not even an argument against chatbots, I have plenty of arguments that are far better substantiated, it's a personal fear about what they're doing.

Show threadIrenes (many)Feb 23

@xgranade the following is pure speculation based solely on personal experience

so speaking for ourselves, we have, like, hard-mode neurology, yeah? don't get us wrong, we love what we are and wouldn't change it, but we have a predisposition towards paranoia and as kids almost all of our conversations were with ourselves (since humans didn't acknowledge us as one of them), which caused us to get pretty far off in the weeds in terms of what we care about and how we talk about it?

Show threadIrenes (many)Feb 23
@xgranade like we had this entire personalized jargon which felt normal to us because everyone we talked to (ie. ourselves) understood it, you know?
Show threadIrenes (many)Feb 23
@xgranade we were very fortunate in that our artistic expression was interacting with computers, which are very rigorous in their demands. use a traditional programming language to tell a computer to do something and you will get nowhere unless you've fully understood what you're asking it to do, so that was a lot of forced practice of our science skills, our ability to test things against measurable reality
Show threadIrenes (many)Feb 23
@xgranade and then later in life, after transition gave us common ground with humans and interacting with them became an option, we had to learn a lot of specific skills to understand the consensus reality that people live in and kind of funnel everything through the stuff we have in common so that there's, like, a mutually intelligible purpose for it? because otherwise people are just confused?
Show threadIrenes (many)Feb 23

@xgranade and because we had to learn all this stuff explicitly through trial and error, we're extremely aware of what the skills consist of

and, like... a generative language model is going to NOT require any of that. none of the science, none of the social skill. it will just mirror people's remarks back to them and it will never admit to not understanding and it doesn't behave any differently when people speak total nonsense to it

Show threadIrenes (many)Feb 23
@xgranade so it feels perfectly clear to us that spending too much time talking to the things would result in atrophy of the trial-and-error parts of social interaction, because people doing that are not exercising that skill but the machine is faking the reward for it anyway
Show threadIrenes (many)Feb 23

@xgranade again, this is total speculation. just because we can identify a plausible mechanism doesn't make this science; somebody would have to do actual research to validate our guess.

... but KNOWING that is kind of the precise thing at issue, yeah?

Show threadCassandra is only carbon nowFeb 23
@ireneista YUP. But I guess why this is a fear for me is because by the time research does validate or disprove any of these guesses, this shit will have done nearly incalculable harm.
Show threadIrenes (many)Feb 23
@xgranade right, absolutely. it's why we have personally been avoiding all interaction with the things. we need our brain. we're using it.
Show threadIrenes (many)Feb 23
@xgranade of course that's an easy decision for us for a variety of reasons, not least that we don't want anything these tools can give us.
Show threadCassandra is only carbon nowFeb 23
@ireneista Yeah, absolutely. It's why I'm careful to not make this shit one of my arguments against LLMs, there's far better and far more substantiated arguments — but it is a personal fear, and that's not nothing, even if fear isn't a good *argument*.
Show threadIrenes (many)Feb 23
@xgranade yeah. fear is delivering an important message - about what matters to you, what you have to lose
Show threadCassandra is only carbon nowFeb 23
@ireneista Honestly? Friends. I have multiple friends who have zero interest in any AI products, but who don't resist when they get added in forced updates. What will happen to them?
Show threadIrenes (many)Feb 23
@xgranade that's our biggest concern, too. we've known people who use the things for therapy, and... well, that terrifies us for a long list of reasons.
Show threadCassandra is only carbon nowFeb 23
@ireneista Yeah, absolutely. But even short of that, the people who would never click the "ask AI" button on purpose are having prompts shoved on them anyway. Fuck, *I* misclicked and hit an "ask AI" button by accident the other day!
Show threadIrenes (many)

@xgranade also, every general-knowledge web search we've done in the last few months has returned almost entirely machine-generated results

(which do not disclose their origin)

Feb 23 at 9:53pmWeb
Show threadCassandra is only carbon nowFeb 23
@ireneista I run my own selfhost instance of a metasearch engine, configured to try and avoid that very problem, with only limited success.