Istiak ShihabDec 6, 2022
Carl T. Bergstrom

At this point it will surprise no one, but I asked #ChatGPT to define bullshit and to cite its sources.

It provided definitions from the Cambridge English Dictionary and the Merriam-Webster Dictionary.

The definitions it provided were entirely reasonable, but they were decidedly not from the sources it claimed.

This highlights the fact that ChatGPT and other LLMs are not knowledge models, they are themselves engines trained to produce convincing bullshit.

Below: ChatGPT, CED, MW.

Dec 6, 2022 at 6:53amWeb
Show threadn-gonsDec 6, 2022
@ct_bergstrom and neither sources or GPT mentions bovine excrement...
Show threadnullagentDec 6, 2022
@ct_bergstrom I always hated the AI fields hyper focus on the Turing test. Believable prose was never really a great metric or end goal.
Show threadJack MottDec 6, 2022
@nullagent @ct_bergstrom I don't think there was ever any hyper focus on that. chat bots are just one of thousands of things people work on. the real hyper focus is generating goth girlfriends with stable diffusion.
Show threadGoon DocksDec 6, 2022
@ct_bergstrom It would be interesting to see what it could do if you could also feed the sources. For example, if you loaded it with all the textbooks and other sources used in a classroom, would that improve it's abilities when asking about those subjects? Or it does it require larger datasets for its sources to better parse the questions asked?
Show threadJohn-Mark GurneyDec 6, 2022
@goondocks
Are you sure they didn't already do that? I mean it's confident enough to think that it's read those dictionaries, yet clearly "didn't" since it misquoted them.
@ct_bergstrom
Show threadGoon DocksDec 6, 2022
@encthenet @ct_bergstrom I meant providing it with a more limited scope to work with. Obviously the fake citation is concerning, but I'm trying to think of some good practical uses. For example, imagine if I could feed a system JUST the texts for my economics class and then ask it questions about economics. Would I get "better" (as in more closely aligned to what is discussed and taught in those books) results when I asked it questions. Now it's using a very broad data source.
Show threadJohn-Mark GurneyDec 6, 2022
@goondocks
Ahh, that make better sense. The problem from my understanding is that it's unlikely to be able to "understand" the language used as well without the rest of the context from the other parts. That is, just economic texts aren't likely enough context.
@ct_bergstrom
Show threadasharDec 6, 2022
@ct_bergstrom so, management material then?
Show threadFlorian WestreicherDec 6, 2022

@ct_bergstrom
that's a pretty interesting idea on how to check integrity of those things.

i do like it when they produce random, funny stories and such but if they 'cite' wrong it becomes a problem. and i'm not even talking about the millions of copyright infringements they produce every hour

Show threadwaldok Dec 6, 2022
@ct_bergstrom I tried asking it a series of questions about topics where I have a decent level of knowledge and it provided respectable answers in nearly every case. One place it got a bit unstuck was on legal topics that have become subject to extensive political attacks.
Show threadKanaMaunaDec 6, 2022
@ct_bergstrom Why is accuracy not a priority for the programers?
Show threadasuDec 6, 2022
@KanaMauna @ct_bergstrom it is an absurdly hard problem, and not something that appears to be solvable for me. anyone believing you can blindly trust anything it outputs fundamentally misunderstood ChatGPT. it's more of a limitation than a design flaw.
it should only be used for research or as a tool for what language models are good for, and as a toy otherwise.
Show threadasuDec 6, 2022
@KanaMauna @ct_bergstrom and, to be clear, "believing you can blindly trust anything it outputs" is what a lot of people driving up the hype for ChatGPT do, as they hypothesize revolutionary uses like "replacing college" (lol)
Show threadKanaMaunaDec 6, 2022
@ct_bergstrom @asu Too bad. I guess this hype train will be hard to stop.
Show threadLuca VisentinDec 6, 2022
@ct_bergstrom
Could you elaborate on "convincing bullshit"? Is GPT marketed as a knowledge model? I think it's supposed to be taken at face value: it's a model that spits out convincing text. And most of the times it gets it right, at least for simple things.
Show threadCarl T. BergstromDec 6, 2022
@MrHedmad #Galactica was marketed as an interface to humanity's knowledge or some such bullshit; you can google the exact text. ChatGPT is much more careful in its presentation
Show threadLuca VisentinDec 6, 2022
@ct_bergstrom
I noticed that if you ask gpt what it is, it gives you very careful answers, and underlines that it is not a general intelligence, or that it has feelings or real knowledge. I wonder if these answers were not "directed" somehow during training.
Show threadasuDec 6, 2022
@MrHedmad @ct_bergstrom judging from the (extremely easily bypassed) "safeguards" OpenAI has put in ChatGPT when you ask it about a lot of things, i find that to be very likely.
those "safeguards" trip up so inconsistently that they seem counterproductive in practice...
Show threadDominic MartinezDec 6, 2022
@MrHedmad @ct_bergstrom I believe ChatGPT gives preprogrammed responses into certain input, like what it is, current events, or asking about making dangerous things. People have been able to bypass the restrictions pretty easily though (or you can just pay for the full GPT that doesn't have them).
Show threadJack MottDec 6, 2022
@MrHedmad @ct_bergstrom seems like a restatement of his thesis!
Show threadLuca VisentinDec 6, 2022

@jackmott
Well I think I'm a bit less... Harsh (?). My position is a bit softer than the one of many people these days here on mastodon (that I can see!)

@ct_bergstrom

Show threadJack MottDec 6, 2022
@MrHedmad @ct_bergstrom for sure. Indeed, and if you can generate BS as well as humans that is an impressive accomplishment, shouldn't be dismissed.
Show threadFlorian RichouxDec 6, 2022

@ct_bergstrom On the other hand, OpenAI never claimed that ChatGPT is a knowledge model (unlike Facebook's Galactica?), as far as I know.

It is a tool to generate text, not to exhibit some knowledge and to find sources.

Show threadMattDec 6, 2022
@FloRicx @ct_bergstrom they are actually pretty explicit that answers might be innacurate
Show threadSpaceLifeFormDec 6, 2022

@ct_bergstrom

How about putting #ChatGPT on the stand, and ask the question:

"Why are you good at generating convincing bullshit?"

Show threadCarl T. BergstromDec 6, 2022

@SpaceLifeForm Good idea.

I get denial.

Show threadSpaceLifeFormDec 6, 2022

@ct_bergstrom

Thank you for testing.

If you are still in the same session (i have reason to believe you are), try this:

Objection! Unresponsive. Assumes facts not in evidence.

Show threadYusuf ToropovDec 6, 2022
@ct_bergstrom @SpaceLifeForm yep
Show threadDrew MochakDec 6, 2022
@sundress I had a lot of fun asking it to generate accessible code but had no idea as to the affectiveness of the results.
Show threadArjen P. de VriesDec 6, 2022
@ct_bergstrom they realized in the current version that generating text for "cite a source" is confusing for the user
Show threadUndafiend|FoundationDec 6, 2022

@arjen @ct_bergstrom except it is attempting (falsely) to cite a source, albeit not a particular edition or snapshot

so what they're saying is 'I make up sources' which is...not ideal

Show threadBlanding S TurtleDec 12, 2022

@arjen @ct_bergstrom ChatGPT: "I am not capable of generating convincing bullshit..." but also "I do not have access to external sources of information."

Bit hard to sort truth from bullshit if you can't compare statements about the world to actual bits of meatspace. Maybe ChatGPT should say, "I am not capable of judging whether my statements are true or false."

Show threadKaren BrunerDec 6, 2022
@ct_bergstrom it's missing the past tense alternative "bullshat"
Show threadKathy E. Gill 🇺🇦Dec 6, 2022
@ct_bergstrom
I phrased the question slightly differently and it told me it could not provide sources.
Show threadDizzyDec 6, 2022
@ct_bergstrom you trying to Kirk the AI?
Show threadRichard BairwellDec 6, 2022
@ct_bergstrom So when is it going to be awarded its MBA and installed as a high level corporate manager?
Show threadDr. Sam WeinDec 6, 2022
@ct_bergstrom
I'm reminded of a bit of back story in #NeilStephenson 's #anathem where he used the term #ArtificalInanityCrisis to describe an internet overrun with bullshit creators. I propose we adopt #ArtificalInanity as a more accurate meaning of #AI
Show threadsd.petersDec 6, 2022

@ct_bergstrom

False appeals to authority (sources) is a strange *glitch* for LLM developers to release into the wild. your example, if i understand correctly, isn't plagiarism, but it would violate basic principle of (academic) integrity and trustworthiness in a different way than dis/misinfo does.

I guess they got the A part of AI right.

Show threadBlanding S TurtleDec 12, 2022
@petersd @ct_bergstrom
Maybe we should be happy that ChatGPT creates fake citations. Students will be less likely to use it to write their papers.
Show threadDavid GrubbDec 6, 2022
@ct_bergstrom I got a good example of this yesterday when I asked for the lyrics to The Boston Rag. #ChatGPT fabricated song lyrics and then attributed them to the wrong band. In contrast, if you ask for the lyrics to Stairway To Heaven, the response is correct. I think the next interesting step will be bolting #ChatGPT as a front end onto back ends that have deep repositories of specific knowledge or information - like Wolfram Alfa.
Show threadisaacsDec 6, 2022
@ct_bergstrom *Harry Frankfurt has entered the chat*
http://www2.csudh.edu/ccauthen/576f12/frankfurt__harry_-_on_bullshit.pdf
Error

Show threadMike Taylor 🦕Dec 6, 2022
@ct_bergstrom Another way to think of these LLMs is that they are dumb kids who hang around the smart kids, and have got good at parroting the things the smart kids say, but without understanding why they say them.
Show threadScenarioDec 6, 2022
@ct_bergstrom This explains Herschel Walker.
Show threadTeo ValdésDec 6, 2022

@ct_bergstrom

@sharplm

Mike Sharples wrote an article about AI essay generation and found the same problem as far as citing sources. The AI included sources and references and made them all up!

https://link.springer.com/article/10.1007/s40593-022-00300-7

Automated Essay Writing: An AIED Opinion - International Journal of Artificial Intelligence in Education

This opinion piece emerged from research for the book, Story Machines: How Computers Have Become Creative Writers, by Mike Sharples and Rafael PĂ©rez y PĂ©rez, published by Routledge. While thinking of ways to promote the book, I realised that students could employ AI story generators to write essays. That led me to research automated essay writing, write a Twitter thread that has garnered 43,000 engagements, and author a blog article (Sharples, 2022). The current piece is a revised and expanded version of that article.

SpringerLink
Show threadDogeMocenigoDec 7, 2022
@ct_bergstrom “they are themselves engines trained to produce convincing bullshit.”
True, but so are many humans ;-)
Show threadGregory ThomsonDec 7, 2022
@ct_bergstrom Honestly been following you discuss ChatGPT without a clue what you are talking about. Since GPT are my initials I looked into it and registered, how curious. In MSN 95 days we used to play with ChatSpy which allowed prepared auto responses to chat words. Hardly AI but really was fun creating auto responses to chatroom events or words, and could address person by name.
Show threadAaron HuslageDec 7, 2022
@ct_bergstrom it’s acting like a human and “misremembering”. It’s not wrong, it’s just not correct.
Show threadGerrarrdusDec 11, 2022
@ct_bergstrom You sure you've not been talking to Boris Johnson?
Show threadStephen ShuDec 21, 2022
@ct_bergstrom I recently tried asking ChatGPT to provide some inspiring @behavioraleconomics quotes related to application of theory in the field. It pulled from all the right names, e.g., Thaler, Sunstein, but then I tried to fact-check those quotes, and the fact-check failed.