@emilymbender Discovering that a chat bot's data is inconsistent with its sources is often trivial. I've had fun asking GPT-3 questions about the rules of Scrabble, especially around how many tiles there are for each letter, and whether you can spell out certain words with and without blank tiles. The answers are internally inconsistent, don't correspond with actual Scrabble rules, and don't even correspond with the source material that GPT-3 pointed me to when I asked it how it "knew".

@markproxy @emilymbender so there is not any setup for direct citation of sources in these models but you can ask it what it's sources are… but it's possibly making up things there, too?!

@mlncn @emilymbender It's not fabricating sources entirely, but the content of the sources I've seen often doesn't match GPT-3's so-called knowledge.