Takeaways: open models are good; popular texts are probably not good barometers of model performance; with the bias toward sci-fi/fantasy, we should be thinking about whose narrative experiences are encoded in these models, and how that influences other behaviors. Work with Kent Chang, Mackenzie Cramer and Sandeep Soni. Details on all books here: https://docs.google.com/spreadsheets/d/1jW7EhsNjIGDMoK2JidyDD7UXH9N0NpEJfWFEj05_LC4 6/6

How do we know? Fill-in-the-name: "Wow. I sit down, fish the questions from my backpack, and go through them, inwardly cursing [MASK] for not providing me with a brief biography. I know nothing about this man I’m about to interview. He could be ninety or he could be thirty." People can't guess the name; it's not knowledge of English you need, but the underlying content to know that "Kate" is the only right answer. (GPT-4 gets this one right, from 50 Shades of Grey.)