Ryan CordellJan 26, 2023

In our lab introducing AI writing to students I encourage them to probe the models—to try & find limit cases & places where the “wires” of the machine become visible. Some things they found:

+ the model will refuse to generate a “diss track” or write insults about either specific celebrities or platypuses
+ when asked to rewrite a paragraph “using words without the letter e” it returned the exact same words with all e’s removed

Lab prompt here:

https://s23bbb.ryancordell.org/labs/Lab1-AI/

Lab 1: Amanuenses to AI

> every figure of speech, snowclone, cliché, joke, proposition, statement, and practically every linguistic structure that can be turned into a template is easily explored with a bot. Every work of literature, every writer’s body of work, every literary movement, national literature, and textual corpus is waiting to be analyzed...

Show threadRyan Cordell

A few other random observations (all about ChatGPT, which I didn’t make clear):

3. Attempts to create an SNL skit were fruitless—it did not seem to have SNL in its training data & just generated generic scripts
4. ChatGPT is famously good at synthesis, but it’s often unable to combine critical ideas in new ways. If asked to produce a Marxist analysis of various texts, it often produces one paragraph of Marxist thought & one about the chosen text, but can’t combine the two concepts

Jan 26, 2023 at 5:09pmWeb
Show threadRyan CordellJan 26, 2023
It did, however, produce a surprising & quite wonderful Marxist haiku about ratatouille (the dish, not the rodent chef movie) so 🤷‍♂️
Show threadJason MittellJan 26, 2023
@ryancordell I wonder if #4 is a byproduct of the training data. So much "critical writing" is effectively a paragraph of theory and a paragraph of textual description, devoid of actual analysis.
Show threadRyan CordellJan 26, 2023
@jmittell it was quite stubborn about keeping the two concepts separate *unless* we asked about texts about which (I assume) there’s more extant critical writing. It will write a marxist analysis of Moby Dick, likely channeling/plagiarizing existing writing in the training set, but if you ask for a marxist analysis of a relatively new TV show you get marxism then show summary/analysis—which makes sense given that it’s just putting words together based on existing words that go together
Show threadAndy FamigliettiJan 26, 2023
@ryancordell @jmittell that's a really interesting observation that deserves further attention... hmmmm
Show threadJason MittellJan 26, 2023
@afamiglietti79 @ryancordell Test case: I just asked ChatGPT to "write an analysis of racial representation in the film Get Out" - the result was a solid if unsophisticated analytic essay. I then asked it the same prompts for Vertigo and Psycho, two films that have SO MUCH written about them, but comparatively little on race. Both were incoherent!
"Critics have pointed out that [Vertigo] perpetuates the stereotype of the exotic and mysterious "other," and that the portrayal of Madeleine as a woman of a different race reinforces the idea that white people are superior to people of other races."
"[Psycho] tells the story of a white man, Norman Bates, who is the proprietor of a hotel, who becomes obsessed with a woman, played by Janet Leigh, whom he believes to be of a different race." (This sentence was also included nearly verbatim for Vertigo, just swapping out character/actor names!)
Show threadJason MittellJan 26, 2023
@ryancordell That bodes well for my field resisting pedagogical short-cuts!