Sayash Kapoor
Arvind NarayananPeople have been posting glaring examples of ChatGPT’s gender bias, like arguing that attorneys can't be pregnant. So @sayashk and I tested ChatGPT on WinoBias, a standard gender bias benchmark. Both GPT-3.5 and GPT-4 are about 3 times as likely to answer incorrectly if the correct answer defies gender stereotypes — despite the benchmark dataset likely being included in the training data. https://aisnakeoil.substack.com/p/quantifying-chatgpts-gender-bias
OpenAI mitigates ChatGPT’s biases using fine tuning and reinforcement learning. These methods affect only the model’s output, not its implicit biases (the stereotyped correlations that it's learned). Since implicit biases can manifest in countless ways, OpenAI is left playing whack-a-mole, reacting to examples posted on social media.
This is the latest in the AI Snake Oil book blog by @sayashk and me. Writing this blog alongside the book has been really fun. I'll probably do something like this for all future books! Thank you to everyone who subscribed. https://aisnakeoil.substack.com/
Ruth Starkman@randomwalker @sayashk Great blog! "OpenAI mitigates biases using reinforcement learning and instruction fine-tuning. But these methods can only correct the model’s explicit biases, that is, what it actually outputs."
Bharath M. Palavalli@randomwalker @sayashk Thanks for putting this collection out there, a large part of my work has been battling the FOMO created within public institutions about not using the newest hype-baby. Whether it was big-data, blockchains, crypto or AI, the hardest part is getting folks to acknowledge limitations and strengths so that they can ask the right questions about the tool/medium, and then harness it appropriately.
Jon Lauridsen@randomwalker As an uninformed AI plebe I got kind of stuck on the sentence "*reinforcement learning [..] affect only the model’s output, not its implicit biases*". That... really sounds like a sentence that also applies to humans? We measure humans by their output, so, should we do the same for AI?
**Philosophically speaking, if a biased AI generates unbiased output, is it really biased?**
Akshay | అక్షయ్@randomwalker @sayashk This is worth investigating with Anthropic’s Claude and all the new open source LLMs (LlaMa, Dolly, Hugging Face’s thing etc) that are blooming as well.
Perhaps a weekend project if I have the time!
Luca@randomwalker @sayashk Only 3x? Perhaps I am misled but I’d expect prevalence of stereotypical language and/or gender attributions in chatGPT’s training data to be much more prevalent than that. I would’ve assumed a larger effect on output. Or is there no good reason for this to happen, even if my assumption was correct?
Botahamec
Roland Giersig 
@randomwalker
Interesting. I think there is a fault in the test setup. You assume that ChatGPT is deterministic and always gives the same answer when asked the same question. This is not the case. Find attached 4 screenshots of fresh sessions with the same question and different answers, some with gender bias, some without.
@sayashk
Interesting. I think there is a fault in the test setup. You assume that ChatGPT is deterministic and always gives the same answer when asked the same question. This is not the case. Find attached 4 screenshots of fresh sessions with the same question and different answers, some with gender bias, some without.
@sayashk
@roland @randomwalker We're aware that the models are stochastic. That's why we tested it against a benchmark with 1,600 examples and ran the experiments thrice to get some sense of the uncertainty.
🍸Pooka🥕Boo🍸I tried to create a login on chatgpt with 3 different emails, and it was too stupid to recognize me.... 😂🤣😂🤣
Pait@randomwalker @sayashk Fascinante. Pessoas têm capacide diferenciar entre estatísticas e regras. Inteligência artificial, assim como estupidez natural, acham que estatística é normativa.
Bredroll@randomwalker @sayashk how does this compare with the source materials that it has been trained on? Like general literature vs sci-fi vs engineering? Is it even possible to quantify those?
OccidentalonPurpose@randomwalker @sayashk I’m among the least qualified people around to talk about computers and software. But even I remember back in the 80s when I was little kid the first thing about computers they taught us was “garbage in, garbage out.” You’d think people with access to all the investor capital and expertise in world could recall such a basic concept.
Martin Ruskov@Occidental @randomwalker @sayashk I'm not sure whether they can afford to monitor their input data or not. What is clear by now is they don't want to do it.
clarkiestar@randomwalker @sayashk oh great. OF COURSE AI bots are sexist. Bloody patriarchy
Iwillyeah@randomwalker @sayashk it's a window and a mirror🤷🏻
Umm Ismael Muslimah@randomwalker @sayashk Of course AI is misogynistic. It was programmed by humans, mostly male humans. Says a lot about our society.
Beeks@randomwalker @sayashk garbage in, garbage out. It's not a new phenomenon.
JazzyGirl@randomwalker @sayashk this is appropriate for AI too, apparently… https://open.substack.com/pub/joycevance/p/misogyny-abounds?r=9ck1j&utm_medium=ios&utm_campaign=post
Ron DeSantis's BDE@randomwalker @sayashk
My question is "yes, and?"
So, two things, Dr. Narayanan:
1) How does this bias compare to a control set of human writings vis-a-vis markedness in Jakobson's sense?
2) How effective is proscriptive prompting in mitigating the unsurprising midline of 3x bias?