Mehrdad YazdaniSep 14, 2023

Im thinking of starting a series where I try to reproduce the claims of Gary Marcus and the like on the limitations of ChatGPT.

This is spurred by coming across one of his posts from the dead bird sites where he brings up some failed reasoning example in GPT, yet when I try to reproduce it succeeds every time.

Show threadDaniel
@crude2refined cynically (conspiratorially?), perhaps OpenAI is reading Marcus and others who are good at finding shortcomings, and correcting them — a good way to undermine their best sceptics?
Dec 2, 2023 at 3:16pmWeb
Show threadMehrdad YazdaniDec 2, 2023

@dang ya maybe they are explicitly following Marcus and patching along the way. The problem is I rarely (ever?) see Marcus share examples with a link, instead it’s always screenshots that are not reproducible.

But I’ve also ran into negative examples brought by folks other than Marcus camp. When I try to reproduce these (certainly more long tailed) negative examples, I can’t (and end up being positive examples).

Show threadMehrdad YazdaniDec 2, 2023

@dang
Supposedly the latest OpenAI updates have a feature to make reproducible outputs but I haven’t tried.

Great to hear from you DG!