Arvind NarayananMar 23, 2023

Language models have become privately controlled research infrastructure. This week, OpenAI deprecated the Codex model that ~100 papers have used—with 3 days’ notice. It has said that newer models will only be stable for 3 months. Goodbye reproducibility!

It'll be interesting to see how developers are going to use these models in production if things are going to break every couple of months.

By @sayashk and me on the AI Snake Oil book blog: https://aisnakeoil.substack.com/p/openais-policies-hinder-reproducible

OpenAI’s policies hinder reproducible research on language models

LLMs have become privately-controlled research infrastructure

AI Snake Oil
Show threadArvind NarayananMar 23, 2023

OpenAI responded to the criticism by saying they'll allow researchers access to Codex. But the application process is opaque: researchers need to fill out a form, and the company decides who gets approved. It is not clear who counts as a researcher, how long they need to wait, or how many people will be approved. Most importantly, Codex is only available through the researcher program “for a limited period of time” (exactly how long is unknown).

https://aisnakeoil.substack.com/p/openais-policies-hinder-reproducible

OpenAI’s policies hinder reproducible research on language models

LLMs have become privately-controlled research infrastructure

AI Snake Oil
Show threadNafnlaus 🇮🇸 🇺🇦Mar 23, 2023

@randomwalker "OpenAI isn't"

We need to happen to LLMs what happened to generative image AIs: an entity like Stability to come along and release (generally) license-unhindered weights to the public for good models that can run on consumer-grade GPUs.

We're "kinda" there with LLMs, in that LLaMA weights were released. Though the license is kind of burdensome, and support to make something ChatGPT-ish out of it is still being hacked together.

(BTW, your substack name is a turnoff to discussion)

Show threadArvind NarayananMar 23, 2023
@nafnlaus Fair point about the name. We use it because it has some history behind it (https://aisnakeoil.substack.com/p/introducing-the-ai-snake-oil-book) and is a bit of a recognized brand at this point, which might be helpful when our book comes out. But perhaps we're weighted down by that baggage. We'll discuss it. Thanks! CC @sayashk
Introducing the AI Snake Oil book project

Something weird happened on November 19, 2019. When a professor shares scholarly slides online, they are usually intended for a niche group of peers. You’d be lucky if 20 people looked at them. But that day, the slides Arvind released went viral. They were downloaded tens of thousands of times and his tweets about them were viewed 2 million times.

AI Snake Oil
Show threadEric Carroll
@randomwalker
I vote keep the name. We are rocketing up the Gartner hype curve faster than we can assess risks. Someone has to call it out.
@nafnlaus @sayashk
Mar 23, 2023 at 10:23pmWeb