thinkbergMar 23, 2023
Prof. Emily M. Bender(she/her)

For those who are still willing to access the birdsite, this promises to be very entertaining:

https://twitter.com/DV2559106965076/status/1638769434763608064

Basically, someone discovered that the latex source "sparks of AGI" paper from MSFT includes a lot of commented out stuff and they're going through it.

[I just downloaded the source too and can confirm that the first thing they point out, at least, checks out.]

Tweet / Twitter

Twitter
Mar 23, 2023 at 6:46pmWeb
Show threadRuth StarkmanMar 23, 2023
@emilymbender yes, OMG oy
Show threadBornachMar 23, 2023
@emilymbender
For those who want to avoid the  someone repackaged the thread here
https://mem.ai/p/Gw4E9TbVgN0aP35S8hBo
Mem

Let AI organize your team’s work—from meeting notes, projects, to knowledge bases. All instantly searchable and readily discoverable.

Show threadDerek | ScriptAutomateMar 23, 2023

@emilymbender

For those interested, here are links:

Sparks of Artificial General Intelligence: Early experiments with GPT-4 (the paper in question)
- https://arxiv.org/abs/2303.12712

Source tar gz archive download (includes commented out sections, some key items in contents/unused dir):
- https://arxiv.org/e-print/2303.12712

Unpublished sections include notes on toxicity (ex. "the model generates toxic content without any prompting.")

Comments also show researchers may not know the cost of running the models.

Sparks of Artificial General Intelligence: Early experiments with GPT-4

Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition. The latest model developed by OpenAI, GPT-4, was trained using an unprecedented scale of compute and data. In this paper, we report on our investigation of an early version of GPT-4, when it was still in active development by OpenAI. We contend that (this early version of) GPT-4 is part of a new cohort of LLMs (along with ChatGPT and Google's PaLM for example) that exhibit more general intelligence than previous AI models. We discuss the rising capabilities and implications of these models. We demonstrate that, beyond its mastery of language, GPT-4 can solve novel and difficult tasks that span mathematics, coding, vision, medicine, law, psychology and more, without needing any special prompting. Moreover, in all of these tasks, GPT-4's performance is strikingly close to human-level performance, and often vastly surpasses prior models such as ChatGPT. Given the breadth and depth of GPT-4's capabilities, we believe that it could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system. In our exploration of GPT-4, we put special emphasis on discovering its limitations, and we discuss the challenges ahead for advancing towards deeper and more comprehensive versions of AGI, including the possible need for pursuing a new paradigm that moves beyond next-word prediction. We conclude with reflections on societal influences of the recent technological leap and future research directions.

arXiv.org
Show threadSebastian LasseMar 23, 2023

@emilymbender

For those who are not
https://nitter.net/DV2559106965076/status/1638769434763608064#m

DV (@DV2559106965076)

You might know that MSFT has released a 154-page paper (https://arxiv.org/abs/2303.12712) on #OpenAI #GPT4 , but do you know they also commented out many parts from the original version? 🧵: A thread of hidden information from their latex source code [1/n]

Nitter
Show threadProtectiveHenMar 24, 2023
@emilymbender Here is a nitter link for those who don't want to give Elon traffic or allow him to surveil you: https://nitter.net/DV2559106965076/status/1638769434763608064
DV (@DV2559106965076)

You might know that MSFT has released a 154-page paper (https://arxiv.org/abs/2303.12712) on #OpenAI #GPT4 , but do you know they also commented out many parts from the original version? 🧵: A thread of hidden information from their latex source code [1/n]

Nitter
Show threadaijooyoomMar 24, 2023

@emilymbender

Here is how you can read it with privacy and without a twitter account:

https://nitter.net/DV2559106965076/status/1638769434763608064

DV (@DV2559106965076)

You might know that MSFT has released a 154-page paper (https://arxiv.org/abs/2303.12712) on #OpenAI #GPT4 , but do you know they also commented out many parts from the original version? 🧵: A thread of hidden information from their latex source code [1/n]

Nitter
Show threadJeroen BaertMar 24, 2023
@emilymbender That's a fun leak
Show threadKnud JahnkeMar 24, 2023

@emilymbender

My mantra: allways sanitize you LaTeX sources. Years ago telescope proposals to ESO had to submit their source. Referees usually just read the generated PDFs but one could also look at the source...

Show threadPsittacus cafeinaMar 25, 2023
@knud @emilymbender
In-thread they mention that arXiv requires posting the entire source code. Might violate rules to scrub it like that
Show threadKnud JahnkeMar 25, 2023

@caffetiel @emilymbender

You need to submit the code that makes the paper. And comments in LaTeX never impact anything. So I don't see how removing every commented bit could 'violate' anything.

Show threadmsgbiMar 24, 2023
@emilymbender @charles_ex
Show threadDavid MankinsMar 25, 2023
@emilymbender
Following the link to the bird site, not too far into the thread I come across tweet about an astrology service. It takes me a few seconds to realize it's an ad, and not itself a commentary on GPT-n hype.
Show threadfaroMar 26, 2023

@emilymbender I started using nitter for link to the other site. Highly recommended!

https://nitter.net/DV2559106965076/status/1638769434763608064

DV (@DV2559106965076)

You might know that MSFT has released a 154-page paper (https://arxiv.org/abs/2303.12712) on #OpenAI #GPT4 , but do you know they also commented out many parts from the original version? 🧵: A thread of hidden information from their latex source code [1/n]

Nitter