Mastodawn

Mark Dingemanse Jul 17, 2023

⚡️New! Opening up ChatGPT
https://dl.acm.org/doi/10.1145/3571884.3604316 ( https://doi.org/10.48550/arXiv.2307.05532 )

TL;DR #chatgpt is unfit for responsible use in science & education. Open instruction-tuned text generators (LLM + RLHF) are on the rise. But how open are they? We track degrees of openness in this #CUI23 paper & live at https://opening-up-chatgpt.github.io

We hope this paper & repository will help more people make mindful choices about tech

work with @andreasliesenfeld and @alianda

Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators | Proceedings of the 5th International Conference on Conversational User Interfaces

ACM Conferences

Show thread

Mark Dingemanse Jul 17, 2023

This paper got more radical as we wrote (and rewrote) it. It started out of a need for a constructive & systematic overview of open LLM+RLHF alternatives. See our repo! https://opening-up-chatgpt.github.io/

Credits to @andreasliesenfeld for coming up with the idea and for observing that the open LLM+RLHF landscape was growing so rapidly that it would be worth a systematic survey.

Looking at all these initiatives made the OpenAI playbook stand out as so cynical & harmful, we had to call it out in print

Opening up ChatGPT: LLM openness leaderboard

Show thread

Mark Dingemanse Jul 18, 2023

We were impressed by the transparency of the BLOOMZ/xmtf models, clearly designed from start to finish with openness and accountability in mind. This is the only player checking virtually *all* the boxes: data collection & curation, training regime, instruction-tuning, codebase, model cards, data sheets: everything meticulously documented and organized. You can look it all up on our site or directly in the repo: https://github.com/opening-up-chatgpt/opening-up-chatgpt.github.io/blob/main/projects/bloomz.yaml

opening-up-chatgpt.github.io/projects/bloomz.yaml at main · opening-up-chatgpt/opening-up-chatgpt.github.io

Liesenfeld, Andreas, Alianda Lopez, and Mark Dingemanse. 2023. “Opening up ChatGPT: Tracking Openness, Transparency, and Accountability in Instruction-Tuned Text Generators.” In ACM Conference on C...

GitHub

Show thread

Mark Dingemanse

And right on cue, Meta releases #llama2. They call it "open source" but it really is not in any sense of the term. All they make available is model weights. The source (training data, RLHF data, code) is nowhere to be found and the whole thing is about the furthest from true FOSS you can get. It ends up on the lowest rung in our overview, just one notch above chatgpt: https://opening-up-chatgpt.github.io

The real question reporters should be asking: What's in it for Meta?

Opening up ChatGPT: LLM openness leaderboard

Show thread

Mark Dingemanse Sep 30, 2023

Some people were using OpenAI's advanced API parameters apparently to derive scientific insights about GPT3.5. This will break in under a week. Thank you OpenAI for making our point so eloquently and efficiently for us

(first image: introduction of our CUI'23 paper https://doi.org/10.1145/3571884.3604316 ; second image: OpenAI announces it will break getting information about prompt/output probabilities for whatever reason)

#OpeningUpChatGPT #proprietary #reproducibility #openscience