AnthonyHaven't read this one yet, but I'm itching to:
https://mastodon.world/@Mer__edith/113197090927589168
Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AI
With the growing attention and investment in recent AI approaches such as large language models, the narrative that the larger the AI system the more valuable, powerful and interesting it is is increasingly seen as common sense. But what is this assumption based on, and how are we measuring value, power, and performance? And what are the collateral consequences of this race to ever-increasing scale? Here, we scrutinize the current scaling trends and trade-offs across multiple axes and refute two common assumptions underlying the 'bigger-is-better' AI paradigm: 1) that improved performance is a product of increased scale, and 2) that all interesting problems addressed by AI require large-scale models. Rather, we argue that this approach is not only fragile scientifically, but comes with undesirable consequences. First, it is not sustainable, as its compute demands increase faster than model performance, leading to unreasonable economic requirements and a disproportionate environmental footprint. Second, it implies focusing on certain problems at the expense of others, leaving aside important applications, e.g. health, education, or the climate. Finally, it exacerbates a concentration of power, which centralizes decision-making in the hands of a few actors while threatening to disempower others in the context of shaping both AI research and its applications throughout society.
Currently this is on #arXiv which, if you've read any of my critiques, is a dubious source. I'd love to see this article appear in a peer-reviewed or otherwise vetted venue, given the importance of its subject.
I've heard through the grapevine that US federal grantmaking agencies like the #NSF (National Science Foundation) are also consolidating around generative AI. This trend is evident if you follow directorates like CISE (Computer and Information Science and Engineering). A friend told me there are several NSF programs that tacitly demand LLMs of some form be used in project proposals, even when doing so is not obviously appropriate. A friend of a friend, who is a university professor, has said "if you're not doing LLMs you're not doing machine learning".
This is an absolutely devastating mindset. While it might be true at a certain cynical, pragmatic level, it's clearly indefensible at an intellectual, scholarly, scientific, and research level. Willingly throwing away the diversity of your own discipline is bizarre, foolish, and dangerous.
#AI #GenAI #GenerativeAI #LLM #ML
https://mastodon.world/@Mer__edith/113197090927589168
Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AI
With the growing attention and investment in recent AI approaches such as large language models, the narrative that the larger the AI system the more valuable, powerful and interesting it is is increasingly seen as common sense. But what is this assumption based on, and how are we measuring value, power, and performance? And what are the collateral consequences of this race to ever-increasing scale? Here, we scrutinize the current scaling trends and trade-offs across multiple axes and refute two common assumptions underlying the 'bigger-is-better' AI paradigm: 1) that improved performance is a product of increased scale, and 2) that all interesting problems addressed by AI require large-scale models. Rather, we argue that this approach is not only fragile scientifically, but comes with undesirable consequences. First, it is not sustainable, as its compute demands increase faster than model performance, leading to unreasonable economic requirements and a disproportionate environmental footprint. Second, it implies focusing on certain problems at the expense of others, leaving aside important applications, e.g. health, education, or the climate. Finally, it exacerbates a concentration of power, which centralizes decision-making in the hands of a few actors while threatening to disempower others in the context of shaping both AI research and its applications throughout society.
Currently this is on #arXiv which, if you've read any of my critiques, is a dubious source. I'd love to see this article appear in a peer-reviewed or otherwise vetted venue, given the importance of its subject.
I've heard through the grapevine that US federal grantmaking agencies like the #NSF (National Science Foundation) are also consolidating around generative AI. This trend is evident if you follow directorates like CISE (Computer and Information Science and Engineering). A friend told me there are several NSF programs that tacitly demand LLMs of some form be used in project proposals, even when doing so is not obviously appropriate. A friend of a friend, who is a university professor, has said "if you're not doing LLMs you're not doing machine learning".
This is an absolutely devastating mindset. While it might be true at a certain cynical, pragmatic level, it's clearly indefensible at an intellectual, scholarly, scientific, and research level. Willingly throwing away the diversity of your own discipline is bizarre, foolish, and dangerous.
#AI #GenAI #GenerativeAI #LLM #ML
Meredith Whittaker (@[email protected])
Attached: 1 image 📣NEW paper! Don’t believe the hype: bigger AI ≠better AI. @SashaMTL, @GaelVaroquaux and me on how the race to bigger, and bigger AI has bad consequences and isn't necessary. 1. Smaller AI models often perform better than big models in context And 2. Obsession with bigness has severe collateral consequences, from climate costs, to concentrated power, to more surveillance, to the capture of AI research. All of this, and what we can do instead 👇 https://arxiv.org/abs/2409.14160
