| Home page | https://boazbarak.org |
| Blog | https://windowsontheory.org/ |
| Bird site | https://twitter.com/boazbaraktcs |
| Group page | https://mlfoundations.org/ |
A paperclip maximizer, a stochastic parrot, king Midas, a jpeg of the web, markets, democracy, an alien, a shoggoth,...
So many metaphors offered for AI. I discuss these and their limitations
https://windowsontheory.org/2023/06/28/metaphors-for-ai-and-why-i-dont-like-them/
New paper on the implicit bias (or lack thereof) of stochastic gradient descent in the online learning setting https://arxiv.org/abs/2306.08590
We show that aside from the computational benefits, SGD noise (larger learning rate or smaller batches) do not offer implicit bias advantages.
With Nikhil Vyas, Depen Morwani, Rosie Zhao, Gal Kaplun, and Sham Kakade.
The success of SGD in deep learning has been ascribed by prior works to the implicit bias induced by finite batch sizes ("SGD noise"). While prior works focused on offline learning (i.e., multiple-epoch training), we study the impact of SGD noise on online (i.e., single epoch) learning. Through an extensive empirical analysis of image and language data, we demonstrate that small batch sizes do not confer any implicit bias advantages in online learning. In contrast to offline learning, the benefits of SGD noise in online learning are strictly computational, facilitating more cost-effective gradient steps. This suggests that SGD in the online regime can be construed as taking noisy steps along the "golden path" of the noiseless gradient descent algorithm. We study this hypothesis and provide supporting evidence in loss and function space. Our findings challenge the prevailing understanding of SGD and offer novel insights into its role in online learning.
Looking forward to talking in Sapienza University of Rome Tuesday June 20 on "The uneasy relation between deep learning and statistics"
https://di.uniroma1.it/it/notizie/seminari/distinguished-lectures
Excited about new work with Nikhil Vyas and Sham Kakade on formally defining and providing rigorous guarantees on potential copyright infringements by generative models. See the blog https://windowsontheory.org/2023/02/21/provable-copyright-protection-for-generative-models/
As we say in paper, there are many legal and ethical issues in generative models, and we are only focused on one of them. However I believe making a formal definition is important, also to ground future discussions.
RT @boazbaraktcs
Statement of concern by Jewish Scientists re proposed "reforms" to weaken the judiciary in Israel is now public. Please circulate it among colleagues, and (if appropriate) consider signing it yourself.
As Jewish scientists within the global science community, we have all felt great satisfaction and taken pride in Israel’s many remarkable accomplishments. We support and value the State of Israel, its pluralistic society, and its vibrant culture. Many of us have friends, family, and scientific
RT @XiXiDu@twitter.com
Your frequent reminder that the world could have defeated climate change in the 1980s if all industrialized nations had followed France's lead.
Teachers in Manatee County, Florida, are being told to make their classroom libraries — and any other "unvetted" book — inaccessible to students, or risk felony prosecution. The new policy is part of an effort to comply with new laws and regulations championed by Governor Ron DeSantis (R). It is based on the premise, promoted by right-wing advocacy groups, that teachers and librarians are using books to "groom" students or indoctrinate them with leftist ideologies.
Not me awkwardly cross-posting my tweets to Mastodon about my new GPT video:
Lance Fortnow
Omar Wasow
Judd Legum
Henry Farrell
karpathy