Boaz Barak

@boazbaraktcs@sigmoid.social
1.2K Followers
127 Following
142 Posts
Working on Theoretical Computer Science and Foundations of Machine Learning. Still also using twitter.
Home pagehttps://boazbarak.org
Bloghttps://windowsontheory.org/
Bird sitehttps://twitter.com/boazbaraktcs
Group pagehttps://mlfoundations.org/

A paperclip maximizer, a stochastic parrot, king Midas, a jpeg of the web, markets, democracy, an alien, a shoggoth,...

So many metaphors offered for AI. I discuss these and their limitations

https://windowsontheory.org/2023/06/28/metaphors-for-ai-and-why-i-dont-like-them/

Metaphors for AI, and why I don’t like them

Photo from National Photo Company Collection; see also (Sobel, 2017) [Cross posted in lesswrong and windows on theory see here for my prior writings]“computer, n. /kəmˈpjuːtə/. One who computes; a …

Windows On Theory

New paper on the implicit bias (or lack thereof) of stochastic gradient descent in the online learning setting https://arxiv.org/abs/2306.08590

We show that aside from the computational benefits, SGD noise (larger learning rate or smaller batches) do not offer implicit bias advantages.
With Nikhil Vyas, Depen Morwani, Rosie Zhao, Gal Kaplun, and Sham Kakade.

Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning

The success of SGD in deep learning has been ascribed by prior works to the implicit bias induced by finite batch sizes ("SGD noise"). While prior works focused on offline learning (i.e., multiple-epoch training), we study the impact of SGD noise on online (i.e., single epoch) learning. Through an extensive empirical analysis of image and language data, we demonstrate that small batch sizes do not confer any implicit bias advantages in online learning. In contrast to offline learning, the benefits of SGD noise in online learning are strictly computational, facilitating more cost-effective gradient steps. This suggests that SGD in the online regime can be construed as taking noisy steps along the "golden path" of the noiseless gradient descent algorithm. We study this hypothesis and provide supporting evidence in loss and function space. Our findings challenge the prevailing understanding of SGD and offer novel insights into its role in online learning.

arXiv.org

Looking forward to talking in Sapienza University of Rome Tuesday June 20 on "The uneasy relation between deep learning and statistics"

https://di.uniroma1.it/it/notizie/seminari/distinguished-lectures

Excited about new work with Nikhil Vyas and Sham Kakade on formally defining and providing rigorous guarantees on potential copyright infringements by generative models. See the blog https://windowsontheory.org/2023/02/21/provable-copyright-protection-for-generative-models/

As we say in paper, there are many legal and ethical issues in generative models, and we are only focused on one of them. However I believe making a formal definition is important, also to ground future discussions.

RT @boazbaraktcs
Statement of concern by Jewish Scientists re proposed "reforms" to weaken the judiciary in Israel is now public. Please circulate it among colleagues, and (if appropriate) consider signing it yourself.

https://sites.google.com/view/scientistsletter

Statement of concern

As Jewish scientists within the global science community, we have all felt great satisfaction and taken pride in Israel’s many remarkable accomplishments. We support and value the State of Israel, its pluralistic society, and its vibrant culture. Many of us have friends, family, and scientific

RT @XiXiDu@twitter.com

Your frequent reminder that the world could have defeated climate change in the 1980s if all industrialized nations had followed France's lead.

🐦🔗: https://twitter.com/XiXiDu/status/1617911860413751296

Alexander Kruel on Twitter

“Your frequent reminder that the world could have defeated climate change in the 1980s if all industrialized nations had followed France's lead.”

Twitter
Florida teachers told to remove books from classroom libraries or risk felony prosecution

Teachers in Manatee County, Florida, are being told to make their classroom libraries — and any other "unvetted" book — inaccessible to students, or risk felony prosecution. The new policy is part of an effort to comply with new laws and regulations championed by Governor Ron DeSantis (R). It is based on the premise, promoted by right-wing advocacy groups, that teachers and librarians are using books to "groom" students or indoctrinate them with leftist ideologies.

Popular Information
Here is a photo of a classroom library at Bayshore High School after the policy was announced.
https://www.chronicle.com/article/after-criticism-harvard-dean-reverses-course-on-fellowship-for-human-rights-leader "Elmendorf has emphasized that his decision was not made in an effort to limit debate at the Kennedy School. He said he plans to ask a faculty committee to develop a process to evaluate future appointments of fellows." They should have done this after the Chelsea Manning affair, and we'll see what the details of the putative committee arrangements are, but it's plausibly progress, and makes similar interventions much less likely in future.
After Criticism, Harvard Dean Reverses Course on Fellowship for Human-Rights Leader

Dean Douglas Elmendorf said he plans to ask a faculty committee to develop a process to evaluate future appointments of fellows.

The Chronicle of Higher Education

Not me awkwardly cross-posting my tweets to Mastodon about my new GPT video:

https://twitter.com/karpathy/status/1615398117683388417

Andrej Karpathy on Twitter

“🔥 New (1h56m) video lecture: "Let's build GPT: from scratch, in code, spelled out." https://t.co/2pKsvgi3dE We build and train a Transformer following the "Attention Is All You Need" paper in the language modeling setting and end up with the core of nanoGPT.”

Twitter