Samuel Vaiter

42 Followers
73 Following
20 Posts
CNRS Researcher in Optimization & Machine learning. Living in Nice, France.
Homepagehttps://samuelvaiter.com

The amount of benchmarks available in benchopt going ↗️ :

https://benchopt.github.io/

Feel free to mention new problems you would like to address, we would love to hear form you!

Interested in helping the optimization community to provide fair comparisons between solvers? Join us on github:

https://github.com/benchopt/benchopt

#benchopt #benchmarks #optimization

Great work with @tommoral @mathusmassias @agramfort @pablin @vaiter @tomdlt @jolars @matdag and many more.

Benchopt: Benchmark repository for optimization — benchopt 1.4.1.dev4 documentation

...look what showed up in the mail today. 🎉

It's about as thick as the previous editions but with slightly bigger pages so we could fit all the new stuff in. Very happy with the print quality.

#ICML2023 : "Papers that include text generated from a large-scale language model (LLM) such as ChatGPT are prohibited unless these produced text is presented as a part of the paper’s experimental analysis."

I'd be happy to know the detector planned to be used...

Learning without backpropagation is really taking off in 2022

First, @BAPearlmutter et al show in "Gradients without Backpropagation" that a single forward pass with perturbed weights is enough to compute unbiased estimate of gradients:
https://arxiv.org/abs/2202.08587

Then, Mengye Ren et al show in "Scaling Forward Gradient With Local Losses" that the variance of doing this is high, but can be reduced by doing activity perturbation (as in Fiete & Seung 2006), but more importantly, having many "local loss" functions:
https://arxiv.org/abs/2210.03310

Then Jeff Hinton takes the "local loss" to another level in "Forward-Forward Algorithm", and connects it to a ton of other ideas e.g. neuromorphic engineering, one shot learning, self supervised learning, ...: https://www.cs.toronto.edu/~hinton/FFA13.pdf

It looks like #MachineLearning and #Neuroscience are really converging.

Gradients without Backpropagation

Using backpropagation to compute gradients of objective functions for optimization has remained a mainstay of machine learning. Backpropagation, or reverse-mode differentiation, is a special case within the general family of automatic differentiation algorithms that also includes the forward mode. We present a method to compute gradients based solely on the directional derivative that one can compute exactly and efficiently via the forward mode. We call this formulation the forward gradient, an unbiased estimate of the gradient that can be evaluated in a single forward run of the function, entirely eliminating the need for backpropagation in gradient descent. We demonstrate forward gradient descent in a range of problems, showing substantial savings in computation and enabling training up to twice as fast in some cases.

arXiv.org
Happy new year!
The (temp?) ban of Paul Graham on twitter is the highlight of the shitstorm since Musk's takeover 😂

Nicolas Courty spoke about spoke about "Optimal transport for graphs: definitions, applications to graph-signal processing".
https://library.cirm-math.fr/Record.htm?idlist=1&record=19280725124910089079

Sophie Achard spoke about "Statistical comparisons of spatio-temporal networks".
https://library.cirm-math.fr/Record.htm?idlist=1&record=19280723124910089059

Haggai Maron spoke about "Subgraph-based networks for expressive, efficient, and domain-independent graph learning".
https://library.cirm-math.fr/Record.htm?idlist=1&record=19280739124910089119

Ulrike von Luxburg and Solveig Klepper spoke about "Clustering with tangles".
https://library.cirm-math.fr/Record.htm?idlist=1&record=19280731124910089139

CIRM - Videos & books Library - Optimal transport for graphs: definitions, applications to graph-signal processing

5 speakers of our workshop on "Signal Processing and Machine Learning on Graphs" in November at were recorded. We had the pleasure to listen to Nicolas Courty, Haggai Maron, Sophie Achard, Ulrike von Luxburg and Solveig Klepper.
Available on the CIRM video library: https://library.cirm-math.fr/ListRecord.htm?list=request&table=3&NumReq=115&cluster_1=2588&confirm=on

#machinelearning #graph #workshop

200 OK

Call for a 2Y postdoc position in data sciences anywhere in @ENS_ULM Retweet like there is no tomorrow! https://data-ens.github.io/jobs/
Laplace Postdoctoral Chair in Data Science - ENS-CFM Data Science Chair

ENS-CFM Data Science Chair

On Thursday I'll be at #NeurIPS2022 presenting a paper on our new system for #autodiff of implicit functions. A 🧵on the paper (https://arxiv.org/abs/2105.15183)
Efficient and Modular Implicit Differentiation

Automatic differentiation (autodiff) has revolutionized machine learning. It allows to express complex computations by composing elementary ones in creative ways and removes the burden of computing their derivatives by hand. More recently, differentiation of optimization problem solutions has attracted widespread attention with applications such as optimization layers, and in bi-level problems such as hyper-parameter optimization and meta-learning. However, so far, implicit differentiation remained difficult to use for practitioners, as it often required case-by-case tedious mathematical derivations and implementations. In this paper, we propose automatic implicit differentiation, an efficient and modular approach for implicit differentiation of optimization problems. In our approach, the user defines directly in Python a function $F$ capturing the optimality conditions of the problem to be differentiated. Once this is done, we leverage autodiff of $F$ and the implicit function theorem to automatically differentiate the optimization problem. Our approach thus combines the benefits of implicit differentiation and autodiff. It is efficient as it can be added on top of any state-of-the-art solver and modular as the optimality condition specification is decoupled from the implicit differentiation mechanism. We show that seemingly simple principles allow to recover many existing implicit differentiation methods and create new ones easily. We demonstrate the ease of formulating and solving bi-level optimization problems using our framework. We also showcase an application to the sensitivity analysis of molecular dynamics.

arXiv.org