Francois Chollet
It's hard to beat a plain SGD optimizer that has a finely tuned learning rate schedule.
Dec 19, 2022 at 6:25amWeb
Show threadMike AndoDec 19, 2022
@fchollet Do you think this is also true for robustness metrics?
Show threadsamuelDec 19, 2022
@fchollet agree with the sprit. I never saw examples for this on Transformers, though. Adam (and it’s derivatives) just seem to be better there.
Show threadMat KelceyDec 19, 2022
@fchollet i did some weirds experiments with a batch size of 170K , and one of the 10 top tuner runs was SGD!