Olivier Grisel

scikit-learn 1.2 will come with a new exact Newton/IRLS solver for binary logistic regression and Poisson/Gamma/Tweedie regression.

It can be much faster than LBFGS on datasets with n_samples>>n_features, in particular with sparse features (e.g. one-hot encoded categorical variables) with different rarity scales.

Full details in the following two PRs:

- https://github.com/scikit-learn/scikit-learn/pull/24767

- https://github.com/scikit-learn/scikit-learn/pull/24637

#pydata #scipy #ml #sklearn

ENH add newton-cholesky solver to LogisticRegression by lorentzenchr · Pull Request #24767 · scikit-learn/scikit-learn

Reference Issues/PRs Completes #16634. Follow-up of #24637. What does this implement/fix? Explain your changes. This adds the solver "newton-cholesky" to the classes LogisticRegression and Logistic...

GitHub
Nov 3, 2022 at 4:32pmWeb
Show threadJed BrownNov 3, 2022
@ogrisel Have you tried LBFGS with Jacobian scaling and Powell restart? It's my go-to for symmetric nonlinear PDE-type problems and I would expect it to transfer here, and provide significant speedups when the Cholesky factorization time starts to dominate your Newton costs.
Show threadOlivier GriselNov 3, 2022
@jedbrown no, do you have a good reference with more details?
Show threadJed BrownNov 4, 2022

@ogrisel This paper describes the method when used with preconditioned Krylov methods as an alternative to Jacobian lagging or JFNK. https://jedbrown.org/files/BrownBrune-LowRankQuasiNewtonRobustJacobianLagging-2013.pdf

The benefit is even larger if you use a sparse direct solver (as you do), but sparse direct is rarely tractable in my problem domain. In PETSc, this algorithm is -snes_type qn -snes_qn_scale_type jacobian. Powell restart is default.

Your benchmark looks pretty easy for Newton so this should only need one factorization.

Show threadEd SuominenNov 30, 2022

Excellent work, @ogrisel and colleagues. I’ve been exploring scikit-learn and have been very impressed with the quality and breadth of this amazing #python #ml package.

It has tons of features and—unlike some other packages I use and love—excellent documentation with a descriptive #API and helpful examples.

And all that with a #freesoftware MIT license!