[mlpack-git] [mlpack] evaluate various changes to L-BFGS optimizer (#370)

[Ryan Curtin]

This looks good; a relative termination condition is a good improvement, in my opinion.  So for the sake of understanding, it seems to me that based on what you've said, L_BFGS gets stuck in some kind of valley with small gradient (but not sufficiently small to terminate), and "walks" down this valley to the R=0 saddle point despite the fact that the objective function improvement at each iteration is very small.

This would imply that either increasing the gradient norm tolerance, or adding a relative objective function improvement termination criterion (wow that's a long set of strung-together nouns), would allow LRSDP to avoid that saddle point.

Just one very minor question before I merge; where does the `factr` name come from?

Your observations with LRSDP and mine in the past seem to further suggest that it's difficult to make LRSDP converge, and we may not be easily able to provide an LRSDP implementation that always converges.

---
Reply to this email directly or view it on GitHub:
https://github.com/mlpack/mlpack/issues/370#issuecomment-69988540
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.cc.gatech.edu/pipermail/mlpack-git/attachments/20150114/a0efd6c0/attachment.html>