[mlpack-git] [mlpack] evaluate various changes to L-BFGS optimizer (#370)

[Stephen Tu]

> One thing I overlooked, though---you've split out the line search termination conditions in order to give better failure messages; but on those failures, it no longer returns false, it just breaks. Is there a particular reason for this? I think that this could cause L_BFGS to iterate forever with no improvement, although the relative improvement termination condition should catch that. Which then implies that there's no need for LineSearch() to return anything at all or check whether or not it failed.

I think I actually needed `break` instead of `return false` for it to work. In my experience, it was `cond3` which was triggering, that is `numIterations >= maxLineSearchTrials`. In this case, I found that it was better to take a (non-zero) step that didn't satisfy the armijo conditions than to give up entirely on the optimization. In other words, it helps to get out of these strange valleys so that on subsequent iterations we can make good progress. I also tried to increase `maxLineSearchTrials`, but then the optimization become quite slow. You do bring up a valid point, which is what happens if the move becomes so small, could it loop infinitely? And that's why we have a relative difference condition, as you pointed it :)

I don't think, however, you should change `LineSearch()` to return `void`. There is still one condition that causes us to terminate entirely (the `initialSearchDirectionDotGradient > 0.0` check above). 

---
Reply to this email directly or view it on GitHub:
https://github.com/mlpack/mlpack/issues/370#issuecomment-70226842
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.cc.gatech.edu/pipermail/mlpack-git/attachments/20150116/3149bc99/attachment.html>