[mlpack-svn] [mlpack] evaluate various changes to L-BFGS optimizer (#370)

[Stephen Tu]

This is a continuation of the thread of discussion in #3. 

The following diff https://github.com/stephentu/mlpack/compare/mlpack:10d90d0...stephentu:8af98d5 allows the lovasz theta SDP test case to pass without removing the last edge that was causing the problem.

Essentially, the pattern of optimization looks like:

  * For the smaller values of sigma (e.g. sigma less than ~1000), the solution tends to R=0
  * As sigma is increased, I saw a discrepancy of behaviors. In the L-BFGS-B implementation, I saw it was able to climb out of the R=0 saddle point and converge towards the correct solution. However, in the mlpack L-BFGS implementation, I saw it get stuck in the saddle point no matter how large sigma got.

In my diff above, I added some logic to the mlpack optimizer to help guide it out of the saddle point. I'm not submitting this as a pull request yet because I'm not sure it is the right thing to do-- I'll defer that to Ryan who implemented L-BFGS in the first place. 

@rcurtin: To respond to your desire to keep out the external fortran L-BFGS-B dependency. That's fine by me if you feel very strongly about it (which you clearly do :smile:). But, I will point out that I was pushing for L-BFGS-B not because I think the specifics of the modified variant of the algorithm (versus what you have implemented) are necessary to get the SDPs to converge, but instead I was just looking for a fast, reliable black box with a self contained implementation which we could easily import. In other words, if you want to invest time in improving an in house optimization routine, I think it's better to go off of the existing implementation. 

Anyways, this patch should be enough for you to decide how to proceed with the optimizer. 

I'm going to turn my attention to writing a few more SDP test cases, and then building a higher level wrapper for matrix completion problems. 

---
Reply to this email directly or view it on GitHub:
https://github.com/mlpack/mlpack/issues/370
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.cc.gatech.edu/pipermail/mlpack-git/attachments/20150101/4e795059/attachment.html>