[mlpack] [GSOC '16] : Student interested in Low rank/sparse optimization using Frank-Wolfe

[Kwon, Jin Kyoung]

> Ah, you should probably look at this documentation instead:
> http://mlpack.org/docs/mlpack-git/doxygen.php
> The other documentation was very old and I've removed it from the
> website.
> If you would like to document the OptimizerType and FunctionType policy
> classes in a document kind of like
> http://mlpack.org/docs/mlpack-git/doxygen.php?doc=metrics.html
> or http://mlpack.org/docs/mlpack-git/doxygen.php?doc=trees.html I'd be
> happy to take a look over a PR and work with you to get it into the main
> codebase.

I think this task is definitely something I'd be interested in! I'd love to learn Doxygen and writing annotated C++ files. To confirm, would the task be adding "optimizers.hpp" and "functions.hpp" under "doc/policies"? I generated local files by running "doxygen" command, and was wondering what our config file that doxygen uses to generate files. Is it the "Doxyfile" in the top directory? I see that it parses all *.hpp files, so I wouldn't need to change the config file, right?

> The neural network code can be found in src/mlpack/methods/ann/.  Looks
> like the changes happened here:
Thank you so much for pointing this out to me! The downloaded copy does not include ann module. Was it left out for a particular purpose? 

I also had a question regarding aug_lagrangian module. The optimizer was wrapped as a function (aug_lagrangian_function) to be usable by simple optimizers like L-BFGS (referring to comments on line 23-37 in aug_lagrangian_function.hpp). But aug_lagrangian.hpp uses L-BFGS optimizer (referring to commends on line 5-7 in aug_lagrangian.hpp), and there doesn't seem to be wrapper functions for lbfgs.hpp... Will you help me understand how do they work in conjunction with another? Also, do you see Frank-Wolfe optimizer being usable by other optimizers? If so, should wrapper classes for it be created? 

Thank you so much for your time again! 

Best Regards,
Jenna Kwon 







On 3/11/16, 7:09 PM, "Ryan Curtin" <ryan at ratml.org> wrote:

>On Fri, Mar 11, 2016 at 05:20:30PM +0000, Kwon, Jin Kyoung wrote:
>> Dear Ryan Curtin,
>> 
>> (This may be a duplicate - I sent the same email from my gmail
>> account, which I suspect got filtered by the mailing list.. so I am
>> resending it from my email registered w/ the mailing list) 
>> 
>> Thank you so much for your quick and thorough reply! I had some
>> follow-up questions, and would really appreciate if you could spend a
>> moment to help me understand the library better.
>> 
>> I saw that there are documentations for optimizers at
>> http://mlpack.org/doxygen/, albeit they are for V1.0.12. Would
>> documentation of optimizer architecture for V2.0.1 be of interest to
>> you as something I can do as a part of the project? Anyhow, looking
>> through doxygen still gave me a good idea of the architecture. I
>> understood that wrapping optimizers in *_function.hpp (e. g.
>> aug_lagrangian_function wrapping aug_lagrangian) enables it to be used
>> by other optimizers. What I noticed were the absenses of test
>> functions for some optimizers (e.g. sa, sgd). Is there a reason for
>> this? Also, is there a reason for naming tests under aug_lagrangian to
>> be “aug_lagrangian_test_functions” versus just “test_functions” (as
>> done under lbfgs and sgd)?
>
>Ah, you should probably look at this documentation instead:
>
>http://mlpack.org/docs/mlpack-git/doxygen.php
>
>The other documentation was very old and I've removed it from the
>website.
>
>If you would like to document the OptimizerType and FunctionType policy
>classes in a document kind of like
>http://mlpack.org/docs/mlpack-git/doxygen.php?doc=metrics.html
>or http://mlpack.org/docs/mlpack-git/doxygen.php?doc=trees.html I'd be
>happy to take a look over a PR and work with you to get it into the main
>codebase.
>
>I think the OptimizerType API is pretty straightforward and so is the
>FunctionType API, so it shouldn't be too much work.
>
>The test functions for the Augmented Lagrangian method are constrained,
>so, they are a little different than the other optimizer test functions.
>If you want to rename that file to test_functions.hpp, feel free.
>
>> I would also like to look at how the artificial neural network code
>> were modified to use the optimizers. This would be help me better
>> understand how to carry out your suggestion for modifying programs
>> like mlpack_softmax_regression to fit the Frank-Wolfe optimizer. From
>> which version to another did the nn code change? Also, where may I
>> find the nn code in the library? I suspect it is under
>> src/mlpack/methods/sparse_coding.. but I also saw that we used to have
>> ann module, which I do not see in the current version.
>
>The neural network code can be found in src/mlpack/methods/ann/.  Looks
>like the changes happened here:
>
>https://github.com/mlpack/mlpack/commit/69f97e50aeddfb83e708475606b74a05bee8034a#diff-c3a396ee194aa467a2e2441643b1aaf7
>https://github.com/mlpack/mlpack/commit/f6c27ed38633f732d1e8dd34e29e4be79b9d9f87#diff-c3a396ee194aa467a2e2441643b1aaf7
>https://github.com/mlpack/mlpack/commit/43f0f13fe84e11709b9d6254ad11827c6eaed80b#diff-c3a396ee194aa467a2e2441643b1aaf7
>
>> I do agree - Charles is a great lecturer! He hasn't mentioned mlpack -
>> what brought me here google queries for "georgia tech machine
>> learning", etc :) I do think he should include mlpack as a part of
>> "Suggest Software" under our class syllabus! We could suggest the
>> idea, as I suspect there must be C++ users in the class. As of now we
>> have an assignment on implementing optimizers such as randomized hill
>> climbing (RHC), MIMIC, SA and genetic algorithms (GA). Would
>> implementing RHC, MIMIC and GA be of future interests for mlpack?
>
>I can't seem to find the syllabus online, but I think it would be a good
>idea if you would suggest to him that mlpack should be put on the
>list... :)
>
>If you have the flexibility in your assignment to implement those
>optimizers to satisfy the mlpack OptimizerType interface, and if you
>write some tests for them, then I'd be happy to accept them into mlpack.
>It shouldn't be too hard to write tests for them.  Writing those
>optimizers for mlpack would probably require much more general code than
>what you'd write for the assignment, though.
>
>Thanks,
>
>Ryan
>
>-- 
>Ryan Curtin    | "Why is it that the landscape is moving... but the boat is
>ryan at ratml.org | still?"  - Train Driver