<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;color:rgb(0,0,0)"><div class="gmail_default">Hi Ryan,</div><div class="gmail_default"><br></div><div class="gmail_default">I read the Hogwild! paper, which to my understanding, gives theoretical convergence guarantees for just parallelizing SGD without worrying about locking, etc in a shared memory model, if the data is large enough and updates happen atomically.</div><div class="gmail_default"><br></div><div class="gmail_default">I also went through your implementations of SGD and mini-batch SGD. I think it would be fairly easy to OpenMP-ize the current implementations along the lines of Hogwild.</div><div class="gmail_default"><br></div><div class="gmail_default">But, in my opinion, if we just use multi-threading, ml-pack might not be very attractive for researchers working with truly large-scale data. </div><div class="gmail_default"><br></div><div class="gmail_default">I think it would be a good idea if we could add support for GPU processing to the existing optimizers. I have prior experience working with CUDA and I think I would be able to add a CUDA version of Hogwild! built on the existing SGD implementations in ml-pack, over the summer. Such that researchers with little knowledge of CUDA can directly use ml-pack to speed their code, without worrying about what's under the hood (much like what Theano does for python).</div><div class="gmail_default"><br></div><div class="gmail_default">Another direction could be to add support for distributed computing, by linking ml-pack to either the Parameter Server by CMU (<a href="http://parameterserver.org">http://parameterserver.org</a>) or integrating the MPI based Parameter Server that I've built, and parallelizing the existing SGD and mini-batch code in ml-pack along the lines of Downpour SGD (similar to Tensor FLow and DistBelief systems developed by Googole).</div><div class="gmail_default"><br></div><div class="gmail_default">The distributed implementation would be a bit more complicated, but I think I should be able to do it over the summer, as that's exactly what the focus of my research is currently.</div><div class="gmail_default"><br></div><div class="gmail_default">I would love to know your thoughts and suggestions.</div><div class="gmail_default"><br></div><div class="gmail_default">Thank You.</div><div class="gmail_default"><br></div><div class="gmail_default">Best,</div><div class="gmail_default">Aditya</div></div></div><div hspace="streak-pt-mark" style="max-height:1px"><img style="width:0px;max-height:0px;overflow:hidden" src="https://mailfoogae.appspot.com/t?sender=aYWRpc2hhcm1hMDc1QGdtYWlsLmNvbQ%3D%3D&type=zerocontent&guid=f552ca67-ab53-4697-861f-81c21bf33e83"><font color="#ffffff" size="1">ᐧ</font></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 1, 2016 at 8:15 PM, Ryan Curtin <span dir="ltr"><<a href="mailto:ryan@ratml.org" target="_blank">ryan@ratml.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Tue, Mar 01, 2016 at 05:39:57PM +0530, Aditya Sharma wrote:<br>
> Hello,<br>
><br>
> I'm Aditya Sharma, a senior at Birla Institute of Technology and Science,<br>
> Pilani, India (BITS Pilani) in the final year of my Bachelor's degree in<br>
> Electrical and Electronics Engineering and I graduate in August 2016.<br>
><br>
</span><span class="">> Owing to my experience in the area, I am really interested in the '<br>
</span>> <<a href="https://github.com/mlpack/mlpack/wiki/SummerOfCodeIdeas#parallel-stochastic-optimization-methods" rel="noreferrer" target="_blank">https://github.com/mlpack/mlpack/wiki/SummerOfCodeIdeas#parallel-stochastic-optimization-methods</a>>Parallel<br>
> stochastic optimisation methods'<br>
> <<a href="https://github.com/mlpack/mlpack/wiki/SummerOfCodeIdeas#parallel-stochastic-optimization-methods" rel="noreferrer" target="_blank">https://github.com/mlpack/mlpack/wiki/SummerOfCodeIdeas#parallel-stochastic-optimization-methods</a>><br>
<span class="">> project<br>
> idea listed on the GSOC 2016 page of mlpack and would love to know more<br>
> details regarding the direction in which the community wants to go with<br>
> regards to this project.<br>
<br>
</span>Hi Aditya,<br>
<br>
This is a pretty open-ended project. The goal, of course, is to<br>
implement parallel stochastic optimization methods. The particular<br>
algorithms that are chosen are less important and they are up to the<br>
student. Hogwild! might be an interesting one to look into, for<br>
instance.<br>
<br>
mlpack has a nice interface for optimizers; take a look at the existing<br>
optimizers in src/mlpack/core/optimizers/ for some examples. Any<br>
optimizers implemented for the project should follow the same API, so<br>
that they work with many mlpack methods, like logistic regression, or<br>
softmax regression, or NCA, and so forth.<br>
<br>
I hope this is helpful; please let me know if I can clarify anything.<br>
<br>
Thanks!<br>
<span class="HOEnZb"><font color="#888888"><br>
Ryan<br>
<br>
--<br>
Ryan Curtin | "But feel, to the very end, the triumph of being<br>
<a href="mailto:ryan@ratml.org">ryan@ratml.org</a> | alive!" - Jöns<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><p dir="ltr"><font face="verdana, sans-serif">Aditya Sharma<br>Fourth Year Undergraduate<br>Department of Electrical and Electronics Engineering,<br>Birla Institute of Technology and Science, Pilani<br>Rajasthan, India - 333031</font></p><p><font face="verdana, sans-serif">WWW: <a href="http://adityasharma.space" target="_blank">http://adityasharma.space</a></font></p><p dir="ltr"><font face="verdana, sans-serif">E-mail: <a href="mailto:adisharma075@gmail.com" style="color:rgb(17,85,204)" target="_blank">adisharma075@gmail.com</a>, <a href="mailto:f2012075@pilani.bits-pilani.ac.in" style="color:rgb(17,85,204)" target="_blank">f2012075@pilani.bits-pilani.ac.in</a><br>LinkedIn: <a href="https://www.linkedin.com/in/adityabits" style="color:rgb(17,85,204)" target="_blank">https://www.linkedin.com/in/adityabits</a></font></p></div></div></div></div>
</div>