[mlpack-git] [mlpack] Add leaky ReLUs (#412)

[Marcus Edel]

Unlike the standard ReL function the leaky rectified linear function has a non-zero gradient over it's entire domain. So instead of having `y = max(0, x)`, you have `y = max(x / a, x)`, where `a` is some constant. This means you still get some sort of non-linearity, but the gradient can flow through in both directions.

For more information see: 
* Andrew L. Maas, Awni Y. Hannun, Andrew Y. Ng, "Rectifier Nonlinearities Improve Neural Network Acoustic Models", 2014

Since the parameter is fixed we could add the leakyness factor as a template parameter. The problem with the idea is that C++ doesn't support double as template parameter. So, we need to figure out a way around this issue.

---
Reply to this email directly or view it on GitHub:
https://github.com/mlpack/mlpack/issues/412
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.cc.gatech.edu/pipermail/mlpack-git/attachments/20150301/eb357d9d/attachment.html>