[mlpack-git] [mlpack] Add Dropout/DropConnect (#413)

[Marcus Edel]

Dropout is a recently introduced algorithm to prevent co-adaptation during training (overfitting). The key idea is to randomly drop units, along with their connections from a neural network during training. Routhly each element of a layer's output is kept with probability p, otherwise it's being set to 0.

For more information see:

* Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov, "Improving neural networks by preventing co-adaptation of feature detectors", 2012
* Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov, "Dropout: A Simple Way to Prevent Neural Networks from Overfitting", 2014

A simple way to implement the technique is to introduce a new function which creates a dropOutMask. Afterwards, we can multiply the dropOutMask with the inputActivation in all layers which should support dropout. Something like:

```
// dropout
if dropoutFraction > 0
{
	ActivationFunction::fn(inputActivation * dropOutMask, outputActivation);
}
else
{
	ActivationFunction::fn(inputActivation, outputActivation);
}
```
DropConnect is a generalization of Dropout that takes the idea a step further. Rather than zeroing each unit activations with probability p, it zeroes the weights/connections with probability p.

For more information see:

* Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, Rob Fergus, "Regularization of Neural Networks using DropConnect", 2013

The idea to implement the technique is similar, except that we need to introduce the feature to all connections which should support dropConnect. The modified code should look something like:

```
// dropConnect
if dropConnectFraction > 0
{
	outputLayer.InputActivation() += (weights * dropOutMask * input);
}
else
{
	outputLayer.InputActivation() += (weights * input);
}
```

---
Reply to this email directly or view it on GitHub:
https://github.com/mlpack/mlpack/issues/413
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.cc.gatech.edu/pipermail/mlpack-git/attachments/20150301/83313fdd/attachment.html>