[mlpack-git] [mlpack] improve speed of SparseAutoencoder and make it more flexible (#451)

[stereomatchingkiss]

>Maybe you can provide more information about the problem with the existing layers?

No problem

1 : Not all of the layers are meaningful for SparseAutoencoder, like PoolingLayer, ConvolutionLayer, multiclass_classification_layer and so on. To make the users life easier, my initial idea is collect meaningful layers at another folder, this could reduce the chances of specify wrong layer type.

pros : 
a : The users would not need to bother about which layer is useable or scratch their head to understand the error messages generated at compile time or runtime
b : The layer could become simpler, since the member functions and data member can designed specifically for the SparseAutoencoder

cons : 
a : Generate duplicate codes

Another solution is using the type traits and static_assert to tell the users "This layer do not support by SparseAutoencoder".

    static_assert(IsValidLayerOfSE<HiddenLayer>, "this hidden layer do not support by SparseAutoencoder");
    static_assert(IsValidLayerOfSE<OutputLayer>, "this output layer do not support by SparseAutoencoder");

pros : 
a : reduce duplicate codes, you do not need to update two kind of similar layers 

cons : 
a :  the users need to deal with the compile time error messages, but I think is should be easy to read with the help of static_assert
b : the layer is a little bit complicated

2 : To make the dropout layer work, we need to add one more template parameter

    template <
        typename ActivationFunction = LogisticFunction,
        typename InputDataType = arma::mat,
        typename OutputDataType = arma::mat
    >
    class DropoutLayer{}

and call the activation after in the Forward function

    template<typename eT>
    void Forward(const arma::Mat<eT>& input, arma::Mat<eT>& output)
    {    
        // Scale with input / (1 - ratio) and set values to zero with probability
        // ratio.
        scale = 1.0 / (1.0 - ratio);      
        mask = arma::randu<arma::Mat<eT> >(input.n_rows, input.n_cols);
        mask.transform( [&](double val) { return (val > ratio); } );
        //following part could use tag dispatch to optimize, when the ActivationFunction
        //equal to EmptyFunction, it should called an empty function
        dropoutInput = input % mask * scale;
        ActivationFunction::fn(dropoutInput, output);
     }    

3 : The other problems of the is that the layout of the input is different with the SparseAutoencoder

SparseAutoencoder assume rows are features, cols are samples; but ann assume one arma::Mat<T> as a sample, and the whole 2D dimension of the arma::Mat<T> as features. However, DropoutLayer and BaseLayer do not affect by this.

Any suggestions?

---
Reply to this email directly or view it on GitHub:
https://github.com/mlpack/mlpack/pull/451#issuecomment-150390803
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.cc.gatech.edu/pipermail/mlpack-git/attachments/20151022/556e4ccb/attachment-0001.html>