[mlpack] queries regarding the implementation of rnn.hpp in ann

[Nikhil Yadala]

Hi,

                           I have looked into,built, coded some algorithms
with  the mlpack's API(especially the ann module), in March .I have also
submitted a proposal (for GSoC) for the essential deep learning module, to
implement API for deep neural nets (RBM, RNN,CNN etc).How ever, I have not
got selected for the GSOC program, But I am still interested in
contributing to MLpack. I would like to know, if any one has  already been
working on implementing these modules(or had already implemented these
modules) either as a part of GSOC project or individually . So that I could
start working on this, If no one has started working on these. Would anyone
here mind suggesting me which network I should start working on?

Thanks,
Nikhil Yadala.

On Mon, Mar 14, 2016 at 7:18 PM, Marcus Edel <marcus.edel at fu-berlin.de>
wrote:

> Hello Nikhil,
>
> > Could you tell me what the variables  ,inputSize, outputSize , seqOut
> specify
>
>
> I'll go and explain the parameter using a simple example. Imagine you have
> a
> sensor, that returns 3 values every second. You record the values for 10
> seconds. Every data point has the same class as label.
>
> input = [4 3 9 5 8 9 1 2 5 9 0 0 0 0 8 ... ] labels = [1]
>
> You use a recurrent neural network which takes 3 values as input and
> returns two
> values (class 1, class 2) as output. Since you are using a recurrent neural
> network you feed the network with a sequence (10 seconds x 3 values). The
> first
> step is to figure out the 'inputSize' of the network, which is the number
> of
> input units (3). We use that parameter to feed the network with the correct
> input at time 't' so (t=0 [4 3 9]) (t=1 [5 8 9]) ...
>
> In this example, every input from the sequence belongs to the same class.
> But
> there are problems that require an output sequence. So you data could look
> like:
>
> input = [4 3 9 5 8 9 1 2 5 9 0 0 0 0 8 ... ] labels = [1 2 1 2 2 ...]
>
> So your input data at (t=0 input=[4 3 9]) (t=1 input=[5 8 9]) ... nothing
> has
> changed. But you expected output data changed (t=0 target=[1]) (t=1
> target=[2])
> (t=2 target=[1]) ..
>
> We use the 'outputSize' to use the right output at each time step.
>
> The 'seqOut' is true if the network is trained to output a sequence (case
> 2) or
> to predict a single class (case 1).
>
> > Also, As per what i understand regarding BPTT,each subsequence(time)
> (typically
> > k=3, at t-1,t,t+1) is considered one layer , and the rest all is similar
> to ffn
> > with a constraint that the weight matrix is same at every layer. But I
> dont
> > understand how the BPTT is implemented in rnn.hpp (If this is not the
> way , its
> > implemented here, could be please direct me to the link, where I could
> get a
> > better understanding of what BPTT does and how it does)
>
> You are right, you feed the network with the input at time t and perform a
> forward pass. At the end of each forward pass you have to save the
> activations
> for the backward pass. You also have to set the input parameter of the
> recurrent
> layer, which is basically the activation of the hidden layer.
>
> The backward pass is similar you first load the activations and then you
> can
> perform the normal backward pass with respect to the recurrent layer.
>
> > Regarding the project proposal, I am planning to implement bidirectional
> deep
> > rnn, So, that there is no need to code the double layer brnn explicitly.
> , and
> > also to give very less time to implement convolutional auto-encoder, as
> the
> > cnn,hpp does almost the same thing , the only tweak that has to be done
> is to
> > hard code the outputlayer to inputs(am I wright? ) .Could you please
> give your
> > views on these?
>
> You are right, you can use the rnn class and merge the two outputs at the
> end.
> It's also true, that the convolutional auto-encoder is straightforward to
> implement, but don't forget you have to write tests for each module.
>
> I hope this is helpful, let me know if I can clarify anything.
>
> Thanks,
> Marcus
>
>
> > On 14 Mar 2016, at 05:01, Nikhil Yadala <nikhil.yadala at gmail.com> wrote:
> >
> > Hi marcus,Ryan,
> >
> >                           I have gone through the complete code of ann,
> I don't get THE exact idea of how rnn is implemented in mlpack.I have a few
> queries
> >
> > Could you tell me what the variables  ,inputSize, outputSize , seqOut
> specify
> >
> > How is the output taken from the network, are we taking output after
> every time instance or is it that we take the output at the end of input
> (time) sequence.?
> >
> > Also, As per what i understand regarding BPTT,each subsequence(time)
> (typically k=3, at t-1,t,t+1) is considered one layer , and the rest all is
> similar to ffn with a constraint that the weight matrix is same at every
> layer. But I dont understand how the BPTT is implemented in rnn.hpp (If
> this is not the way , its implemented here, could be please direct me to
> the link, where I could get a better understanding of what BPTT does and
> how it does)
> >
> > Regarding the project proposal, I am planning to implement bidirectional
> deep rnn, So, that there is no need to code the double layer brnn
> explicitly. , and also to give very less time to implement convolutional
> auto-encoder, as the cnn,hpp does almost the same thing , the only tweak
> that has to be done is to hard code the outputlayer to inputs(am I wright?
> ) .Could you please give your views on these?
> >
> >
> > thanks,
> > Nikhil Yadala.
> > _______________________________________________
> > mlpack mailing list
> > mlpack at cc.gatech.edu
> > https://mailman.cc.gatech.edu/mailman/listinfo/mlpack
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.cc.gatech.edu/pipermail/mlpack/attachments/20160527/41a07c4f/attachment.html>