[mlpack] disadvantages of using tuple as framework for neural network

[Shangtong Zhang]

Hi,


I'm not sure about the number of connection. Maybe it is a better idea to write
a new layer type and a new connection. Which will reduce the problem to 5 layer.


The structure of LeNet5:
With an input layer, 6 convolutional kernel is used to generate 6 feature maps(C1).
I modled each feature map as a layer.
then apply pooling on each feature maps in C1
and get 6 feature maps in S2 (denoting them as S2_1, S2_2, …, S2_6)
And C3 has 16 feature maps (denoting them as C3_1, C3_2, …., C3_16).
There are many kinds of connection strategies between S2 and C3.
Maybe C3_1 is connected with S2_2 only, but C3_1 may also be connected with all 6 feature maps in S2.
I learn that an implementation in theano use full connections.
I just want to imitate it so 6 * 16 = 96 connections is need.
I can’t design one ConnectionType to include all these connections,
as the number of connections and the structure of connections are uncertain.


Best
Regards


Shangtong Zhang,
Third Year Undergraduate,
School of Computer Science,
Fudan University, China.


原始邮件
发件人:Marcus Edelmarcus.edel at fu-berlin.de
收件人:Ryan Curtinryan at ratml.org
抄送:Shangtong Zhangzhangshangtong.cpp at qq.com; mlpackmlpack at cc.gatech.edu
发送时间:2015年2月10日(周二) 05:58
主题:Re: [mlpack] disadvantages of using tuple as framework for neural network


Hello,


Using a base class for all ConnectionType classes would mean virtual
functions, which in this context would incur non-negligible slowdown
(since the ConnectionType's functions will be used so much).


This is absolutely right, using inheritance would results in a measurable
slowdown, which is non-negligible. I couldn't think of another solution as using
tuples in combination with variadic templates to support different layers. Which
doesn't mean there isn't a better solution.


But it seems I can't generate it automatically
or I need some very complicate c++ technique to do this.
http://stackoverflow.com/questions/28410697/c-convert-vector-to-tuple


I think Barry's solution is quite neat, I've done something similar in the RNN
and FFN class. Originally there was the plan to write a 'BuildNetwork' function
which simply creates a network with k units and n layer (layer with the same
type), but I couldn't find the time to write such a function. However, the problem
with the idea is that often we need a different layer type between two layer. Maybe
it makes more sense to write a function which generates a block of layers, like
you asked at stackoverflo. What do you think?


This means there are 6 * 16 = 96 connections,
then I need to instantiates 96 class of ConnectionType and store them in a tuple.
so I need to get a tuple like c1, c2, c3, ………, c96, it’s too large that I can’t write it manually.


I'm not sure about the number of connection. Maybe it is a better idea to write
a new layer type and a new connection. Which will reduce the problem to 5 layer.


Thank you for according time and effort into the code,


Marcus






On 09 Feb 2015, at 17:36, Ryan Curtin ryan at ratml.org wrote:


On Mon, Feb 09, 2015 at 10:17:39PM +0800, Shangtong Zhang wrote:

Hi,


I implement cnn based on current framework of NN.
I meet some problem when I try to test my cnn on mnist dataset.
To do classification on mnist dataset, I try to realize LeNet5,
but LeNet5 has so many connections.
In one layer, there are 6 feature maps, and in next layer there are 16 feature maps.
And an implementation in theano connects these two layers use full connection.
This means there are 6 * 16 = 96 connections,
then I need to instantiates 96 class of ConnectionType and store them in a tuple.
so I need to get a tuple like c1, c2, c3, ………, c96, it’s too large that I can’t write it manually.
But it seems I can't generate it automatically
or I need some very complicate c++ technique to do this.
http://stackoverflow.com/questions/28410697/c-convert-vector-to-tuple

So I think tuple isn't appropriate for this work.
I suggest that we use vector to replace tuple and have a base class for all ConnectionType class.
In this way, ConnectionTraits may also be not used.


Using a base class for all ConnectionType classes would mean virtual
functions, which in this context would incur non-negligible slowdown
(since the ConnectionType's functions will be used so much).

Although I agree that making a std::tuple with 96 objects in it is
unwieldy, we should be careful to consider the drawbacks of inheritance
in a library that focuses on speed.

I've barely looked at the code in methods/ann/, so I certainly don't
have any better ideas, but I at least wanted to point out this
perspective.

--
Ryan Curtin | "You got to stick with your principles."
ryan at ratml.org| - Harry Waters
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.cc.gatech.edu/pipermail/mlpack/attachments/20150210/05d10a48/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1365562321_2400.jpg
Type: image/jpeg
Size: 34032 bytes
Desc: not available
URL: <https://mailman.cc.gatech.edu/pipermail/mlpack/attachments/20150210/05d10a48/attachment-0001.jpg>