<p>I like that this uses the existing optimizer API, but a couple comments:</p>
<p>I think that <code>std::vector<arma::mat*></code> is a little bit awkward; could you possibly instead take the instantiated network as the input and extract/modify the parameters from there?</p>
<p>What does the function <code>Deriv(arma::mat const&, arma::mat&)</code> do? How is that different from <code>Gradient()</code>?</p>
<blockquote>
<p>Besides, the SoftmaxFunction need to cache the probabilities value, else you have to recalculate the probabilites two more times when fine tune the parameters, any suggestions?</p>
</blockquote>
<p>This is a common problem in the various optimizers; often an optimizer will make a call to <code>Evaluate()</code> followed directly by a call to <code>Gradient()</code> with the same parameters. I don't think this can be fixed unless we modify the API for the optimizers. One way to do this might be to add an <code>EvaluateWithGradient()</code> function that both calculates the objective and also calculates the gradient. It's relatively straightforward to use template metaprogramming inside of the optimizers in order to call either <code>Evaluate()</code> then <code>Gradient()</code> or <code>EvaluateWithGradient()</code> when available. So this way, the <code>EvaluateWithGradient()</code> function would be optional, but when supplied it could accelerate the computation. What do you think of this idea?</p>
<p>I can't comment on how well this proposal fits with the rest of the neural network code; I don't know if there are already any plans for fine tuning.</p>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br>Reply to this email directly or <a href="https://github.com/mlpack/mlpack/issues/458#issuecomment-145542827">view it on GitHub</a>.<img alt="" height="1" src="https://github.com/notifications/beacon/AJ4bFOOGWo58cb5WpYs8hZ1FTzohNGF2ks5o4n3JgaJpZM4GIjrs.gif" width="1" /></p>
<div itemscope itemtype="http://schema.org/EmailMessage">
<div itemprop="action" itemscope itemtype="http://schema.org/ViewAction">
<link itemprop="url" href="https://github.com/mlpack/mlpack/issues/458#issuecomment-145542827"></link>
<meta itemprop="name" content="View Issue"></meta>
</div>
<meta itemprop="description" content="View this Issue on GitHub"></meta>
</div>