<blockquote>
<p>this could mean lots of memory allocation if the dataset is of any significant size</p>
</blockquote>
<p>I agree with that, this solution is not memory efficient. </p>
<blockquote>
<p>we read a row at a time, and if one of the dimensions has to be mapped, then we need to just read through the file again, getting the right columns, and applying the mappings and updating the matrix we have</p>
</blockquote>
<p>This solution will save much more memory, drawbacks are slower speed(IO manipulation is expensive) and more complicated file manipulation. </p>
<p>I think the problem is, should we apply this optimization?Realistically, If the files are big, the users should not use CSV from the beginning.</p>
<blockquote>
<p>Although realistically, if the user wants fast loading, they should not be using CSV</p>
</blockquote>
<p>Agree</p>
<blockquote>
<p>If the matrix is not to be transposed, we only need to read through the row we are looking at</p>
</blockquote>
<p>This part already did, I only store the whole string into vector of vector when the users want to transpose the file.</p>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/mlpack/mlpack/pull/660#issuecomment-222888565">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe/AJ4bFPxIhCZ2H9v0yM73poE5IavYdmxdks5qHQgAgaJpZM4Iq0a1">mute the thread</a>.<img alt="" height="1" src="https://github.com/notifications/beacon/AJ4bFBAuyHtGz7PrdfzP9UMRsa2u8G-Hks5qHQgAgaJpZM4Iq0a1.gif" width="1" /></p>
<div itemscope itemtype="http://schema.org/EmailMessage">
<div itemprop="action" itemscope itemtype="http://schema.org/ViewAction">
<link itemprop="url" href="https://github.com/mlpack/mlpack/pull/660#issuecomment-222888565"></link>
<meta itemprop="name" content="View Pull Request"></meta>
</div>
<meta itemprop="description" content="View this Pull Request on GitHub"></meta>
</div>