[mlpack-svn] [MLPACK] #376: CosineTreeTest/CosineNodeCosineSplit test sometimes fails, only on i386

[MLPACK Trac]

#376: CosineTreeTest/CosineNodeCosineSplit test sometimes fails, only on i386
---------------------+------------------------------------------------------
 Reporter:  rcurtin  |        Owner:                                       
     Type:  defect   |       Status:  new                                  
 Priority:  minor    |    Milestone:  mlpack 1.1.0                         
Component:  mlpack   |     Keywords:  cosine tree, cosinenode, cosine, test
 Blocking:           |   Blocked By:                                       
---------------------+------------------------------------------------------
 I have noticed in the nightly matrix build
 (http://big.cc.gt.atl.ga.us/job/mlpack%20-%20nightly%20matrix%20build/)
 that the test `CosineTreeTest/CosineNodeCosineSplit` sometimes fails, but
 only on i386:

 {{{
 [Error] - check cosineMax - cosines(i) < cosines(i) - cosineMin failed
 [0.2321881278243989 >= 0.050073702602425718]
  == [File] - /home/jenkins/workspace/mlpack - nightly matrix
 build/arch/linux-i386/armadillo/armadillo-4.400.1/buildmode/release/repo/src/mlpack/tests/cosine_tree_test.cpp
  == [Line] - 112

 [Error] - check cosineMax - cosines(i) < cosines(i) - cosineMin failed
 [0.23254797225922919 >= 0.049713858167595437]
  == [File] - /home/jenkins/workspace/mlpack - nightly matrix
 build/arch/linux-i386/armadillo/armadillo-4.400.1/buildmode/release/repo/src/mlpack/tests/cosine_tree_test.cpp
  == [Line] - 112

 [Error] - check cosineMax - cosines(i) < cosines(i) - cosineMin failed
 [0.24496212097773529 >= 0.037299709449089336]
  == [File] - /home/jenkins/workspace/mlpack - nightly matrix
 build/arch/linux-i386/armadillo/armadillo-4.400.1/buildmode/release/repo/src/mlpack/tests/cosine_tree_test.cpp
  == [Line] - 112

 [Error] - check cosineMax - cosines(i) < cosines(i) - cosineMin failed
 [0.24307624422077645 >= 0.039185586206048173]
  == [File] - /home/jenkins/workspace/mlpack - nightly matrix
 build/arch/linux-i386/armadillo/armadillo-4.400.1/buildmode/release/repo/src/mlpack/tests/cosine_tree_test.cpp
  == [Line] - 112

 [Error] - check cosineMax - cosines(i) < cosines(i) - cosineMin failed
 [0.23793054672682101 >= 0.04433128370000361]
  == [File] - /home/jenkins/workspace/mlpack - nightly matrix
 build/arch/linux-i386/armadillo/armadillo-4.400.1/buildmode/release/repo/src/mlpack/tests/cosine_tree_test.cpp
  == [Line] - 112
 }}}

 To reproduce it, do this:

  * Get an i386 system (a VM should work)
  * Modify cosine_tree_test.cpp and add the following to the beginning of
 CosineTreeTest:

 {{{
 const size_t seed = std::time(NULL);
 math::RandomSeed(seed);
 std::cout << "Seed: " << seed << std::endl;
 }}}

  * Now compile and run the test in a bash loop, to find a random seed that
 causes the issue:

 {{{
 $ while(true); do bin/mlpack_test -t CosineTreeTest/CosineNodeCosineSplit;
 sleep 1; done
 }}}

 Then you can hardcode the failing random seed for the purposes of
 debugging.

 The cosine tree is built by choosing a basis vector (that is, a point from
 the dataset), and calculating the cosine distance between that basis
 vector and all other points.  Points with large cosines go to the left
 child, and points with small cosines go to the right child; the current
 implemented split is (I think) the median cosine -- this results in a tree
 where the left and right child have the same number of points.  This
 process is repeated iteratively.  (The details are in the paper ``QUIC-
 SVD: Fast SVD Using Cosine Trees`` by Michael Holmes, Charles Isbell, and
 Alex Gray; I've only described it here just to give an idea.)

 The tests appear to be failing because in some cases, points in the right
 child appear to have larger cosine to the basis vector than points in the
 left child.  I've checked for memory leaks and other issues with valgrind
 that may only show up on i386 and fixed what I found (missing destructor)
 in r17485, but this did not fix the issue.

 So, the bug may be either in the cosine tree code or the test itself; I
 don't know enough to say.  But anyone wishing to solve this bug should
 spend a little time understanding the basics of the cosine tree, debugging
 what (or if) there is a problem for cosine trees that fail the test, and
 at that point maybe the solution will be clear.  For the 1.0.11 release
 I've commented the test out and noted that the bug is present (r17488).

 Siddharth, I've CC'ed you just in case my description is incorrect or
 anything like that.  If you have thoughts, feel free, but if not, don't
 feel obligated.  I would call this relatively low priority since i386 is
 far less important these days. :)

-- 
Ticket URL: <http://trac.research.cc.gatech.edu/fastlab/ticket/376>
MLPACK <www.fast-lab.org>
MLPACK is an intuitive, fast, and scalable C++ machine learning library developed at Georgia Tech.