[mlpack-svn] [MLPACK] #361: kernel pca unexpected behaviour

[MLPACK Trac]

#361: kernel pca unexpected behaviour
-----------------------+----------------------------------------------------
  Reporter:  ftrovato  |        Owner:     
      Type:  defect    |       Status:  new
  Priority:  major     |    Milestone:     
 Component:  mlpack    |   Resolution:     
  Keywords:            |     Blocking:     
Blocked By:            |  
-----------------------+----------------------------------------------------

Comment (by ftrovato):

 Hi Marcus,
 thanks for your detailed answer. I changed naive_method.hpp exactly as I
 found in the revision r17032.

 Concerning your commments, the only one I do not really understand well is
 that about the centering.
 What I see from my kernel_pca tests is that on the x-axis (first principal
 component) the plot is not centered, while it is on the y-axis (second
 PC).
 I am wondering, does centering means that final PCs are centered like in
 regular pca or not? To illustrate this I performed a fist test. I have
 attached two files: in "pca.png" you see the results of regular pca and in
 "kernel_linear_nystroem.png" you see the results of kernel pca (options -k
 liner -n -c) .
 Please note that I did not report the scatter plot of PC1 vs PC2 but
 instead I plotted the so called Free Energy Surface (FES), which I
 obtained form the PC1,Pc2 scatter plot by using the formula
 FES=-0.6ln(Prob(PC1,PC2)), where Prob(PC1,PC2) is the probability of
 finding PC1,PC2 (the last one evaluated calculating a 2D histogram).

 As you can see there is no centering with kernel pca.

 What I like in these plots is that they have very similar shapes. Probably
 it is not so evident because of the scales used, but changing the sign of
 the PC1 values of kernel_linear_nystroem.png does the trick. However the
 PC1 absolute values and center are different from pca.png.
 Is the change of sign connected to your previous observation? If yes could
 you send me the code to determine automatically the sign?

 I tried a second test, this time using the full kernel. I have attached
 the file "kernel_linear.png" to illustrate the results.

 1) Data are centered this time!
 2) There is no need to change the sign of the x-axis (PC1) since the
 orientation of the figure is the same as in the regular pca
 3) The scale of the axes are orders of magnitude different form pca and
 kernel pca using the nystroem method.

 Comments on point (3): although I can rescale the axes to get
 correspondence with the other methods (I guess that the subspace described
 by the rescaled PCs will remain invariant), is it expected that the
 absolute values differ by order of magnitudes in all 3 methods?


 I have performed a third test in which I used kernel_pca (revised version
 r17032) to see if the results changed by varying some parameters of the
 kernels.

 4) Using nystroem method: both with the polynomial and the laplacian
 kernels I see no differences when varying degree and bandwidth in the
 range 0.0001 to 1000. The output files do not differ.

 5) using the full kernel evaluation: with the laplacian kernel there are
 differences when using bandwidth=0.0001 or 1000. The absolute values are
 on the order of 0.01, 0.001 for both. I have attached the two files
 reporting only pc1 (first col) and pc2 (second col) for both bandwidths.
 Hopefully the long file names should be self-explanatory.

 I believe that (4) could be a bug, potentially.

 Concerning the very large differences in the absolute values of PC1 and
 PC2 found with the different methods (comparing pca with kernel_pca or the
 different kernel_pca), I have to say that I do not understand them. Are
 they expected?

 Thanks,
 Fabio

-- 
Ticket URL: <http://trac.research.cc.gatech.edu/fastlab/ticket/361#comment:3>
MLPACK <www.fast-lab.org>
MLPACK is an intuitive, fast, and scalable C++ machine learning library developed at Georgia Tech.