[mlpack-svn] [MLPACK] #253: MATLAB bindings, when installed, do not work
MLPACK Trac
trac at coffeetalk-1.cc.gatech.edu
Wed Oct 17 02:38:17 EDT 2012
#253: MATLAB bindings, when installed, do not work
-------------------------------+--------------------------------------------
Reporter: rcurtin | Owner: rcurtin
Type: defect | Status: new
Priority: major | Milestone: mlpack 1.1.0
Component: mlpack | Resolution:
Keywords: matlab, bindings | Blocking: 252
Blocked By: |
-------------------------------+--------------------------------------------
Comment (by rcurtin):
I am collecting information on this ticket since it is complex and
generally undocumented (thanks, !MathWorks!).
First,
[http://www.mathworks.com/matlabcentral/newsreader/view_thread/253412 this
MathWorks thread] (which has no !MathWorks commenters) suggests that
LD_LIBRARY_PATH changes within a MATLAB shell are not respected and
disregarded, which commenters James and Thomas Clark frustratedly point
out.
Second, the actual issue stems from using an unsupported compiler
according to MATLAB's
[http://www.mathworks.com/support/compilers/R2012a/glnxa64.html list of
supported compilers]. I won't comment on how I feel about how a product
released in March 2012 does not support a compiler version released in
April 2010 (that's the GCC 4.5.0 release), but I will say there are zero
valid excuses for this, especially with the fact that !MathWorks' income
in 2011 was $700 million
[http://www.mathworks.com/company/aboutus/?s_cid=wiki_mathworks_1
according to their own reports].
Third, one solution which is unacceptable for our purposes is to simply
remove the system libraries that ship with MATLAB in
${MATLAB_ROOT}/bin/glnxa64/, thereby forcing the loading of the correct
system libraries in the system default locations, as suggested repeatedly
in different threads such as [http://stackoverflow.com/questions/9959306
/how-to-tell-mex-to-link-with-the-libstdc-so-6-in-usr-lib-instead-of-the-
one-i here] and [http://synaptic-activity.blogspot.com/2011/05/glibcxx34
-not-found-problem-in-matlab.html here in Steve-o's marginally informed
blog]. However, this is Not A Good Idea (TM) and not something I am
willing to implement with CMake to solve this problem.
One of the threads I have read has suggested that the issue is caused
because mex is not linking first against the system libraries, and needs
some help to convince it to link against system libraries first with -L.
That's easy to do and I can show that the produced executable was linked
against the system library because the MATLAB-provided libstdc++.so.6 does
not contain GLIBCXX_3.4.15 yet both the produced .mexa64 and the system
libraries have it:
{{{
:[ ryan @ trevelyan ]: $ ldd -v matlab/emst.mexa64 | grep GLIBCXX_3.4.15
libstdc++.so.6 (GLIBCXX_3.4.15) => /usr/lib/x86_64-linux-
gnu/libstdc++.so.6
libstdc++.so.6 (GLIBCXX_3.4.15) => /usr/lib/x86_64-linux-
gnu/libstdc++.so.6
:[ ryan @ trevelyan ]: $ strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6
| grep GLIBCXX_3.4.15
GLIBCXX_3.4.15
:[ ryan @ trevelyan ]: $ strings /opt/matlab/bin/glnxa64/libstdc++.so.6 |
grep GLIBCXX_3.4.15
:[ ryan @ trevelyan ]: $
}}}
Okay, so I should be able to then start MATLAB, set LD_LIBRARY_PATH
smartly, and have success, right? No.
{{{
:[ ryan @ trevelyan ]: $ LD_LIBRARY_PATH="/usr/lib/x86_64-linux-
gnu:/lib/x86_64-linux-gnu" /opt/matlab/bin/matlab -nodesktop -nosplash
}}}
does not work when I then try to run the emst() function, and this does
not work either:
{{{
:[ ryan @ trevelyan ]: $ /opt/matlab/bin/matlab
/opt/matlab/bin/matlab: 1: /opt/matlab/bin/util/oscheck.sh:
/lib64/libc.so.6: not found
< M A T L A B (R) >
Copyright 1984-2012 The MathWorks, Inc.
R2012a (7.14.0.739) 64-bit (glnxa64)
February 9, 2012
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
>> x = rand(50,50);
>> setenv('LD_LIBRARY_PATH', ['/usr/lib/x86_64-linux-gnu:/lib/x86_64
-linux-gnu' getenv('LD_LIBRARY_PATH')])
>> emst(x)
Invalid MEX-file '/opt/matlab/toolbox/mlpack/mex_emst.mexa64':
/opt/matlab/bin/glnxa64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found
(required by
/home/ryan/work/fastlab/mlpack/trunk/build-matlab/lib/libmlpack.so.1)
Error in emst (line 36)
result = mex_emst(dataPoints', 1, parsed.leafSize);
>>
}}}
Okay, so what happens if we set an RPATH in the mex file to tell it
explicitly to look for the system libraries first?
{{{
:[ ryan @ trevelyan ]: $ sudo chrpath -r '/usr/lib/x86_64-linux-
gnu:/lib/x86_64-linux-gnu' mex_emst.mexa64
mex_emst.mexa64: RPATH=/home/ryan/work/fastlab/mlpack/trunk/build-
matlab/lib:
mex_emst.mexa64: new RPATH: /usr/lib/x86_64-linux-gnu:/lib/x86_64-linux-
gnu
:[ ryan @ trevelyan ]: $ /opt/matlab/bin/matlab -nodesktop -nosplash
/opt/matlab/bin/matlab: 1: /opt/matlab/bin/util/oscheck.sh:
/lib64/libc.so.6: not found
< M A T L A B (R) >
Copyright 1984-2012 The MathWorks, Inc.
R2012a (7.14.0.739) 64-bit (glnxa64)
February 9, 2012
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
>> x = rand(50,50);
>> emst(x)
Invalid MEX-file '/opt/matlab/toolbox/mlpack/mex_emst.mexa64':
/opt/matlab/bin/glnxa64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found
(required by /usr/lib/libmlpack.so.1)
Error in emst (line 36)
result = mex_emst(dataPoints', 1, parsed.leafSize);
}}}
And unsurprisingly the rpath is being ignored. This suggests that either
(a) MATLAB is doing something it isn't supposed to be doing with search
path ordering or (b) MATLAB is setting LD_LIBRARY_PATH to its own
conglomeration of directories and, as earlier, I seem to be unable to
prepend anything to that list via `getenv()`/`setenv()`. The interns must
have implemented this functionality in MATLAB because it is well known
that [http://xahlee.info/UnixResource_dir/_/ldpath.html LD_LIBRARY_PATH]
[https://blogs.oracle.com/ali/entry/avoiding_ld_library_path_the is]
[http://linuxmafia.com/faq/Admin/ld-lib-path.html bad]. And if you, dear
reader, are one of the !MathWorks programmers who made this design
decision, then I do not have words which convey what I feel about your
competency or general computer literacy.
Rants aside, let's try and find out what MATLAB is actually doing when it
loads libraries. First let's look at how it loads libmlpack.so using
strace.
{{{
[pid 27863] open("/usr/lib/x86_64-linux-gnu/libmlpack.so.1", O_RDONLY) =
-1 ENOENT (No such file or directory)
[pid 27863] open("/lib/x86_64-linux-gnu/libmlpack.so.1", O_RDONLY) = -1
ENOENT (No such file or directory)
[pid 27863] open("/opt/matlab/bin/glnxa64/libmlpack.so.1", O_RDONLY) = -1
ENOENT (No such file or directory)
[pid 27863]
open("/opt/matlab/bin/glnxa64/../../sys/os/glnxa64/libmlpack.so.1",
O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 27863] open("/opt/matlab/sys/os/glnxa64/libmlpack.so.1", O_RDONLY) =
-1 ENOENT (No such file or directory)
[pid 27863] open("/opt/matlab/bin/glnxa64/libmlpack.so.1", O_RDONLY) = -1
ENOENT (No such file or directory)
[pid 27863] open("/opt/matlab/extern/lib/glnxa64/libmlpack.so.1",
O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 27863] open("/opt/matlab/runtime/glnxa64/libmlpack.so.1", O_RDONLY) =
-1 ENOENT (No such file or directory)
[pid 27863]
open("/opt/matlab/sys/java/jre/glnxa64/jre/lib/amd64/native_threads/libmlpack.so.1",
O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 27863]
open("/opt/matlab/sys/java/jre/glnxa64/jre/lib/amd64/server/libmlpack.so.1",
O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 27863]
open("/opt/matlab/sys/java/jre/glnxa64/jre/lib/amd64/libmlpack.so.1",
O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 27863] open("/etc/ld.so.cache", O_RDONLY) = 288
[pid 27863] fstat(288, {st_mode=S_IFREG|0644, st_size=83769, ...}) = 0
[pid 27863] mmap(NULL, 83769, PROT_READ, MAP_PRIVATE, 288, 0) =
0x7f5705cdb000
[pid 27863] close(288) = 0
[pid 27863] access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file
or directory)
[pid 27863] open("/lib/x86_64-linux-gnu/libmlpack.so.1", O_RDONLY) = -1
ENOENT (No such file or directory)
[pid 27863] open("/usr/lib/x86_64-linux-gnu/libmlpack.so.1", O_RDONLY) =
-1 ENOENT (No such file or directory)
[pid 27863] open("/lib/libmlpack.so.1", O_RDONLY) = -1 ENOENT (No such
file or directory)
[pid 27863] open("/usr/lib/libmlpack.so.1", O_RDONLY) = 288
}}}
Interesting, so it's as if it actually checking `/usr/lib/x86_64-linux-
gnu` and `/lib/x86_64-linux-gnu` first, which is odd because I did not
specify LD_LIBRARY_PATH either from bash when MATLAB was started or use
`setenv()` inside MATLAB before calling `emst()`. For now I will regard
the fact that it checked those two directories as some artifact of my
setup that I have mangled or accidentally modified during my testing
process, and therefore that is perhaps not stock MATLAB behavior.
Regardless, if MATLAB checks those two directories for libstdc++,so, then
it should load the correct one because of the search order, right? Nope!
That's because libstdc++.so is already loaded by MATLAB so there is no
need to reload it. Wait, but what happens if we attempt to load MATLAB
when specifying LD_LIBRARY_PATH? Does it work then, when we use the
following command?
{{{
:[ ryan @ trevelyan ]: $ LD_LIBRARY_PATH="/usr/lib/x86_64-linux-
gnu:/lib/x86_64-linux-gnu" /opt/matlab/bin/matlab -nodesktop -nosplash
}}}
I grabbed the syscalls of all that with strace and parsing the output, we
get to see when MATLAB loads libstdc++.so.6 and the checking order it
uses.
{{{
:[ ryan @ trevelyan ]: $ grep 'libstdc++.so.6' matlab-out
27924 open("/opt/matlab/bin/glnxa64/libstdc++.so.6", O_RDONLY) = 3
}}}
Weird. It's as if it's ignoring the LD_LIBRARY_PATH. Oh wait. Because
it is. Opening /opt/matlab/bin/matlab we see it's not actually an
executable but a shell script which isn't preserving LD_LIBRARY_PATH but
instead setting it using a file called .matlab7rc.sh. Wonderful! In
fact, there is a `-n` option which the matlab script accepts that will
list all of the environment variables used to start the program. Which,
by the way, irritatingly invokes the pager, unnecessarily. This means
that some !MathWorks script kiddie took the time to write code to make
that happens, which better explains MATLAB's ignorance of standards if
they are spending their $700 million income hiring "programmers" who are
wasting their time finding automated ways to invoke tools whose invocation
should unarguably be at only the user's discretion. Nonetheless, let's
take a look.
{{{
:[ ryan @ trevelyan ]: $ LD_LIBRARY_PATH="/usr/lib/x86_64-linux-
gnu:/lib/x86_64-linux-gnu" /opt/matlab/bin/matlab -n
-> DIR = /opt/matlab/bin
------------------------------------------------------------------------
a = argument e = environment r = rcfile s = script
------------------------------------------------------------------------
-> r MATLAB = /opt/matlab
-> LM_LICENSE_FILE = (variable not defined)
-> MLM_LICENSE_FILE = (variable not defined)
-> s AUTOMOUNT_MAP =
-> e DISPLAY = :0.0
-> r ARCH = glnxa64
-> s TOOLBOX = /opt/matlab/toolbox
-> r XAPPLRESDIR = /opt/matlab/X11/app-defaults
-> r XKEYSYMDB = /opt/matlab/X11/app-defaults/XKeysymDB
-> e MAX_OPEN_FILES = 1024
-> s _JVM_THREADS_TYPE =
-> e MATLAB_JAVA =
-> s MATLAB_MEM_MGR =
-> s MATLAB_DEBUG =
-> rs LD_LIBRARY_PATH =
/opt/matlab/sys/os/glnxa64:/opt/matlab/bin/glnxa64:/opt/matlab/extern/lib/glnxa64:/opt/matlab/runtime/glnxa64:/opt/matlab/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/opt/matlab/sys/java/jre/glnxa64/jre/lib/amd64/server:/opt/matlab/sys/java/jre/glnxa64/jre/lib/amd64:/usr/lib/x86_64
-linux-gnu:/lib/x86_64-linux-gnu
-> a arglist =
-> e SHELL = /bin/bash
-> e PATH =
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
-> s MATLABPATH = (initial version)
/opt/matlab/toolbox/local
-> $MATLAB/bin/glnxa64/MATLAB shared library information -
|-----------------------------------------------------------------------
| linux-vdso.so.1 => (0x00007fff838a1000)
| libut.so => /opt/matlab/bin/glnxa64/libut.so (0x00007f59b0be3000)
| libmwfl.so => /opt/matlab/bin/glnxa64/libmwfl.so
(0x00007f59b08eb000)
| libmx.so => /opt/matlab/bin/glnxa64/libmx.so (0x00007f59b0631000)
| libmwmpath.so => /opt/matlab/bin/glnxa64/libmwmpath.so
(0x00007f59b03ee000)
| libmwservices.so => /opt/matlab/bin/glnxa64/libmwservices.so
(0x00007f59aff5b000)
| libmwjmi.so => /opt/matlab/bin/glnxa64/libmwjmi.so
(0x00007f59afcb3000)
| libmwbridge.so => /opt/matlab/bin/glnxa64/libmwbridge.so
(0x00007f59afa78000)
| libmwmcr.so => /opt/matlab/bin/glnxa64/libmwmcr.so
(0x00007f59af6f8000)
| libmwm_dispatcher.so =>
/opt/matlab/bin/glnxa64/libmwm_dispatcher.so (0x00007f59af45d000)
| libmwm_interpreter.so =>
/opt/matlab/bin/glnxa64/libmwm_interpreter.so (0x00007f59aebb3000)
| libmwi18n.so => /opt/matlab/bin/glnxa64/libmwi18n.so
(0x00007f59ae91e000)
| libstdc++.so.6 => /opt/matlab/bin/glnxa64/libstdc++.so.6
(0x00007f59ae60f000)
| libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f59ae38d000)
| libgcc_s.so.1 => /opt/matlab/bin/glnxa64/libgcc_s.so.1
(0x00007f59ae176000)
| libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f59adf5a000)
| libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f59adbd3000)
| librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
(0x00007f59ad9ca000)
| libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
(0x00007f59ad7c6000)
| libboost_date_time.so.1.44.0 =>
/opt/matlab/bin/glnxa64/libboost_date_time.so.1.44.0 (0x00007f59ad5b4000)
| libboost_system.so.1.44.0 =>
/opt/matlab/bin/glnxa64/libboost_system.so.1.44.0 (0x00007f59ad3b0000)
| libboost_thread.so.1.44.0 =>
/opt/matlab/bin/glnxa64/libboost_thread.so.1.44.0 (0x00007f59ad199000)
| libexpat.so.1 => /opt/matlab/bin/glnxa64/libexpat.so.1
(0x00007f59acf71000)
| libicudata.so.44 => /opt/matlab/bin/glnxa64/libicudata.so.44
(0x00007f59abf30000)
| libicuuc.so.44 => /opt/matlab/bin/glnxa64/libicuuc.so.44
(0x00007f59abbcc000)
| libicui18n.so.44 => /opt/matlab/bin/glnxa64/libicui18n.so.44
(0x00007f59ab7cf000)
| libicuio.so.44 => /opt/matlab/bin/glnxa64/libicuio.so.44
(0x00007f59ab5c1000)
| libtbb.so.2 => /opt/matlab/bin/glnxa64/libtbb.so.2
(0x00007f59ab482000)
| libtbbmalloc.so.2 => /opt/matlab/bin/glnxa64/libtbbmalloc.so.2
(0x00007f59ab359000)
| libmwresource_core.so =>
/opt/matlab/bin/glnxa64/libmwresource_core.so (0x00007f59ab156000)
| libmwMATLAB_res.so => /opt/matlab/bin/glnxa64/libmwMATLAB_res.so
(0x00007f59aac5a000)
.., (it just goes on and on)
}}}
Awesome, so our command-line specified LD_LIBRARY_PATH was, as I
suspected, entirely ignored. But this isn't the fault of the
.matlab7rc.sh file. In fact that file even contains hilarious meaningless
lines to preserve LD_LIBRARY_PATH, which work as expected.
{{{
if [ "$LD_LIBRARY_PATH" != "" ]; then
LD_LIBRARY_PATH=$LD_LIBRARY_PATH
else
LD_LIBRARY_PATH=
fi
}}}
Besides being a wonderful instance of apparent
[http://en.wikipedia.org/wiki/Cargo_cult_programming cargo cult
programming], we can verify that LD_LIBRARY_PATH is not annihilated at the
end of the sourcing of .matlab7rc.sh. So at least there's that. Looking
into the MATLAB script we find more wonderful cargo cult code:
{{{
LD_LIBRARY_PATH="`eval echo $LD_LIBRARY_PATH`"
}}}
and finally we discover that actually, the LD_LIBRARY_PATH we set on the
command line is being appended (*not* prepended) to the final
LD_LIBRARY_PATH the actual MATLAB executable gets. But even further
inspection reveals that the LDPATH_PREFIX variable gets prepended, so we
can just set that, right? Nope! That's annihilated by .matlab7rc.sh
without exception:
{{{
LDPATH_PREFIX=''
}}}
Cool, so we can comment that out and then set LDPATH_PREFIX on the command
line and then it will work, right? Well, sort of, but no. The following
output is from `matlab -n`:
{{{
...
-> $MATLAB/bin/glnxa64/MATLAB shared library information -
|-----------------------------------------------------------------------
| linux-vdso.so.1 => (0x00007fff346a7000)
| libut.so => /opt/matlab/bin/glnxa64/libut.so (0x00007fc08697f000)
| libmwfl.so => /opt/matlab/bin/glnxa64/libmwfl.so
(0x00007fc086687000)
| libmx.so => /opt/matlab/bin/glnxa64/libmx.so (0x00007fc0863cd000)
| libmwmpath.so => /opt/matlab/bin/glnxa64/libmwmpath.so
(0x00007fc08618a000)
| libmwservices.so => /opt/matlab/bin/glnxa64/libmwservices.so
(0x00007fc085cf7000)
| libmwjmi.so => /opt/matlab/bin/glnxa64/libmwjmi.so
(0x00007fc085a4f000)
| libmwbridge.so => /opt/matlab/bin/glnxa64/libmwbridge.so
(0x00007fc085814000)
| libmwmcr.so => /opt/matlab/bin/glnxa64/libmwmcr.so
(0x00007fc085494000)
| libmwm_dispatcher.so =>
/opt/matlab/bin/glnxa64/libmwm_dispatcher.so (0x00007fc0851f9000)
| libmwm_interpreter.so =>
/opt/matlab/bin/glnxa64/libmwm_interpreter.so (0x00007fc08494f000)
| libmwi18n.so => /opt/matlab/bin/glnxa64/libmwi18n.so
(0x00007fc0846ba000)
| libstdc++.so.6 => /opt/matlab/bin/glnxa64/libstdc++.so.6
(0x00007fc0843ab000)
| libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc084129000)
...
}}}
So it got `libm.so.6` right, but strangely not `libstdc++.so.6`. Why is
that? A look shows that bin/glnxa64/MATLAB has an RPATH set:
{{{
:[ ryan @ trevelyan ]: $ objdump -x glnxa64/MATLAB | grep RPATH
RPATH $ORIGIN:$ORIGIN/../../sys/os/glnxa64
}}}
and that is indeed our problem. GNU ld.so chooses to read RPATH first,
and LD_LIBRARY_PATH later, meaning we simply cannot override the RPATH
without removing it entirely. Sure enough, removing it causes the whole
thing to work.
This leaves us at an impasse. There are maybe two ways for us to fix the
problem:
* Strip the RPATH from the MATLAB executable, then set LD_LIBRARY_PATH by
hand.
* Possibly set /etc/ld.so.preload to force the system libstdc++.so.6 to
be loaded.
Because neither of these can be done without serious system modification,
which is far outside the scope of what MLPACK's CMake install scripts
should be doing, we cannot overcome this limitation. This is why most
distributions suggest the avoidance of RPATHs entirely, and this is a
great example of that. However, as has been shown repeatedly throughout
the course of this discussion (although in length it's getting closer to a
dissertation), !MathWorks clearly does not hire competent developers who
are in tune with the standards of the Linux community. Perhaps they would
do better on Windows -- if there was any serious standardization for
things like this, or clear documentation from Microsoft on how things
should be done -- but, oh wait, that's certainly not how things are.
Regardless, these poor design decisions mean that we realistically have
two solutions for MATLAB bindings to work properly:
1. Recompile libmlpack.so against the system libraries MATLAB provides,
then distribute that in ${MATLAB_ROOT}/libmlpack.so and use either
LD_LIBRARY_PATH hackery or an RPATH to ensure that the correct
libmlpack.so is loaded.
2. Force users to have a "MATLAB supported compiler" installed on their
system, and fail otherwise. That will ensure that the compiled mex file
does not contain symbols not present in the MATLAB distribution of system
libraries.
So, now that I've wasted hours of my life digging to the bottom of an
amateur spaghetti code expletivefest just to meet a dead-end caused by the
ignorance (possibly willful ignorance) of Linux system standards by
!MathWorks "professionals", I can go to sleep with just another anecdote
of how closed-source companies shit all over standards, don't communicate
effectively with their community to learn the needs of their userbase,
ignore critical bugs (I promise if I filed this it would never be touched,
at least in part because nobody is competent enough), do not operate
inside the open-source ecosystem, do not devote resources to
intercompatibility with other existing packages, reinvent the wheel
continually and poorly, and in general produce unpolished, half-finished,
buggy software that creates countless headaches for systems administrators
and users of their software, MATLAB, Mathematica, Maple, Xilinx ISE,
MathCAD, it's all the same. And they make money for this. At least with
open-source projects, we can file a bug which has a lower chance of being
routed directly to /dev/null.
--
Ticket URL: <http://trac.research.cc.gatech.edu/fastlab/ticket/253#comment:1>
MLPACK <www.fast-lab.org>
MLPACK is an intuitive, fast, and scalable C++ machine learning library developed by the FASTLAB at Georgia Tech under Dr. Alex Gray.
More information about the mlpack-svn
mailing list