MATLAB-MEX-CULA failure

Support for issues specific to the Linux operating systems.

MATLAB-MEX-CULA failure

Postby vogel » Mon Jan 10, 2011 12:41 pm

Operating System: Linux UBUNTU 10.04
Kernel Linux 2.6.32-27-generic
CUDA Version: 3.2
GPU Model: GTX-480
MATLAB Version 7.11.0.584 (R2010b)

I'm trying to compile and run MATLAB-MEX code that calls CULA. The sample code below performs no function other than to call the CULA initialization function culaInitialize. It gives me the following message:
CUDA runtime or driver is not supported

I've tried various mex/compiler options. The options that gave me the above message are in the comments section of the listing below. The code still compiles if I drop the CUDA library link and the '-lcuda', but I get the same initialization error message.


// testCula.cpp -- call to CULA initialization function from MATLAB-MEX
//
// MATLAB-MEX compilation and linking is implemented as follows:
//
// >> mex -I/usr/local/cula/include -L/usr/local/cula/lib64 -lcula ...
// -L/usr/local/cuda/lib64 -lcudart testCula.cpp

#include "mex.h"
#include <culapack.h>

void checkStatus(culaStatus status);

// MATLAB Gateway Function
void mexFunction(int nlhs, mxArray* plhs[], int nrhs, const mxArray* prhs[])
{

// CULA initialization
culaStatus status = culaInitialize();

// print out status
char buf[80];
culaGetErrorInfoString(status, culaGetErrorInfo(), buf, sizeof(buf));
mexPrintf("%s\n", buf);

}
vogel
 
Posts: 7
Joined: Wed Jan 05, 2011 2:24 pm

Re: MATLAB-MEX-CULA failure

Postby john » Tue Jan 11, 2011 11:23 am

Which CULA version are you running? And (please be precise), which CUDA toolkit and which CUDA driver are you running? Both of the latter must be 3.2 final. Version 260.19.26 or greater for the driver and 3.2.16 for the CUDA toolkit.

Also if you're only compiling in host CULA functionality, your command line can be shortened to:
mex -I/usr/local/cula/include -L/usr/local/cula/lib64 -lcula testCula.cpp
john
Administrator
 
Posts: 587
Joined: Thu Jul 23, 2009 2:31 pm

Re: MATLAB-MEX-CULA failure

Postby vogel » Tue Jan 11, 2011 2:07 pm

Hi John,

I'm using CULA R10 (December 13, 2010) Linux 64-bit.

I'm also using NVIDIA driver version 260.19.26, which I just downloaded and installed (I had been running version 260.19.12) and CUDA toolkit version 3.2. Installing the new driver made no difference. When I compile testCula.cpp with your suggested (shortened) mex script, I still get the output

>> testCula
CUDA runtime or driver is not supported

Incidentally, I can successfully compile and run the stand-alone c-language demo code gesvd.c. The problem seems to be specific to MATLAB and the way it interfaces with CUDA and CULA.

--Curt
vogel
 
Posts: 7
Joined: Wed Jan 05, 2011 2:24 pm

Re: MATLAB-MEX-CULA failure

Postby kyle » Thu Jan 13, 2011 9:42 am

Are you using MATLAB's parallel computing toolbox by chance?
kyle
Administrator
 
Posts: 301
Joined: Fri Jun 12, 2009 7:47 pm

Re: MATLAB-MEX-CULA failure

Postby vogel » Thu Jan 13, 2011 10:05 am

The MATLAB Parallel Computing Toolbox is installed on my machine. I'm not using it (yet) in the CUDA/CULA code that I've written, but the toolbox is not disabled. If this is the problem, can I disable it without uninstalling it?
vogel
 
Posts: 7
Joined: Wed Jan 05, 2011 2:24 pm

Re: MATLAB-MEX-CULA failure

Postby john » Thu Jan 13, 2011 10:29 am

Not sure, but it's likely that your program is loading Matlab's CUDA runtime, which is (I believe) older than the one CULA requires. CULA R10 is based on CUDA 3.2, and there isn't great backwards compatibility for CUDA programs. You might have some success with an older version of CULA, but performance and features will suffer. It might be worth disabling or removing the Matlab toolbox if you're not using it.
john
Administrator
 
Posts: 587
Joined: Thu Jul 23, 2009 2:31 pm

Re: MATLAB-MEX-CULA failure

Postby vogel » Mon Jan 24, 2011 2:07 pm

I just uninstalled the MATLAB distributed computing toolbox. This made no difference. I'm getting the same error.
vogel
 
Posts: 7
Joined: Wed Jan 05, 2011 2:24 pm

Re: MATLAB-MEX-CULA failure

Postby Boxed Cylon » Tue Jan 25, 2011 10:28 am

The test code you posted above worked for me without a problem, using the test compile line you provided. I am using Suse linux 11.1 and matlab R2010b. I'm wondering if it isn't a library search path issue - the $LD_LIBRARY_PATH variable. It may also be that your latest version of matlab includes some cuda libraries that are conflicting with the executable - move them out of the way; or I believe the $LD_LIBRARY_PATH variable takes precedence.

Executing:
>> !ldd testCula.mexa64
will tell you where matlab is getting its libraries from, I believe.
Boxed Cylon
 
Posts: 48
Joined: Fri Oct 16, 2009 8:57 pm

Re: MATLAB-MEX-CULA failure

Postby kyle » Tue Jan 25, 2011 12:07 pm

Thanks for the help, Boxed Cylon. We unfortunately do not have MATLAB 2010b in house so we cannot reproduce this exact condition.
kyle
Administrator
 
Posts: 301
Joined: Fri Jun 12, 2009 7:47 pm

Re: MATLAB-MEX-CULA failure

Postby vogel » Wed Jan 26, 2011 3:18 pm

I ran ldd on both the mex binary that's giving me the error and on a C-language binary, generated from the svd example you provided on your website.

Here is the result for my mex binary:

desktop:~/CUDA-LinearAlgebra/svdCode/mex$ ldd testCula.mexa64
linux-vdso.so.1 => (0x00007fffe84d7000)
libcula.so => /usr/local/cula/lib64/libcula.so (0x00007f3c7e4ab000)
libmx.so => not found
libmex.so => not found
libmat.so => not found
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f3c7e17b000)
libm.so.6 => /lib/libm.so.6 (0x00007f3c7def7000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f3c7dce0000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f3c7dac3000)
libc.so.6 => /lib/libc.so.6 (0x00007f3c7d73f000)
libcublas.so.3 => /usr/local/cula/lib64/libcublas.so.3 (0x00007f3c78364000)
libcudart.so.3 => /usr/local/cula/lib64/libcudart.so.3 (0x00007f3c78103000)
libgomp.so.1 => /usr/lib/libgomp.so.1 (0x00007f3c77ef4000)
librt.so.1 => /lib/librt.so.1 (0x00007f3c77cec000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3c8004c000)
libdl.so.2 => /lib/libdl.so.2 (0x00007f3c77ae7000)

and here is the result for the C-language binary, which runs without error:

desktop:~/CUDA-LinearAlgebra/svdCode/CCodes$ ldd gesvd
linux-vdso.so.1 => (0x00007fffd2bff000)
libcula.so => /usr/local/cula/lib64/libcula.so (0x00007f15f5469000)
libcublas.so.3 => /usr/local/cula/lib64/libcublas.so.3 (0x00007f15f008e000)
libcudart.so.3 => /usr/local/cula/lib64/libcudart.so.3 (0x00007f15efe2c000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f15efbf4000)
libc.so.6 => /lib/libc.so.6 (0x00007f15ef871000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f15ef55c000)
libgomp.so.1 => /usr/lib/libgomp.so.1 (0x00007f15ef34e000)
librt.so.1 => /lib/librt.so.1 (0x00007f15ef146000)
libm.so.6 => /lib/libm.so.6 (0x00007f15eeec2000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f15eecab000)
/lib64/ld-linux-x86-64.so.2 (0x00007f15f6e08000)
libdl.so.2 => /lib/libdl.so.2 (0x00007f15eeaa7000)

As far as I can tell, in both binaries libcudart and libcublas point to the same the same places in /usr/local/cula/lib64.
vogel
 
Posts: 7
Joined: Wed Jan 05, 2011 2:24 pm

Re: MATLAB-MEX-CULA failure

Postby Boxed Cylon » Wed Jan 26, 2011 3:26 pm

Ah - I meant that ">> !ldd xxx" should be done from within matlab.... The theory is that matlab's libraries may be superceding the ordinary cuda/cula libraries...
Boxed Cylon
 
Posts: 48
Joined: Fri Oct 16, 2009 8:57 pm

Re: MATLAB-MEX-CULA failure

Postby vogel » Wed Jan 26, 2011 8:45 pm

Looks like this is it. Thanks for your help!

Below are the 1st 3 lines from ldd testCula.mexa64, when invoked from MATLAB.

>> !ldd testCula.mexa64
linux-vdso.so.1 => (0x00007fffcd1ff000)
libcula.so => /usr/local/cula/lib64/libcula.so (0x00007f87bbfe2000)
libcudart.so.3 => /usr/local/MATLAB/R2010b/bin/glnxa64/libcudart.so.3 (0x00007f87bbdae000)

/usr/local/MATLAB/R210b/bin/glnxa64/libcudart.so.3 points to /usr/local/MATLAB/R2010b/bin/glnxa64/libcudart.so.3.1.12. If I rename the latter file, my testCula code compiles and runs correctly.

Here is the load library path:
echo $LD_LIBRARY_PATH
/usr/local/cula/lib64:/usr/local/cuda/lib64:

MATLAB is apparently overriding $LD_LIBRARY_PATH. Any suggestions on more benign ways of taking care of this problem? I had installed the MATLAB Distributed Computing Toolbox on my system (I uninstalled it as part of the debugging process). I'm concerned that changing a file name in the MATLAB bin directory may have undesirable consequences in the future. Also, this fix is likely to be wiped out when I update MATLAB to version 2011a.
vogel
 
Posts: 7
Joined: Wed Jan 05, 2011 2:24 pm

Re: MATLAB-MEX-CULA failure

Postby Boxed Cylon » Wed Jan 26, 2011 10:46 pm

Try:

export LD_PRELOAD=/usr/local/cula/lib64:/usr/local/cuda/lib64

before starting matlab - this will likely fix the problem, maybe...
Boxed Cylon
 
Posts: 48
Joined: Fri Oct 16, 2009 8:57 pm

Re: MATLAB-MEX-CULA failure

Postby vogel » Thu Jan 27, 2011 9:57 am

This didn't work; see error message below.

[1] 3471
ERROR: ld.so: object '/usr/local/cula/lib64' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object '/usr/local/cuda/lib64' from LD_PRELOAD cannot be preloaded: ignored.
vogel
 
Posts: 7
Joined: Wed Jan 05, 2011 2:24 pm

Re: MATLAB-MEX-CULA failure

Postby Boxed Cylon » Thu Jan 27, 2011 10:04 am

Alas, I can't suggest a solution - I am out of ideas... It seems to me that matlab should not be providing the cuda libraries in its installation for this reason. Matlab always lags the cutting edge, so those of us who live on the edge have to figure out ways around Matlab's luddicity...

There may be a way to fix this problem, I just have no alternate suggestions...
Boxed Cylon
 
Posts: 48
Joined: Fri Oct 16, 2009 8:57 pm

Next

Return to Linux Support

Who is online

Users browsing this forum: No registered users and 1 guest

cron