Resolved: Bug in SetCudaDevicePlatform with multiple GPU?

[blgolden]

Hi Developers,

UPDATE3: So I got cula to use my deviceId=1. The PlatformOpts.deviceId=1 is completely ignored as in the code below. However, if I export CUDA_VISIBLE_DEVICE=1 then cula uses deviceId=1 ok. In other words DeviceId=0 needs to be invisible to cula. Or am I just misinterpreting the the cula manual?

UPDATE2: Selecting a device does not work at all for me. I tested my C code on a colleague's system with 2 k20x's and a k40. It only uses the k40. Has anyone else got this working?

UPDATE: After more testing I have found I was mistaken and that I am not able to successfully direct cula to my second device at all. Am I doing something wrong? Here's how I am telling cula to use device 1:

int device = 1;
culaSparseCudaOptions platformOpts;
platformOpts.deviceId = device;
culaSparseCudaOptionsInit(handle, &platformOpts);
platformOpts.deviceId = device; //Did it twice just to be sure
culaSparseSetCudaPlatform(handle, plan, &platformOpts);

What am I doing wrong? Watching the job using nvidia-smi clearly shows the job allocating memory on device 0. Anyone else have this working? My GTX760 (device 1) should have no problem with cula? I am able to get my cuda code onto device 1 no problems.

ORIGINAL POST:

First: Thanks for such a fantastic library!

I'm pretty sure I have found a bug in the SetCudaDevicePlatform.

My OS: Linux ubuntu 12.04

Synopsis: Selecting device 1 when data have been preloaded to device 1 and using cudaSparseSetCudaDeviceOptions, the culaSparseExecutePlan still tries to use device 0 and throws a cublas runtime error 1 (cublas not initialized). I am using the Cg solver.

Detailed description: I am performing an MCMC analysis, so I want to avoid moving the A and b of A=xb to the device from the host for each sample. I have successfully implemented the problem using my device 0 and using the SetCudaDeviceOptions strategy over the SetCudaOptions strategy (i.e., data are already on the device 0 versus letting CULA move it) works great - no problems. However, when I select device 1 CULA throws a cublas runtime error 1 (cublas not initialized).

I have verified that my data are all on device 1 and that I have set platformOpts.deviceId = 1;

Also, I have verified that it works great with Device 1 when I use the SetCudaOptions strategy (let cula move the data back and forth).

Cuda installed version 5.5

GPU Models:

Device id 0 -> 01:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX Titan] (rev a1)
Device id 1 -> 02:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 760] (rev a1)