cublas: very slow init

[safars]

Hi!
I'm trying to get CULAtools working, it's almost OK, but some strange things happened.

As I called culaInitialize(), it took 22 s to return. (Looked a little too much for me.) After looking around on the forums, I tried to measure cublasInit() runtimes separately. By the way, some dll dependency issues occured, too.

According to the docs, the 2.3 CUDA toolbox (including CUBLAS) is needed for running CULA. However, if I use the toolbox cublas.dll, I get runtime errors stating that the cublasCtrmv() function couldn't be found. Comparing this version to the one packed with CULA, I discovered that the latter is 5 MB bigger (20 MB for the CULA one, 15 for the nVidia dll), and indeed, the smaller dll didn't contain the aforementioned code.

Both were 64 bit and had the same version number (6.14.11.2030).

So here are the measurement results:
- nvidia original CUBLAS, without culaInitialize: 1.7 s (340 ms without debugger)
- CULA CUBLAS, without culaInitialize: 2.3 s (360 ms without debugger)
- nvidia original CUBLAS, with cudaInitialize: 22 s (5.8 s without debugger)

cudaInitialize() didn't even need to run, it was enough to be compiled in (volatile int i = 3; if(i!=3) culaInitialize(); ).

Any ideas why this happens? (And why has the debugger so much influence on the runtimes?) Or... how to make this faster? And what's the difference between the two CUBLAS dll versions? (Better yet: why do they differ and where do they come from?)

Thanks!

(BTW: GTX 260, Vista 64bit, CUDA 2.3, CULA Basic, VS2008).

[john]

Hello Safars,
I'm going to answer your questions over a couple of posts. The one I can answer immediately is the difference in the CULA cublas.dll file, which as you have noted is 5 MB larger than standard 2.3. Our collaborations with Nvidia have resulted in them making additions and modifications to CUBLAS, which they then provided to us for our CULA work. We have been specifically permitted to release these to our users so that we could maintain a release schedule that is independent of CUDA releases. These enhanced versions will become publicly available in an upcoming CUDA release.

For the other issues, I'm working on getting an answer for those. We haven't seen them in our lab here, but it's certainly been reported by our users (as you have seen.) I will reply here when I have more of an answer.

Thank you,
John

[kolonel]

Hi,
Would you please let me know what to do to get rid of that runtime errors stating that the cublasCtrmv() function couldn't be found??
I am using XP64, CUDA 2.3, CULA BASIC, VS 2005.
Thank you

[safars]

Hi,
try to use the cublas.dll found in <WHERE_CULA_IS>/bin (or, in your case, bin64) dir! Windows looks for dlls in that directory first where your executables are, so if you copy the dll near them it will be the default choice instead of the CUDA toolkit 2.3 one.

Hope that helps :)

[kolonel]

great! thanks for your help!
there is not any setting I could do instead of copying the dll file?

[safars]

might be useful (it's about how Windows looks for DLLs).

By the way, try changing the order CUDA and CULA are listed in your $PATH environment variable (put CULA first). (Never tried that one, but maybe...)

[wburger]

Hi John,

on a Q9400 quad core, GTS 250, running Win XP 64-bit, I have also noticed the six second delay. Running the initialization function for the first time in a program takes approximately six seconds. After this first delay, calling the initialize function again is very quick (after calling shutdown first, of course).

The standard CUBLAS dll's do not show this behavior. Any ideas yet on how to remove the delay? Thanks in advance for the response.

[dan]

I wanted to follow up this thread to report that we've solved this issue in CULA 1.1 final, which we will be releasing shortly.

[morris]

Could you briefly describe why there was a 6 second delay and what was done to remove it?

I ask as, after having updated from Cula Basic 1.1 to 1.1b, I'm now also seeing a constant 5.5 second delay for culaDevice routines svd, qrf, trf and sgels. I noticed the delay when the timing values kept on being constant for 64^2 to 512^2 input matrices.

Note that I'm seeing this when calling through Matlab R2007b and 1.1 did not cause such a delay.

The benchmarks run through a dosbox look like those posted in the thread "sgev 1.1 slow", so it doens't show:

-- SGEQRF Benchmark --

Size CULA (s) MKL (s) Speedup
------ ---------- ---------- ---------
4096 0.90 0.00 0.0000
5120 1.18 0.00 0.0000
6144 2.02 0.00 0.0000
7168 3.00 0.00 0.0000
8192 3.42 0.00 0.0000

Note that I'd don't have MKL and thus avoided calling it.

I'm running on Windows Vista 64 on an Intel Quad Q9650 and turned off Defender (still 5.5 seconds delay). There was no Matlab update, no Cuda update (2.3), and I don't have Jacket as it does not do logical indexing yet.

Knowing why there had been such a long delay in version < 1.0 final would help me find a solution.

Thank you,

Morris

[dan]

Hi Morris,

Your benchmark times are in line with what is to be expected, and as you said, they don't show any delay. The issue that caused the long init times would have shown up here and so I believe that issue is unrelated to the delay you're seeing in Matlab.

Without knowing how you're calling the device functions in matlab, it's hard to say where the delay is coming from. Are you using mex files? If so, have you tried putting a few fine-grained timings in your mex to see if the delay is wholly confined to the Device call?

With some more information it would be easier to help you debug this problem.

Dan

[morris]

After looking into my problem, it's thes same 5 second delay that was reported by others. This delay appeared when I uninstalled 1.1 and installed 1.1b. Here is the c file and Makefile to demonstrate the problem:

My system BTW:
Intel Core 2 Quad Q9650 3GHz
Windows Vista 64
Cula 1.1b
Cuda 2.3
Driver 190.38 for Windows Vista 64-bit

#include <culapackdevice.h>
#ifdef _MSC_VER
# pragma comment(lib, "cudart.lib")
#endif

int main(void)
{
culaStatus status;
status = culaInitialize();
return 0;
}

# Makefile for cuda projects
#
# Run this makefile from
# WindowsStart>VS>Startup>Tools>Prompts>VisualStudio2008x64Win64CommandPrompt

# Compiler
NVCC=C:\CUDA\bin\nvcc.exe
CC=cl.exe
INCLUDES4=/I "C:\Cula\include" /I "C:\Cuda\include"
CFLAGS=/D WIN32 /D "_CONSOLE" /D "_VC80_UPGRADE=0x0710" /D "_MBCS" /FD /EHsc /MT /W3 /c /TC

# Linker
LINK=link.exe
LDFLAGS=/INCREMENTAL:NO /MACHINE:X64 /SUBSYSTEM:CONSOLE /OPT:REF /OPT:NOICF /DYNAMICBASE:NO
LIBPATH4=/LIBPATH:"C:\Cula\lib64" /LIBPATH:"C:\Cuda\lib64" # order does not affect 5 second delay
LIBSCULA=cudart.lib cula.lib # order does not affect 5 second delay
LIBS=kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib

OBJ=cula_5sec.obj
all: cula_5sec.exe

# $@ the target
# $< the first dependency
# $* The part of a filename which matched a suffix rule

cula_5sec.exe: $(OBJ)
$(LINK) /OUT:$@ $(LIBPATH4) $(LIBSCULA) $(LIBS) $(LDFLAGS) $(OBJ)
nvcc --version # Cuda compilation tools, release 2.3, V0.2.1221
del *.obj *.idb

# all c files in the pwd
%.obj: %.c
$(CC) /O2 $(INCLUDES4) $(CFLAGS) $<

[kyle]

What GPU do you have as well, Morris?

[morris]

Sorry, I forgot: it an GTX 275 with 896 MB.

[morris]

I noticed that there is no delay when running (by double cliking) benchmark.exe from

C:\Cula\examples\benchmark\x64\Release

which includes cula.dll, cudart.dll and cublas.dll.

But when I move benchmark.exe up to C:\Cula\examples\benchmark, I get the 5 second delay. The exe finds the dlls as they are specified in the VisualStudio benchmark_vc9.sln file.

My path first lists C:\Cula and then C:\Cuda.

I have a feeling this is Vista related.

Morris

[dan]

My guess is that you're somehow linking to the older version of a CULA library. I noticed that you mentioned C:\CULA, which is the not the default path, is it possible that you've installed CULA to more than one location and you're linking to an older copy (e.g. C:/Program Files/CULA)? Note that a dll in the same path as an executable will override any other PATH, which will explain why there is no delay when running from the Release directory.

I recommend a tool called Dependency Walker (http://dependencywalker.com/) to help you try and debug this problem. It'll show libraries and the paths to which your executable is linking.

Dan