cublas: very slow init

Support for issues specific to the Windows operating systems.

cublas: very slow init

Postby safars » Mon Oct 05, 2009 6:14 am

Hi!
I'm trying to get CULAtools working, it's almost OK, but some strange things happened.

As I called culaInitialize(), it took 22 s to return. (Looked a little too much for me.) After looking around on the forums, I tried to measure cublasInit() runtimes separately. By the way, some dll dependency issues occured, too.

According to the docs, the 2.3 CUDA toolbox (including CUBLAS) is needed for running CULA. However, if I use the toolbox cublas.dll, I get runtime errors stating that the cublasCtrmv() function couldn't be found. Comparing this version to the one packed with CULA, I discovered that the latter is 5 MB bigger (20 MB for the CULA one, 15 for the nVidia dll), and indeed, the smaller dll didn't contain the aforementioned code.

Both were 64 bit and had the same version number (6.14.11.2030).

So here are the measurement results:
- nvidia original CUBLAS, without culaInitialize: 1.7 s (340 ms without debugger)
- CULA CUBLAS, without culaInitialize: 2.3 s (360 ms without debugger)
- nvidia original CUBLAS, with cudaInitialize: 22 s (5.8 s without debugger)

cudaInitialize() didn't even need to run, it was enough to be compiled in (volatile int i = 3; if(i!=3) culaInitialize(); ).

Any ideas why this happens? (And why has the debugger so much influence on the runtimes?) Or... how to make this faster? And what's the difference between the two CUBLAS dll versions? (Better yet: why do they differ and where do they come from?)

Thanks!

(BTW: GTX 260, Vista 64bit, CUDA 2.3, CULA Basic, VS2008).
safars
 
Posts: 3
Joined: Mon Oct 05, 2009 10:48 am

Re:cublas: very slow init

Postby john » Tue Oct 06, 2009 11:03 am

Hello Safars,
I'm going to answer your questions over a couple of posts. The one I can answer immediately is the difference in the CULA cublas.dll file, which as you have noted is 5 MB larger than standard 2.3. Our collaborations with Nvidia have resulted in them making additions and modifications to CUBLAS, which they then provided to us for our CULA work. We have been specifically permitted to release these to our users so that we could maintain a release schedule that is independent of CUDA releases. These enhanced versions will become publicly available in an upcoming CUDA release.

For the other issues, I'm working on getting an answer for those. We haven't seen them in our lab here, but it's certainly been reported by our users (as you have seen.) I will reply here when I have more of an answer.

Thank you,
John
john
Administrator
 
Posts: 587
Joined: Thu Jul 23, 2009 2:31 pm

Re: cublas: very slow init

Postby kolonel » Wed Oct 28, 2009 12:26 pm

Hi,
Would you please let me know what to do to get rid of that runtime errors stating that the cublasCtrmv() function couldn't be found??
I am using XP64, CUDA 2.3, CULA BASIC, VS 2005.
Thank you
University of Alberta
kolonel
 
Posts: 14
Joined: Mon Oct 26, 2009 9:18 pm
Location: Canada

Re: cublas: very slow init

Postby safars » Wed Oct 28, 2009 12:48 pm

Hi,
try to use the cublas.dll found in <WHERE_CULA_IS>/bin (or, in your case, bin64) dir! Windows looks for dlls in that directory first where your executables are, so if you copy the dll near them it will be the default choice instead of the CUDA toolkit 2.3 one.

Hope that helps :)
safars
 
Posts: 3
Joined: Mon Oct 05, 2009 10:48 am

Re: cublas: very slow init

Postby kolonel » Wed Oct 28, 2009 1:36 pm

great! thanks for your help!
there is not any setting I could do instead of copying the dll file?
University of Alberta
kolonel
 
Posts: 14
Joined: Mon Oct 26, 2009 9:18 pm
Location: Canada

Re: cublas: very slow init

Postby safars » Wed Oct 28, 2009 1:52 pm

This MSDN article might be useful (it's about how Windows looks for DLLs).

By the way, try changing the order CUDA and CULA are listed in your $PATH environment variable (put CULA first). (Never tried that one, but maybe...)
safars
 
Posts: 3
Joined: Mon Oct 05, 2009 10:48 am

Re:cublas: very slow init

Postby wburger » Mon Nov 09, 2009 1:33 am

Hi John,

on a Q9400 quad core, GTS 250, running Win XP 64-bit, I have also noticed the six second delay. Running the initialization function for the first time in a program takes approximately six seconds. After this first delay, calling the initialize function again is very quick (after calling shutdown first, of course).

The standard CUBLAS dll's do not show this behavior. Any ideas yet on how to remove the delay? Thanks in advance for the response.
(CULA basic Release 1.0 Final (September 30, 2009))
wburger
 
Posts: 1
Joined: Fri Nov 06, 2009 9:06 am

Re:cublas: very slow init

Postby dan » Mon Nov 23, 2009 9:38 am

I wanted to follow up this thread to report that we've solved this issue in CULA 1.1 final, which we will be releasing shortly.
dan
Administrator
 
Posts: 61
Joined: Thu Jul 23, 2009 2:29 pm

Re:cublas: very slow init

Postby morris » Thu Jan 28, 2010 7:38 am

Could you briefly describe why there was a 6 second delay and what was done to remove it?

I ask as, after having updated from Cula Basic 1.1 to 1.1b, I'm now also seeing a constant 5.5 second delay for culaDevice routines svd, qrf, trf and sgels. I noticed the delay when the timing values kept on being constant for 64^2 to 512^2 input matrices.

Note that I'm seeing this when calling through Matlab R2007b and 1.1 did not cause such a delay.

The benchmarks run through a dosbox look like those posted in the thread "sgev 1.1 slow", so it doens't show:

-- SGEQRF Benchmark --

Size CULA (s) MKL (s) Speedup
------ ---------- ---------- ---------
4096 0.90 0.00 0.0000
5120 1.18 0.00 0.0000
6144 2.02 0.00 0.0000
7168 3.00 0.00 0.0000
8192 3.42 0.00 0.0000

Note that I'd don't have MKL and thus avoided calling it.

I'm running on Windows Vista 64 on an Intel Quad Q9650 and turned off Defender (still 5.5 seconds delay). There was no Matlab update, no Cuda update (2.3), and I don't have Jacket as it does not do logical indexing yet.

Knowing why there had been such a long delay in version < 1.0 final would help me find a solution.

Thank you,

Morris
morris
 
Posts: 9
Joined: Wed Sep 16, 2009 1:30 pm

Re:cublas: very slow init

Postby dan » Thu Jan 28, 2010 8:59 am

Hi Morris,

Your benchmark times are in line with what is to be expected, and as you said, they don't show any delay. The issue that caused the long init times would have shown up here and so I believe that issue is unrelated to the delay you're seeing in Matlab.

Without knowing how you're calling the device functions in matlab, it's hard to say where the delay is coming from. Are you using mex files? If so, have you tried putting a few fine-grained timings in your mex to see if the delay is wholly confined to the Device call?

With some more information it would be easier to help you debug this problem.

Dan
dan
Administrator
 
Posts: 61
Joined: Thu Jul 23, 2009 2:29 pm

Re:cublas: very slow init

Postby morris » Fri Jan 29, 2010 3:17 am

After looking into my problem, it's thes same 5 second delay that was reported by others. This delay appeared when I uninstalled 1.1 and installed 1.1b. Here is the c file and Makefile to demonstrate the problem:

My system BTW:
Intel Core 2 Quad Q9650 3GHz
Windows Vista 64
Cula 1.1b
Cuda 2.3
Driver 190.38 for Windows Vista 64-bit

#include <culapackdevice.h>
#ifdef _MSC_VER
# pragma comment(lib, "cudart.lib")
#endif

int main(void)
{
culaStatus status;
status = culaInitialize();
return 0;
}

# Makefile for cuda projects
#
# Run this makefile from
# WindowsStart>VS>Startup>Tools>Prompts>VisualStudio2008x64Win64CommandPrompt

# Compiler
NVCC=C:\CUDA\bin\nvcc.exe
CC=cl.exe
INCLUDES4=/I "C:\Cula\include" /I "C:\Cuda\include"
CFLAGS=/D WIN32 /D "_CONSOLE" /D "_VC80_UPGRADE=0x0710" /D "_MBCS" /FD /EHsc /MT /W3 /c /TC

# Linker
LINK=link.exe
LDFLAGS=/INCREMENTAL:NO /MACHINE:X64 /SUBSYSTEM:CONSOLE /OPT:REF /OPT:NOICF /DYNAMICBASE:NO
LIBPATH4=/LIBPATH:"C:\Cula\lib64" /LIBPATH:"C:\Cuda\lib64" # order does not affect 5 second delay
LIBSCULA=cudart.lib cula.lib # order does not affect 5 second delay
LIBS=kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib

OBJ=cula_5sec.obj
all: cula_5sec.exe

# $@ the target
# $< the first dependency
# $* The part of a filename which matched a suffix rule

cula_5sec.exe: $(OBJ)
$(LINK) /OUT:$@ $(LIBPATH4) $(LIBSCULA) $(LIBS) $(LDFLAGS) $(OBJ)
nvcc --version # Cuda compilation tools, release 2.3, V0.2.1221
del *.obj *.idb

# all c files in the pwd
%.obj: %.c
$(CC) /O2 $(INCLUDES4) $(CFLAGS) $<
morris
 
Posts: 9
Joined: Wed Sep 16, 2009 1:30 pm

Re:cublas: very slow init

Postby kyle » Fri Jan 29, 2010 6:22 am

What GPU do you have as well, Morris?
kyle
Administrator
 
Posts: 301
Joined: Fri Jun 12, 2009 7:47 pm

Re:cublas: very slow init

Postby morris » Fri Jan 29, 2010 6:57 am

Sorry, I forgot: it an GTX 275 with 896 MB.
morris
 
Posts: 9
Joined: Wed Sep 16, 2009 1:30 pm

Re:cublas: very slow init

Postby morris » Mon Feb 01, 2010 1:28 am

I noticed that there is no delay when running (by double cliking) benchmark.exe from

C:\Cula\examples\benchmark\x64\Release

which includes cula.dll, cudart.dll and cublas.dll.

But when I move benchmark.exe up to C:\Cula\examples\benchmark, I get the 5 second delay. The exe finds the dlls as they are specified in the VisualStudio benchmark_vc9.sln file.

My path first lists C:\Cula and then C:\Cuda.

I have a feeling this is Vista related.

Morris
morris
 
Posts: 9
Joined: Wed Sep 16, 2009 1:30 pm

Re:cublas: very slow init

Postby dan » Mon Feb 01, 2010 8:25 am

My guess is that you're somehow linking to the older version of a CULA library. I noticed that you mentioned C:\CULA, which is the not the default path, is it possible that you've installed CULA to more than one location and you're linking to an older copy (e.g. C:/Program Files/CULA)? Note that a dll in the same path as an executable will override any other PATH, which will explain why there is no delay when running from the Release directory.

I recommend a tool called Dependency Walker (http://dependencywalker.com/) to help you try and debug this problem. It'll show libraries and the paths to which your executable is linking.

Dan
dan
Administrator
 
Posts: 61
Joined: Thu Jul 23, 2009 2:29 pm

Next

Return to Windows Support

Who is online

Users browsing this forum: No registered users and 1 guest

cron