Long application startup times (90 seconds)?

Support for issues specific to the Linux operating systems.

Long application startup times (90 seconds)?

Postby brooksmoses » Mon Jul 23, 2012 11:48 pm

Hi!

I'm trying to debug an issue where our applications are taking a long time (90 seconds or more) to start up when I link them against the full CULA Dense libraries, but only a few seconds when linked against CULA Dense Free Edition. This time is pretty much all CPU-spin; top reports 100% CPU usage, and the time differs between my two CUDA machines in inverse proportion to their CPU clock speed.

I'm using CULA Dense R14 in both cases, on a couple of different systems (one RHEL 6.1, one Ubuntu 12.04) running the CUDA 4.2 SDK, which I know may be part of the issue since R14 is only supposed to be used with CUDA 4.1.

Anyway, is this a known issue? Anyone else seeing something like it? I figured I should ask before spending a lot of time trying to see if it's something deep in dark corners of my own code!

Thanks!
- Brooks
brooksmoses
CULA Premium
 
Posts: 4
Joined: Mon May 23, 2011 2:53 pm

Re: Long application startup times (90 seconds)?

Postby john » Tue Jul 24, 2012 10:56 am

Hi Brooks, this is likely CUBLAS jitting. These are Kepler machines, I assume?
john
Administrator
 
Posts: 587
Joined: Thu Jul 23, 2009 2:31 pm

Re: Long application startup times (90 seconds)?

Postby brooksmoses » Tue Jul 24, 2012 2:03 pm

Yes, these are both Kepler (GTX-680) machines.

Is there any way we can avoid this? (Perhaps it will go away when we upgrade versions, if the jitting is because of an absence of sm_30 kernels?) With several hundred programs in our testsuite, 90 seconds/program adds up a bit....

Thanks!
brooksmoses
CULA Premium
 
Posts: 4
Joined: Mon May 23, 2011 2:53 pm

Re: Long application startup times (90 seconds)?

Postby john » Wed Jul 25, 2012 5:56 am

Yes, this is the startup jitting many hundreds of SM30 kernels. We'll have you something soon.
john
Administrator
 
Posts: 587
Joined: Thu Jul 23, 2009 2:31 pm

Re: Long application startup times (90 seconds)?

Postby brooksmoses » Wed Jul 25, 2012 12:15 pm

Sounds good. Thanks!
brooksmoses
CULA Premium
 
Posts: 4
Joined: Mon May 23, 2011 2:53 pm

Re: Long application startup times (90 seconds)?

Postby brooksmoses » Sun Aug 05, 2012 11:32 am

An update -- I happened to notice that my .nv/ComputeCache directory was overflowing my home-directory quota. (For arcane reasons, the home directory on my test machine is a tiny NFS-mounted directory, and not supposed to be used for much.) I reset that to point to a fast local disk with a lot more space, mostly just so my home directory wouldn't be entirely full, and then was pleasantly surprised to notice that my CULA tests weren't timing out any more!

I guess what was happening was that the ComputeCache was completely full, and so each application would re-JIT each kernel because they'd keep getting evicted by the previously-compiled one. And thus it had to re-JIT them on every test application launch, whereas now they're cached and things are a lot faster.
brooksmoses
CULA Premium
 
Posts: 4
Joined: Mon May 23, 2011 2:53 pm


Return to Linux Support

Who is online

Users browsing this forum: No registered users and 2 guests

cron