Long application startup times (90 seconds)?

[brooksmoses]

Hi!

I'm trying to debug an issue where our applications are taking a long time (90 seconds or more) to start up when I link them against the full CULA Dense libraries, but only a few seconds when linked against CULA Dense Free Edition. This time is pretty much all CPU-spin; top reports 100% CPU usage, and the time differs between my two CUDA machines in inverse proportion to their CPU clock speed.

I'm using CULA Dense R14 in both cases, on a couple of different systems (one RHEL 6.1, one Ubuntu 12.04) running the CUDA 4.2 SDK, which I know may be part of the issue since R14 is only supposed to be used with CUDA 4.1.

Anyway, is this a known issue? Anyone else seeing something like it? I figured I should ask before spending a lot of time trying to see if it's something deep in dark corners of my own code!

Thanks!
- Brooks

[john]

Hi Brooks, this is likely CUBLAS jitting. These are Kepler machines, I assume?

[brooksmoses]

Yes, these are both Kepler (GTX-680) machines.

Is there any way we can avoid this? (Perhaps it will go away when we upgrade versions, if the jitting is because of an absence of sm_30 kernels?) With several hundred programs in our testsuite, 90 seconds/program adds up a bit....

Thanks!

[john]

Yes, this is the startup jitting many hundreds of SM30 kernels. We'll have you something soon.

[brooksmoses]

Sounds good. Thanks!

[brooksmoses]

An update -- I happened to notice that my .nv/ComputeCache directory was overflowing my home-directory quota. (For arcane reasons, the home directory on my test machine is a tiny NFS-mounted directory, and not supposed to be used for much.) I reset that to point to a fast local disk with a lot more space, mostly just so my home directory wouldn't be entirely full, and then was pleasantly surprised to notice that my CULA tests weren't timing out any more!

I guess what was happening was that the ComputeCache was completely full, and so each application would re-JIT each kernel because they'd keep getting evicted by the previously-compiled one. And thus it had to re-JIT them on every test application launch, whereas now they're cached and things are a lot faster.