Doubt about CULA on multiple GPUs

General CULA Dense (LAPACK & BLAS) support and troubleshooting. Use this forum if you are having a general problem or have encountered a bug.

Doubt about CULA on multiple GPUs

Postby cding » Fri Jul 22, 2011 11:52 am

I'm using pgfotran compiler for the CUDA fortran code, mpif90 for mpi code.
If I want to use CULA routines on multiple GPUs controlled by same amount of CPUs, I need to select device for each CPU. Can I use cudasetdevice() instead of cula_selectdevice()?


Code: Select all
cuda_status = cudasetdevice(gpuid)
call check_status_cuda(cuda_status )

cula_status = cula_initialize()
call check_status(cula_status)


Or

Code: Select all
cula_status = cula_selectdevice(gpuid)
call check_status(cula_status)

cula_status = cula_initialize()
call check_status(cula_status)


Which is correct, or both of them is good?
cding
 
Posts: 15
Joined: Tue Sep 14, 2010 8:25 pm

Re: Doubt about CULA on multiple GPUs

Postby john » Fri Jul 22, 2011 12:07 pm

Either should work. culaSelectDevice is merely a passthrough for cudaSelectDevice so that you are not forced to invoke the CUDA toolkit if you only need the Host interface in CULA.
john
Administrator
 
Posts: 587
Joined: Thu Jul 23, 2009 2:31 pm

Re: Doubt about CULA on multiple GPUs

Postby cding » Mon Jul 25, 2011 10:54 am

john wrote:Either should work. culaSelectDevice is merely a passthrough for cudaSelectDevice so that you are not forced to invoke the CUDA toolkit if you only need the Host interface in CULA.


Thank you john. I need to use Device interface in CULA. It makes any difference?

And I more question about running CULA on multiple devices:

If I use cudasetdevice() to select device, then initialize cula, then call cula_routines, I really want to make sure I'm using the devices correctly. So I call cula_getexecutingdevice(). But the device ID I get back is alway 0 no matter which device I chose (There are 4 devices on my node). However, if I call cudagetdeviceProperties(prop,gpuid), I get the device ID which is the one I chose.

Do you think even I didn't get correct ID back using cula_getexecutingdevice() after select device by calling cudasetdevice(), I'm still good to use the right one I set?
cding
 
Posts: 15
Joined: Tue Sep 14, 2010 8:25 pm

Re: Doubt about CULA on multiple GPUs

Postby john » Mon Jul 25, 2011 1:59 pm

Just to make sure, you're calling it as:

Code: Select all
cula_status = cula_getexecutingdevice(gpuid)


?
john
Administrator
 
Posts: 587
Joined: Thu Jul 23, 2009 2:31 pm

Re: Doubt about CULA on multiple GPUs

Postby cding » Tue Jul 26, 2011 9:49 am

john wrote:Just to make sure, you're calling it as:

Code: Select all
cula_status = cula_getexecutingdevice(gpuid)


?


Yes. Below is my code:

more_mpi.f
Code: Select all
module more_mpi
   include 'mpif.h'
   integer :: ierr,cpuid,numprocs,namelen !mpi
   character(len=100) processor_name
end module



cula_module.cuf
Code: Select all
module cula_module
    use cudafor
    use more_mpi
! ***** some CULA related variables *****
    INTEGER CULA_STATUS
    INTEGER DEVICE_ID
    CHARACTER(len=100) BUF
    INTEGER BUF_SIZE
    PARAMETER (BUF_SIZE=100)

    EXTERNAL CULA_SELECTDEVICE
    EXTERNAL CULA_INITIALIZE
    EXTERNAL cula_device_cgesvd
    EXTERNAL CULA_SHUTDOWN
    EXTERNAL CULA_GETDEVICEINFO
    EXTERNAL CULA_GETEXECUTINGDEVICE


    INTEGER CULA_SELECTDEVICE
    INTEGER CULA_INITIALIZE
    INTEGER cula_device_cgesvd
    INTEGER CULA_GETDEVICEINFO
    INTEGER CULA_GETEXECUTINGDEVICE

    INTEGER CULA_CGESV !cula
    integer cuda_status
    integer :: gpuid,numdevices !gpu
    integer :: info
    type(cudadeviceprop) :: prop

    contains

    subroutine init_cula2()

       gpuid=cpuid
       ! gpuid=1
       write(*,*) 'cpuid: ', cpuid
       write(*,*) 'GPU device ', gpuid, ' should be selected'

       write(*,*) 'Selecting Device From CUDA'
       cuda_status = cudasetdevice(gpuid)
       call check_status_cuda(cuda_status)

       WRITE(*,*) 'Initializing CULA'
       cula_status = cula_initialize()
       call check_status(cula_status)

       WRITE(*,*) 'Getting Device ID From CULA'
       cula_status = CULA_GETEXECUTINGDEVICE(device_id)
       call check_status(cula_status)
       WRITE(*,*) "Device ID: ",device_id

       WRITE(*,*) 'Getting Device Info From CULA'
       cula_status = CULA_GETDEVICEINFO(device_id, buf, buf_size)
       call check_status(cula_status)
       WRITE(*,*) "BUF: ",BUF

    end subroutine init_cula2
end module cula_module


main.f
Code: Select all
program main
   use more_mpi
   use cula_module
   call mpi_init(ierr)
   call mpi_comm_rank(mpi_comm_world,cpuid,ierr)
   call mpi_comm_size(mpi_comm_world,numprocs,ierr)
   call mpi_get_processor_name(processor_name,namelen,ierr)

   call init_cula2()

   call mpi_finalize(ierr)
end program


Result:
cpuid: 1
GPU device 1 should be selected
Selecting Device From CUDA
cpuid: 0
GPU device 0 should be selected
Selecting Device From CUDA
cpuid: 2
GPU device 2 should be selected
Selecting Device From CUDA
cpuid: 3
GPU device 3 should be selected
Selecting Device From CUDA
Initializing CULA
Initializing CULA
Initializing CULA
Initializing CULA
Getting Device ID From CULA
Device ID: 0
Getting Device Info From CULA
BUF:
Tesla T10 Processor (4 GB)
Getting Device ID From CULA
Device ID: 0
Getting Device Info From CULA
BUF:
Tesla T10 Processor (4 GB)
Getting Device ID From CULA
Device ID: 0
Getting Device Info From CULA
BUF:
Tesla T10 Processor (4 GB)
Getting Device ID From CULA
Device ID: 0
Getting Device Info From CULA
BUF:
Tesla T10 Processor (4 GB)
cding
 
Posts: 15
Joined: Tue Sep 14, 2010 8:25 pm

Re: Doubt about CULA on multiple GPUs

Postby suzannepk » Tue Mar 20, 2012 11:31 am

I am trying to use some of the cula fucntions listed in the code given above, for example culaGetExecutingDevice, but my compiler (pgi) does not recognize them even though I have other cula functions, for example cula_dgesv, compiling and work in my code. I have a file cula_dense.h which defines culaGetExecutingDevice. Do I need to include this file in the fortran file to get use of functions such as culaGetExecutingDevice or link to it specifically or is there a flag I need to set in my makefile? --for example I need -lcula_lapack and -lcula_lapack_fortran to use cual_dgesv.

The list of cula and cuda flags that i am currently using is -lcula_core -lcula_lapack -lcula_lapack_fortran -lcublas -lcudart -lpthread -Mcuda

I have tried this both with the host and device interface approaches. I am still learning cuda and would appreciate any suggestions or advice.
suzannepk
 
Posts: 9
Joined: Tue Feb 07, 2012 2:16 pm

Re: Doubt about CULA on multiple GPUs

Postby john » Tue Mar 20, 2012 11:47 am

Suzanne,
I sent an update package to your system administrators (Oscar and Richard) which contains many updates and new examples for Fortran users (especially PGI-Fortran). Was this packet forwarded to you by these gentlemen?

John
john
Administrator
 
Posts: 587
Joined: Thu Jul 23, 2009 2:31 pm

Re: Doubt about CULA on multiple GPUs

Postby suzannepk » Tue Mar 20, 2012 12:59 pm

Yes, thanks. I'll look there. The multi-gpu example is in C, but I am leaning to translate.
suzannepk
 
Posts: 9
Joined: Tue Feb 07, 2012 2:16 pm

Re: Doubt about CULA on multiple GPUs

Postby john » Wed Mar 21, 2012 7:52 am

You'll want to check out the fortranDeviceInterface example - it will be very helpful for this effort.
john
Administrator
 
Posts: 587
Joined: Thu Jul 23, 2009 2:31 pm


Return to CULA Dense Support

Who is online

Users browsing this forum: No registered users and 1 guest