Doubt about CULA on multiple GPUs

[cding]

I'm using pgfotran compiler for the CUDA fortran code, mpif90 for mpi code.
If I want to use CULA routines on multiple GPUs controlled by same amount of CPUs, I need to select device for each CPU. Can I use cudasetdevice() instead of cula_selectdevice()?

Code: Select all

cuda_status = cudasetdevice(gpuid)
call check_status_cuda(cuda_status )

cula_status = cula_initialize()
call check_status(cula_status)

Or

Code: Select all

cula_status = cula_selectdevice(gpuid)
call check_status(cula_status)

cula_status = cula_initialize()
call check_status(cula_status)

Which is correct, or both of them is good?

[john]

Either should work. culaSelectDevice is merely a passthrough for cudaSelectDevice so that you are not forced to invoke the CUDA toolkit if you only need the Host interface in CULA.

[cding]

Either should work. culaSelectDevice is merely a passthrough for cudaSelectDevice so that you are not forced to invoke the CUDA toolkit if you only need the Host interface in CULA.

Thank you john. I need to use Device interface in CULA. It makes any difference?

And I more question about running CULA on multiple devices:

If I use cudasetdevice() to select device, then initialize cula, then call cula_routines, I really want to make sure I'm using the devices correctly. So I call cula_getexecutingdevice(). But the device ID I get back is alway 0 no matter which device I chose (There are 4 devices on my node). However, if I call cudagetdeviceProperties(prop,gpuid), I get the device ID which is the one I chose.

Do you think even I didn't get correct ID back using cula_getexecutingdevice() after select device by calling cudasetdevice(), I'm still good to use the right one I set?

[john]

Just to make sure, you're calling it as:

Code: Select all

cula_status = cula_getexecutingdevice(gpuid)

?

[cding]

Just to make sure, you're calling it as:

Code: Select all

cula_status = cula_getexecutingdevice(gpuid)

?

Yes. Below is my code:

more_mpi.f

Code: Select all

module more_mpi
   include 'mpif.h'
   integer :: ierr,cpuid,numprocs,namelen !mpi
   character(len=100) processor_name
end module

cula_module.cuf

Code: Select all

module cula_module
    use cudafor
    use more_mpi
! ***** some CULA related variables *****
    INTEGER CULA_STATUS
    INTEGER DEVICE_ID
    CHARACTER(len=100) BUF
    INTEGER BUF_SIZE
    PARAMETER (BUF_SIZE=100)

    EXTERNAL CULA_SELECTDEVICE
    EXTERNAL CULA_INITIALIZE
    EXTERNAL cula_device_cgesvd
    EXTERNAL CULA_SHUTDOWN
    EXTERNAL CULA_GETDEVICEINFO
    EXTERNAL CULA_GETEXECUTINGDEVICE


    INTEGER CULA_SELECTDEVICE
    INTEGER CULA_INITIALIZE
    INTEGER cula_device_cgesvd
    INTEGER CULA_GETDEVICEINFO
    INTEGER CULA_GETEXECUTINGDEVICE

    INTEGER CULA_CGESV !cula
    integer cuda_status
    integer :: gpuid,numdevices !gpu
    integer :: info
    type(cudadeviceprop) :: prop

    contains

    subroutine init_cula2()

       gpuid=cpuid
       ! gpuid=1
       write(*,*) 'cpuid: ', cpuid
       write(*,*) 'GPU device ', gpuid, ' should be selected'

       write(*,*) 'Selecting Device From CUDA'
       cuda_status = cudasetdevice(gpuid)
       call check_status_cuda(cuda_status)

       WRITE(*,*) 'Initializing CULA'
       cula_status = cula_initialize()
       call check_status(cula_status)

       WRITE(*,*) 'Getting Device ID From CULA'
       cula_status = CULA_GETEXECUTINGDEVICE(device_id)
       call check_status(cula_status)
       WRITE(*,*) "Device ID: ",device_id

       WRITE(*,*) 'Getting Device Info From CULA'
       cula_status = CULA_GETDEVICEINFO(device_id, buf, buf_size)
       call check_status(cula_status)
       WRITE(*,*) "BUF: ",BUF

    end subroutine init_cula2
end module cula_module

main.f

Code: Select all

program main
   use more_mpi
   use cula_module
   call mpi_init(ierr)
   call mpi_comm_rank(mpi_comm_world,cpuid,ierr)
   call mpi_comm_size(mpi_comm_world,numprocs,ierr)
   call mpi_get_processor_name(processor_name,namelen,ierr)

   call init_cula2()

   call mpi_finalize(ierr)
end program


Result:
cpuid: 1
GPU device 1 should be selected
Selecting Device From CUDA
cpuid: 0
GPU device 0 should be selected
Selecting Device From CUDA
cpuid: 2
GPU device 2 should be selected
Selecting Device From CUDA
cpuid: 3
GPU device 3 should be selected
Selecting Device From CUDA
Initializing CULA
Initializing CULA
Initializing CULA
Initializing CULA
Getting Device ID From CULA
Device ID: 0
Getting Device Info From CULA
BUF:
Tesla T10 Processor (4 GB)
Getting Device ID From CULA
Device ID: 0
Getting Device Info From CULA
BUF:
Tesla T10 Processor (4 GB)
Getting Device ID From CULA
Device ID: 0
Getting Device Info From CULA
BUF:
Tesla T10 Processor (4 GB)
Getting Device ID From CULA
Device ID: 0
Getting Device Info From CULA
BUF:
Tesla T10 Processor (4 GB)

[suzannepk]

I am trying to use some of the cula fucntions listed in the code given above, for example culaGetExecutingDevice, but my compiler (pgi) does not recognize them even though I have other cula functions, for example cula_dgesv, compiling and work in my code. I have a file cula_dense.h which defines culaGetExecutingDevice. Do I need to include this file in the fortran file to get use of functions such as culaGetExecutingDevice or link to it specifically or is there a flag I need to set in my makefile? --for example I need -lcula_lapack and -lcula_lapack_fortran to use cual_dgesv.

The list of cula and cuda flags that i am currently using is -lcula_core -lcula_lapack -lcula_lapack_fortran -lcublas -lcudart -lpthread -Mcuda

I have tried this both with the host and device interface approaches. I am still learning cuda and would appreciate any suggestions or advice.

[john]

Suzanne,
I sent an update package to your system administrators (Oscar and Richard) which contains many updates and new examples for Fortran users (especially PGI-Fortran). Was this packet forwarded to you by these gentlemen?

John

[suzannepk]

Yes, thanks. I'll look there. The multi-gpu example is in C, but I am leaning to translate.

[john]

You'll want to check out the fortranDeviceInterface example - it will be very helpful for this effort.