Yade

Setting Up GPU

Asked by Nima Goudarzi

Hi there,

I'm trying to set up GPU following https://yade-dev.gitlab.io/trunk/GPUacceleration.html#install-suitesparse. I have encountered doing so. Two main issues are:

1- After installation of Cuda, the samples folder is not generated within the Cuda directory (/usr/local/Cuda/Samples). Therefore, I have to download the Cuda Samples by git clone https://github.com/NVIDIA/cuda-samples.git and then compiling within Samples folder. Running ./deviceQuery inside'/Samples/1_Utilities/deviceQuery' seems satisfactory giving:
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "Quadro P2200"
  CUDA Driver Version / Runtime Version 11.7 / 11.7
  CUDA Capability Major/Minor version number: 6.1
  Total amount of global memory: 5051 MBytes (5296029696 bytes)
  (010) Multiprocessors, (128) CUDA Cores/MP: 1280 CUDA Cores
  GPU Max Clock rate: 1493 MHz (1.49 GHz)
  Memory Clock rate: 5005 Mhz
  Memory Bus Width: 160-bit
  L2 Cache Size: 1310720 bytes
  Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
  Total amount of constant memory: 65536 bytes
  Total amount of shared memory per block: 49152 bytes
  Total shared memory per multiprocessor: 98304 bytes
  Total number of registers available per block: 65536
  Warp size: 32
  Maximum number of threads per multiprocessor: 2048
  Maximum number of threads per block: 1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch: 2147483647 bytes
  Texture alignment: 512 bytes
  Concurrent copy and kernel execution: Yes with 2 copy engine(s)
  Run time limit on kernels: Yes
  Integrated GPU sharing Host Memory: No
  Support host page-locked memory mapping: Yes
  Alignment requirement for Surfaces: Yes
  Device has ECC support: Disabled
  Device supports Unified Addressing (UVA): Yes
  Device supports Managed Memory: Yes
  Device supports Compute Preemption: Yes
  Supports Cooperative Kernel Launch: Yes
  Supports MultiDevice Co-op Kernel Launch: Yes
  Device PCI Domain ID / Bus ID / location ID: 0 / 101 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.7, CUDA Runtime Version = 11.7, NumDevs = 1
Result = PASS

Is this the right approach for testing the Cuda installation for the most recent version? I'm not aware if the Samples folder had been generated for eralier versions inside the Cuda directory itself.

I, then, run
sudo apt-get install libopenblas-dev liblapack-dev

2- Following the instruction, I download the latest version of SuiteSparse package (5.13.0) and extracted it to usr/local folder using sudo (Question: Is extracting to usr/local mandatory?). Therefore, there is a /usr/local/SuiteSparse-5.13.0 directory from which I run make config within this directory gives:

----------------------------------------------------------------
SuiteSparse package compilation options:
----------------------------------------------------------------

SuiteSparse Version: 5.13.0
SuiteSparse top folder: /usr/local/SuiteSparse-5.13.0
Package: LIBRARY= PackageNameWillGoHere
Version: VERSION= x.y.z
SO version: SO_VERSION= x
System: UNAME= Linux
Install directory: INSTALL= /usr/local/SuiteSparse-5.13.0
Install libraries in: INSTALL_LIB= /usr/local/SuiteSparse-5.13.0/lib
Install include files in: INSTALL_INCLUDE= /usr/local/SuiteSparse-5.13.0/include
Install documentation in: INSTALL_DOC= /usr/local/SuiteSparse-5.13.0/share/doc/suitesparse-5.13.0
Optimization level: OPTIMIZATION= -O3
parallel make jobs: JOBS= 8
BLAS library: BLAS= -lblas
LAPACK library: LAPACK= -llapack
Other libraries: LDLIBS= -lm -lrt
static library: AR_TARGET= PackageNameWillGoHere.a
shared library (full): SO_TARGET= PackageNameWillGoHere.so.x.y.z
shared library (main): SO_MAIN= PackageNameWillGoHere.so.x
shared library (short): SO_PLAIN= PackageNameWillGoHere.so
shared library options: SO_OPTS= -L/usr/local/SuiteSparse-5.13.0/lib -Wl,-rpath=/usr/local/SuiteSparse-5.13.0/lib -shared -Wl,-soname -Wl,PackageNameWillGoHere.so.x -Wl,--no-undefined -Wl,-rpath, -Wl,-z,origin
shared library name tool: SO_INSTALL_NAME= echo
ranlib, for static libs: RANLIB= ranlib
static library command: ARCHIVE= ar rv
copy file: CP= cp -f
move file: MV= mv -f
remove file: RM= rm -f
pretty (for Tcov tests): PRETTY= grep -v "^#" | indent -bl -nce -bli0 -i4 -sob -l120
C compiler: CC= cc
C++ compiler: CXX= g++
CUDA enabled: CUDA= auto
CUDA compiler: NVCC= echo
CUDA root directory: CUDA_PATH=
OpenMP flags: CFOPENMP= -fopenmp
C/C++ compiler flags: CF= -O3 -fexceptions -fPIC -fopenmp
LD flags: LDFLAGS= -L/usr/local/SuiteSparse-5.13.0/lib -Wl,-rpath=/usr/local/SuiteSparse-5.13.0/lib
Fortran compiler: F77= f77
Fortran flags: F77FLAGS=
Intel MKL root: MKLROOT=
Auto detect Intel icc: AUTOCC= no
UMFPACK config: UMFPACK_CONFIG=
CHOLMOD config: CHOLMOD_CONFIG=
SuiteSparseQR config: SPQR_CONFIG=
CUDA library: CUDART_LIB=
CUBLAS library: CUBLAS_LIB=
METIS and CHOLMOD/Partition configuration:
Your METIS library: MY_METIS_LIB=
Your metis.h is in: MY_METIS_INC=
METIS is used via the CHOLMOD/Partition module, configured as follows.
If the next line has -DNPARTITION then METIS will not be used:
CHOLMOD Partition config:
CHOLMOD Partition libs: -lccolamd -lcamd -lmetis
CHOLMOD Partition include: -I/usr/local/SuiteSparse-5.13.0/CCOLAMD/Include -I/usr/local/SuiteSparse-5.13.0/CAMD/Include -I/usr/local/SuiteSparse-5.13.0/metis-5.1.0/include
MAKE: make
CMake options: -DCMAKE_INSTALL_PREFIX=/usr/local/SuiteSparse-5.13.0 -DCMAKE_CXX_COMPILER=g++ -DCMAKE_C_COMPILER=cc

As observed, the paths are blank for CUDART_LIB= and CUBLAS_LIB= which is not a good sign.
My effort for manual pointing to the Cuda directory in /usr/local/SuiteSparse/SuiteSparse_config/SuiteSparse_config.mk were not successful.

What is wrong with my implementation?

Cheers

Question information

Language:
English Edit question
Status:
Answered
For:
Yade Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Robert Caulk (rcaulk) said : 		#1

Hey,

Have you tried downloading the versions of CUDA and suitesparse mentioned in [1]?

Notably:

Suite-sparse 4.6.0-beta
CUDA 9.0
Nvidia 384.11 GPU drivers

In the CUDA install you will find the samples folder indicated in [2].

Your question:

>> Question: Is extracting to usr/local mandatory?

No it is in-fact not mandatory. Try extracting to your /home/ folder instead.

>> the paths are blank for CUDART_LIB= and CUBLAS_LIB= which is not a good sign

Yes, it is not a good sign. But you should be able to navigate and find these libraries manually inside the cuda-9.0/lib64 folder (please verify that they do exist ). Then you can put that path in SuiteSparse_config.mk directly.

Please keep me updated.

Cheers,

Robert

[1] https://www.sciencedirect.com/science/article/abs/pii/S0010465519303340
[2] https://yade-dev.gitlab.io/trunk/GPUacceleration.html#accelerating-yade-s-flowengine-with-gpu

Revision history for this message
Nima Goudarzi (nimagoudarzi58) said : 		#2

Hi Robert,

Thanks. I had been able to check the Cuda capability (Cuda 11.7) and compile SuitSparse (5.13.0). I even compiled YADE with GPU enabled.

(1) I guess, there is an inconsistency in the Cuda installation instruction (YADE manual) for adding CUDA library to the path:

Yade manual:

# Add the CUDA library to your path
export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda/lib64\ ${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

After installation of Cuda, three different Cuda folders are generated in my usr/local directory
1- Cuda
2- Cuda 11.7
3-Cuda 11

Original instruction manual:

The PATH variable needs to include export PATH=/usr/local/cuda-11.8/bin${PATH:+:${PATH}}. Nsight Compute has moved to /opt/nvidia/nsight-compute/ only in rpm/deb installation method. When using .run installer it is still located under /usr/local/cuda-11.8/.

To add this path to the PATH variable:

export PATH=/usr/local/cuda-11.8/bin${PATH:+:${PATH}}
In addition, when using the runfile installation method, the LD_LIBRARY_PATH variable needs to contain /usr/local/cuda-11.8/lib64 on a 64-bit system, or /usr/local/cuda-11.8/lib on a 32-bit system

To change the environment variables for 64-bit operating systems:

export LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64\
                         ${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

This is different from Yade manual. Which one to use?
However neither resolve the blank path issue unless the CUDAPATH is introduced (in the terminal)

For example if Cuda 11.7 is installed:

CUDAPATH=/usr/local/cuda- 11.7
export PATH=/usr/local/cuda- 11.7/bin${PATH: + ${PATH}}
export_LD_LIBRARY_PATH=/usr/local/cuda- 11.7/lib64\ ${LD_LIBRARY_PATH: + :${LD_LIBRARY_PATH}}

and make config within cd '/usr/local/SuiteSparse-5.13.0'

gives

CUDA root directory: CUDA PATH=/usr/local/cuda-11.7
.
.
.
Cuda library: CUDART_LIB=/usr/local/cuda-11.7/lib64/libcudart.so
CUBLAS library: CUBLAS_LIB=/usr/local/cuda-11.7/lib64/libcublas.so

I am able to compile SuiteSparse-5.13.0 by make after this but am not sure if the provided paths are correct.

(2) Regarding manual entry of CUDA_PATH in SuitSparse_Config.mk if this is the first few lines for #NVIDIA CUDA configuration for CHOLMOD and SPQR

CUDA=auto

if fneq ($(CUDA), no)
           CUDA_PATH= $(shell which nvcc 2>/dev/null | sed "s/ \ /bin\ /nvcc//")
else
          CUDA_PATH=

Manual entry of the CUDA_PATH (enforcing all conditions to point to the location of installed Cuda)
CUDA=auto

if fneq ($(CUDA), no)
           CUDA_PATH= /usr/local/cuda-11.7
else
          CUDA_PATH=/usr/local/cuda-11.7

Leads to the similar result as (1)
Is this the appropriate approach for manual change of SuitSparse_Config.mk or I need to point to another directory (for example cuda itself-- not cuda 11.7? Please advise

I have been able to verify the accuracy of compilation by executing sh gpu.sh within SuiteSparse/CHOLMOD/Demo.

3- Here is the cmake command I use for compiling yade:
cmake -DCMAKE_INSTALL_PREFIX=../install ../trunk -DCHOLMOD_GPU=ON -DSUITESPARSEPATH=/usr/local/SuiteSparse-5.13.0

Cmake finds the paths as recommended in YADE manual but the paths are not the ones I introduce either in SuitSparse_Config.mk or in the terminal. Also, multiple identical paths are reported for the same CHOLMOD (and dependencies such as AMD), SuiteSparse, CuBlas, and Metis. I feel something is wrong here.

BTW, I can compile YADE with this approach but when I run YADE executable, I get segmentation fault (core dumped).
Similar deployment on an Azure VM gives another error.

3- I'm now working on your comment for Cuda 9.0 but need to know which trunk version (for YADE compilation) is compatible with this installation. Also, how important is finding Nvidia 384.11 GPU drivers? Basically, how can I enforce to install this specific version?

Thanks so much

Nima

Can you help with this problem?

Provide an answer of your own, or ask Nima Goudarzi for more information if necessary.

To post a message you must log in.