error in siesta compiling by the Openblas for openmpi and openmp

Asked by 翟明星 on 2020-07-23

Dear Siesta Users/Developers,

I would like to ask your kind help.

I successfully compiled siesta-4.1-b4 by using openMP+ MPI with openblas library, but when I run the input file h2o.fdf in the example, an error turn up as follows:

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
************************************************************************
No. of atoms with KB's overlaping orbs in proc 0. Max # of overlaps: 3 23
New grid distribution: 1
           1 1: 40 1: 30 1: 15
           2 1: 40 1: 30 16: 30

InitMesh: MESH = 80 x 60 x 60 = 288000
InitMesh: (bp) = 40 x 30 x 30 = 36000
InitMesh: Mesh cutoff (required, used) = 300.000 301.251 Ry
ExtMesh (bp) on 0 = 100 x 86 x 71 = 610600
New grid distribution: 2
           1 13: 40 1: 30 1: 30
           2 1: 12 1: 30 1: 30
New grid distribution: 3
           1 18: 40 1: 30 1: 30
           2 1: 17 1: 30 1: 30
Setting up quadratic distribution...
ExtMesh (bp) on 0 = 88 x 86 x 86 = 650848
PhiOnMesh: Number of (b)points on node 0 = 25200
PhiOnMesh: nlist on node 0 = 88187
rdiag: Error in Cholesky factorisation
Stopping Program from Node: 0
rdiag: Error in Cholesky factorisation
Stopping Program from Node: 1
          23
rdiag: Error in Cholesky factorisation
Stopping Program from Node: 1
          23
rdiag: Error in Cholesky factorisation
Stopping Program from Node: 0
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI COMMUNICATOR 3 CREATE FROM 0
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[siying-vbox:27368] 1 more process has sent help message help-mpi-api.txt / mpi-abort
[siying-vbox:27368] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

Question information

Language:
English Edit question
Status:
Solved
For:
Siesta Edit question
Assignee:
No assignee Edit question
Solved by:
Nick Papior
Solved:
2020-08-19
Last query:
2020-08-19
Last reply:
2020-07-27
Nick Papior (nickpapior) said : #1

Which versions of the library did you use?

And also, please share your arch.make.

翟明星 (mxzhai) said : #3

1. library :
    (1) gcc version 5.4.0;
    (2) OpenBLAS version 0.3.9;
    (3) version of scalapack is scalapack-2.1.0 .
2. the arch.make is as follows
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
#
.SUFFIXES:
.SUFFIXES: .f .F .o .c .a .f90 .F90

SIESTA_ARCH = unknown

CC = gcc

FPP = $(FC) -E -P -x c

FC = mpif90

FC_SERIAL = gfortran

# Here we should put mainly optimization flags
FFLAGS = -O2 -fPIC -ftree-vectorize -march=native
FFLAGS += -DMPI
FFLAGS += -fopenmp

AR = ar
ARFLAGS_EXTRA =

RANLIB = ranlib

SYS = nag

DEFS_PREFIX =
LDFLAGS =

FCFLAGS_fixed_f =
FCFLAGS_free_f90 =
FPPFLAGS_fixed_F =
FPPFLAGS_free_F90 =

   LIBS += -L/opt/libs/OpenBLAS/lib -lopenblas
# LIBS += -L/opt/libs/OpenBLAS/lib -lopenblas
   LIBS += /opt/libs/openblas-scalapack-2-1/libscalapack.a

INCFLAGS += -I/opt/build/netcdf/4.7.4/include
LDFLAGS += -L/opt/build/zlib/1.2.11/lib -Wl,-rpath=/opt/build/zlib/1.2.11/lib
LDFLAGS += -L/opt/build/hdf5/1.10.4/lib -Wl,-rpath=/opt/build/hdf5/1.10.4/lib
LDFLAGS += -L/opt/build/netcdf/4.7.4/lib -Wl,-rpath=/opt/build/netcdf/4.7.4/lib
LIBS += -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz
COMP_LIBS += libncdf.a libfdict.a
FPPFLAGS += -DCDF -DNCDF -DNCDF_4

  MPI_INTERFACE = libmpi_f90.a
  MPI_INCLUDE = .

FPPFLAGS = $(DEFS_PREFIX)-DFC_HAVE_ABORT

LIBS += $(NETCDF_LIBS) $(SCALAPACK_LIBS) $(LAPACK_LIBS) $(MPI_LIBS) $(COMP_LIBS)
LIBS += -fopenmp

atom.o: atom.F
        $(FC) -c $(FFLAGS_DEBUG) $(INCFLAGS) $(FPPFLAGS) $(FPPFLAGS_fixed_F) $<

.c.o:
        $(CC) -c $(CFLAGS) $(INCFLAGS) $(CPPFLAGS) $<
.F.o:
        $(FC) -c $(FFLAGS) $(INCFLAGS) $(FPPFLAGS) $(FPPFLAGS_fixed_F) $<
.F90.o:
        $(FC) -c $(FFLAGS) $(INCFLAGS) $(FPPFLAGS) $(FPPFLAGS_free_F90) $<
.f.o:
        $(FC) -c $(FFLAGS) $(INCFLAGS) $(FCFLAGS_fixed_f) $<
.f90.o:
        $(FC) -c $(FFLAGS) $(INCFLAGS) $(FCFLAGS_free_f90) $<

Nick Papior (nickpapior) said : #4

Could you try to recompile without openmp?

Please ensure you do a make clean first.

On Sat, 25 Jul 2020, 03:55 翟明星, <email address hidden>
wrote:

> Question #692016 on Siesta changed:
> https://answers.launchpad.net/siesta/+question/692016
>
> Status: Needs information => Open
>
> 翟明星 gave more information on the question:
> 1. library :
> (1) gcc version 5.4.0;
> (2) OpenBLAS version 0.3.9;
> (3) version of scalapack is scalapack-2.1.0 .
> 2. the arch.make is as follows
>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> #
> .SUFFIXES:
> .SUFFIXES: .f .F .o .c .a .f90 .F90
>
> SIESTA_ARCH = unknown
>
> CC = gcc
>
> FPP = $(FC) -E -P -x c
>
> FC = mpif90
>
> FC_SERIAL = gfortran
>
> # Here we should put mainly optimization flags
> FFLAGS = -O2 -fPIC -ftree-vectorize -march=native
> FFLAGS += -DMPI
> FFLAGS += -fopenmp
>
> AR = ar
> ARFLAGS_EXTRA =
>
> RANLIB = ranlib
>
> SYS = nag
>
> DEFS_PREFIX =
> LDFLAGS =
>
> FCFLAGS_fixed_f =
> FCFLAGS_free_f90 =
> FPPFLAGS_fixed_F =
> FPPFLAGS_free_F90 =
>
> LIBS += -L/opt/libs/OpenBLAS/lib -lopenblas
> # LIBS += -L/opt/libs/OpenBLAS/lib -lopenblas
> LIBS += /opt/libs/openblas-scalapack-2-1/libscalapack.a
>
> INCFLAGS += -I/opt/build/netcdf/4.7.4/include
> LDFLAGS += -L/opt/build/zlib/1.2.11/lib
> -Wl,-rpath=/opt/build/zlib/1.2.11/lib
> LDFLAGS += -L/opt/build/hdf5/1.10.4/lib
> -Wl,-rpath=/opt/build/hdf5/1.10.4/lib
> LDFLAGS += -L/opt/build/netcdf/4.7.4/lib
> -Wl,-rpath=/opt/build/netcdf/4.7.4/lib
> LIBS += -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz
> COMP_LIBS += libncdf.a libfdict.a
> FPPFLAGS += -DCDF -DNCDF -DNCDF_4
>
> MPI_INTERFACE = libmpi_f90.a
> MPI_INCLUDE = .
>
> FPPFLAGS = $(DEFS_PREFIX)-DFC_HAVE_ABORT
>
> LIBS += $(NETCDF_LIBS) $(SCALAPACK_LIBS) $(LAPACK_LIBS) $(MPI_LIBS)
> $(COMP_LIBS)
> LIBS += -fopenmp
>
> atom.o: atom.F
> $(FC) -c $(FFLAGS_DEBUG) $(INCFLAGS) $(FPPFLAGS)
> $(FPPFLAGS_fixed_F) $<
>
> .c.o:
> $(CC) -c $(CFLAGS) $(INCFLAGS) $(CPPFLAGS) $<
> .F.o:
> $(FC) -c $(FFLAGS) $(INCFLAGS) $(FPPFLAGS) $(FPPFLAGS_fixed_F) $<
> .F90.o:
> $(FC) -c $(FFLAGS) $(INCFLAGS) $(FPPFLAGS) $(FPPFLAGS_free_F90) $<
> .f.o:
> $(FC) -c $(FFLAGS) $(INCFLAGS) $(FCFLAGS_fixed_f) $<
> .f90.o:
> $(FC) -c $(FFLAGS) $(INCFLAGS) $(FCFLAGS_free_f90) $<
>
> --
> You received this question notification because you are an answer
> contact for Siesta.
>

翟明星 (mxzhai) said : #5

Thank you for that information. I comment the openmp lines. However, I still got the same error. I feel so lost.

Nick Papior (nickpapior) said : #6

Perhaps you have 2 atoms too close? Please check this.

On Sun, 26 Jul 2020, 05:25 翟明星, <email address hidden>
wrote:

> Question #692016 on Siesta changed:
> https://answers.launchpad.net/siesta/+question/692016
>
> Status: Answered => Open
>
> 翟明星 is still having a problem:
> Thank you for that information. I comment the openmp lines. However, I
> still got the same error. I feel so lost.
>
> --
> You received this question notification because you are an answer
> contact for Siesta.
>

翟明星 (mxzhai) said : #7

Thank you your answer. Fisrtly, the example I run is provided in the Examples directory of your software, and I checked it out. Secondly, when I compiled siesta with COMP_LIBS = libsiestaLAPACK.a libsiestaBLAS instead of openblas, the example runs complete. So, I think that it maybe the openblas library make the program abort. But I make sure that I compile correctly the openblas library, and make a test.

Best Nick Papior (nickpapior) said : #8

Yeah, ok, that clearly indicates a problem with the openblas + your architecture.

Could you please try openblas 0.3.10?

翟明星 (mxzhai) said : #9

Thanks! I have the same error using siesta compiled with Openblas 0.3.10. Maybe the problem is an exception in my compute. Let's forget it.

翟明星 (mxzhai) said : #10

Thanks Nick Papior, that solved my question.