[Condor-users] Condor, mpi, icc with HPL

[Philipp Kolmann <kolmann@xxxxxxxxxxxxxxxx>]

Hi!

maybe someone can help me with this issue.

I have condor set up so far and it works like a charme.

I have HPL 
http://www.netlib.org/benchmark/hpl/
compiled with gcc and MPICH 1.2.4 which runs fine on condor.

When using icc 8.1 the program starts executing, but then dies :

[...]
[2] Ending MPI_Send
[2] Starting MPI_Comm_size...
[2] Ending MPI_Comm_size
[2] Starting MPI_Comm_rank...
[2] Ending MPI_Comm_rank
[2] Starting MPI_Recv with count = 1, source = 0, tag = 9001...
[2] Ending MPI_Recv from 0 with tag 9001
[2] Starting MPI_Send with count = 1, dest = 3, tag = 9001...
[2] Ending MPI_Send
p2_4992:  p4_error: interrupt SIGSEGV: 11
(END) 

Attached is the Makefile for hpl.

Thanks for any help,
kind regards
Philipp Kolmann
University of Technology, Vienna
Austria

-- 
If you have problems in Windows: REBOOT
If you have problems in Linux:   BE ROOT

#  
# ----------------------------------------------------------------------
# - shell --------------------------------------------------------------
# ----------------------------------------------------------------------
#
SHELL        = /bin/sh
#
CD           = cd
CP           = cp
LN_S         = ln -s
MKDIR        = mkdir
RM           = /bin/rm -f
TOUCH        = touch
#
# ----------------------------------------------------------------------
# - Platform identifier ------------------------------------------------
# ----------------------------------------------------------------------
#
ARCH         = Linux_WINZIG-mpi_icc
#
# ----------------------------------------------------------------------
# - HPL Directory Structure / HPL library ------------------------------
# ----------------------------------------------------------------------
#
TOPdir       = $(HOME)/hpl
INCdir       = $(TOPdir)/include
BINdir       = $(TOPdir)/bin/$(ARCH)
LIBdir       = $(TOPdir)/lib/$(ARCH)
#
HPLlib       = $(LIBdir)/libhpl.a 
#
# ----------------------------------------------------------------------
# - Message Passing library (MPI) --------------------------------------
# ----------------------------------------------------------------------
# MPinc tells the  C  compiler where to find the Message Passing library
# header files,  MPlib  is defined  to be the name of  the library to be
# used. The variable MPdir is only used for defining MPinc and MPlib.
#
MPdir        = /grid/home/pkolmann/SW/mpich
MPinc        = -I$(MPdir)/include
MPlib        = $(MPdir)/lib/libmpich.a
#
# ----------------------------------------------------------------------
# - Linear Algebra library (BLAS or VSIPL) -----------------------------
# ----------------------------------------------------------------------
# LAinc tells the  C  compiler where to find the Linear Algebra  library
# header files,  LAlib  is defined  to be the name of  the library to be
# used. The variable LAdir is only used for defining LAinc and LAlib.
#
#LAdir        = $(HOME)/netlib/ARCHIVES/Linux_PII
#LAinc        =
#LAlib        = $(LAdir)/libcblas.a $(LAdir)/libatlas.a
LAdir        = /opt/intel/mkl72/lib/32
LAinc        = -I/opt/intel/mkl72/include -I/usr/include \
		-I/opt/intel_cc_80/include \
		-I/usr/lib/gcc-lib/i486-linux/3.3.5/include
LAlib        = $(LAdir)/libmkl_ia32.a  \
		$(LAdir)/libguide.a  \
		$(LAdir)/libmkl_lapack.a \
		$(LAdir)/libmkl_solver.a \
		-L/opt/intel_fc_80/lib \
		-i-static -static-libcxa \
		-lpthread
#
# ----------------------------------------------------------------------
# - F77 / C interface --------------------------------------------------
# ----------------------------------------------------------------------
# You can skip this section  if and only if  you are not planning to use
# a  BLAS  library featuring a Fortran 77 interface.  Otherwise,  it  is
# necessary  to  fill out the  F2CDEFS  variable  with  the  appropriate
# options.  **One and only one**  option should be chosen in **each** of
# the 3 following categories:
#
# 1) name space (How C calls a Fortran 77 routine)
#
# -DAdd_              : all lower case and a suffixed underscore  (Suns,
#                       Intel, ...),                           [default]
# -DNoChange          : all lower case (IBM RS6000),
# -DUpCase            : all upper case (Cray),
# -DAdd__             : the FORTRAN compiler in use is f2c.
#
# 2) C and Fortran 77 integer mapping
#
# -DF77_INTEGER=int   : Fortran 77 INTEGER is a C int,         [default]
# -DF77_INTEGER=long  : Fortran 77 INTEGER is a C long,
# -DF77_INTEGER=short : Fortran 77 INTEGER is a C short.
#
# 3) Fortran 77 string handling
#
# -DStringSunStyle    : The string address is passed at the string loca-
#                       tion on the stack, and the string length is then
#                       passed as  an  F77_INTEGER  after  all  explicit
#                       stack arguments,                       [default]
# -DStringStructPtr   : The address  of  a  structure  is  passed  by  a
#                       Fortran 77  string,  and the structure is of the
#                       form: struct {char *cp; F77_INTEGER len;},
# -DStringStructVal   : A structure is passed by value for each  Fortran
#                       77 string,  and  the  structure is  of the form:
#                       struct {char *cp; F77_INTEGER len;},
# -DStringCrayStyle   : Special option for  Cray  machines,  which  uses
#                       Cray  fcd  (fortran  character  descriptor)  for
#                       interoperation.
#
F2CDEFS      =
#
# ----------------------------------------------------------------------
# - HPL includes / libraries / specifics -------------------------------
# ----------------------------------------------------------------------
#
HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) $(LAinc) $(MPinc)
HPL_LIBS     = $(HPLlib) $(LAlib) $(MPlib)
#
# - Compile time options -----------------------------------------------
#
# -DHPL_COPY_L           force the copy of the panel L before bcast;
# -DHPL_CALL_CBLAS       call the cblas interface;
# -DHPL_CALL_VSIPL       call the vsip  library;
# -DHPL_DETAILED_TIMING  enable detailed timers;
#
# By default HPL will:
#    *) not copy L before broadcast,
#    *) call the BLAS Fortran 77 interface,
#    *) not display detailed timing information.
#
HPL_OPTS     = -DHPL_CALL_CBLAS
#
# ----------------------------------------------------------------------
#
HPL_DEFS     = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)
#
# ----------------------------------------------------------------------
# - Compilers / linkers - Optimization flags ---------------------------
# ----------------------------------------------------------------------
#
CC           = $(MPdir)/bin/mpicc -config=icc
CCNOOPT      = $(HPL_DEFS)
CCFLAGS      = $(HPL_DEFS) -O3 
#
# On some platforms,  it is necessary  to use the Fortran linker to find
# the Fortran internals used in the BLAS library.
#
LINKER       = $(MPdir)/bin/mpicc -config=icc -mpitrace
#LINKFLAGS    = $(CCFLAGS) -L/opt/intel_cc_80/lib -lguide -lguide_stats
LINKFLAGS    = $(CCFLAGS)
#
ARCHIVER     = ar
ARFLAGS      = r
RANLIB       = echo
#
# ----------------------------------------------------------------------