[Condor-users] Open-MPI in Condor: wrapper + submitfile

["J.S. van Bethlehem" <j.s.van.bethlehem@xxxxxxxxxxxx>]

It seems this message was blocked because I changed the alias of my
email address. My apologies if I'm wrong and this is the second time the
message is send.
--------------------------------------------------------------------------

Recently there have been some questions on this list regarding the combo
Open-MPI with Condor, including some problems I mentioned myself. After
fiddling around quite a bit I now have a setup that works for me. I
attached the condor submit file and the shell script that wraps the
MPI-job that seems to work for me. Hopefully others find this useful as
well.
Btw: this is for a system with a seperate fileserver, so I don't need
any filetransfer to be done by Condor. AFAICS it should be trivial to
let Condor do the filetransfer by adding the standard commands in the
submit file

Some questions remain though (these may be suited better for the
developer-list though):
- often I get the following message:
 /bin/rm: cannot remove `sshd.out': No such file or directory
This seems to be caused by the sshd_cleanup function, which is called by
all nodes. But in the case of a filesystem that is accessible from all
nodes (ie: no Condor filetransfer) all nodes will try to remove the
_same_ file and therefore they fail as soon as one node performed the
rm-command.
So: it would be nice to be able to tell from within the script whether
Condor is using filetransfer or not. In the former case all nodes must
call sshd_cleanup, in the latter only one, let's say node 0 is sufficient.
(Another solution would be to write something like if [ -f sshd.out ];
then rm sshd.out; fi in the sshd.sh script; that would work correctly
always and doesn't need a new method do detect whether Condor is doing
filetransfers from inside the script)

- the MPI-wrappers for MPICH(2) en LAMMPI make a call to /bin/rm to
remove the 'machines' file. This will fail in a (probably kind of
academic) scenario where some non-standard installation has put the 'rm'
command at a different place. Unless I'm overlooking something, I think
it's better to just write
'rm -f machines'
(like it's done for all other calls to systemcommands)

- the exit code $TMP that the script returns, is _not_ passed on to
Condor. How can I recover that exit code?

Yours sincerely,
Jakob van Bethlehem

## without using a shared file system
######################################
universe = parallel
executable = ompiscript
arguments = executable
log = log/logfile
output = output/outfile.$(NODE)
error = output/errfile.$(NODE)
machine_count = 4
environment = LD_LIBRARY_PATH=/usr/lib/openmpi/1.2.5-gcc/lib:/opt/intel_fc_80/lib;
queue 


#!/bin/sh
##**************************************************************
##
## Copyright (C) 1990-2007, Condor Team, Computer Sciences Department,
## University of Wisconsin-Madison, WI.
## 
## Licensed under the Apache License, Version 2.0 (the "License"); you
## may not use this file except in compliance with the License.  You may
## obtain a copy of the License at
## 
##    http://www.apache.org/licenses/LICENSE-2.0
## 
## Unless required by applicable law or agreed to in writing, software
## distributed under the License is distributed on an "AS IS" BASIS,
## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
## See the License for the specific language governing permissions and
## limitations under the License.
##
##**************************************************************

# Set this to the bin directory of your ompi installation
OMPIDIR=/usr/lib/openmpi/1.2.5-gcc/bin

###### FROM HERE YOU SHOULDN'T HAVE TO CHANGE STUFF ######
# get out some CONDOR-data
_CONDOR_PROCNO=$_CONDOR_PROCNO
_CONDOR_NPROCS=$_CONDOR_NPROCS
_CONDOR_REMOTE_SPOOL_DIR=$_CONDOR_REMOTE_SPOOL_DIR

# find the location of the shell-scripts with condor-ssh stuff
SSHD_SH=`condor_config_val libexec`
SSHD_SH=$SSHD_SH/sshd.sh

CONDOR_SSH=`condor_config_val libexec`
CONDOR_SSH=$CONDOR_SSH/condor_ssh

# source the condor-ssh script
. $SSHD_SH $_CONDOR_PROCNO $_CONDOR_NPROCS 

# If not the head node, just sleep forever, to let the
# sshds run
if [ $_CONDOR_PROCNO -ne 0 ]; then
  wait
  sshd_cleanup
  exit 0
fi

#### only node 0 goes here ####
# get the name of the mpi-executable
EXECUTABLE=$1
shift

# the binary is copied but the executable flag is cleared.
# so take care of this
chmod +x $EXECUTABLE

# Set this to the bin directory of your ompi installation
PATH=$OMPIDIR:$PATH
export PATH

# when a job is killed by the user, this script will get sigterm
# This script has to catch it
finalize() {
  sshd_cleanup
  exit
}
trap finalize TERM

# Next get out the machines that CONDOR has allocated for the job
# The second field in the contact file is the machine name that
# condor_ssh uses:
CONDOR_CONTACT_FILE=$_CONDOR_SCRATCH_DIR/contact
export CONDOR_CONTACT_FILE
sort -n < $CONDOR_CONTACT_FILE | awk '{print $2}' > machines

## now run the actual mpijob
mpirun -machinefile machines $EXECUTABLE $@ &

## wait for it to finish
CHILD=$!
TMP=130
while [ $TMP -gt 128 ] ; do
	wait $CHILD
	TMP=$?;
done

# clean up files
sshd_cleanup
rm -f machines

exit $TMP