PBS with ALPS Launch Example

Below is a sample PBS script that uses ALPS (i.e. aprun) to launch applications. In this example, one application is using 4 nodes, another is using 1 node, and the SOS aggregator is running on 1 node. In this example, the applications were compiled with TAU measurement support, and TAU was configured to send data to SOS. The TAU and SOS integration is such that TAU will fork-exec the sosd listener daemons on each node if there isn't one running already. This is necessary because ALPS will only allow 1 application to be launched with aprun on each node.

Note - this example assumes that all executables are in the current working directory. Not all filesystems are accessible from the compute nodes on Cray systems, and not all accessible filesystems are writable.

Another Note - ALPS only allows 1 application to be launched through aprun per node, so the nodes in the allocation will be automatically split. This is not true for mpirun on other systems!

#!/bin/bash -l
#PBS -N test 
#PBS -q debug 
#PBS -A ACCOUNTID
#PBS -l nodes=6,walltime=00:10:00 
#PBS -j oe
#PBS -o both.out

# echo all commands - for debugging purposes
set -x

###
# -------------------- set the environment -------------------- #
###

# change to the directory where we were launched
cd $PBS_O_WORKDIR

# get the current working directory
export cwd=`pwd`
echo ${cwd}

# load modules
module swap PrgEnv-pgi PrgEnv-gnu
module load flexpath/1.12
module load adios/1.12.0
module load python/2.7.9

# set some output file names
export xmainOut="xmain.out" 
export readOut="read2.out"

# SOS_FORK_COMMAND is used when TAU can't connect an SOS client to an existing SOS listener.
# The application MPI ranks will self-organize, and only 1 rank from each node will
# attempt to launch the SOS listener on that node, using this command.
# @LISTENER_RANK@ will get subsitited for the node rank within all MPI ranks for
# that application, also using an SOS_LISTENER_RANK_OFFSET value. 
export sos_cmd="${cwd}/sosd -l 5 -a 1 -w ${cwd}"
export SOS_FORK_COMMAND="${sos_cmd} -k @LISTENER_RANK@ -r listener"

# Set the TCP port that the listener will listen to, and the port that clients will
# attempt to connect to.
export SOS_CMD_PORT=22500

# Set the directory where the SOS listeners and aggregators will use to establish
# EVPath links to each other.  This needs to be a shared filesystem path, and it
# needs to be writable, of course.
export SOS_EVPATH_MEETUP=${cwd}

# Tell TAU that it should connect to SOS and send TAU data to SOS when adios_close(),
# adios_advance_step() calls are made, and when the application terminates.
export TAU_SOS=1

# If Verbose TAU output is required for debugging.
# export TAU_VERBOSE=1
# Make sure sosd can find libenet.so, if rpath wasn't used
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/sw/xk6/flexpath/1.12/cle5.2_gnu4.9.3/lib

# clean up old files, if necessary
rm -rf sosd.* profile* *.out

###
# -------------------- Launch the SOS aggregator -------------- #
###

# launch the aggregator - ALPS will take the first node of the allocation
# and the aggregator will be "rank" 0 in the SOS processes. Launch
# in the background, so we can continue launching other aprun calls.
aprun -n 1 -N 1 ${sos_cmd} -k 0 -r aggregator &

# sleep a few seconds, just to allow the aggregator to safely boot
sleep 5 

###
# --------- Launch the reader app in the pipeline ----------- #
###

# 1 node doing reader
# ALPS will take the second node of the allocation
# Tell SOS how many application ranks per node there are
export SOS_APP_RANKS_PER_NODE=16
# Tell SOS what "rank" it's listeners should start with - the
# aggregator was "rank" 0, so this node's listener will be 1
export SOS_LISTENER_RANK_OFFSET=1
# Where should TAU write the profile data?
export PROFILEDIR=profiles_reader2
# Go!  in the background, so we can continue launching aprun calls
aprun -n 16 -N 16 ./reader2 5 x z > ${readOut}  2>&1 &

# Wait a bit.  Just to be safe...
sleep 3

###
# --------- Launch the writer app in the pipeline ----------- #
###

# last 4 nodes doing xmain
# ALPS will take the third, fourth, fifth, sixth nodes of the allocation
# Tell SOS how many application ranks per node there are
export SOS_APP_RANKS_PER_NODE=16
# Tell SOS what "rank" it's listeners should start with - the
# aggregator was "rank" 0, and the reader node was 1, 
# so this node's listeners will start at 2 and be 2,3,4,5
export SOS_LISTENER_RANK_OFFSET=2
# Where should TAU write the profile data?
export PROFILEDIR=profiles_xmain
# Go!  in the foreground, so when xmain exits, the PBS allocation will exit.
aprun -n 64 -N 16 ./xmain > ${xmainOut} 2>&1

# sleep a few seconds...
sleep 3

# wait for clean exit - i.e. all aprun calls have exited
wait