Running Marlin Jobs on the PPE compute farm

It's very easy to run Marlin jobs on the PPE Torque/PBS compute farm: the AFS token is taken care of automatically, and access to the software packages, job input and output is via AFS. Attached to this page is a tarball containing the relevant scripts and template files mentioned in this wiki.

PBS Script

We found it easier to run processors 1 to 6 on batches of 10 input files , and save the outputs as /afs/phas.gla.ac.uk/data/ilc/datasets01/1.4tev/ZVRES_out/runX_out.slcio. The PBS script used is very simple:

#PBS -N ZVRES_Job_Group_22
#PBS -m abe
#PBS -M my.email@example.com
#PBS -q long5

echo -n "Worker node: "
hostname

echo "Running ZVRES Job Group 22 ..." 

/afs/phas.gla.ac.uk/user/p/protopop/ILC/analysis/Marlin/ZZfusion/zvresRunPBS.sh 22

echo "Marlin done!"
echo

where ZVRES_Job_Group_22 is the name under which the job is listed when typing qstat, and zvresRunPBS.sh is our custom script (attached). We actually used another script to produce the individual PBS scripts from a template: sqbq.sh (attached).

Wrapper script

The wrapper script zvresRunPBS.sh (contained in the tarball attached) selects 10 input files, sets the Marlin environment, and runs processors 1 to 6. In addition, it prepares the steering files for the FTI and FT steps (which we ran on our desktops). This is not the only or the simplest way to do this, but it simplified our work:

#!/bin/bash
# Run the most time consuming LCFI processors on a subset of files
# by protopop@cern.ch - Jul 2013 (PBS version)

events="-1"

if [ "$1" -eq "$1" 2> /dev/null ]; then
  dstmp=`date | awk '{print $2$3}'`
  tag="$1"
  echo "Using run tag $tag"
else
  echo "Please provide a numeric tag!"
  exit 1
fi

cd /afs/phas.gla.ac.uk/user/p/protopop/ILC/analysis/Marlin/ZZfusion/batch

steer="zvres$tag.xml"
lcout="/afs/phas.gla.ac.uk/data/ilc/datasets01/1.4tev/ZVRES_out/run$tag-out.slcio"
gearx="../gear_ldc.xml"
flist="../steer/lcio_files.xml"

let "n1=$1*10"
let "n2=n1+9"
files=`sed -n "$n1,$n2"p < $flist`

cp ../steer/template_01.xml "$steer"
ls -lA

echo "Customising template for this batch run ..."
perl -p -i -e "s#MYNEVENTS#$events#g" $steer
perl -p -i -e "s#MYGEARFILE#$gearx#g" $steer
perl -p -i -e "s#MYLCIOFILES#$files#g" $steer
perl -p -i -e "s#MYVERBOSITY#SILENT#g" $steer
perl -p -i -e "s#MYOUTPUTLCIOFILE#$lcout#g" $steer

echo "Your steering file is $steer "

if [ "1" -eq "1" ]; then
 source /afs/phas.gla.ac.uk/data/ilc/software/ilcsoft/v01-14/init_ilcsoft.sh   
 Marlin $steer >/dev/null 2>&1  
else
 echo "Will not run Marlin"
fi

echo "Run $tag done."

nxtst="fti-$tag-steer.xml"
echo "Creating $nxtst steering file ..."
cp fti-3-pbs-steer.xml $nxtst
perl -p -i -e "s#run3#run$tag#g" $nxtst
perl -p -i -e "s#fti3#fti$tag#g" $nxtst

ftstr="ft$tag-gnets.xml"
echo "Creating $ftstr steering file ..."
cp ft1-pbs-gnets.xml $ftstr
perl -p -i -e "s#fti1#fti$tag#g" $ftstr
perl -p -i -e "s#flavourtag1#flavourtag$tag#g" $ftstr

echo "Bye!"
echo

How to run it ?

For the bunch shown in the example above, we used the command subq.sh 22. This produced the PBS script, and submitted it. If other processors are needed, or other paths are preferred, then please feel free to download and modify our scripts as suitable.

Background set

For the background files, we used slightly modified version of the scripts, which can be found in this tarball.