Overview

Queues

Job Prioritisation

The cluster is configured with a fair-share scheduler, which aims to distribute compute time fairly among users. When multiple users are competing for resources, preference will be shown to users whose recent usage has been lower. Short jobs are also generally given priority over longer jobs.

Jobs running in the vlong* queues can be pre-empted by jobs in the short* and medium* queues. A pre-empted job is placed in the suspended state; it remains in memory on the compute node, but is no longer being executed. Once the pre-empting job has finished, the pre-empted job will be allowed to continue.

The PBS headnode is offler.ppe.gla.ac.uk, and you will see this name in the output of various PBS commands.

$ qsub test.pbs
1000150.offler.ppe.gla.ac.uk

$ qstat

offler.ppe.gla.ac.uk:
                                                                         Req'd  Req'd   Elap
Job ID               Username Queue    Jobname          SessID NDS   TSK Memory Time  S Time
-------------------- -------- -------- ---------------- ------ ----- --- ------ ----- - -----
1000151.offler.p     rrabbit  medium6  maus_sim_814      56289     1   1    --  05:59 R 03:21   node034
1000152.offler.p     bbunny   long6    test_job          29669     1   1    --  24:00 R 01:24   node007

This

Delete a job

Jobs are deleted using the qdel command:

$ qsub <JOB_ID>

To delete the job with ID 12345:

$ qdel 12345

References

• qsub
• qdel

Using PBS

Create a submission script

Submit a job

Jobs are submitted using the qsub command:

$ qsub <FILENAME>

To submit a job defined by the submission script test.pbs:

$ qsub test.pbs

More details can be found in the qsub man page.

Show running jobs

Killing a job

Jobs may be terminated by executing qdel JOBID where the JOBID is the numerical ID code returned in the qstat listing.

• There is a constant weighting given to short jobs and smaller weighting given to medium and long jobs. So that if all other factors are equal short jobs will have priority.
• The primary weighting is user fairshare. As a users jobs run their usage increases and the priority of their queued jobs decreases. This is balanced so that a user who uses exactly their fairshare allotment (currently 20% of the cpu averaged over the previous 48 days) will have their medium job priority decreased such that the medium job priority is equal to someone else's vlong job priority who has not used the batch system in the previous 48 days.
• Waiting jobs priority slowly increases as a function of time waiting in the queue. Currently a vlong job would have to wait several weeks to match the priority of a medium queue job all other things being equal.

To see the current priorities of waiting jobs use the command showq -i.

Job Pre-emption

Job Priority

The priority of a job is the sum of several weighting factors.

• There is a constant weighting given to short jobs and smaller weighting given to medium and long jobs. So that if all other factors are equal short job will have priority.
• The primary weighting is user fairshare. As a users jobs run their usage increases and the priority of their queued jobs decreases. This is balanced so that a user who uses exactly their fairshare allotment (currently 20% of the cpu averaged over the previous week) will have their medium job priority decreased such that the medium job priority is equal to someone elses vlong job priority who has not used the batch system in the previous week.
• A waiting jobs priority slowly increases as a function of time waiting in the queue. Currently a vlong job would have to wait several weeks to match the priority of a medium queue job all other things being equal.

Jobs in the vlong4 and vlong5 queues can be preempted by jobs waiting in the short4, short5, medium4 or medium5 queues. A preempted job is placed in the suspended state - it remains in memory but is not longer being executed. Once the preempting job has finished the preempted job starts executing again.

While it is possible to view your own jobs with qstat, the command will not display all jobs. To display all jobs use the Maui client command showq

uname -m

a 64 bit machine will return x86_64 and 32 bit machine i686.

Batch System

The PPE group has limited resources for batch computing. The ppepbs batch system is provided for running a small number of jobs. If a large number of jobs are needed then please use the Grid or try the Compute Cluster

PPEPBS

The PPE group has a small batch system that is managed via the TORQUE Resource Manager (based on OpenPBS) and the Maui scheduler. Each of the nodes within this batch system has 64 bit SL 4.4 installed. Currently the LINUX desktops are a mixture of SL4x and SL5x. To run on the batch system executables should therefore be built on an SL4x PPE LINUX desktop or PPELX (see LoginServices). To check the version scientific linux installed:

cat /etc/redhat-release

Job submission

ssh ppepbs
qsub test.job

where test.job might contain

#PBS -N TestJob
#PBS -l walltime=1,mem=1024Mb
#PBS -m abe
#PBS -M user@machine
#
echo "This is a test..."

More documentation is given in the qsub man page. The TORQUE documentation pages installed on ppepbs can be listed via

ssh ppepbs
rpm -ql torque-docs

Queues

Queue            Memory CPU Time Walltime Node  Run Que Lm  State
---------------- ------ -------- -------- ----  --- --- --  -----
short              --      --    01:00:00   --    0   0 --   E R
medium             --      --    06:00:00   --    0   0 --   E R
long               --      --    24:00:00   --    0   0 --   E R
vlong              --      --    120:00:0   --    0   0 --   E R

where short is the default queue and Walltime is the maximum walltime allowed on each queue. While it is possible to view your own jobs with qstat, the command will not display all jobs. To display all jobs use the Maui client command showq

-- AndrewPickford - 12 Jan 2009