ce01.gla.scotgrid.ac.uk
ce02.gla.scotgrid.ac.uk
ce03.gla.scotgrid.ac.uk
ce04.gla.scotgrid.ac.uk

$ arcsub -c ce03.gla.scotgrid.ac.uk test.xrsl
Job submitted with jobid: gsiftp://ce03.gla.scotgrid.ac.uk:2811/jobs/NCKLDmEQkwrnZ4eC5pmRAbBiTBFKDmABFKDmpMFKDmABFKDmQffBxy

$ arcsub -j test.dat -c ce03.gla.scotgrid.ac.uk test.xrsl

$ arcstat gsiftp://ce03.gla.scotgrid.ac.uk:2811/jobs/NCKLDmEQkwrnZ4eC5pmRAbBiTBFKDmABFKDmpMFKDmABFKDmQffBxy
Job: gsiftp://ce03.gla.scotgrid.ac.uk:2811/jobs/NCKLDmEQkwrnZ4eC5pmRAbBiTBFKDmABFKDmpMFKDmABFKDmQffBxy

$ arcget gsiftp://ce03.gla.scotgrid.ac.uk:2811/jobs/NCKLDmEQkwrnZ4eC5pmRAbBiTBFKDmABFKDmpMFKDmABFKDmQffBxy

export X509_VOMSES=/etc/vomses

$ arcproxy -S <VO_ALIAS> -N

$ arcproxy -S vo.scotgrid.ac.uk -N

&

svr009.gla.scotgrid.ac.uk

$ arcsub -c <CE_HOSTNAME> <XRSL_FILENAME>

$ arcsub -c svr011.gla.scotgrid.ac.uk test.xrsl

$ arcsub -j <JOBLIST_FILENAME> -c <CE_HOSTNAME> <XRSL_FILENAME>

$ arcsub -j test.dat -c svr011.gla.scotgrid.ac.uk test.xrsl

$ arcstat <JOB_ID>

$ arcstat gsiftp://svr011.gla.scotgrid.ac.uk:2811/jobs/NCKLDmEQkwrnZ4eC5pmRAbBiTBFKDmABFKDmpMFKDmABFKDmQffBxy

$ arcstat -j <JOBLIST_FILENAME>

$ arcget <JOB_ID>

$ arcget gsiftp://svr011.gla.scotgrid.ac.uk:2811/jobs/NCKLDmEQkwrnZ4eC5pmRAbBiTBFKDmABFKDmpMFKDmABFKDmQffBxy

&

("<FILENAME>" "<URL>")

("index.html" "http://www.example.org/index.html")

Copying input and output files ("staging")

You can tell ARC to copy input and output files to and from the compute element by including additional attributes in your xRSL file:

&
(executable = "test.sh")
(arguments = "")
(jobName = "TestJob")
(inputFiles = ("input.dat" ""))
(outputFiles = ("output.txt" "")
               ("results.tgz" "")
)
(stdout = "stdout")
(stderr = "stderr")
(gmlog = "gmlog")
(walltime="60")

Files used in the exectuable, stdout and stderr attributes are transferred automatically, but other files should be listed in the inputFiles or outputFiles attribute as necessary. The inputFiles and outputFiles attributes each take one or more values like this:

("<FILENAME>" "<URL>")

Where <URL> is left blank, ARC transfers the file to or from the submission machine (this would be the case for input.dat, output.txt and results.tgz in the example xRSL above). Alternatively, a URL may be provided to copy the file to or from a remote resource:

("index.html" "http://www.example.org/index.html")
("rabbits.zip" "ftp://ftp.example.org/rabbits.zip")
("values.dat" "gsiftp://gridftp.example.org/data/values.dat")

Various protocols are supported, including Rucio and SRM, and details can be found in the ARC reference manual:

http://www.nordugrid.org/documents/xrsl.pdf

Due to the slightly convoluted path files follow to make their way from the submission machine through the CE to the compute node, it is easiest to avoid using paths when specifying outputFiles. Instead, if files are created in subdirectories, it may be simpler to copy these files back to $HOME at the end of the script (this is a working directory belonging to your job, and is not related to your home directory). You may also wish to add multiple output files or directories to an archive, in order to simplify the process of retrieving results further.

$ arcsub -j <JOBLIST_FILENAME> -c <CE_HOSTNAME> <XRSL_FILENAME>

For example:

$ arcsub -j test.dat -c svr011.gla.scotgrid.ac.uk test.xrsl

Querying the status of a job

You can obtain information about the status of jobs using the arcstat command:

$ arcstat <JOB_ID>

For example, to obtain information about the job submitted in the previous step:

$ arcstat gsiftp://svr011.gla.scotgrid.ac.uk:2811/jobs/NCKLDmEQkwrnZ4eC5pmRAbBiTBFKDmABFKDmpMFKDmABFKDmQffBxy
Job: gsiftp://svr011.gla.scotgrid.ac.uk:2811/jobs/NCKLDmEQkwrnZ4eC5pmRAbBiTBFKDmABFKDmpMFKDmABFKDmQffBxy
 Name: StageJob
 State: Queuing

Status of 1 jobs was queried, 1 jobs returned information

You may have to wait a few minutes after submitting a job before status information becomes available.

You can also query the status of all jobs in a job list file:

$ arcstat -j <JOBLIST_FILENAME>

Retrieving job output

Output and log files for a job can be retrieved using the arcget command. As when querying the status of a job, you can use either a job ID or a job list file with this command:

$ arcget <JOB_ID>
$ arcget -j <JOBLIST_FILENAME>

For example, to get the output of the job submitted above:

$ arcget gsiftp://svr011.gla.scotgrid.ac.uk:2811/jobs/NCKLDmEQkwrnZ4eC5pmRAbBiTBFKDmABFKDmpMFKDmABFKDmQffBxy
Results stored at: p6vLDmj3kwrnZ4eC3pmXXsQmABFKDmABFKDm9pFKDmABFKDmtVM1wm
Jobs processed: 1, successfully retrieved: 1, successfully cleaned: 1

You will only be able to retrieve job output once the job has finished.

export X509_VOMSES=/etc/vomses

svr009.gla.scotgrid.ac.uk
svr010.gla.scotgrid.ac.uk
svr011.gla.scotgrid.ac.uk
svr019.gla.scotgrid.ac.uk

It does not matter which CE you choose to submit to. (If you've looked at the tutorial linked above, you'll see that Durham gave their CEs the sensible names ce1, ce2, etc. We thought that would be too easy.)

Jobs are submitted using the arcsub command:

arcsub -j <DATABASE_FILENAME> -c <CE_HOSTNAME> <XRSL_FILENAME>

For example, to submit test.xrsl to svr011 at Glasgow:

arcsub -j test.db -c svr011.gla.scotgrid.ac.uk test.xrsl

introductory tutorial

ARC tools

Certificates and proxies

To use grid resources, you will need a certificate, from which you can generate a proxy certificate. The proxy certificate has a relatively short lifetime, and is used to actually submit the job. A proxy is associated with a particular Virtual Organisation (VO), for example vo.scotgrid.ac.uk, which is selected when it is created. You can generate a proxy using the arcproxy command:

arcproxy -S <VO_ALIAS> -N

For example, to generate a proxy for the vo.scotgrid.ac.uk VO:

arcproxy -S vo.scotgrid.ac.uk -N

Job description (xRSL)

Before submitting a job, you need to create a file which describes the features of the job for ARC (its executable, the names of input and output files, what to do with logs, etc.). This file is written in the Extended Resource Specification Language (xRSL). A simple job description which runs a script called test.sh could look like this:

&
(executable = "test.sh")
(arguments = "")
(jobName = "TestJob")
(stdout = "stdout")
(stderr = "stderr")
(gmlog = "test.log")
(walltime="60")

A full description of xRSL can be found in the ARC reference manual:

http://www.nordugrid.org/documents/xrsl.pdf

Submitting a job

setupATLAS
localSetupEmi
 

setupATLAS
localSetupDQ2Client
 

Using the UI

Source the appropriate setup file depending on your current shell and sl version of your machine. If you are using the dq2 client this needs to be done after sourcing the dq2 client setup file.

To check your current shell use: echo $SHELL

To check the version of SL use: cat /etc/redhat-release

Using the UI

To check the version of SL use: cat /etc/redhat-release

Using the dq2-client

source /data/ppe01/sl44/i386/grid/glite-ui/latest/external/etc/profile.d/grid-env.csh 

Using GANGA for ATLAS

then follow the Physics Analysis Workbook (Batch 2) https://twiki.cern.ch/twiki/bin/view/AtlasProtected/PhysicsAnalysisWorkBookBatch2

and https://twiki.cern.ch/twiki/bin/view/Atlas/FullGangaAtlasTutorial

Now try these simple commands:

dq2-ls alpgen.105762.ttbarlnlnNp2_lowQ

and to show the individual files (This may not work on SL5 machines)

dq2-ls -f alpgen.105762.ttbarlnlnNp2_lowQ

Grid Services

Grid User Interface

Every PPE SL4x desktop can act as a EGEE gLite UI. The User Interface (UI) tools, dq2-client and GANGA are installed in /data/ppe01/sl44/i386/grid

Using the UI

Check your current shell

echo $SHELL 

If you are using bash or zsh then

source /data/ppe01/sl44/i386/grid/glite-ui/latest/external/etc/profile.d/grid-env.sh 

If you are using tcsh or csh then

source /data/ppe01/sl44/i386/grid/glite-ui/latest/external/etc/profile.d/grid-env.csh 

Using the dq2-client

Setup the UI as mentioned in the previous section.

If you are using bash or zsh then

source /data/ppe01/sl44/i386/grid/dq2-client/setup.sh 

If you are using tcsh or csh then

source /data/ppe01/sl44/i386/grid/dq2-client/setup.csh 

Then create a voms proxy

voms-proxy-init -voms atlas 

and use the dq2 Tools as needed.

Using GANGA

Setup the UI as mentioned in the previous section.

If you are using bash or zsh then

source /data/ppe01/sl44/i386/grid/ganga/setup.sh 

If you are using tcsh or csh then

source /data/ppe01/sl44/i386/grid/ganga/setup.csh 

Refer to the ATLAS, LHCb or core GANGA documentation.

The release installed contains GangaAtlas, GangaLHCb, GangaNG, GangaCronus, GangaGUI, GangaPlotter

-- WilliamBell - 08 Aug 2008