Difference: GridServices (16 vs. 17)

Revision 172019-06-04 - GordonStewart

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Grid Services

Line: 13 to 12
 alias setupATLAS='source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh' setupATLAS
Changed:
<
<
lsetup emi

If you plan to submit jobs to the ScotGrid VO, at present you must also amend the X509_VOMSES environment variable as follows:

>
>
lsetup emi
 
Changed:
<
<
export X509_VOMSES=/etc/vomses

>
>
If you plan to submit jobs to the ScotGrid VO, at present you must also amend the X509_VOMSES environment variable as follows:
export X509_VOMSES=/etc/vomses

 

Certificates and proxies

To use grid resources, you will need a certificate, from which you can generate a proxy certificate. The proxy certificate has a relatively short lifetime, and is used to actually submit the job. A proxy is associated with a particular Virtual Organisation (VO), for example vo.scotgrid.ac.uk, which is selected when it is created. You can generate a proxy using the arcproxy command:

Changed:
<
<
$ arcproxy -S <VO_ALIAS> -N

>
>
$ arcproxy -S <VO_ALIAS> -N

 

For example, to generate a proxy for the vo.scotgrid.ac.uk VO:

Changed:
<
<
$ arcproxy -S vo.scotgrid.ac.uk -N

>
>
$ arcproxy -S vo.scotgrid.ac.uk -N

 Your identity: /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=bugs bunny Contacting VOMS server (named vo.scotgrid.ac.uk): voms.gridpp.ac.uk on port: 15509 Proxy generation succeeded
Line: 43 to 35
 

Job description (xRSL)

Before submitting a job, you need to create a file which describes the features of the job for ARC (its executable, the names of input and output files, what to do with logs, etc.). This file is written in the Extended Resource Specification Language (xRSL). A simple job description which runs a script called test.sh could look like this:

Changed:
<
<
&

>
>
&

 (executable = "test.sh") (arguments = "") (jobName = "TestJob")
Line: 61 to 51
 

Submitting a job

Changed:
<
<
Jobs are submitted to a Compute Element (CE). The ScotGrid site at Glasgow has four CEs:

svr009.gla.scotgrid.ac.uk

>
>
Jobs are submitted to a Compute Element (CE). The ScotGrid site at Glasgow has four CEs:
svr009.gla.scotgrid.ac.uk

 svr010.gla.scotgrid.ac.uk svr011.gla.scotgrid.ac.uk svr019.gla.scotgrid.ac.uk
Line: 73 to 61
 It does not matter which CE you choose to submit to. (If you've looked at the tutorial linked above, you'll see that Durham gave their CEs the sensible names ce1, ce2, etc. We thought that would be too easy.)

Jobs are submitted using the arcsub command:

Changed:
<
<
$ arcsub -c <CE_HOSTNAME> <XRSL_FILENAME>

>
>
$ arcsub -c <CE_HOSTNAME> <XRSL_FILENAME>

 

For example, to submit test.xrsl to svr011 at Glasgow:

Changed:
<
<
$ arcsub -c svr011.gla.scotgrid.ac.uk test.xrsl

>
>
$ arcsub -c svr011.gla.scotgrid.ac.uk test.xrsl

 Job submitted with jobid: gsiftp://svr011.gla.scotgrid.ac.uk:2811/jobs/NCKLDmEQkwrnZ4eC5pmRAbBiTBFKDmABFKDmpMFKDmABFKDmQffBxy

When a job is submitted successfully, you will be presented with its job ID which can be used to refer to the job later. Information about submitted jobs is also recorded in a job list file; by default, this file is ~/.arc/jobs.dat (~/.arc/jobs.xml with some earlier versions of ARC), but you can choose a different location by supplying the -j argument to arcsub:

Changed:
<
<
$ arcsub -j <JOBLIST_FILENAME> -c <CE_HOSTNAME> <XRSL_FILENAME>

>
>
$ arcsub -j <JOBLIST_FILENAME> -c <CE_HOSTNAME> <XRSL_FILENAME>

 

For example:

Changed:
<
<
$ arcsub -j test.dat -c svr011.gla.scotgrid.ac.uk test.xrsl

>
>
$ arcsub -j test.dat -c svr011.gla.scotgrid.ac.uk test.xrsl

 

Querying the status of a job

You can obtain information about the status of jobs using the arcstat command:

Changed:
<
<
$ arcstat <JOB_ID>

>
>
$ arcstat <JOB_ID>

 

For example, to obtain information about the job submitted in the previous step:

Changed:
<
<
$ arcstat gsiftp://svr011.gla.scotgrid.ac.uk:2811/jobs/NCKLDmEQkwrnZ4eC5pmRAbBiTBFKDmABFKDmpMFKDmABFKDmQffBxy

>
>
$ arcstat gsiftp://svr011.gla.scotgrid.ac.uk:2811/jobs/NCKLDmEQkwrnZ4eC5pmRAbBiTBFKDmABFKDmpMFKDmABFKDmQffBxy

 Job: gsiftp://svr011.gla.scotgrid.ac.uk:2811/jobs/NCKLDmEQkwrnZ4eC5pmRAbBiTBFKDmABFKDmpMFKDmABFKDmQffBxy Name: StageJob State: Queuing
Line: 119 to 95
 You may have to wait a few minutes after submitting a job before status information becomes available.

You can also query the status of all jobs in a job list file:

Changed:
<
<
$ arcstat -j <JOBLIST_FILENAME>

>
>
$ arcstat -j <JOBLIST_FILENAME>

 

Retrieving job output

Output and log files for a job can be retrieved using the arcget command. As when querying the status of a job, you can use either a job ID or a job list file with this command:

Changed:
<
<
$ arcget <JOB_ID>

>
>
$ arcget <JOB_ID>

 $ arcget -j <JOBLIST_FILENAME>

For example, to get the output of the job submitted above:

Changed:
<
<
$ arcget gsiftp://svr011.gla.scotgrid.ac.uk:2811/jobs/NCKLDmEQkwrnZ4eC5pmRAbBiTBFKDmABFKDmpMFKDmABFKDmQffBxy

>
>
$ arcget gsiftp://svr011.gla.scotgrid.ac.uk:2811/jobs/NCKLDmEQkwrnZ4eC5pmRAbBiTBFKDmABFKDmpMFKDmABFKDmQffBxy

 Results stored at: p6vLDmj3kwrnZ4eC3pmXXsQmABFKDmABFKDm9pFKDmABFKDmtVM1wm Jobs processed: 1, successfully retrieved: 1, successfully cleaned: 1
Line: 146 to 116
 

Copying input and output files ("staging")

You can tell ARC to copy input and output files to and from the compute element by including additional attributes in your xRSL file:

Changed:
<
<
&

>
>
&

 (executable = "test.sh") (arguments = "") (jobName = "TestJob")
Line: 163 to 131
 

Files used in the exectuable, stdout and stderr attributes are transferred automatically, but other files should be listed in the inputFiles or outputFiles attribute as necessary. The inputFiles and outputFiles attributes each take one or more values like this:

Changed:
<
<
("<FILENAME>" "<URL>")

>
>
("<FILENAME>" "<URL>")

 

Where <URL> is left blank, ARC transfers the file to or from the submission machine (this would be the case for input.dat, output.txt and results.tgz in the example xRSL above). Alternatively, a URL may be provided to copy the file to or from a remote resource:

Changed:
<
<
("index.html" "http://www.example.org/index.html")

>
>
("index.html" "http://www.example.org/index.html")

 ("rabbits.zip" "ftp://ftp.example.org/rabbits.zip") ("values.dat" "gsiftp://gridftp.example.org/data/values.dat")
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback