Difference: RunningGangaWithPanda (1 vs. 15)

Revision 152015-01-07 - GraemeStewart

Line: 1 to 1
Changed:
<
<
-- ThomasDoherty - 2009-10-26
>
>

Deprecated

This page is now only of historical interest. For up to date ganga and pathena instructions, please see the information in the ATLAS Software Tutorial.

 

Using Ganga to submit jobs to the Panda backend on lxplus

References:

Line: 8 to 10
  Data preparation reprocessing - using Ganga
Changed:
<
<
https://twiki.cern.ch/twiki/bin/view/Atlas/RegularComputingTutorial#DAY_4_USING_THE_GRID
>
>
https://twiki.cern.ch/twiki/bin/view/Atlas/RegularComputingTutorial#DAY_4_USING_THE_GRID
  1. In a clean lxplus afs shell, setup Ganga.
Changed:
<
<
  source /afs/cern.ch/sw/ganga/install/etc/setup-atlas.sh
>
>
  source /afs/cern.ch/sw/ganga/install/etc/setup-atlas.sh
  2. Setup the athena release.

NOTE: To set up for any release one must be familar with using CMT (bootstrap procedures and requirement files) - see here for more information. In this example (for reprocessing see the reference page above for the necessary release) once the requirements file is set up and a release directory in your test area is created then try:

Changed:
<
<
  source ~/cmthome/setup.sh -tag=14.5.2.6,32,AtlasProduction
>
>
  source ~/cmthome/setup.sh -tag=14.5.2.6,32,AtlasProduction
  3. Setup any checked out packages you use in your code.

For this example check out (and compile) the UserAnalysis package as in the "HelloWorld"example here or here.

Changed:
<
<
  cd $TESTAREA/14.5.2.6/PhysicsAnalysis/AnalysisCommon/UserAnalysis/cmt
  source setup.sh
>
>
  cd $TESTAREA/14.5.2.6/PhysicsAnalysis/AnalysisCommon/UserAnalysis/cmt
  source setup.sh
  NOTE: For demonstration purposes (to show that this setup does indeed pull in the code changes made to a checked out package) - I have appended to a comment in AnalysisSkeleton.cxx (i.e I changed "No AOD MC truth particle container found in TDS" to "No AOD MC truth particle container found in TDS - This comment changed by Tom")

4. Go to run directory and start ganga (once the code is compiled).NOTE ganga picks up your grid cert/key from your .globus directory on lxplus - if you have not created these files please follow the instructions here under the preparing the certificate section. Also you must create a file called 'vomses' in a new directory called .glite (again note the '.') and enter this line for the ATLAS VO - "atlas" "voms.cern.ch" "15001" "/C=CH/O=CERN/OU=GRID/CN=host/voms.cern.ch" "atlas" into this file.

Deleted:
<
<
   cd ../run
   ganga
 
Changed:
<
<
5. Before you prepare/run your Ganga JO you must prepare the Athena JO it points to.For this you can use the top job options copied from your UserAnalysis packages share directory.
>
>
   cd ../run
   ganga
 
Added:
>
>
5. Before you prepare/run your Ganga JO you must prepare the Athena JO it points to.For this you can use the top job options copied from your UserAnalysis packages share directory.
 
cp ../share/AnalysisSkeleton_topOptions.py .

BUT to prepare your code for running on the Grid there are some changes needed for this Athena JO - please add these lines:

Line: 54 to 52
 from DBReplicaSvcConf import DBReplicaSvc ServiceMgr+=DBReplicaSvc(UseCOOLSQLite=False)
Changed:
<
<
_________________________________________________________________________
>
>
_____________________________________________________________________________
  Also remember to remove (or comment out) the input data line and if you are running a reprocessing job change the geometry tag and the conditions DB tag to match those used in the reprocessing cycle (see details for each reprocessing campaign on this page here. For example:
Changed:
<
<
globalflags.ConditionsTag.set_Value_and_Lock('COMCOND-REPC-002-13')
>
>
globalflags.ConditionsTag.set_Value_and_Lock('COMCOND-REPC-002-13')
  6. Execute your Ganga job script while Ganga is running (where an example of what the 'pandaBackend_test.py' would look like is below in other words have this file in your run directory) and type:
Deleted:
<
<
    execfile('pandaBackend_test.py') 
 
Changed:
<
<
or simply from the command line run ganga with the name of the Ganga JO appended:
>
>
    execfile('pandaBackend_test.py') 
 
Added:
>
>
or simply from the command line run ganga with the name of the Ganga JO appended:
 
    ganga pandaBackend_test.py

6. You can monitor your job's progress by typing jobs inside Ganga or, if you submitted to the Panda backend by http://panda.cern.ch:25880/server/pandamon/query.

7. Once your job has finished you can copy the output data using the dq2 tools.

Changed:
<
<
          
source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.sh
   dq2-get "your_dataset_name"
>
>
source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.sh
   dq2-get "your_dataset_name"
  Where "your_dataset_name" is given to you by Ganga once the job completes. Also once the job completes Panda in particular sends you an email like this. In the email if you click on the 'PandaMonURL' link - in my case for job id 81.Then click on any of the sub-job pandaID's for example ' 1026613035' then scroll down to the link 'Find and view log files' - you can look at the Ganga log for your subjob which is named 'athena_stdout.txt' (only look at 'athena_stderr.txt if your job does not complete). I can then find that"No AOD MC truth particle container found in TDS - This comment changed by Tom" appears in this log.

The Ganga JO 'pandaBackend_test.py' could look like this (without line numbers):

Changed:
<
<
1    j = Job()
>
>
1    j = Job()
 2 j.application = Athena() 3 j.application.atlas_dbrelease = 'ddo.000001.Atlas.Ideal.DBRelease.v06060101:DBRelease-6.6.1.1.tar.gz' 4 j.application.option_file = 'AnalysisSkeleton_topOptions.py'
Line: 95 to 91
 13 j.submit()

Deleted:
<
<
  For the LCG() backend you might also need
j.outputdata.outputdata=['AnalysisSkeleton.aan.root']
which should match exactly the output file name from your jobs.
Line: 182 to 177
 

Reusing the environment

Changed:
<
<
>
>
 
  If you're running the same code over multiple datasets, you can save time by telling jobs after the first job to use the same environment. This means that panda doesn't have to run a new `build' job every time, and can make things much faster. This works as long as you don't recompile anything in your environment between jobs, you can change the joboptions.

Revision 142013-12-17 - AndrewPickford

Line: 1 to 1
 -- ThomasDoherty - 2009-10-26

Using Ganga to submit jobs to the Panda backend on lxplus

Line: 32 to 32
  NOTE: For demonstration purposes (to show that this setup does indeed pull in the code changes made to a checked out package) - I have appended to a comment in AnalysisSkeleton.cxx (i.e I changed "No AOD MC truth particle container found in TDS" to "No AOD MC truth particle container found in TDS - This comment changed by Tom")
Changed:
<
<
4. Go to run directory and start ganga (once the code is compiled).NOTE ganga picks up your grid cert/key from your .globus directory on lxplus - if you have not created these files please follow the instructions here under the preparing the certificate section. Also you must create a file called 'vomses' in a new directory called .glite (again note the '.') and enter this line for the ATLAS VO - "atlas" "voms.cern.ch" "15001" "/C=CH/O=CERN/OU=GRID/CN=host/voms.cern.ch" "atlas" into this file.
>
>
4. Go to run directory and start ganga (once the code is compiled).NOTE ganga picks up your grid cert/key from your .globus directory on lxplus - if you have not created these files please follow the instructions here under the preparing the certificate section. Also you must create a file called 'vomses' in a new directory called .glite (again note the '.') and enter this line for the ATLAS VO - "atlas" "voms.cern.ch" "15001" "/C=CH/O=CERN/OU=GRID/CN=host/voms.cern.ch" "atlas" into this file.
 
   cd ../run
   ganga

Revision 132010-06-11 - NickEdwards

Line: 1 to 1
 -- ThomasDoherty - 2009-10-26

Using Ganga to submit jobs to the Panda backend on lxplus

Line: 62 to 62
 

6. Execute your Ganga job script while Ganga is running (where an example of what the 'pandaBackend_test.py' would look like is below in other words have this file in your run directory) and type:

Changed:
<
<
    execfile('pandaBackend_test.py') 
>
>
    execfile('pandaBackend_test.py') 
  or simply from the command line run ganga with the name of the Ganga JO appended:
Line: 96 to 97
 
Changed:
<
<
For the LCG() backend you might also need
j.outputdata.outputdata=['AnalysisSkeleton.aan.root']
which should match exactly the output file name from your jobs.

Also for the LCG() backend, a site can be specified:
j.backend.requirements.other=['other.GlueSiteUniqueID=="UKI-SCOTGRID-GLASGOW"']

>
>
For the LCG() backend you might also need
j.outputdata.outputdata=['AnalysisSkeleton.aan.root']
which should match exactly the output file name from your jobs.
 
Changed:
<
<
To submit to the UK Cloud add
j.backend.requirements.cloud='UK'
>
>
Also for the LCG() backend, a site can be specified:
j.backend.requirements.other=['other.GlueSiteUniqueID=="UKI-SCOTGRID-GLASGOW"']
 
Changed:
<
<
>
>
To submit to the UK Cloud add
j.backend.requirements.cloud='UK'
 
Changed:
<
<
NOTE: Line 3 is an example of overriding a database release to match the one needed to read ESD/DPD. In the case of the spring cosmic reprocessing,the DB release is 6.6.1.1. If the database releases don't match the jobs fail on the Grid ( remove this line if it is not necessary). Line 4 corresponds to your Athena jobOptions. Line 5 is set to False because we have already compiled the packages locally if you want your job to compile your checked out code before submitting then simply change this to True Line 6 tells Ganga to tar your user area and send it with the job. Line 10 specifies the backend to which you are sending your job. There are three options: LCG, Panda and NorduGrid. In the example above Panda was chosen because the data existed only in BNLPANDA, a site in the US cloud. Line 12 corresponds to the number of subjobs you want to split your job into. Finally in Line 13 you submit your job
>
>
</pre>
 
Added:
>
>
NOTE: Line 3 is an example of overriding a database release to match the one needed to read ESD/DPD. In the case of the spring cosmic reprocessing,the DB release is 6.6.1.1. If the database releases don't match the jobs fail on the Grid ( remove this line if it is not necessary). Line 4 corresponds to your Athena jobOptions. Line 5 is set to False because we have already compiled the packages locally if you want your job to compile your checked out code before submitting then simply change this to True Line 6 tells Ganga to tar your user area and send it with the job. Line 10 specifies the backend to which you are sending your job. There are three options: LCG, Panda and NorduGrid. In the example above Panda was chosen because the data existed only in BNLPANDA, a site in the US cloud. Line 12 corresponds to the number of subjobs you want to split your job into. Finally in Line 13 you submit your job
 
Changed:
<
<
The Ganga output looks something like this (Note the output is a dataset: Output dataset user09.chriscollins.ganga.2.20091210 ):
run% ganga pandaBackend_test.py

>
>
The Ganga output looks something like this (Note the output is a dataset: Output dataset user09.chriscollins.ganga.2.20091210 ):
run% ganga pandaBackend_test.py

  * Welcome to Ganga * Version: Ganga-5-4-3
Line: 171 to 158
 Ganga.GPIDev.Lib.Job : INFO job 2 status changed to "submitted"
Changed:
<
<
Helpful commands inside Ganga:
jobs lists your jobs
jobs(1) lists the content of job 1
help() goes into help mode ( quit to leave help)
j=jobs(1) and j.kill() will kill job 1.
>
>
Helpful commands inside Ganga:
jobs lists your jobs
jobs(1) lists the content of job 1
help() goes into help mode ( quit to leave help)
j=jobs(1) and j.kill() will kill job 1.
 
Changed:
<
<
Your output will be in the dq2 registered dataset. For me this was user09.chriscollins.ganga.2.20091210 Again this is available from jobs(x)
>
>
Your output will be in the dq2 registered dataset. For me this was user09.chriscollins.ganga.2.20091210 Again this is available from jobs(x)
 

Running Executable on panda Backend

Line: 198 to 177
 j.submit()

_________________________________________________________________________

Added:
>
>

Reusing the environment

 
Added:
>
>
If you're running the same code over multiple datasets, you can save time by telling jobs after the first job to use the same environment. This means that panda doesn't have to run a new `build' job every time, and can make things much faster. This works as long as you don't recompile anything in your environment between jobs, you can change the joboptions.


First you need to find the name of the previous library. Say we want to reuse the library built in ganga job 323: given a job ID:


Either: in ganga:

In [19]:jobs(323).backend.libds

Out[19]: user.nedwards.20100610.323.lib


or use dq2:

$ dq2-ls %nedwards%323*

user.nedwards.20100610.323

user.nedwards.20100610.323.lib

user.nedwards.20100610.323_sub07797849


The library is user.nedwards.20100610.323.lib. In subsequent job options you can set:

 
Added:
>
>
j.backend.libds = 'user.nedwards.20100610.323.lib'

Revision 122010-03-29 - MichaelWright

Line: 1 to 1
 -- ThomasDoherty - 2009-10-26

Using Ganga to submit jobs to the Panda backend on lxplus

Line: 179 to 179
  Your output will be in the dq2 registered dataset. For me this was user09.chriscollins.ganga.2.20091210 Again this is available from jobs(x)
Added:
>
>

Running Executable on panda Backend

This has been said to work on panda backend. It has not been tested. If found to be successful please UPDATE!

______________________________________________________________________________

j = Job()
j.application=Executable()
j.application.exe='myscript.sh'
j.backend=Panda()
j.backend.site='ANALY_FZK'
j.inputsandbox=['/path/to/myscript.sh']
j.submit()

_____________________________________________________________________________

Revision 112010-02-03 - ChrisCollins

Line: 1 to 1
 -- ThomasDoherty - 2009-10-26

Using Ganga to submit jobs to the Panda backend on lxplus

Line: 98 to 98
  For the LCG() backend you might also need
j.outputdata.outputdata=['AnalysisSkeleton.aan.root']
Changed:
<
<
which sould match exaclt y the output file name from your jobs.
>
>
which should match exactly the output file name from your jobs.

Also for the LCG() backend, a site can be specified:
j.backend.requirements.other=['other.GlueSiteUniqueID=="UKI-SCOTGRID-GLASGOW"']

  To submit to the UK Cloud add
j.backend.requirements.cloud='UK'

Revision 102010-02-03 - ChrisCollins

Line: 1 to 1
 -- ThomasDoherty - 2009-10-26

Using Ganga to submit jobs to the Panda backend on lxplus

Line: 96 to 96
 
Changed:
<
<
you might also need
>
>
For the LCG() backend you might also need
 j.outputdata.outputdata=['AnalysisSkeleton.aan.root']
which sould match exaclt y the output file name from your jobs.

Revision 92010-02-01 - MichaelWright

Line: 1 to 1
 -- ThomasDoherty - 2009-10-26

Using Ganga to submit jobs to the Panda backend on lxplus

Line: 100 to 100
 j.outputdata.outputdata=['AnalysisSkeleton.aan.root']
which sould match exaclt y the output file name from your jobs.
Added:
>
>
To submit to the UK Cloud add
j.backend.requirements.cloud='UK'
 

Revision 82009-12-10 - ChrisCollins

Line: 1 to 1
 -- ThomasDoherty - 2009-10-26

Using Ganga to submit jobs to the Panda backend on lxplus

Line: 8 to 8
  Data preparation reprocessing - using Ganga
Added:
>
>
https://twiki.cern.ch/twiki/bin/view/Atlas/RegularComputingTutorial#DAY_4_USING_THE_GRID
 1. In a clean lxplus afs shell, setup Ganga.
  source /afs/cern.ch/sw/ganga/install/etc/setup-atlas.sh
Line: 94 to 96
 
Added:
>
>
you might also need
j.outputdata.outputdata=['AnalysisSkeleton.aan.root']
which sould match exaclt y the output file name from your jobs.

 NOTE: Line 3 is an example of overriding a database release to match the one needed to read ESD/DPD. In the case of the spring cosmic reprocessing,the DB release is 6.6.1.1. If the database releases don't match the jobs fail on the Grid ( remove this line if it is not necessary). Line 4 corresponds to your Athena jobOptions. Line 5 is set to False because we have already compiled the packages locally if you want your job to compile your checked out code before submitting then simply change this to True Line 6 tells Ganga to tar your user area and send it with the job.
Line: 102 to 111
 a site in the US cloud. Line 12 corresponds to the number of subjobs you want to split your job into. Finally in Line 13 you submit your job
Added:
>
>

The Ganga output looks something like this (Note the output is a dataset: Output dataset user09.chriscollins.ganga.2.20091210 ):

run% ganga pandaBackend_test.py

*** Welcome to Ganga ***
Version: Ganga-5-4-3
Documentation and support: http://cern.ch/ganga
Type help() or help('index') for online help.

This is free software (GPL), and you are welcome to redistribute it
under certain conditions; type license() for details.


For help visit the ATLAS Distributed Analysis Help eGroup:
  https://groups.cern.ch/group/hn-atlas-dist-analysis-help/

GangaAtlas                         : INFO     Found 0 tasks
Ganga.GPIDev.Lib.JobRegistry       : INFO     Found 2 jobs in "jobs", completed in 0 seconds
Ganga.GPIDev.Lib.JobRegistry       : INFO     Found 0 jobs in "templates", completed in 0 seconds

********************************************************************
New in 5.2.0: Change the configuration order w.r.t. Athena.prepare()
              New Panda backend schema - not backwards compatible
For details see the release notes or the wiki tutorials
********************************************************************

GangaAtlas.Lib.Athena              : WARNING  New prepare() method has been called. The old prepare method is c   alled now prepare_old()
GangaAtlas.Lib.Athena              : INFO     Found Working Directory /home/chrisc/atlas/GANGA-TEST-15.3.0.1/15   .3.0.1
GangaAtlas.Lib.Athena              : INFO     Found ATLAS Release 15.3.0
GangaAtlas.Lib.Athena              : INFO     Found ATLAS Production Release 15.3.0.1
GangaAtlas.Lib.Athena              : INFO     Found ATLAS Project AtlasProduction
GangaAtlas.Lib.Athena              : INFO     Found ATLAS CMTCONFIG i686-slc4-gcc34-opt
GangaAtlas.Lib.Athena              : INFO     Using run directory: PhysicsAnalysis/HiggsPhys/HiggsAssocTop/TtHH   bbDPDBasedAnalysis/run/
GangaAtlas.Lib.Athena              : INFO     Extracting athena run configuration ...
GangaAtlas.Lib.Athena              : INFO     Detected Athena run configuration: {'input': {'noInput': True}, '   other': {}, 'output': {'outNtuple': ['FILE1'], 'alloutputs': ['D3PD.root']}}
GangaAtlas.Lib.Athena              : INFO     Creating /tmp/chrisc/sources.f3f6d811-f7cd-42d3-8d50-47c47f58ae78   .tar ...
GangaAtlas.Lib.Athena              : INFO     Option athena_compile=False. Adding InstallArea to /tmp/chrisc/so   urces.f3f6d811-f7cd-42d3-8d50-47c47f58ae78.tar ...
Ganga.GPIDev.Lib.Job               : INFO     submitting job 2
Ganga.GPIDev.Lib.Job               : INFO     job 2 status changed to "submitting"
GangaPanda.Lib.Panda               : INFO     Panda brokerage results: cloud UK, site ANALY_GLASGOW
GangaAtlas.Lib.ATLASDataset        : WARNING  Dataset mc08.106314.Pythia_ttH120_2l2nu4b.merge.AOD.e364_s462_r63   5_t53_tid065132 has 8 locations
GangaAtlas.Lib.ATLASDataset        : WARNING  Please be patient - waiting for site-index update at site UKI-SCO   TGRID-GLASGOW_LOCALGROUPDISK ...
GangaAtlas.Lib.Athena              : WARNING  You are using DQ2JobSplitter.filesize or the backend used support   s only a maximum dataset size of 10000 MB per subjob - job splitting has been adjusted accordingly.
GangaPanda.Lib.Athena              : INFO     Input dataset(s) ['mc08.106314.Pythia_ttH120_2l2nu4b.merge.AOD.e3   64_s462_r635_t53_tid065132']
GangaPanda.Lib.Athena              : INFO     Output dataset user09.chriscollins.ganga.2.20091210
GangaPanda.Lib.Athena              : INFO     Running job options: TtAnalysis-ttHSignalSMALL-GRID.py
GangaPanda.Lib.Panda               : INFO     Uploading source tarball sources.f3f6d811-f7cd-42d3-8d50-47c47f58   ae78.tar.gz in /tmp/chrisc to Panda...
Ganga.GPIDev.Lib.Job               : INFO     job 2.0 status changed to "submitting"
Ganga.GPIDev.Lib.Job               : INFO     job 2.0 status changed to "submitted"
Ganga.GPIDev.Lib.Job               : INFO     job 2 status changed to "submitted"

Helpful commands inside Ganga:
jobs lists your jobs
jobs(1) lists the content of job 1
help() goes into help mode ( quit to leave help)
j=jobs(1) and j.kill() will kill job 1.

Your output will be in the dq2 registered dataset. For me this was user09.chriscollins.ganga.2.20091210 Again this is available from jobs(x)

Revision 72009-11-16 - ThomasDoherty

Line: 1 to 1
 -- ThomasDoherty - 2009-10-26
Changed:
<
<

Using Ganga to submit jobs to the Panda backend

>
>

Using Ganga to submit jobs to the Panda backend on lxplus

  References:
Line: 8 to 8
  Data preparation reprocessing - using Ganga
Changed:
<
<
1. In a clean afs shell, setup Ganga.
>
>
1. In a clean lxplus afs shell, setup Ganga.
 
  source /afs/cern.ch/sw/ganga/install/etc/setup-atlas.sh
Line: 30 to 30
  NOTE: For demonstration purposes (to show that this setup does indeed pull in the code changes made to a checked out package) - I have appended to a comment in AnalysisSkeleton.cxx (i.e I changed "No AOD MC truth particle container found in TDS" to "No AOD MC truth particle container found in TDS - This comment changed by Tom")
Changed:
<
<
4. Go to run directory and start ganga (once the code is compiled).
>
>
4. Go to run directory and start ganga (once the code is compiled).NOTE ganga picks up your grid cert/key from your .globus directory on lxplus - if you have not created these files please follow the instructions here under the preparing the certificate section. Also you must create a file called 'vomses' in a new directory called .glite (again note the '.') and enter this line for the ATLAS VO - "atlas" "voms.cern.ch" "15001" "/C=CH/O=CERN/OU=GRID/CN=host/voms.cern.ch" "atlas" into this file.
 
   cd ../run
   ganga
Changed:
<
<
5. Execute your Ganga job script while Ganga is running (where an example of what the 'pandaBackend_test.py' would look like is below in other words have this file in your run directory) and type:
    execfile('pandaBackend_test.py')
>
>
5. Before you prepare/run your Ganga JO you must prepare the Athena JO it points to.For this you can use the top job options copied from your UserAnalysis packages share directory.
 
Added:
>
>
cp ../share/AnalysisSkeleton_topOptions.py .

BUT to prepare your code for running on the Grid there are some changes needed for this Athena JO - please add these lines:

______________________________________________________________________________

include("RecExCommission/RecExCommissionFlags_jobOptions.py" )
ATLASCosmicFlags.useLocalCOOL  = True
# setup DBReplicaSvc to choose closest Oracle replica, configurables style
from AthenaCommon.AppMgr import ServiceMgr
from PoolSvc.PoolSvcConf import PoolSvc
ServiceMgr+=PoolSvc(SortReplicas=True)
from DBReplicaSvc.DBReplicaSvcConf import DBReplicaSvc
ServiceMgr+=DBReplicaSvc(UseCOOLSQLite=False) 
 
Added:
>
>
_________________________________________________________________________
 
Added:
>
>
Also remember to remove (or comment out) the input data line and if you are running a reprocessing job change the geometry tag and the conditions DB tag to match those used in the reprocessing cycle (see details for each reprocessing campaign on this page here. For example:
globalflags.ConditionsTag.set_Value_and_Lock('COMCOND-REPC-002-13')
 
Added:
>
>
6. Execute your Ganga job script while Ganga is running (where an example of what the 'pandaBackend_test.py' would look like is below in other words have this file in your run directory) and type:
    execfile('pandaBackend_test.py') 
 or simply from the command line run ganga with the name of the Ganga JO appended:

    ganga pandaBackend_test.py
Line: 75 to 94
 
Changed:
<
<
NOTE: Line 3 is an example of overriding a database release to match the one needed to read ESD/DPD. In the case of the spring cosmic reprocessing,the DB release is 6.6.1.1. If the database releases don't match the jobs fail on the Grid ( remove this line if it is not necessary). Line 4 corresponds to your Athena jobOptions. You can use the top job options copied from your UserAnalysis packages share directory.
cp ../share/AnalysisSkeleton_topOptions.py .

BUT to prepare your code for running on the Grid there are some changes needed for this Athena JO - please add these lines:

______________________________________________________________________________

include("RecExCommission/RecExCommissionFlags_jobOptions.py" )
ATLASCosmicFlags.useLocalCOOL  = True
# setup DBReplicaSvc to choose closest Oracle replica, configurables style
from AthenaCommon.AppMgr import ServiceMgr
from PoolSvc.PoolSvcConf import PoolSvc
ServiceMgr+=PoolSvc(SortReplicas=True)
from DBReplicaSvc.DBReplicaSvcConf import DBReplicaSvc
ServiceMgr+=DBReplicaSvc(UseCOOLSQLite=False) 

_____________________________________________________________________________

Also remember to remove (or comment out) the input data line and if you are running a reprocessing job change the geometry tag and the conditions DB tag to match those used in the reprocessing cycle (see details for each reprocessing campaign on this page here. For example:

globalflags.ConditionsTag.set_Value_and_Lock('COMCOND-REPC-002-13')

Back to the Ganga JO script:

Line 5 is set to False because we have already compiled the packages locally if you want your job to compile 
>
>
NOTE: Line 3 is an example of overriding a database release to match the one needed to read ESD/DPD. In the case of the spring cosmic reprocessing,the DB release is 6.6.1.1. If the database releases don't match the jobs fail on the Grid ( remove this line if it is not necessary). Line 4 corresponds to your Athena jobOptions. Line 5 is set to False because we have already compiled the packages locally if you want your job to compile
 your checked out code before submitting then simply change this to True Line 6 tells Ganga to tar your user area and send it with the job. Line 10 specifies the backend to which you are sending your job. There are three options: LCG, Panda and NorduGrid. In the example above Panda was chosen because the data existed only in BNLPANDA, a site in the US cloud. Line 12 corresponds to the number of subjobs you want to split your job into.
Changed:
<
<
Finally in Line 13 you submit your job.
>
>
Finally in Line 13 you submit your job

Revision 62009-10-30 - ThomasDoherty

Line: 1 to 1
 -- ThomasDoherty - 2009-10-26

Using Ganga to submit jobs to the Panda backend

Line: 8 to 8
  Data preparation reprocessing - using Ganga
Changed:
<
<
1. In a clean shell, setup Ganga.
>
>
1. In a clean afs shell, setup Ganga.
 
  source /afs/cern.ch/sw/ganga/install/etc/setup-atlas.sh

2. Setup the athena release.

Changed:
<
<
NOTE: To set up for any release one must be familar with using CMT (bootstrap procedures and requirement files) - see here for more information. In this case once your requirements file is set up and a directory in your test area for 14.5.2.6 is created for example (for reprocessing see the reference page above for the necessary release) then try:
>
>
NOTE: To set up for any release one must be familar with using CMT (bootstrap procedures and requirement files) - see here for more information. In this example (for reprocessing see the reference page above for the necessary release) once the requirements file is set up and a release directory in your test area is created then try:
 
  source ~/cmthome/setup.sh -tag=14.5.2.6,32,AtlasProduction
Line: 28 to 28
 
Changed:
<
<
4. Go to run directory and start ganga.
>
>
NOTE: For demonstration purposes (to show that this setup does indeed pull in the code changes made to a checked out package) - I have appended to a comment in AnalysisSkeleton.cxx (i.e I changed "No AOD MC truth particle container found in TDS" to "No AOD MC truth particle container found in TDS - This comment changed by Tom")

4. Go to run directory and start ganga (once the code is compiled).

 
   cd ../run
   ganga
Line: 39 to 43
 
Added:
>
>
or simply from the command line run ganga with the name of the Ganga JO appended:

    ganga pandaBackend_test.py
 6. You can monitor your job's progress by typing jobs inside Ganga or, if you submitted to the Panda backend by http://panda.cern.ch:25880/server/pandamon/query.

7. Once your job has finished you can copy the output data using the dq2 tools.

Line: 47 to 55
 
Changed:
<
<
Where "your_dataset_name" is given to you by Ganga once the job completes. And 'pandaBackend_test.py' could look like this (without line numbers):
>
>
Where "your_dataset_name" is given to you by Ganga once the job completes. Also once the job completes Panda in particular sends you an email like this. In the email if you click on the 'PandaMonURL' link - in my case for job id 81.Then click on any of the sub-job pandaID's for example ' 1026613035' then scroll down to the link 'Find and view log files' - you can look at the Ganga log for your subjob which is named 'athena_stdout.txt' (only look at 'athena_stderr.txt if your job does not complete). I can then find that"No AOD MC truth particle container found in TDS - This comment changed by Tom" appears in this log.

The Ganga JO 'pandaBackend_test.py' could look like this (without line numbers):

 
1    j = Job()
2    j.application = Athena()
3    j.application.atlas_dbrelease = 'ddo.000001.Atlas.Ideal.DBRelease.v06060101:DBRelease-6.6.1.1.tar.gz'
Line: 88 to 98
 

Back to the Ganga JO script:

Changed:
<
<
Line 5 is set to False because we have already compiled the packages locally.
>
>
Line 5 is set to False because we have already compiled the packages locally if you want your job to compile 
your checked out code before submitting then simply change this to True
 Line 6 tells Ganga to tar your user area and send it with the job. Line 10 specifies the backend to which you are sending your job. There are three options: LCG, Panda and NorduGrid. In the example above Panda was chosen because the data existed only in BNLPANDA,

Revision 52009-10-29 - ThomasDoherty

Line: 1 to 1
 -- ThomasDoherty - 2009-10-26

Using Ganga to submit jobs to the Panda backend

Line: 15 to 15
  2. Setup the athena release.
Changed:
<
<
NOTE: To set up for any release one must be familar with using CMT (bootstrap procedures and requirement files) - see here for more information. In this case once your requirements file is set up and a directory in your test area for 14.5.2.6 is created then try:
  source cmthome/setup.sh -tag=14.5.2.6,32,AtlasProduction
>
>
NOTE: To set up for any release one must be familar with using CMT (bootstrap procedures and requirement files) - see here for more information. In this case once your requirements file is set up and a directory in your test area for 14.5.2.6 is created for example (for reprocessing see the reference page above for the necessary release) then try:
  source ~/cmthome/setup.sh -tag=14.5.2.6,32,AtlasProduction
 

3. Setup any checked out packages you use in your code.

For this example check out (and compile) the UserAnalysis package as in the "HelloWorld"example here or here.

Changed:
<
<
  cd $TEST/PhysicsAnalysis/AnalysisCommon/UserAnalysis/cmt
>
>
  cd $TESTAREA/14.5.2.6/PhysicsAnalysis/AnalysisCommon/UserAnalysis/cmt
  source setup.sh

Revision 42009-10-27 - ThomasDoherty

Line: 1 to 1
 -- ThomasDoherty - 2009-10-26

Using Ganga to submit jobs to the Panda backend

Line: 22 to 22
  3. Setup any checked out packages you use in your code.
Changed:
<
<
For example check out (and compile) the UserAnalysis package as in the "HelloWorld"example here or here.
>
>
For this example check out (and compile) the UserAnalysis package as in the "HelloWorld"example here or here.
 
  cd $TEST/PhysicsAnalysis/AnalysisCommon/UserAnalysis/cmt
  source setup.sh
Line: 51 to 51
 
1    j = Job()
2    j.application = Athena()
3    j.application.atlas_dbrelease = 'ddo.000001.Atlas.Ideal.DBRelease.v06060101:DBRelease-6.6.1.1.tar.gz'
Changed:
<
<
4 j.application.option_file = 'Data_jobOptions_cosmic.py'
>
>
4 j.application.option_file = 'AnalysisSkeleton_topOptions.py'
 5 j.application.athena_compile = False 6 j.application.prepare() 7 j.inputdata = DQ2Dataset()
Line: 65 to 65
 
Changed:
<
<
NOTE: Line 3 is overriding the database release to match the one needed to read ESD/DPD. In the case of the spring cosmic reprocessing,the DB release is 6.6.1.1. If the database releases don't match the jobs fail on the Grid. Line 4 corresponds to your Athena jobOptions. You can use the top job options copied from your UserAnalysis packages share directory.
>
>
NOTE: Line 3 is an example of overriding a database release to match the one needed to read ESD/DPD. In the case of the spring cosmic reprocessing,the DB release is 6.6.1.1. If the database releases don't match the jobs fail on the Grid ( remove this line if it is not necessary). Line 4 corresponds to your Athena jobOptions. You can use the top job options copied from your UserAnalysis packages share directory.
 
cp ../share/AnalysisSkeleton_topOptions.py .

BUT to prepare your code for running on the Grid there are some changes needed for this Athena JO - please add these lines:

Line: 83 to 83
 _________________________________________________________________________
Changed:
<
<
Also remember to remove (or comment out) the input data line and change the geometry tag and the conditions DB tag to match those used in the reprocessing cycle (see details for each reprocessing campaign on this page here. For example:
>
>
Also remember to remove (or comment out) the input data line and if you are running a reprocessing job change the geometry tag and the conditions DB tag to match those used in the reprocessing cycle (see details for each reprocessing campaign on this page here. For example:
 
globalflags.ConditionsTag.set_Value_and_Lock('COMCOND-REPC-002-13')
Deleted:
<
<

 
Deleted:
<
<
The athena JO's used for this specific example (Data_jobOptions_cosmic.py and Settings_DepletionDepth.py) can be found here and here - so if you wish to use them - copy them into your run directory.
 Back to the Ganga JO script:
Line 5 is set to False because we have already compiled the packages locally.
Line 6 tells Ganga to tar your user area and send it with the job.
Line 10 specifies the backend to which you are sending your job. There are three options: LCG, 
Changed:
<
<
Panda and NorduGrid. I chose Panda because my data existed only in BNLPANDA, a site in the US cloud.
>
>
Panda and NorduGrid. In the example above Panda was chosen because the data existed only in BNLPANDA, a site in the US cloud.
 Line 12 corresponds to the number of subjobs you want to split your job into. Finally in Line 13 you submit your job.

Revision 32009-10-26 - ThomasDoherty

Line: 1 to 1
 -- ThomasDoherty - 2009-10-26

Using Ganga to submit jobs to the Panda backend

Line: 8 to 8
  Data preparation reprocessing - using Ganga
Changed:
<
<
1. In a clean shell, setup Ganga. It is important to setup Ganga before Athena.
>
>
1. In a clean shell, setup Ganga.
 
  source /afs/cern.ch/sw/ganga/install/etc/setup-atlas.sh

2. Setup the athena release.

Changed:
<
<
NOTE: To set up for any release one must be familar with using CMT (bootstrap procedures and requirement files) - see here for more info:
>
>
NOTE: To set up for any release one must be familar with using CMT (bootstrap procedures and requirement files) - see here for more information. In this case once your requirements file is set up and a directory in your test area for 14.5.2.6 is created then try:
 
  source cmthome/setup.sh -tag=14.5.2.6,32,AtlasProduction

3. Setup any checked out packages you use in your code.

Changed:
<
<
  cd $TEST/PhysicsAnalysis/AnalysisCommon/UserAnalysis/cmt
>
>
For example check out (and compile) the UserAnalysis package as in the "HelloWorld"example here or here.
  cd $TEST/PhysicsAnalysis/AnalysisCommon/UserAnalysis/cmt
  source setup.sh
 
Changed:
<
<
> source setup.sh
4. Go to run directory and start ganga.
   > cd ../run
   > ganga
5. Execute your job script.
   In[0]: execfile('cbarrera_test2.py')
6. You can monitor your job's progress by typing *jobs* inside Ganga or, if you submitted to the Panda backend by http://panda.cern.ch:25880/server/pandamon/query.
7. Once your job has finished you can copy the output data using the dq2 tools.
   > source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.sh
   > dq2-get 
>
>
4. Go to run directory and start ganga.
   cd ../run
   ganga

5. Execute your Ganga job script while Ganga is running (where an example of what the 'pandaBackend_test.py' would look like is below in other words have this file in your run directory) and type:

    execfile('pandaBackend_test.py')
 
Changed:
<
<

yourData

Obviously it's not as simple as that... 

>
>
6. You can monitor your job's progress by typing jobs inside Ganga or, if you submitted to the Panda backend by http://panda.cern.ch:25880/server/pandamon/query.
 
Changed:
<
<
Here is the job file I submitted using ganga:
>
>
7. Once your job has finished you can copy the output data using the dq2 tools.
          
source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.sh
   dq2-get "your_dataset_name"
 
Changed:
<
<
1 j = Job()
>
>
Where "your_dataset_name" is given to you by Ganga once the job completes. And 'pandaBackend_test.py' could look like this (without line numbers):
1    j = Job()
 2 j.application = Athena() 3 j.application.atlas_dbrelease = 'ddo.000001.Atlas.Ideal.DBRelease.v06060101:DBRelease-6.6.1.1.tar.gz' 4 j.application.option_file = 'Data_jobOptions_cosmic.py'
Line: 54 to 62
 12 j.splitter.numsubjobs = 20 13 j.submit()
Changed:
<
<
A few comments on some of these lines. Line 3 is overriding the database release to match the one needed to read ESD/DPD. In the case of the spring cosmic reprocessing, the DB release is 6.6.1.1. If the database releases don't match the jobs fail on the Grid. I don't understand why it works locally though (maybe a question for Graeme). Line 4 corresponds to your jobOptions. There were some changes I needed to do in mine as well, to prepare my code for running on the Grid:
>
>

NOTE: Line 3 is overriding the database release to match the one needed to read ESD/DPD. In the case of the spring cosmic reprocessing,the DB release is 6.6.1.1. If the database releases don't match the jobs fail on the Grid. Line 4 corresponds to your Athena jobOptions. You can use the top job options copied from your UserAnalysis packages share directory.

cp ../share/AnalysisSkeleton_topOptions.py .
 
Changed:
<
<
__________________________________________________________________________
>
>
BUT to prepare your code for running on the Grid there are some changes needed for this Athena JO - please add these lines:
______________________________________________________________________________
  include("RecExCommission/RecExCommissionFlags_jobOptions.py" ) ATLASCosmicFlags.useLocalCOOL = True
Line: 69 to 80
 from DBReplicaSvcConf import DBReplicaSvc ServiceMgr+=DBReplicaSvc(UseCOOLSQLite=False)
Changed:
<
<
globalflags.ConditionsTag.set_Value_and_Lock('COMCOND-REPC-002-13') __________________________________________________________________________
>
>
_________________________________________________________________________
 
Changed:
<
<
Also remember to remove the input data line in the original JO's (from second Reference webpage). Line 5 is set to False because we have already compiled the packages locally. Line 6 tells Ganga to tar your user area and send it with the job. Line 10 specifies the backend to which you are sending your job. There are three options: LCG, Panda and NorduGrid. I chose Panda because my data existed only in BNLPANDA, a site in the US cloud. And apparently, as Graeme told us yesterday, that's the way to go: prepare your jobs with Ganga and then submit to PanDa. Line 12 corresponds to the number of subjobs you want to split your job into. I read in this page https://twiki.cern.ch/twiki/bin/view/Atlas/DAGangaFAQ#DQ2JobSplitter_won_t_submit_when that if you are using DQ2Splitter you should not specify a site, that you should let it decide. But I believe specifying a cloud is permitted. Finally in Line 13 you submit your job.
>
>
Also remember to remove (or comment out) the input data line and change the geometry tag and the conditions DB tag to match those used in the reprocessing cycle (see details for each reprocessing campaign on this page here. For example:
globalflags.ConditionsTag.set_Value_and_Lock('COMCOND-REPC-002-13')
 
Changed:
<
<
This instructions are just guidance. I am sure unexpected things will emerge when dealing with different cases.
>
>
The athena JO's used for this specific example (Data_jobOptions_cosmic.py and Settings_DepletionDepth.py) can be found here and here - so if you wish to use them - copy them into your run directory.
 
Added:
>
>
Back to the Ganga JO script:
Line 5 is set to False because we have already compiled the packages locally.
Line 6 tells Ganga to tar your user area and send it with the job.
Line 10 specifies the backend to which you are sending your job. There are three options: LCG, 
Panda and NorduGrid. I chose Panda because my data existed only in BNLPANDA, a site in the US cloud. 
Line 12 corresponds to the number of subjobs you want to split your job into. 
Finally in Line 13 you submit your job.
 

Revision 22009-10-26 - ThomasDoherty

Line: 1 to 1
 -- ThomasDoherty - 2009-10-26

Using Ganga to submit jobs to the Panda backend

Line: 6 to 6
  Full Ganga Atlas Tutorial
Changed:
<
<
Data preparation reporocessing - using Ganga
>
>
Data preparation reprocessing - using Ganga
  1. In a clean shell, setup Ganga. It is important to setup Ganga before Athena.
  source /afs/cern.ch/sw/ganga/install/etc/setup-atlas.sh
Line: 14 to 14
 

2. Setup the athena release.

Changed:
<
<
   
> source cmthome/setup.sh -tag=14.5.2.6,AtlasProduction

>
>
NOTE: To set up for any release one must be familar with using CMT (bootstrap procedures and requirement files) - see here for more info:
  source cmthome/setup.sh -tag=14.5.2.6,32,AtlasProduction
 3. Setup any checked out packages you use in your code.
Changed:
<
<
> cd $TEST/PhysicsAnalysis/AnalysisCommon/UserAnalysis/cmt
>
>
  cd $TEST/PhysicsAnalysis/AnalysisCommon/UserAnalysis/cmt
  > source setup.sh
Changed:
<
<
4. Go to run directory and start ganga.
>
>
4. Go to run directory and start ganga.

  > cd ../run > ganga 5. Execute your job script.

Revision 12009-10-26 - ThomasDoherty

Line: 1 to 1
Added:
>
>
-- ThomasDoherty - 2009-10-26

Using Ganga to submit jobs to the Panda backend

References:

Full Ganga Atlas Tutorial

Data preparation reporocessing - using Ganga

1. In a clean shell, setup Ganga. It is important to setup Ganga before Athena.

  source /afs/cern.ch/sw/ganga/install/etc/setup-atlas.sh

2. Setup the athena release.

   
> source cmthome/setup.sh -tag=14.5.2.6,AtlasProduction
3. Setup any checked out packages you use in your code.
   > cd $TEST/PhysicsAnalysis/AnalysisCommon/UserAnalysis/cmt
   > source setup.sh
4. Go to run directory and start ganga.
   > cd ../run
   > ganga
5. Execute your job script.
   In[0]: execfile('cbarrera_test2.py')
6. You can monitor your job's progress by typing *jobs* inside Ganga or, if you submitted to the Panda backend by http://panda.cern.ch:25880/server/pandamon/query.
7. Once your job has finished you can copy the output data using the dq2 tools.
   > source /afs/cern.ch/atlas/offline/external/GRID/ddm/DQ2Clients/setup.sh
   > dq2-get 

yourData

Obviously it's not as simple as that... 

Here is the job file I submitted using ganga:

1    j = Job()
2    j.application = Athena()
3    j.application.atlas_dbrelease = 'ddo.000001.Atlas.Ideal.DBRelease.v06060101:DBRelease-6.6.1.1.tar.gz'
4    j.application.option_file = 'Data_jobOptions_cosmic.py'
5    j.application.athena_compile = False
6    j.application.prepare()
7    j.inputdata = DQ2Dataset()
8    j.inputdata.dataset = "data08_cos.00092051.physics_IDCosmic.recon.ESD.o4_r653/"
9    j.outputdata = DQ2OutputDataset()
10   j.backend = Panda()
11   j.splitter = DQ2JobSplitter()
12   j.splitter.numsubjobs = 20
13   j.submit()

A few comments on some of these lines. Line 3 is overriding the database release to match the one needed to read ESD/DPD. In the case of the spring cosmic reprocessing, 
the DB release is 6.6.1.1. If the database releases don't match the jobs fail on the Grid. I don't understand why it works locally though (maybe a question for Graeme). 
Line 4 corresponds to your jobOptions. There were some changes I needed to do in mine as well, to prepare my code for running on the Grid:

______________________________________________________________________________

include("RecExCommission/RecExCommissionFlags_jobOptions.py" )
ATLASCosmicFlags.useLocalCOOL  = True
# setup DBReplicaSvc to choose closest Oracle replica, configurables style
from AthenaCommon.AppMgr import ServiceMgr
from PoolSvc.PoolSvcConf import PoolSvc
ServiceMgr+=PoolSvc(SortReplicas=True)
from DBReplicaSvc.DBReplicaSvcConf import DBReplicaSvc
ServiceMgr+=DBReplicaSvc(UseCOOLSQLite=False) 

globalflags.ConditionsTag.set_Value_and_Lock('COMCOND-REPC-002-13')
______________________________________________________________________________

Also remember to remove the input data line in the original JO's (from second Reference webpage). Line 5 is set to False because we have already compiled the packages 
locally. Line 6 tells Ganga to tar your user area and send it with the job. Line 10 specifies the backend to which you are sending your job. There are three options: LCG, 
Panda and NorduGrid. I chose Panda because my data existed only in BNLPANDA, a site in the US cloud. And apparently, as Graeme told us yesterday, that's the way to go: 
prepare your jobs with Ganga and then submit to PanDa. Line 12 corresponds to the number of subjobs you want to split your job into. I read in this page 
https://twiki.cern.ch/twiki/bin/view/Atlas/DAGangaFAQ#DQ2JobSplitter_won_t_submit_when that if you are using DQ2Splitter you should not specify a site, that you should let 
it decide. But I believe specifying a cloud is permitted. Finally in Line 13 you submit your job.

This instructions are just guidance. I am sure unexpected things will emerge when dealing with different cases.

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback