(r16) RunMokkaLocally < LinearCollider

LinearCollider Web>RunMokkaLocally (revision 16) (raw view)~~EditAttach~~
---+ Running Mokka simulations locally

Extensive sets of simulations are already available and many types of physics can be studied using these sets. However, if a customised detector configuration is needed, it is important to be able to run our own simulations locally, using the PPE batch farm. This wiki explains how to do this, step by step.

%TOC%

---++ Testing Mokka

This is described in the [[ILCSoft]] wiki. One can try quickly:

<verbatim>
. /afs/phas.gla.ac.uk/data/ilc/software/ilcsoft/v01-14/init_ilcsoft.sh
cd /afs/phas.gla.ac.uk/data/ilc/software/validation/StandardConfig/current
Mokka -M ILD_o1_v05 -e ./particle.tbl bbudsc_3evt.steer
</verbatim>

---++ Changing certain detector parameters

Detector parameters are taken from (a default) database. There are many versions of the ILC detectors. One can browse this database via the [[http://www-flc.desy.de/ldcoptimization/tools/mokkamodels.php?model=ILD_o1_v05][Mokka Detector Model Database Browser]].

To change for example the number of layers in the =Hcal=, and the sensitive Si layer thickness in =Ecal=, one would add in the steering file
<verbatim>
/Mokka/init/globalModelParameter Hcal_nlayers 36
/Mokka/init/globalModelParameter Ecal_Si_thickness 0.45
</verbatim>

to the steering file and use the "-U" command line option to enable the change:
<verbatim>
Mokka -M ILD_o1_v05 -U -e ./particle.tbl my_bbudsc_3evt.steer
</verbatim>

---++ Visualisation

To visualise the detector geometry set in the steering file for example:
<verbatim>
/Mokka/init/BatchMode false
</verbatim>

and then in Geant4 prompt type 
<verbatim>
/vis/open OGL
/vis/scene/create 
/vis/drawVolume
/vis/viewer/set/viewpointThetaPhi 60 45
</verbatim>

<img width="449" alt="ILD_G4_GLS.png" src="%ATTACHURLPATH%/ILD_G4_GLS.png" height="421" />

You can further manipulate the view, for example
<verbatim>
/vis/viewer/set/viewpointThetaPhi 30 40
/vis/viewer/panTo -5 -1
/vis/viewer/zoom 4.
</verbatim>

To draw trajectories:
<verbatim>
/vis/scene/add/trajectories 
/vis/scene/add/hits 
/vis/viewer/flush
</verbatim>

A good tutorial can be found [[http://geant4.slac.stanford.edu/Presentations/vis/G4OpenGLTutorial/G4OpenGLTutorial.html][here]].

---++ Running from your own work directory

To be able to customise your scripts and better keep track of the outputs, you should run =Mokka= from your own directory, e.g.
<verbatim>
cd ~
mkdir ILC_Simulations
cd ILC_Simulations
cp /afs/phas.gla.ac.uk/data/ilc/software/validation/StandardConfig/current/bbudsc_3evt* .
</verbatim>

Edit now these files, rename and customise them and run (after you've sourced the correct environment as before)
<verbatim>
Mokka -M ILD_o1_v05 -U -e ./particle.tbl my_sim.steer
</verbatim>

---++ Running Mokka on the PPE batch farm

Here's a simple script (named =mokka_test.pbs=) that runs the Mokka example on the PPE batch farm:
<verbatim>
#PBS -N Mokka_test_1
#PBS -m abe
#PBS -M gary.smythe@glasgow.ac.uk
#PBS -q long5

echo "Running Mokka test 1 ..." 

. /afs/phas.gla.ac.uk/data/ilc/software/ilcsoft/v01-14/init_ilcsoft.sh
cd /afs/phas.gla.ac.uk/user/p/protopop/ILC_Simulations
Mokka -M ILD_o1_v05 -e ./particle.tbl my_sim.steer

echo "Mokka test 1 done!"
</verbatim>

To submit the job from any PPE linux account, use the command
<verbatim>
qsub mokka_test.pbs
</verbatim>

To learn more about PBS commands type =man pbs= in your terminal.

---++ Using a particle gun

The example above needs around 3 minutes per event. For certain purposes, it is enough to use a simple particle gun instead, and this will run much faster. To do this, replace in the example steering file
<verbatim>
/Mokka/init/initialMacroFile bbudsc_3evt.g4macro
</verbatim>

with
<verbatim>
/Mokka/init/initialMacroFile e-gun.g4macro
</verbatim>

where the file =e-gun.g4macro= contains the following instructions:
<verbatim>
/generator/generator particleGun
/gun/particle e-
/gun/energy 100 GeV
/gun/momentumSmearing 5 GeV
/gun/position 0 0 0
/gun/direction 1 0 1
/run/beamOn 3000
exit
</verbatim>
Check gun settings with:
 <verbatim>
/gun/info                 
Particle:       e- (m = 0.000510999 GeV)
Kinetic Energy: 100 GeV (momentum = 100.001 GeV)
Position:       0 0 0 cm
Direction:      0.707107 0 0.707107 (theta = 45 deg, phi = 0 deg)
(Steps and smearing are not displayed.)
</verbatim>

However, it is more useful to  have the electron gun 'spray' particles in all regions of the detector, instead of a set direction. This can be obtained for example with: 
<verbatim>
/gun/direction 0 0 1
/gun/directionSmearingMode uniform
/gun/phiSmearing 360 deg
/gun/thetaSmearing 180 deg
</verbatim>
This is documented in section 'VI. New Commands for the Particle Gun' of Mokka [[http://llrforge.in2p3.fr/svn/Mokka/trunk/ReleaseNotes/release_notes_mokka-05-02][release notes 05-02]].

---++ Re-running a set of events

Say we want to run a set of simulations with modified detector configurations, while using the same generated set, for example:
<verbatim>
/ilc/prod/clic/350gev/hzee/gen/00001335/000/hzee_gen_1335_16.stdhep
</verbatim>
Our =steer= file, =Zee_cdr500.steer= will contain something like this (compare to http://lcd-data.web.cern.ch/lcd-data/doc/clic_ild_cdr.steer)
<verbatim>
/Mokka/init/lcioFilename hzee_10evt_CLIC_ILD_CDR500.slcio
/Mokka/init/lcioWriteMode WRITE_NEW
/Mokka/init/MokkaGearFileName GearCLIC_ILD_CDR500.xml

/Mokka/init/user consult
/Mokka/init/dbPasswd consult

/Mokka/init/detectorModel CLIC_ILD_CDR500
/Mokka/init/lorentzTransformationAngle 10 mrad 
/Mokka/init/physicsListName QGSP_BERT

/Mokka/init/rangeCut 0.1 mm        # Geant4 production range cut [default is 0.005 mm]

/Mokka/init/TPCCut 10 MeV          # Default is 10MeV

/Mokka/init/lcioDetailedTRKHitMode VXDCollection
/Mokka/init/lcioDetailedTRKHitMode FTDCollection
/Mokka/init/lcioDetailedTRKHitMode ETDCollection
/Mokka/init/lcioDetailedTRKHitMode SETCollection
/Mokka/init/lcioDetailedTRKHitMode SITCollection
/Mokka/init/lcioDetailedTRKHitMode TPCCollection 

# To run with modified detector parameters use command line option '-U' 
# For a list of specs visit the Mokka Detector Model DB Browser at 
# http://www-flc.desy.de/ldcoptimization/tools/mokkamodels.php?model=CLIC_ILD_CDR500
# /Mokka/init/globalModelParameter Ecal_nlayers1 17
# /Mokka/init/globalModelParameter Ecal_nlayers2 8
# /Mokka/init/globalModelParameter Ecal_radiator_layers_set1_thickness 2.4
# /Mokka/init/globalModelParameter Ecal_radiator_layers_set2_thickness 4.8
# (above numbers from http://agenda.linearcollider.org/getFile.py/access?contribId=50&sessionId=9&resId=0&materialId=slides&confId=5686)

/Mokka/init/randomSeed 16
/Mokka/init/mcRunNumber 1335
/Mokka/init/initialMacroFile hzee_1335_16.g4macro

/Mokka/init/printLevel 1
/Mokka/init/BatchMode true
</verbatim>
where the =G4 macro file=, =hzee_1335_16.g4macro= contains:
<verbatim>
/run/verbose 0
/event/verbose 0
/tracking/verbose 0
/generator/generator /afs/phas.gla.ac.uk/data/ilc/datasets01/350gev/hzee/gen/00001335/000/hzee_gen_1335_16.stdhep
/run/beamOn 10
exit
</verbatim>
Remember to change both the LCIO and gear output file names for each detector configuration. The =G4 macro= stays the same. For example I have:
| *Steering file* | *Output file* | *Gear file* |
| Zee_cdr500.steer | hzee_10evt_CLIC_ILD_CDR500.slcio | !GearCLIC_ILD_CDR500.xml  |
| Zee_cdr500_mdf1.steer | hzee_10evt_CLIC_ILD_CDR500_mdf1.slcio | !GearCLIC_ILD_CDR500_mdf1.xml |

Run with =Mokka Zee_cdr500.steer= and =Mokka -U Zee_cdr500_mdf1.steer=.

---++ Automating job submission

There's a script to simplify running Mokka simulations on the PPE compute farm. You can find it here:
<verbatim>
/afs/phas.gla.ac.uk/data/ilc/datasets01/350gev/hzee/mokka_dp/run-ecal-run.sh
</verbatim>
I run it like this
<verbatim>
    ppepc102>  ./run-ecal-run.sh <#events> <geom_version> <gen_set>
</verbatim>
for example =./run-ecal-run.sh 500 mdf1 1335_16= will run 500 events, using template =hzee-steer-mdf1.tpl= for the Ecal geometry, and =hzee_gen_1335_16.stdhep= as input.

One will have to customise a few settings before using it, and download the stdhep files for input. Can also define a few more Ecal geometries (see templates/) to play with. 

The script will create the relevant mac, steer and PBS script files, and submit the job straight away. If you modify the script, it’s best to test your settings before actually submitting the jobs.
 
---+++ How to find the optimal number of events

One can use the =stdhepjob= utility from LCIO to count the number of events in the input =.stdhep= file, e.g.
<verbatim>
     ppepc102> stdhepjob hzee_gen_1335_16.stdhep /tmp/out.slcio -1 
</verbatim>
This will simply run over all events and print out how many were processed.

We found out that on our farm a 500 events jobs runs approximately 15h, so we could split the input in 2-3 runs by making use of =/generator/skipEvents=.
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	100118_MokkaMarlinTutorial.pdf	r1	manage	93.7 K	2013-11-01 - 13:54	DanProtopopescu	Mokka and Marlin tutorial for ILD - Jan 2009
png	ILD_G4_GLS.png	r1	manage	59.7 K	2013-11-05 - 15:48	DanProtopopescu	ILD visualisation example (see text)
tar	MokkaTutorial.tar	r1	manage	9420.0 K	2013-11-05 - 11:44	DanProtopopescu	Mokka tutorial files (tar archive)
pdf	lucaci_ecal.pdf	r1	manage	614.4 K	2014-01-22 - 14:32	DanProtopopescu	ECAL Studies (with links to other talks) - A. Lucaci, Oct 2012
Topic revision: r16 - 2014-02-03 - DanProtopopescu
LinearCollider