Running Mokka simulations locally

Extensive sets of simulations are already available and many types of physics can be studied using these sets. However, if a customised detector configuration is needed, it is important to be able to run our own simulations locally, using the PPE batch farm. This wiki explains how to do this, step by step.

Testing Mokka

This is described in the ILCSoft wiki. One can try quickly:

. /afs/phas.gla.ac.uk/data/ilc/software/ilcsoft/v01-14/init_ilcsoft.sh
cd /afs/phas.gla.ac.uk/data/ilc/software/validation/StandardConfig/current
Mokka -M ILD_o1_v05 -e ./particle.tbl bbudsc_3evt.steer

Changing certain detector parameters

Detector parameters are taken from (a default) database. There are many versions of the ILC detectors. One can browse this database via the Mokka Detector Model Database Browser.

To change for example the number of layers in the Hcal, and the sensitive Si layer thickness in Ecal, one would add in the steering file

/Mokka/init/globalModelParameter Hcal_nlayers 36
/Mokka/init/globalModelParameter Ecal_Si_thickness 0.45

to the steering file and use the "-U" command line option to enable the change:

Mokka -M ILD_o1_v05 -U -e ./particle.tbl my_bbudsc_3evt.steer

Visualisation

To visualise the detector geometry set in the steering file for example:

/Mokka/init/BatchMode false

and then in Geant4 prompt type

/vis/open OGL
/vis/scene/create 
/vis/drawVolume
/vis/viewer/set/viewpointThetaPhi 60 45

You can further manipulate the view, for example

/vis/viewer/set/viewpointThetaPhi 30 40
/vis/viewer/panTo -5 -1
/vis/viewer/zoom 4.
/vis/viewer/set/background white

To draw trajectories:

/vis/scene/add/trajectories 
/vis/scene/add/hits 
/vis/viewer/flush

A good tutorial can be found here.

Running from your own work directory

To be able to customise your scripts and better keep track of the outputs, you should run Mokka from your own directory, e.g.

cd ~
mkdir ILC_Simulations
cd ILC_Simulations
cp /afs/phas.gla.ac.uk/data/ilc/software/validation/StandardConfig/current/bbudsc_3evt* .

Edit now these files, rename and customise them and run (after you've sourced the correct environment as before)

Mokka -M ILD_o1_v05 -U -e ./particle.tbl my_sim.steer

Running Mokka on the PPE batch farm

Here's a simple script (named mokka_test.pbs) that runs the Mokka example on the PPE batch farm:

#PBS -N Mokka_test_1
#PBS -m abe
#PBS -M gary.smythe@glasgow.ac.uk
#PBS -q long5

echo "Running Mokka test 1 ..." 

. /afs/phas.gla.ac.uk/data/ilc/software/ilcsoft/v01-14/init_ilcsoft.sh
cd /afs/phas.gla.ac.uk/user/p/protopop/ILC_Simulations
Mokka -M ILD_o1_v05 -e ./particle.tbl my_sim.steer

echo "Mokka test 1 done!"

To submit the job from any PPE linux account, use the command

qsub mokka_test.pbs

To learn more about PBS commands type man pbs in your terminal.

Using a particle gun

The example above needs around 3 minutes per event. For certain purposes, it is enough to use a simple particle gun instead, and this will run much faster. To do this, replace in the example steering file

/Mokka/init/initialMacroFile bbudsc_3evt.g4macro

with

/Mokka/init/initialMacroFile e-gun.g4macro

where the file e-gun.g4macro contains the following instructions:

/generator/generator particleGun
/gun/particle e-
/gun/energy 100 GeV
/gun/momentumSmearing 5 GeV
/gun/position 0 0 0
/gun/direction 1 0 1
/run/beamOn 3000
exit

Check gun settings with:

/gun/info                 
Particle:       e- (m = 0.000510999 GeV)
Kinetic Energy: 100 GeV (momentum = 100.001 GeV)
Position:       0 0 0 cm
Direction:      0.707107 0 0.707107 (theta = 45 deg, phi = 0 deg)
(Steps and smearing are not displayed.)

However, it is more useful to have the electron gun 'spray' particles in all regions of the detector, instead of a set direction. This can be obtained for example with:

/gun/direction 0 0 1
/gun/directionSmearingMode uniform
/gun/phiSmearing 360 deg
/gun/thetaSmearing 180 deg

This is documented in section 'VI. New Commands for the Particle Gun' of Mokka release notes 05-02.

Re-running a set of events

Say we want to run a set of simulations with modified detector configurations, while using the same generated set, for example:

/ilc/prod/clic/350gev/hzee/gen/00001335/000/hzee_gen_1335_16.stdhep

Our steer file, Zee_cdr500.steer will contain something like this (compare to http://lcd-data.web.cern.ch/lcd-data/doc/clic_ild_cdr.steer)

/Mokka/init/lcioFilename hzee_10evt_CLIC_ILD_CDR500.slcio
/Mokka/init/lcioWriteMode WRITE_NEW
/Mokka/init/MokkaGearFileName GearCLIC_ILD_CDR500.xml

/Mokka/init/user consult
/Mokka/init/dbPasswd consult

/Mokka/init/detectorModel CLIC_ILD_CDR500
/Mokka/init/lorentzTransformationAngle 10 mrad 
/Mokka/init/physicsListName QGSP_BERT

/Mokka/init/rangeCut 0.1 mm        # Geant4 production range cut [default is 0.005 mm]

/Mokka/init/TPCCut 10 MeV          # Default is 10MeV

/Mokka/init/lcioDetailedTRKHitMode VXDCollection
/Mokka/init/lcioDetailedTRKHitMode FTDCollection
/Mokka/init/lcioDetailedTRKHitMode ETDCollection
/Mokka/init/lcioDetailedTRKHitMode SETCollection
/Mokka/init/lcioDetailedTRKHitMode SITCollection
/Mokka/init/lcioDetailedTRKHitMode TPCCollection 

# To run with modified detector parameters use command line option '-U' 
# For a list of specs visit the Mokka Detector Model DB Browser at 
# http://www-flc.desy.de/ldcoptimization/tools/mokkamodels.php?model=CLIC_ILD_CDR500
# /Mokka/init/globalModelParameter Ecal_nlayers1 17
# /Mokka/init/globalModelParameter Ecal_nlayers2 8
# /Mokka/init/globalModelParameter Ecal_radiator_layers_set1_thickness 2.4
# /Mokka/init/globalModelParameter Ecal_radiator_layers_set2_thickness 4.8
# (above numbers from http://agenda.linearcollider.org/getFile.py/access?contribId=50&sessionId=9&resId=0&materialId=slides&confId=5686)

/Mokka/init/randomSeed 16
/Mokka/init/mcRunNumber 1335
/Mokka/init/initialMacroFile hzee_1335_16.g4macro

/Mokka/init/printLevel 1
/Mokka/init/BatchMode true

where the G4 macro file, hzee_1335_16.g4macro contains:

/run/verbose 0
/event/verbose 0
/tracking/verbose 0
/generator/generator /afs/phas.gla.ac.uk/data/ilc/datasets01/350gev/hzee/gen/00001335/000/hzee_gen_1335_16.stdhep
/run/beamOn 10
exit

Remember to change both the LCIO and gear output file names for each detector configuration. The G4 macro stays the same. For example I have:

Steering file	Output file	Gear file
Zee_cdr500.steer	hzee_10evt_CLIC_ILD_CDR500.slcio	GearCLIC_ILD_CDR500.xml
Zee_cdr500_mdf1.steer	hzee_10evt_CLIC_ILD_CDR500_mdf1.slcio	GearCLIC_ILD_CDR500_mdf1.xml

Run with Mokka Zee_cdr500.steer and Mokka -U Zee_cdr500_mdf1.steer.

Automating job submission

There's a script to simplify running Mokka simulations on the PPE compute farm. You can find it here:

/afs/phas.gla.ac.uk/data/ilc/datasets01/350gev/hzee/mokka_dp/run-ecal-run.sh

I run it like this

    ppepc102>  ./run-ecal-run.sh <#events> <geom_version> <gen_set>

for example ./run-ecal-run.sh 500 mdf1 1335_16 will run 500 events, using template hzee-steer-mdf1.tpl for the Ecal geometry, and hzee_gen_1335_16.stdhep as input.

One will have to customise a few settings before using it, and download the stdhep files for input. Can also define a few more Ecal geometries (see templates/) to play with.

The script will create the relevant mac, steer and PBS script files, and submit the job straight away. If you modify the script, it’s best to test your settings before actually submitting the jobs.

How to find the optimal number of events

One can use the stdhepjob utility from LCIO to count the number of events in the input .stdhep file, e.g.

     ppepc102> stdhepjob hzee_gen_1335_16.stdhep /tmp/out.slcio -1 

This will simply run over all events and print out how many were processed.

We found out that on our farm a 500 events jobs runs approximately 15h, so we could split the input in 2-3 runs by making use of /generator/skipEvents.