Difference: WilliamBreadenMaddenSandbox (1 vs. 12)

Revision 122015-06-16 - WilliamBreadenMadden

Line: 1 to 1
 
META TOPICPARENT name="Main.WilliamBreadenMadden"
Changed:
<
<
>
>

This is some mathematics: . It is inline.

 
META FILEATTACHMENT attachment="2014-06-30T1358.gif" attr="" comment="parallel job processor" date="1404133307" name="2014-06-30T1358.gif" path="2014-06-30T1358.gif" size="862869" user="WilliamBreadenMadden" version="1"
META FILEATTACHMENT attachment="2014-06-30T1401.jpg" attr="" comment="parallel job processor logging" date="1404133353" name="2014-06-30T1401.jpg" path="2014-06-30T1401.jpg" size="1185878" user="WilliamBreadenMadden" version="1"

Revision 112015-03-30 - WilliamBreadenMadden

Line: 1 to 1
 
META TOPICPARENT name="Main.WilliamBreadenMadden"
Changed:
<
<

Availability of mc14_13TeV samples compared to 8 TeV defaults in H to bb

  8 TeV 13 TeV
process dataset number generator number of events process dataset number generator number of events AMI link
ttH 125 GeV, allhad 169887 PowHel Py8 999999 ttH, 0-150 GeV slice 189529 Pythia8 50000 link
ttH, 150-350 GeV slice 189530 Pythia8 50000 link
ttH, > 350 GeV slice 189531 Pythia8 50000 link
ttH 125 GeV, l+jets 169888 PowHel Py8 999990 ttH, 0-150 GeV slice 189526 Pythia8 50000 link
ttH, 150-350 GeV slice 189527 Pythia8 50000 link
ttH, >350 GeV slice 189528 Pythia8 50000 link
ttH 125 GeV, dilepton 169889 PowHel Py8 998892 ttH, 0-150 GeV slice 189523 Pythia8 50000 link
ttH, 150-350 GeV slice 189524 Pythia8 50000 link
ttH, >350 GeV slice 189525 Pythia8 50000 link
ttbar nonallhadronic 117050 Powheg+Pythia6 48450 ttbar_nonallhad 110401 Powheg+Pythia+Photos+Tauola 400000  
single top tch. top 110090 Powheg+Pythia6 1000000 singletop_tchan_lept_top 110070 Powheg+Pythia+Photos+Tauola 999000  
single top tch, antitop 110091 Powheg+Pythia6 5000000 singletop_tchan_lept_antitop 110071 Powheg+Pythia+Photos+Tauola 1000000  
single top schan 110119 Powheg+Pythia6 1199895 - - - -  
single top, Wt chan. inclusive 110140 Powheg+Pythia6 999692 Wtchan_incl_DR 110305 Powheg+Pythia+Photos+Tauola 958500  
single top, Wt chan. dilepton 110141 Powheg+Pythia6 999894 -   - -  
ttbar+W 119353 Madgraph+Py6   ttbarW 119353 MadGraph +Pythia+Photos+Tauola 399500  
ttbar+Z 119355 Madgraph+Py6   ttbarZ 119355 MadGraph +Pythia+Photos+Tauola 400000  
ttbar+WW 158813 Madgraph+Py6   ttbarWW 119583 MadGraph +Pythia+Photos+Tauola 10000  
ttbar+Wj Excl 174830 Madgraph+Py6   ttbarWjExcl 174830 MadGraph +Pythia+Photos+Tauola 399500  
ttbar+Wjj Incl. 174831 Madgraph+Py6   ttbarWjjIncl 174831 MadGraph +Pythia+Photos+Tauola 395000  
ttbar+Zj Excl. 174832 Madgraph+Py6   ttbarZjExcl 174832 MadGraph +Pythia+Photos+Tauola 399500  
ttbar+Zjj Incl. 174833 Madgraph+Py6   ttbarZjjIncl 174833 MadGraph +Pythia+Photos+Tauola 399500  
WZ to enu qq 183735 Sherpa   -   - -  
WZ to munu qq 183737 Sherpa   -   - -  
WZ to taunu qq 183739 Sherpa   -   - -  
WW lnulnu np0 107100 AlpgenJimmy   -   - -  
WW lnulnu np1 107101 AlpgenJimmy   -   - -  
WW lnulnu np2 107102 AlpgenJimmy   -   - -  
WW lnulnu np3 107103 AlpgenJimmy   -   - -  
WZ incl ll np0 107104 AlpgenJimmy   -   - -  
WZ incl ll np1 107105 AlpgenJimmy   -   - -  
WZ incl ll np2 107106 AlpgenJimmy   -   - -  
WZ incl ll np3 107107 AlpgenJimmy   -   - -  
ZZ incl ll np0 107108 AlpgenJimmy   -   - -  
ZZ incl ll np1 107109 AlpgenJimmy   -   - -  
ZZ incl ll np2 107110 AlpgenJimmy   -   - -  
ZZ incl ll np3 107111 AlpgenJimmy   -   - -  
WW qq lnue np0 110829 AlpgenJimmy   -   - -  
WW qq lnue np1 110830 AlpgenJimmy   -   - -  
WW qq lnue np2 110831 AlpgenJimmy   -   - -  
WW qq lnue np3 110832 AlpgenJimmy   -   - -  
Wenu np0 147025 AlpgenPythia   -   - -  
Wenu np1 147026 AlpgenPythia   -   - -  
Wenu np2 147027 AlpgenPythia   -   - -  
Wenu np3 147028 AlpgenPythia   -   - -  
Wenu np4 147029 AlpgenPythia   -   - -  
Wenu np5 incl 147030 AlpgenPythia   -   - -  
Wmunu np0 147033 AlpgenPythia   -   - -  
Wmunu np1 147034 AlpgenPythia   -   - -  
Wmunu np2 147035 AlpgenPythia   -   - -  
Wmunu np3 147036 AlpgenPythia   -   - -  
Wmunu np4 147037 AlpgenPythia   -   - -  
Wmunu np5 incl 147038 AlpgenPythia   -   - -  
Wtaunu np0 147041 AlpgenPythia   -   - -  
Wtaunu np1 147042 AlpgenPythia   -   - -  
Wtaunu np2 147043 AlpgenPythia   -   - -  
Wtaunu np3 147044 AlpgenPythia   -   - -  
Wtaunu np4 147045 AlpgenPythia   -   - -  
Wtaunu np5 inc 147046 AlpgenPythia   -   - -  
Wcc Np0 200156 AlpgenPythia   -   - -  
Wcc Np1 200157 AlpgenPythia   -   - -  
Wcc Np2 200158 AlpgenPythia   -   - -  
Wcc Np3 incl 200159 AlpgenPythia   -   - -  
Wc Np0 200056 AlpgenPythia   -   - -  
Wc Np1 200057 AlpgenPythia   -   - -  
Wc Np2 200058 AlpgenPythia   -   - -  
Wc Np3 200059 AlpgenPythia   -   - -  
Wc Np4 incl 200060 AlpgenPythia   -   - -  
Wbb Np0 200156 AlpgenPythia   -   - -  
Wbb Np1 200257 AlpgenPythia   -   - -  
Wbb Np2 200258 AlpgenPythia   -   - -  
Wbb Np3 inc 200259 AlpgenPythia   -   - -  


Availability of mc14_13TeV samples compared to 8 TeV defaults in H to bb

>
>
 
META FILEATTACHMENT attachment="2014-06-30T1358.gif" attr="" comment="parallel job processor" date="1404133307" name="2014-06-30T1358.gif" path="2014-06-30T1358.gif" size="862869" user="WilliamBreadenMadden" version="1"
META FILEATTACHMENT attachment="2014-06-30T1401.jpg" attr="" comment="parallel job processor logging" date="1404133353" name="2014-06-30T1401.jpg" path="2014-06-30T1401.jpg" size="1185878" user="WilliamBreadenMadden" version="1"

Revision 102015-03-27 - WilliamBreadenMadden

Line: 1 to 1
 
META TOPICPARENT name="Main.WilliamBreadenMadden"

Availability of mc14_13TeV samples compared to 8 TeV defaults in H to bb

Line: 16 to 16
 
ttbar nonallhadronic 117050 Powheg+Pythia6 48450 ttbar_nonallhad 110401 Powheg+Pythia+Photos+Tauola 400000  
single top tch. top 110090 Powheg+Pythia6 1000000 singletop_tchan_lept_top 110070 Powheg+Pythia+Photos+Tauola 999000  
single top tch, antitop 110091 Powheg+Pythia6 5000000 singletop_tchan_lept_antitop 110071 Powheg+Pythia+Photos+Tauola 1000000  
Changed:
<
<
single top schan 110119 Powheg+Pythia6 - - - - -  
single top, Wt chan. inclusive 110140 Powheg+Pythia6 110305 Wtchan_incl_DR 110305 Powheg+Pythia+Photos+Tauola 958500  
single top, Wt chan. dilepton 110141 Powheg+Pythia6 - -   - -  
ttbar+W 119353 Madgraph+Py6 119353 ttbarW 119353 MadGraph +Pythia+Photos+Tauola 399500  
ttbar+Z 119355 Madgraph+Py6 119355 ttbarZ 119355 MadGraph +Pythia+Photos+Tauola 400000  
ttbar+WW 158813 Madgraph+Py6 119583 ttbarWW 119583 MadGraph +Pythia+Photos+Tauola 10000  
ttbar+Wj Excl 174830 Madgraph+Py6 174830 ttbarWjExcl 174830 MadGraph +Pythia+Photos+Tauola 399500  
ttbar+Wjj Incl. 174831 Madgraph+Py6 174831 ttbarWjjIncl 174831 MadGraph +Pythia+Photos+Tauola 395000  
ttbar+Zj Excl. 174832 Madgraph+Py6 174832 ttbarZjExcl 174832 MadGraph +Pythia+Photos+Tauola 399500  
ttbar+Zjj Incl. 174833 Madgraph+Py6 174833 ttbarZjjIncl 174833 MadGraph +Pythia+Photos+Tauola 399500  
>
>
single top schan 110119 Powheg+Pythia6 1199895 - - - -  
single top, Wt chan. inclusive 110140 Powheg+Pythia6 999692 Wtchan_incl_DR 110305 Powheg+Pythia+Photos+Tauola 958500  
single top, Wt chan. dilepton 110141 Powheg+Pythia6 999894 -   - -  
ttbar+W 119353 Madgraph+Py6   ttbarW 119353 MadGraph +Pythia+Photos+Tauola 399500  
ttbar+Z 119355 Madgraph+Py6   ttbarZ 119355 MadGraph +Pythia+Photos+Tauola 400000  
ttbar+WW 158813 Madgraph+Py6   ttbarWW 119583 MadGraph +Pythia+Photos+Tauola 10000  
ttbar+Wj Excl 174830 Madgraph+Py6   ttbarWjExcl 174830 MadGraph +Pythia+Photos+Tauola 399500  
ttbar+Wjj Incl. 174831 Madgraph+Py6   ttbarWjjIncl 174831 MadGraph +Pythia+Photos+Tauola 395000  
ttbar+Zj Excl. 174832 Madgraph+Py6   ttbarZjExcl 174832 MadGraph +Pythia+Photos+Tauola 399500  
ttbar+Zjj Incl. 174833 Madgraph+Py6   ttbarZjjIncl 174833 MadGraph +Pythia+Photos+Tauola 399500  
 
WZ to enu qq 183735 Sherpa   -   - -  
WZ to munu qq 183737 Sherpa   -   - -  
WZ to taunu qq 183739 Sherpa   -   - -  
Line: 77 to 77
 
Wbb Np2 200258 AlpgenPythia   -   - -  
Wbb Np3 inc 200259 AlpgenPythia   -   - -  
Added:
>
>

Availability of mc14_13TeV samples compared to 8 TeV defaults in H to bb

 
META FILEATTACHMENT attachment="2014-06-30T1358.gif" attr="" comment="parallel job processor" date="1404133307" name="2014-06-30T1358.gif" path="2014-06-30T1358.gif" size="862869" user="WilliamBreadenMadden" version="1"
META FILEATTACHMENT attachment="2014-06-30T1401.jpg" attr="" comment="parallel job processor logging" date="1404133353" name="2014-06-30T1401.jpg" path="2014-06-30T1401.jpg" size="1185878" user="WilliamBreadenMadden" version="1"

Revision 92015-03-27 - WilliamBreadenMadden

Line: 1 to 1
 
META TOPICPARENT name="Main.WilliamBreadenMadden"
Changed:
<
<
>
>

Availability of mc14_13TeV samples compared to 8 TeV defaults in H to bb

 
Changed:
<
<
>
>
  8 TeV 13 TeV
process dataset number generator number of events process dataset number generator number of events AMI link
ttH 125 GeV, allhad 169887 PowHel Py8 999999 ttH, 0-150 GeV slice 189529 Pythia8 50000 link
ttH, 150-350 GeV slice 189530 Pythia8 50000 link
ttH, > 350 GeV slice 189531 Pythia8 50000 link
ttH 125 GeV, l+jets 169888 PowHel Py8 999990 ttH, 0-150 GeV slice 189526 Pythia8 50000 link
ttH, 150-350 GeV slice 189527 Pythia8 50000 link
ttH, >350 GeV slice 189528 Pythia8 50000 link
ttH 125 GeV, dilepton 169889 PowHel Py8 998892 ttH, 0-150 GeV slice 189523 Pythia8 50000 link
ttH, 150-350 GeV slice 189524 Pythia8 50000 link
ttH, >350 GeV slice 189525 Pythia8 50000 link
ttbar nonallhadronic 117050 Powheg+Pythia6 48450 ttbar_nonallhad 110401 Powheg+Pythia+Photos+Tauola 400000  
single top tch. top 110090 Powheg+Pythia6 1000000 singletop_tchan_lept_top 110070 Powheg+Pythia+Photos+Tauola 999000  
single top tch, antitop 110091 Powheg+Pythia6 5000000 singletop_tchan_lept_antitop 110071 Powheg+Pythia+Photos+Tauola 1000000  
single top schan 110119 Powheg+Pythia6 - - - - -  
single top, Wt chan. inclusive 110140 Powheg+Pythia6 110305 Wtchan_incl_DR 110305 Powheg+Pythia+Photos+Tauola 958500  
single top, Wt chan. dilepton 110141 Powheg+Pythia6 - -   - -  
ttbar+W 119353 Madgraph+Py6 119353 ttbarW 119353 MadGraph +Pythia+Photos+Tauola 399500  
ttbar+Z 119355 Madgraph+Py6 119355 ttbarZ 119355 MadGraph +Pythia+Photos+Tauola 400000  
ttbar+WW 158813 Madgraph+Py6 119583 ttbarWW 119583 MadGraph +Pythia+Photos+Tauola 10000  
ttbar+Wj Excl 174830 Madgraph+Py6 174830 ttbarWjExcl 174830 MadGraph +Pythia+Photos+Tauola 399500  
ttbar+Wjj Incl. 174831 Madgraph+Py6 174831 ttbarWjjIncl 174831 MadGraph +Pythia+Photos+Tauola 395000  
ttbar+Zj Excl. 174832 Madgraph+Py6 174832 ttbarZjExcl 174832 MadGraph +Pythia+Photos+Tauola 399500  
ttbar+Zjj Incl. 174833 Madgraph+Py6 174833 ttbarZjjIncl 174833 MadGraph +Pythia+Photos+Tauola 399500  
WZ to enu qq 183735 Sherpa   -   - -  
WZ to munu qq 183737 Sherpa   -   - -  
WZ to taunu qq 183739 Sherpa   -   - -  
WW lnulnu np0 107100 AlpgenJimmy   -   - -  
WW lnulnu np1 107101 AlpgenJimmy   -   - -  
WW lnulnu np2 107102 AlpgenJimmy   -   - -  
WW lnulnu np3 107103 AlpgenJimmy   -   - -  
WZ incl ll np0 107104 AlpgenJimmy   -   - -  
WZ incl ll np1 107105 AlpgenJimmy   -   - -  
WZ incl ll np2 107106 AlpgenJimmy   -   - -  
WZ incl ll np3 107107 AlpgenJimmy   -   - -  
ZZ incl ll np0 107108 AlpgenJimmy   -   - -  
ZZ incl ll np1 107109 AlpgenJimmy   -   - -  
ZZ incl ll np2 107110 AlpgenJimmy   -   - -  
ZZ incl ll np3 107111 AlpgenJimmy   -   - -  
WW qq lnue np0 110829 AlpgenJimmy   -   - -  
WW qq lnue np1 110830 AlpgenJimmy   -   - -  
WW qq lnue np2 110831 AlpgenJimmy   -   - -  
WW qq lnue np3 110832 AlpgenJimmy   -   - -  
Wenu np0 147025 AlpgenPythia   -   - -  
Wenu np1 147026 AlpgenPythia   -   - -  
Wenu np2 147027 AlpgenPythia   -   - -  
Wenu np3 147028 AlpgenPythia   -   - -  
Wenu np4 147029 AlpgenPythia   -   - -  
Wenu np5 incl 147030 AlpgenPythia   -   - -  
Wmunu np0 147033 AlpgenPythia   -   - -  
Wmunu np1 147034 AlpgenPythia   -   - -  
Wmunu np2 147035 AlpgenPythia   -   - -  
Wmunu np3 147036 AlpgenPythia   -   - -  
Wmunu np4 147037 AlpgenPythia   -   - -  
Wmunu np5 incl 147038 AlpgenPythia   -   - -  
Wtaunu np0 147041 AlpgenPythia   -   - -  
Wtaunu np1 147042 AlpgenPythia   -   - -  
Wtaunu np2 147043 AlpgenPythia   -   - -  
Wtaunu np3 147044 AlpgenPythia   -   - -  
Wtaunu np4 147045 AlpgenPythia   -   - -  
Wtaunu np5 inc 147046 AlpgenPythia   -   - -  
Wcc Np0 200156 AlpgenPythia   -   - -  
Wcc Np1 200157 AlpgenPythia   -   - -  
Wcc Np2 200158 AlpgenPythia   -   - -  
Wcc Np3 incl 200159 AlpgenPythia   -   - -  
Wc Np0 200056 AlpgenPythia   -   - -  
Wc Np1 200057 AlpgenPythia   -   - -  
Wc Np2 200058 AlpgenPythia   -   - -  
Wc Np3 200059 AlpgenPythia   -   - -  
Wc Np4 incl 200060 AlpgenPythia   -   - -  
Wbb Np0 200156 AlpgenPythia   -   - -  
Wbb Np1 200257 AlpgenPythia   -   - -  
Wbb Np2 200258 AlpgenPythia   -   - -  
Wbb Np3 inc 200259 AlpgenPythia   -   - -  
 
META FILEATTACHMENT attachment="2014-06-30T1358.gif" attr="" comment="parallel job processor" date="1404133307" name="2014-06-30T1358.gif" path="2014-06-30T1358.gif" size="862869" user="WilliamBreadenMadden" version="1"
META FILEATTACHMENT attachment="2014-06-30T1401.jpg" attr="" comment="parallel job processor logging" date="1404133353" name="2014-06-30T1401.jpg" path="2014-06-30T1401.jpg" size="1185878" user="WilliamBreadenMadden" version="1"

Revision 82015-03-25 - WilliamBreadenMadden

Line: 1 to 1
 
META TOPICPARENT name="Main.WilliamBreadenMadden"
Changed:
<
<

link testing:

with HTML-style link:

ALL

with TWiki-style link:

ALL

8 TeV 13 TeV
process dataset number generator number of events process dataset number generator number of events AMI link
ttH 125 GeV, allhad 169887 PowHel Py8 189529 ttH, 0-150 GeV slice 189529 Pythia8 50000 link
    189530 ttH, 150-350 GeV slice 189530 Pythia8 50000 link
    189531 ttH, >350 GeV slice 189531 Pythia8 50000 link
ttH 125 GeV, l+jets 169888 PowHel Py8 189526 ttH, 0-150 GeV slice 189526 Pythia8 50000 link
    189527 ttH, 150-350 GeV slice 189527 Pythia8 50000 link
    189528 ttH, >350 GeV slice 189528 Pythia8 50000 link
ttH 125 GeV, dilepton 169889 PowHel Py8 189523 ttH, 0-150 GeV slice 189523 Pythia8 50000 link
    189524 ttH, 150-350 GeV slice 189524 Pythia8 50000 link
    189525 ttH, >350 GeV slice 189525 Pythia8 50000 link
ttbar nonallhadronic 117050 Powheg+Pythia6 110401 ttbar_nonallhad 110401 Powheg+Pythia+Photos+Tauola 400000  
single top tch. top 110090 Powheg+Pythia6 110070 singletop_tchan_lept_top 110070 Powheg+Pythia+Photos+Tauola 999000  
single top tch, antitop 110091 Powheg+Pythia6 110071 singletop_tchan_lept_antitop 110071 Powheg+Pythia+Photos+Tauola 1000000  
single top schan 110119 Powheg+Pythia6 - - - - -  
single top, Wt chan. inclusive 110140 Powheg+Pythia6 110305 Wtchan_incl_DR 110305 Powheg+Pythia+Photos+Tauola 958500  
single top, Wt chan. dilepton 110141 Powheg+Pythia6 - -   - -  
ttbar+W 119353 Madgraph+Py6 119353 ttbarW 119353 MadGraph+Pythia+Photos+Tauola 399500  
ttbar+Z 119355 Madgraph+Py6 119355 ttbarZ 119355 MadGraph+Pythia+Photos+Tauola 400000  
ttbar+WW 158813 Madgraph+Py6 119583 ttbarWW 119583 MadGraph+Pythia+Photos+Tauola 10000  
ttbar+Wj Excl 174830 Madgraph+Py6 174830 ttbarWjExcl 174830 MadGraph+Pythia+Photos+Tauola 399500  
ttbar+Wjj Incl. 174831 Madgraph+Py6 174831 ttbarWjjIncl 174831 MadGraph+Pythia+Photos+Tauola 395000  
ttbar+Zj Excl. 174832 Madgraph+Py6 174832 ttbarZjExcl 174832 MadGraph+Pythia+Photos+Tauola 399500  
ttbar+Zjj Incl. 174833 Madgraph+Py6 174833 ttbarZjjIncl 174833 MadGraph+Pythia+Photos+Tauola 399500  
WZ to enu qq 183735 Sherpa   -   - -  
WZ to munu qq 183737 Sherpa   -   - -  
WZ to taunu qq 183739 Sherpa   -   - -  
WW lnulnu np0 107100 AlpgenJimmy   -   - -  
WW lnulnu np1 107101 AlpgenJimmy   -   - -  
WW lnulnu np2 107102 AlpgenJimmy   -   - -  
WW lnulnu np3 107103 AlpgenJimmy   -   - -  
WZ incl ll np0 107104 AlpgenJimmy   -   - -  
WZ incl ll np1 107105 AlpgenJimmy   -   - -  
WZ incl ll np2 107106 AlpgenJimmy   -   - -  
WZ incl ll np3 107107 AlpgenJimmy   -   - -  
ZZ incl ll np0 107108 AlpgenJimmy   -   - -  
ZZ incl ll np1 107109 AlpgenJimmy   -   - -  
ZZ incl ll np2 107110 AlpgenJimmy   -   - -  
ZZ incl ll np3 107111 AlpgenJimmy   -   - -  
WW qq lnue np0 110829 AlpgenJimmy   -   - -  
WW qq lnue np1 110830 AlpgenJimmy   -   - -  
WW qq lnue np2 110831 AlpgenJimmy   -   - -  
WW qq lnue np3 110832 AlpgenJimmy   -   - -  
Wenu np0 147025 AlpgenPythia   -   - -  
Wenu np1 147026 AlpgenPythia   -   - -  
Wenu np2 147027 AlpgenPythia   -   - -  
Wenu np3 147028 AlpgenPythia   -   - -  
Wenu np4 147029 AlpgenPythia   -   - -  
Wenu np5 incl 147030 AlpgenPythia   -   - -  
Wmunu np0 147033 AlpgenPythia   -   - -  
Wmunu np1 147034 AlpgenPythia   -   - -  
Wmunu np2 147035 AlpgenPythia   -   - -  
Wmunu np3 147036 AlpgenPythia   -   - -  
Wmunu np4 147037 AlpgenPythia   -   - -  
Wmunu np5 incl 147038 AlpgenPythia   -   - -  
Wtaunu np0 147041 AlpgenPythia   -   - -  
Wtaunu np1 147042 AlpgenPythia   -   - -  
Wtaunu np2 147043 AlpgenPythia   -   - -  
Wtaunu np3 147044 AlpgenPythia   -   - -  
Wtaunu np4 147045 AlpgenPythia   -   - -  
Wtaunu np5 inc 147046 AlpgenPythia   -   - -  
Wcc Np0 200156 AlpgenPythia   -   - -  
Wcc Np1 200157 AlpgenPythia   -   - -  
Wcc Np2 200158 AlpgenPythia   -   - -  
Wcc Np3 incl 200159 AlpgenPythia   -   - -  
Wc Np0 200056 AlpgenPythia   -   - -  
Wc Np1 200057 AlpgenPythia   -   - -  
Wc Np2 200058 AlpgenPythia   -   - -  
Wc Np3 200059 AlpgenPythia   -   - -  
Wc Np4 incl 200060 AlpgenPythia   -   - -  
Wbb Np0 200156 AlpgenPythia   -   - -  
Wbb Np1 200257 AlpgenPythia   -   - -  
Wbb Np2 200258 AlpgenPythia   -   - -  
Wbb Np3 inc 200259 AlpgenPythia   -   - -  
>
>
 
META FILEATTACHMENT attachment="2014-06-30T1358.gif" attr="" comment="parallel job processor" date="1404133307" name="2014-06-30T1358.gif" path="2014-06-30T1358.gif" size="862869" user="WilliamBreadenMadden" version="1"
META FILEATTACHMENT attachment="2014-06-30T1401.jpg" attr="" comment="parallel job processor logging" date="1404133353" name="2014-06-30T1401.jpg" path="2014-06-30T1401.jpg" size="1185878" user="WilliamBreadenMadden" version="1"

Revision 72015-02-23 - WilliamBreadenMadden

Line: 1 to 1
 
META TOPICPARENT name="Main.WilliamBreadenMadden"
Added:
>
>

link testing:

with HTML-style link:

ALL

with TWiki-style link:

ALL

 
8 TeV 13 TeV
process dataset number generator number of events process dataset number generator number of events AMI link
ttH 125 GeV, allhad 169887 PowHel Py8 189529 ttH, 0-150 GeV slice 189529 Pythia8 50000 link

Revision 62015-02-18 - WilliamBreadenMadden

Line: 1 to 1
 
META TOPICPARENT name="Main.WilliamBreadenMadden"
8 TeV 13 TeV
Changed:
<
<
process dataset number 8 TeV generator number of events process dataset number mc14_13TeV generator number of events AMI link
>
>
process dataset number generator number of events process dataset number generator number of events AMI link
 
ttH 125 GeV, allhad 169887 PowHel Py8 189529 ttH, 0-150 GeV slice 189529 Pythia8 50000 link
Changed:
<
<
    189530 ttH, 150-350 GeV slice 189530 Pythia8 50000  
    189531 ttH, >350 GeV slice 189531 Pythia8 50000  
ttH 125 GeV, l+jets 169888 PowHel Py8 189526 ttH, 0-150 GeV slice 189526 Pythia8 50000  
    189527 ttH, 150-350 GeV slice 189527 Pythia8 50000  
    189528 ttH, >350 GeV slice 189528 Pythia8 50000  
ttH 125 GeV, dilepton 169889 PowHel Py8 189523 ttH, 0-150 GeV slice 189523 Pythia8 50000  
    189524 ttH, 150-350 GeV slice 189524 Pythia8 50000  
    189525 ttH, >350 GeV slice 189525 Pythia8 50000  
>
>
    189530 ttH, 150-350 GeV slice 189530 Pythia8 50000 link
    189531 ttH, >350 GeV slice 189531 Pythia8 50000 link
ttH 125 GeV, l+jets 169888 PowHel Py8 189526 ttH, 0-150 GeV slice 189526 Pythia8 50000 link
    189527 ttH, 150-350 GeV slice 189527 Pythia8 50000 link
    189528 ttH, >350 GeV slice 189528 Pythia8 50000 link
ttH 125 GeV, dilepton 169889 PowHel Py8 189523 ttH, 0-150 GeV slice 189523 Pythia8 50000 link
    189524 ttH, 150-350 GeV slice 189524 Pythia8 50000 link
    189525 ttH, >350 GeV slice 189525 Pythia8 50000 link
 
ttbar nonallhadronic 117050 Powheg+Pythia6 110401 ttbar_nonallhad 110401 Powheg+Pythia+Photos+Tauola 400000  
single top tch. top 110090 Powheg+Pythia6 110070 singletop_tchan_lept_top 110070 Powheg+Pythia+Photos+Tauola 999000  
single top tch, antitop 110091 Powheg+Pythia6 110071 singletop_tchan_lept_antitop 110071 Powheg+Pythia+Photos+Tauola 1000000  
single top schan 110119 Powheg+Pythia6 - - - - -  
Changed:
<
<
single top, Wt chan. inclusive 110140 Powheg+Pythia6 110305 Wtchan_incl_DR 110305   958500  
>
>
single top, Wt chan. inclusive 110140 Powheg+Pythia6 110305 Wtchan_incl_DR 110305 Powheg+Pythia+Photos+Tauola 958500  
 
single top, Wt chan. dilepton 110141 Powheg+Pythia6 - -   - -  
ttbar+W 119353 Madgraph+Py6 119353 ttbarW 119353 MadGraph+Pythia+Photos+Tauola 399500  
ttbar+Z 119355 Madgraph+Py6 119355 ttbarZ 119355 MadGraph+Pythia+Photos+Tauola 400000  

Revision 52015-02-18 - WilliamBreadenMadden

Line: 1 to 1
 
META TOPICPARENT name="Main.WilliamBreadenMadden"
Changed:
<
<

parallel file validation

threading and multiprocessing

In computing, a process is an instance of a computer program that is being executed. It contains the program code and its current activity. A thread is a stream of instructions in a process and multithreading consists of multiple threads running within a process concurrently. Multiprocessing is the instantiation of multiple processes by the operating system. An error in one thread can bring down all the threads of a process while an error in a process can not (easily) bring down another process.

Python Global Interpreter Lock

The Python Global Interpreter Lock (GIL) is a mutual exclusion lock held by a programming language interpreter thread in order to avoid sharing code that is not thread-safe with other threads. Programs written in programming languages with a GIL can use separate processes to achieve parallelism.

The Python interpreter is not fully thread safe and, so, a GIL is implemented in order to support multithreaded Python programs. Without the GIL, even many simple operations are problematic (e.g. incrementing the reference count of some common object simultaneously in two threads could result in a single increment, rather than the intended two increments). As a rule in Python, only the thread that has acquired the GIL may operate on Python objects or call API functions. The Python interpreter attempts to switch threads regularly (using set.checkinterval()) in order to emulate concurrency.

Python multiprocessing

The Python module multiprocessing is a process-based 'threading' interface. It

  • supports spawning multiple processes,
  • offers local and remote concurrency (subprocesses are used instead of threads in order to avoid the GIL),
  • good control of multiple processors of a computer and
  • acceptable control of multiple processors on multiple computers, for example, via SSH.

The multiprocessing module has many built-in options for building a parallel program, but the three most basic are the Process, Queue and Lock classes.

The Process is an abstraction that sets up another Python process, provides it code to run and a way for the parent program to control its execution. Processes are spawned using a Process object and then called using its start() method.

%CODE{"python"}% from multiprocessing import Process

def function1(text): print text

if name == '__main__': process1 = Process(target=function1, args=('hello world',)) process1.start() process1.join() %ENDCODE%

In this example, the Process class is imported, a function for the process to run is created and then a Process object is instantiated with the function it is destined to run. The process is started. It runs and then returns a result. The process is completed using the method join (without this, the child process becomes a zombie process).

multiprocessing communications

Objects may be exchanged between processes through two main types of communication:

  • queues
  • pipes

queues

Queues are simple-to-implement objects that are essentially a thread/process safe first in, first out data structure. They can store any pickleable (serialisable) Python object and and are straightforward to use for sharing data between processes.

%CODE{"python"}% from multiprocessing import Queue

queue1 = Queue()

queue1.put('text') queue1.put(['data1', 1, {'data2': 2}])

queue1.get() # returns 'text' queue1.get() # returns ['data1', 1, {'data2': 2}] %ENDCODE%

pipes

The Pipe() function returns a pair of connection objects connected by a pipe allowing for duplex (two way) communications. The two connection objects returned by Pipe() represent the two 'ends' of the pipe. Each end has methods such as send() and recv().

%CODE{"python"}% from multiprocessing import Process, Pipe

def function1(connection1): connection1.send([1, 2, 3]) connection1.close()

if name == '__main__': parent_connection1, child_connection1 = Pipe() p = Process(target=function1, args=(child_connection1,)) p.start() print parent_connection1.recv() # prints "[1, 2, 3]" p.join() %ENDCODE%

locks

Locks are straightforward ways of allowing code to claim a lock, blocking other processes from executing similar code until the process is complete. The lock then may be released.

%CODE{"python"}% from multiprocessing import Lock

lock1 = Lock()

lock1.acquire() print 'I AM THE ONE!' lock1.release() %ENDCODE%

simple programs

Using Process, Queue and Lock instances, simple programs may be implemented. A rough idea for a program may have a work queue (data to be processed in some way) and result queue (answers produced from the data) used as communications. Processes may be created, as required for performing the work, and then started. The processes may be controlled using a list or a Pool. After all processes have been started, processes may be stopped by telling the processes to join. Each process may store the result of its work in the result queue and this queue could be printed after the child processes have exited.

Job Transforms file validation

Job Transforms can allow most of the complexity of standard Athena jobs to be hidden and to be updated by experts as necessary. As part of managing the processing of the job, Job Transforms offers the ability to test and validate input and output files. This is because errors in file access can be difficult to diagnose in the online software and, once a file has been written, it is useful to check that it reads the file back off the storage system.

The transforms should offer the ability to do both fast (e.g., event count checks) and deep (full ROOT scans) of files. (As the value of such checks must be balanced against the cost of doing them, in practice ATLAS makes a fast check on input files and a deeper check of output files.)

The legacy method of validating files produced by Job Transforms is to complete file validations one by one, serially. Parallelisation of processes such as file validations offers a means of reducing running time and is useful for the purposes of AthenaMP.

Amdhal's law

Amdhal's law asserts that the speedup of a program using multiple processors in parallel computing is limited by the sequential fraction of the program.

parallel job processor

The parallel job processor is, as its name suggests, the system that manages the processing of jobs in parallel. Core to its functioning is the Python multiprocessing module. Multiprocessing is a Python module that supports the spawning of processes using an interface similar to the threading module. In effect, multiprocessing offers an approach to bypassing the Python Global Interpreter Lock by using subprocesses instead of threads.

Here is described visually a non-exhaustive illustration of the parallel job processor and associated concepts. The figure below illustrates the basic structure of the parallel job processor. Of particular note are the objects

parallel job processor

There are a number of steps involved in the typical use and operation of the parallel job processor.

Job and JobGroup definitions

A Job for the parallel job processor consists of

  • a work function,
  • its arguments and
  • other information (such as its timeout specification).

submission

A Job or a JobGroup is submitted to the parallel job processor. The parallel job processor submits all jobs to the process pool it created on initialisation, recording the submission time.

checking for results, timeouts and failures

Submissions are monitored for timeouts and failures, with exceptions being raised when appropriate. Timeouts are measured at the JobGroup level, rather than at the Job level.

getting results

Results of jobs returned by the process pool are propagated to the Jobs and to the JobGroup as they become available.

return results

Results of all jobs are stored in order in the results list of the JobGroup. This list is returned by the parallel job processor.

usage

The code below depicts an example job group for the parallel job processor. In this case, it is a unit test for the system. Here, a job group of two jobs is defined. One of the jobs specifies a hello world function with a sleep time argument and the other specifies a multiplication function with multiplicand arguments. There is a timeout specified for each job.

%CODE{"python"}% ## @brief unit test for working functions # @detail This method is a unit test of the parallel job processor # testing the processing of two simple, working functions. def test_working(self): msg.info("\n\n\n\nPARALLEL JOB PROCESSOR WORKING TEST") jobGroup1 = JobGroup( name = "working functions test", jobs = [ Job( name = "hello world function", workFunction = helloWorld, workFunctionKeywordArguments = {

'sleepTime'
1, }, workFunctionTimeout = 10 ), Job( name = "multiplication function", workFunction = multiply, workFunctionKeywordArguments = {
'multiplicand1'
2,
'multiplicand2'
3 }, workFunctionTimeout = 10 ) ] ) parallelJobProcessor1 = ParallelJobProcessor() parallelJobProcessor1.submit(jobSubmission = jobGroup1) results = parallelJobProcessor1.getResults() self.assertEquals(results, ['hello world', 6])
%ENDCODE%

The figure below illustrates example (abbreviated) parallel job processor terminal output (for the parallel validation of typical physics data).

parallel job processor logging

The video below illustrates some benefits of the use of the parallel job processor -- in this case, for the purpose of making a Monte Carlo estimation of pi.

in a pickle: the current difficulties of serial communications between processes

Serialisation is the process of converting an object to a bytestream. Serialisation of objects for the purposes of communications is provided by the Python pickle module. Certain objects important to file validations cannot be pickled. Specifically, the legacy validation functions were not workable. The solution was to make new validation functions.

The serialisation limitations are likely to be addressed in due course in the shorter term with more advanced serialisation methods such as those offered by https://pypi.python.org/pypi/dill and in the longer term with improvements in Python parallelisation and GIL handling. Dill provides the user with the same interface as pickle and also includes additional features. Its primary use is to send Python objects across a network as a bytestream. It is quite flexible and allows arbitrary user-defined classes and functions to be serialised. Dill is part of https://pypi.python.org/pypi/pathos, a Python framework for heterogeneous computing. Dill is in early development stages.

data validation

The standard validation procedure for output files is defined by the function performStandardFileValidation of the module PyJobTransforms.trfValidation. This function oversees the identification of a validation function appropriate for a given data type and the management of work for the parallel job processor.

In order to address the limitations of pickle, separate data validation functions, such as returnIntegrityOfPOOLFile, are provided by the module PyJobTransforms.trfFileValidationFunctions. The integrity functions defined in this package are associated with the classes of the various data types in the module PyJobTransforms.trfArgClasses. Specifically, the name of the validation function appropriate for a type of data is defined as a data member of the class for the data type. In the case of POOL files, the specification is as follows:

%CODE{"python"}% class argPOOLFile(argAthenaFile):

integrityFunction = "returnIntegrityOfPOOLFile" %ENDCODE%

unit tests

The unit tests for the parallel job processor are defined in test_trfUtilsParallelJobProcessor.py and the unit tests for validation of data using the parallel job processor are defined in test_trfUtilsParallelJobProcessorData.py. The unit tests for the parallel job processor include testing of work functions that are valid, that time out and that fail. The unit tests for the data validation using the parallel job processor include testing of AOD, corrupted AOD, ESD, histogram, TAG and bytestream data.

>
>
8 TeV 13 TeV
process dataset number 8 TeV generator number of events process dataset number mc14_13TeV generator number of events AMI link
ttH 125 GeV, allhad 169887 PowHel Py8 189529 ttH, 0-150 GeV slice 189529 Pythia8 50000 link
    189530 ttH, 150-350 GeV slice 189530 Pythia8 50000  
    189531 ttH, >350 GeV slice 189531 Pythia8 50000  
ttH 125 GeV, l+jets 169888 PowHel Py8 189526 ttH, 0-150 GeV slice 189526 Pythia8 50000  
    189527 ttH, 150-350 GeV slice 189527 Pythia8 50000  
    189528 ttH, >350 GeV slice 189528 Pythia8 50000  
ttH 125 GeV, dilepton 169889 PowHel Py8 189523 ttH, 0-150 GeV slice 189523 Pythia8 50000  
    189524 ttH, 150-350 GeV slice 189524 Pythia8 50000  
    189525 ttH, >350 GeV slice 189525 Pythia8 50000  
ttbar nonallhadronic 117050 Powheg+Pythia6 110401 ttbar_nonallhad 110401 Powheg+Pythia+Photos+Tauola 400000  
single top tch. top 110090 Powheg+Pythia6 110070 singletop_tchan_lept_top 110070 Powheg+Pythia+Photos+Tauola 999000  
single top tch, antitop 110091 Powheg+Pythia6 110071 singletop_tchan_lept_antitop 110071 Powheg+Pythia+Photos+Tauola 1000000  
single top schan 110119 Powheg+Pythia6 - - - - -  
single top, Wt chan. inclusive 110140 Powheg+Pythia6 110305 Wtchan_incl_DR 110305   958500  
single top, Wt chan. dilepton 110141 Powheg+Pythia6 - -   - -  
ttbar+W 119353 Madgraph+Py6 119353 ttbarW 119353 MadGraph+Pythia+Photos+Tauola 399500  
ttbar+Z 119355 Madgraph+Py6 119355 ttbarZ 119355 MadGraph+Pythia+Photos+Tauola 400000  
ttbar+WW 158813 Madgraph+Py6 119583 ttbarWW 119583 MadGraph+Pythia+Photos+Tauola 10000  
ttbar+Wj Excl 174830 Madgraph+Py6 174830 ttbarWjExcl 174830 MadGraph+Pythia+Photos+Tauola 399500  
ttbar+Wjj Incl. 174831 Madgraph+Py6 174831 ttbarWjjIncl 174831 MadGraph+Pythia+Photos+Tauola 395000  
ttbar+Zj Excl. 174832 Madgraph+Py6 174832 ttbarZjExcl 174832 MadGraph+Pythia+Photos+Tauola 399500  
ttbar+Zjj Incl. 174833 Madgraph+Py6 174833 ttbarZjjIncl 174833 MadGraph+Pythia+Photos+Tauola 399500  
WZ to enu qq 183735 Sherpa   -   - -  
WZ to munu qq 183737 Sherpa   -   - -  
WZ to taunu qq 183739 Sherpa   -   - -  
WW lnulnu np0 107100 AlpgenJimmy   -   - -  
WW lnulnu np1 107101 AlpgenJimmy   -   - -  
WW lnulnu np2 107102 AlpgenJimmy   -   - -  
WW lnulnu np3 107103 AlpgenJimmy   -   - -  
WZ incl ll np0 107104 AlpgenJimmy   -   - -  
WZ incl ll np1 107105 AlpgenJimmy   -   - -  
WZ incl ll np2 107106 AlpgenJimmy   -   - -  
WZ incl ll np3 107107 AlpgenJimmy   -   - -  
ZZ incl ll np0 107108 AlpgenJimmy   -   - -  
ZZ incl ll np1 107109 AlpgenJimmy   -   - -  
ZZ incl ll np2 107110 AlpgenJimmy   -   - -  
ZZ incl ll np3 107111 AlpgenJimmy   -   - -  
WW qq lnue np0 110829 AlpgenJimmy   -   - -  
WW qq lnue np1 110830 AlpgenJimmy   -   - -  
WW qq lnue np2 110831 AlpgenJimmy   -   - -  
WW qq lnue np3 110832 AlpgenJimmy   -   - -  
Wenu np0 147025 AlpgenPythia   -   - -  
Wenu np1 147026 AlpgenPythia   -   - -  
Wenu np2 147027 AlpgenPythia   -   - -  
Wenu np3 147028 AlpgenPythia   -   - -  
Wenu np4 147029 AlpgenPythia   -   - -  
Wenu np5 incl 147030 AlpgenPythia   -   - -  
Wmunu np0 147033 AlpgenPythia   -   - -  
Wmunu np1 147034 AlpgenPythia   -   - -  
Wmunu np2 147035 AlpgenPythia   -   - -  
Wmunu np3 147036 AlpgenPythia   -   - -  
Wmunu np4 147037 AlpgenPythia   -   - -  
Wmunu np5 incl 147038 AlpgenPythia   -   - -  
Wtaunu np0 147041 AlpgenPythia   -   - -  
Wtaunu np1 147042 AlpgenPythia   -   - -  
Wtaunu np2 147043 AlpgenPythia   -   - -  
Wtaunu np3 147044 AlpgenPythia   -   - -  
Wtaunu np4 147045 AlpgenPythia   -   - -  
Wtaunu np5 inc 147046 AlpgenPythia   -   - -  
Wcc Np0 200156 AlpgenPythia   -   - -  
Wcc Np1 200157 AlpgenPythia   -   - -  
Wcc Np2 200158 AlpgenPythia   -   - -  
Wcc Np3 incl 200159 AlpgenPythia   -   - -  
Wc Np0 200056 AlpgenPythia   -   - -  
Wc Np1 200057 AlpgenPythia   -   - -  
Wc Np2 200058 AlpgenPythia   -   - -  
Wc Np3 200059 AlpgenPythia   -   - -  
Wc Np4 incl 200060 AlpgenPythia   -   - -  
Wbb Np0 200156 AlpgenPythia   -   - -  
Wbb Np1 200257 AlpgenPythia   -   - -  
Wbb Np2 200258 AlpgenPythia   -   - -  
Wbb Np3 inc 200259 AlpgenPythia   -   - -  
 
META FILEATTACHMENT attachment="2014-06-30T1358.gif" attr="" comment="parallel job processor" date="1404133307" name="2014-06-30T1358.gif" path="2014-06-30T1358.gif" size="862869" user="WilliamBreadenMadden" version="1"
META FILEATTACHMENT attachment="2014-06-30T1401.jpg" attr="" comment="parallel job processor logging" date="1404133353" name="2014-06-30T1401.jpg" path="2014-06-30T1401.jpg" size="1185878" user="WilliamBreadenMadden" version="1"

Revision 42014-07-29 - WilliamBreadenMadden

Line: 1 to 1
 
META TOPICPARENT name="Main.WilliamBreadenMadden"
Changed:
<
<

parallel file validation

>
>

parallel file validation

 
Changed:
<
<

threading and multiprocessing

>
>

threading and multiprocessing

  In computing, a process is an instance of a computer program that is being executed. It contains the program code and its current activity. A thread is a stream of instructions in a process and multithreading consists of multiple threads running within a process concurrently. Multiprocessing is the instantiation of multiple processes by the operating system. An error in one thread can bring down all the threads of a process while an error in a process can not (easily) bring down another process.
Changed:
<
<

Python Global Interpreter Lock

>
>

Python Global Interpreter Lock

  The Python Global Interpreter Lock (GIL) is a mutual exclusion lock held by a programming language interpreter thread in order to avoid sharing code that is not thread-safe with other threads. Programs written in programming languages with a GIL can use separate processes to achieve parallelism.
Changed:
<
<
The Python interpreter is not fully thread safe and, so, a GIL is implemented in order to support multithreaded Python programs. Without the GIL, even many simple operations are problematic (e.g. incrementing the reference count of some common object simultaneously in two threads could result in a single increment, rather than the intended two increments). As a rule in Python, Only the thread that has acquired the GIL may operate on Python objects or call API functions. The Python interpreter attempts to switch threads regularly (using set.checkinterval()) in order to emulate concurrency.
>
>
The Python interpreter is not fully thread safe and, so, a GIL is implemented in order to support multithreaded Python programs. Without the GIL, even many simple operations are problematic (e.g. incrementing the reference count of some common object simultaneously in two threads could result in a single increment, rather than the intended two increments). As a rule in Python, only the thread that has acquired the GIL may operate on Python objects or call API functions. The Python interpreter attempts to switch threads regularly (using set.checkinterval()) in order to emulate concurrency.
 
Changed:
<
<

Python multiprocessing

>
>

Python multiprocessing

 
Changed:
<
<
The Python module "multiprocessing" is a process-based 'threading' interface. It
>
>
The Python module multiprocessing is a process-based 'threading' interface. It
 
  • supports spawning multiple processes,
  • offers local and remote concurrency (subprocesses are used instead of threads in order to avoid the GIL),
Line: 39 to 39
  In this example, the Process class is imported, a function for the process to run is created and then a Process object is instantiated with the function it is destined to run. The process is started. It runs and then returns a result. The process is completed using the method join (without this, the child process becomes a zombie process).
Changed:
<
<

multiprocessing communications

>
>

multiprocessing communications

  Objects may be exchanged between processes through two main types of communication:

  • queues
  • pipes
Changed:
<
<

a taste of Python multiprocessing: queues

>
>

queues

 
Changed:
<
<
Queues are simple to implement objects that are essentially a thread/process safe first in, first out data structure. They can store any pickleable (serialisable) Python object and and are straightforward to use for sharing data between processes.
>
>
Queues are simple-to-implement objects that are essentially a thread/process safe first in, first out data structure. They can store any pickleable (serialisable) Python object and and are straightforward to use for sharing data between processes.
  %CODE{"python"}% from multiprocessing import Queue
Line: 62 to 62
 queue1.get() # returns ['data1', 1, {'data2': 2}] %ENDCODE%
Changed:
<
<

pipes

>
>

pipes

  The Pipe() function returns a pair of connection objects connected by a pipe allowing for duplex (two way) communications. The two connection objects returned by Pipe() represent the two 'ends' of the pipe. Each end has methods such as send() and recv().
Line: 81 to 81
  p.join() %ENDCODE%
Changed:
<
<

locks

>
>

locks

  Locks are straightforward ways of allowing code to claim a lock, blocking other processes from executing similar code until the process is complete. The lock then may be released.
Line: 95 to 95
 lock1.release() %ENDCODE%
Changed:
<
<

simple programs

>
>

simple programs

 
Changed:
<
<
Using Processs, Queues and Locks, simple programs may be implemented. A rough idea for a program may have a work queue (data to be processed in some way) and result queue (answers produced from the data) used as communications. Processes may be created, as required for performing the work, and then started. The processes may be controlled using a list or a Pool. After all processes have been started, processes may be stopped by telling the processes to join. Each process may store the result of its work in the result queue and this queue could be printed after the child processes have exited.
>
>
Using Process, Queue and Lock instances, simple programs may be implemented. A rough idea for a program may have a work queue (data to be processed in some way) and result queue (answers produced from the data) used as communications. Processes may be created, as required for performing the work, and then started. The processes may be controlled using a list or a Pool. After all processes have been started, processes may be stopped by telling the processes to join. Each process may store the result of its work in the result queue and this queue could be printed after the child processes have exited.
 
Changed:
<
<
multiprocessing in Job Transforms

future: implementation of multiprocessing in Job Transforms (travelling to CERN soon to work more directly on project)

Job Transforms file validation

>
>

Job Transforms file validation

  Job Transforms can allow most of the complexity of standard Athena jobs to be hidden and to be updated by experts as necessary. As part of managing the processing of the job, Job Transforms offers the ability to test and validate input and output files. This is because errors in file access can be difficult to diagnose in the online software and, once a file has been written, it is useful to check that it reads the file back off the storage system.
Changed:
<
<
The transforms should offer the ability to do both fast (e.g., event count checks) and deep (full ROOT scans) of files. (As the value of such checks must be balanced against the cost of diong them, in practice ATLAS makes a fast check on input files and a deeper check of output files.)
>
>
The transforms should offer the ability to do both fast (e.g., event count checks) and deep (full ROOT scans) of files. (As the value of such checks must be balanced against the cost of doing them, in practice ATLAS makes a fast check on input files and a deeper check of output files.)
  The legacy method of validating files produced by Job Transforms is to complete file validations one by one, serially. Parallelisation of processes such as file validations offers a means of reducing running time and is useful for the purposes of AthenaMP.
Changed:
<
<

Amdhal's law

>
>

Amdhal's law

  Amdhal's law asserts that the speedup of a program using multiple processors in parallel computing is limited by the sequential fraction of the program.
Changed:
<
<

parallel job processor

>
>

parallel job processor

 
Changed:
<
<
The parallel job processor is a central part of the service task. It is, as its name suggests, the system that manages the processing of jobs in parallel. Core to its functioning is the Python multiprocessing module. Multiprocessing is a Python module that supports the spawning of processes using an interface similar to the threading module. In effect, multiprocessing offers an approach to bypassing the Python Global Interpreter Lock by using subprocesses instead of threads.
>
>
The parallel job processor is, as its name suggests, the system that manages the processing of jobs in parallel. Core to its functioning is the Python multiprocessing module. Multiprocessing is a Python module that supports the spawning of processes using an interface similar to the threading module. In effect, multiprocessing offers an approach to bypassing the Python Global Interpreter Lock by using subprocesses instead of threads.
  Here is described visually a non-exhaustive illustration of the parallel job processor and associated concepts. The figure below illustrates the basic structure of the parallel job processor. Of particular note are the objects
Line: 129 to 125
  parallel job processor
Changed:
<
<
There are a number of steps involved in the typical usage and operation of the parallel job processor.
>
>
There are a number of steps involved in the typical use and operation of the parallel job processor.
 
Changed:
<
<

Job and JobGroup definitions

>
>
Job and JobGroup definitions
  A Job for the parallel job processor consists of
Line: 139 to 135
 
  • its arguments and
  • other information (such as its timeout specification).
Changed:
<
<

submission

>
>
submission
  A Job or a JobGroup is submitted to the parallel job processor. The parallel job processor submits all jobs to the process pool it created on initialisation, recording the submission time.
Changed:
<
<

checking for results, timeouts and failures

>
>
checking for results, timeouts and failures
  Submissions are monitored for timeouts and failures, with exceptions being raised when appropriate. Timeouts are measured at the JobGroup level, rather than at the Job level.
Changed:
<
<

getting results

>
>
getting results
  Results of jobs returned by the process pool are propagated to the Jobs and to the JobGroup as they become available.
Changed:
<
<

return results

>
>
return results
  Results of all jobs are stored in order in the results list of the JobGroup. This list is returned by the parallel job processor.
Changed:
<
<

usage

>
>

usage

  The code below depicts an example job group for the parallel job processor. In this case, it is a unit test for the system. Here, a job group of two jobs is defined. One of the jobs specifies a hello world function with a sleep time argument and the other specifies a multiplication function with multiplicand arguments. There is a timeout specified for each job.
Line: 201 to 197
 
Changed:
<
<

in a pickle: the current difficulties of serial communications between processes

>
>

in a pickle: the current difficulties of serial communications between processes

 
Changed:
<
<
Serialisation is the process of converting an object to a bytestream.
>
>
Serialisation is the process of converting an object to a bytestream. Serialisation of objects for the purposes of communications is provided by the Python pickle module. Certain objects important to file validations cannot be pickled. Specifically, the legacy validation functions were not workable. The solution was to make new validation functions.
 
Changed:
<
<
Serialisation of objects for the purposes of communications is provided by the Python pickle module. Certain objects important to file validations cannot be pickled. Specifically, the legacy validation functions were not workable. The solution was to make new validation functions. While this extended the service task somewhat, it proved to be the most obviously reliable and validated approach -- and was successful.
>
>
The serialisation limitations are likely to be addressed in due course in the shorter term with more advanced serialisation methods such as those offered by https://pypi.python.org/pypi/dill and in the longer term with improvements in Python parallelisation and GIL handling. Dill provides the user with the same interface as pickle and also includes additional features. Its primary use is to send Python objects across a network as a bytestream. It is quite flexible and allows arbitrary user-defined classes and functions to be serialised. Dill is part of https://pypi.python.org/pypi/pathos, a Python framework for heterogeneous computing. Dill is in early development stages.
 
Changed:
<
<
The serialisation limitations are likely to be addressed in due course in the shorter term with more advanced serialisation methods such as those offered by dill (https://pypi.python.org/pypi/dill) and in the longer term with improvements in Python parallelisation and GIL handling.
>
>

data validation

 
Changed:
<
<
Dill provides the user with the same interface as pickle and also includes additional features. Its primary use is to send Python objects across a network as a bytestream. It is quite flexible and allows arbitrary user-defined classes and functions to be serialised. Dill is part of pathos, a Python framework for heterogeneous computing. Dill is in the early development states.
>
>
The standard validation procedure for output files is defined by the function performStandardFileValidation of the module PyJobTransforms.trfValidation. This function oversees the identification of a validation function appropriate for a given data type and the management of work for the parallel job processor.
 
Changed:
<
<
In order to address the limitations of pickle, separate data validation functions, such as returnIntegrityOfPOOLFile, are provided by the module PyJobTransforms.trfFileValidationFunctions. The integrity functions defined in this package are associated with the arg classes for the various data types in the module PyJobTransforms.trfArgClasses. Specifically, the name of the validation function appropriate for a type of data are defined as data members of the class for the data type. In the case of POOL files, the specification is as follows:
>
>
In order to address the limitations of pickle, separate data validation functions, such as returnIntegrityOfPOOLFile, are provided by the module PyJobTransforms.trfFileValidationFunctions. The integrity functions defined in this package are associated with the classes of the various data types in the module PyJobTransforms.trfArgClasses. Specifically, the name of the validation function appropriate for a type of data is defined as a data member of the class for the data type. In the case of POOL files, the specification is as follows:
  %CODE{"python"}% class argPOOLFile(argAthenaFile):
Line: 219 to 215
  integrityFunction = "returnIntegrityOfPOOLFile" %ENDCODE%
Changed:
<
<
\section{characterising impact}

Figures~\ref{figure:impact_1},~\ref{figure:impact_1} and \ref{figure:impact_1} depict tentative graphs of processing POOL files. Figure~\ref{figure:impact_1} illustrates the processing time against number of files validated in parallel while figure~\ref{figure:impact_2}illustrates the ratio of processing time to processing time for one file.

>
>

unit tests

 
Changed:
<
<
\begin{figure}[H] \begin{center} \includegraphics[width=\imageWidthASpecification]{figures/2014-04-10_1.eps} \end{center} \caption{parallel job processor: large efficiency improvement as a result of parallelisation} \label{figure:impact_1} \end{figure}

\begin{figure}[H] \begin{center} \includegraphics[width=\imageWidthASpecification]{figures/2014-04-10_2.eps} \end{center} \caption{parallel job processor: large efficiency improvement as a result of parallelisation} \label{figure:impact_2} \end{figure}

\noindent Figure~\ref{figure:impact_3} illustrates the performance for the unusual task of parallel validation of up to 40 POOL files.

\begin{figure}[H] \begin{center} \includegraphics[width=\imageWidthASpecification]{figures/2014-04-11_1.eps} \end{center} \caption{parallel job processor: large efficiency improvement as a result of parallelisation} \label{figure:impact_3} \end{figure}

>
>
The unit tests for the parallel job processor are defined in test_trfUtilsParallelJobProcessor.py and the unit tests for validation of data using the parallel job processor are defined in test_trfUtilsParallelJobProcessorData.py. The unit tests for the parallel job processor include testing of work functions that are valid, that time out and that fail. The unit tests for the data validation using the parallel job processor include testing of AOD, corrupted AOD, ESD, histogram, TAG and bytestream data.
 
META FILEATTACHMENT attachment="2014-06-30T1358.gif" attr="" comment="parallel job processor" date="1404133307" name="2014-06-30T1358.gif" path="2014-06-30T1358.gif" size="862869" user="WilliamBreadenMadden" version="1"
META FILEATTACHMENT attachment="2014-06-30T1401.jpg" attr="" comment="parallel job processor logging" date="1404133353" name="2014-06-30T1401.jpg" path="2014-06-30T1401.jpg" size="1185878" user="WilliamBreadenMadden" version="1"

Revision 32014-07-29 - WilliamBreadenMadden

Line: 1 to 1
 
META TOPICPARENT name="Main.WilliamBreadenMadden"

parallel file validation

threading and multiprocessing

Changed:
<
<
  • thread: stream of instructions in a process
  • multithreading: multiple threads running within a process
  • multiprocessing: multiple processes instantiated by the operating system (an error in one process can not bring down another process)

  • multithreading versus multiprocessing: An error in one thread can bring down all the threads of a process while an error in a process can not (easily) bring down another process.
>
>
In computing, a process is an instance of a computer program that is being executed. It contains the program code and its current activity. A thread is a stream of instructions in a process and multithreading consists of multiple threads running within a process concurrently. Multiprocessing is the instantiation of multiple processes by the operating system. An error in one thread can bring down all the threads of a process while an error in a process can not (easily) bring down another process.
 

Python Global Interpreter Lock

Line: 123 to 119
  The parallel job processor is a central part of the service task. It is, as its name suggests, the system that manages the processing of jobs in parallel. Core to its functioning is the Python multiprocessing module. Multiprocessing is a Python module that supports the spawning of processes using an interface similar to the threading module. In effect, multiprocessing offers an approach to bypassing the Python Global Interpreter Lock by using subprocesses instead of threads.
Changed:
<
<
Here is described visually a non-exhaustive illustration of the parallel job processor and associated concepts.

parallel job processor

Figure~\ref{figure:PJP_step_0} illustrates the basic structure of the parallel job processor. Of particular note are the objects

>
>
Here is described visually a non-exhaustive illustration of the parallel job processor and associated concepts. The figure below illustrates the basic structure of the parallel job processor. Of particular note are the objects
 
  • function,
  • Job,
Line: 137 to 127
 
Changed:
<
<
There are a number of steps involved in the typical usage and operation of the parallel job processor.
>
>
parallel job processor
 
Changed:
<
<
\subsection{Job and JobGroup definitions}
>
>
There are a number of steps involved in the typical usage and operation of the parallel job processor.
 
Changed:
<
<
\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/2.jpg} \end{center} \caption{parallel job processor: Job and JobGroup definitions} \label{figure:PJP_step_1} \end{figure}
>
>

Job and JobGroup definitions

  A Job for the parallel job processor consists of
Changed:
<
<
\begin{itemize} \item a work function, \item its arguments and \item other information (such as its timeout specification). \end{itemize}
>
>
  • a work function,
  • its arguments and
  • other information (such as its timeout specification).
 
Changed:
<
<
\subsection{submission}

\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/3.jpg} \end{center} \caption{parallel job processor: submission} \label{figure:PJP_step_2} \end{figure}

>
>

submission

  A Job or a JobGroup is submitted to the parallel job processor. The parallel job processor submits all jobs to the process pool it created on initialisation, recording the submission time.
Changed:
<
<
\subsection{checking for results, timeouts and failures}

\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/4.jpg} \end{center} \caption{parallel job processor: checking for results, timeouts and failures} \label{figure:PJP_step_3} \end{figure}

>
>

checking for results, timeouts and failures

  Submissions are monitored for timeouts and failures, with exceptions being raised when appropriate. Timeouts are measured at the JobGroup level, rather than at the Job level.
Changed:
<
<
\subsection{getting results}

\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/5.jpg} \end{center} \caption{parallel job processor: getting results} \label{figure:PJP_step_4} \end{figure}

>
>

getting results

  Results of jobs returned by the process pool are propagated to the Jobs and to the JobGroup as they become available.
Changed:
<
<
\subsection{return results}

\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/6.jpg} \end{center} \caption{parallel job processor: return results} \label{figure:PJP_step_5} \end{figure}

>
>

return results

  Results of all jobs are stored in order in the results list of the JobGroup. This list is returned by the parallel job processor.
Changed:
<
<
\subsection{usage}

\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/7.jpg} \end{center} \caption{parallel job processor: job specification in Python} \label{figure:PJP_job_specification} \end{figure}

>
>

usage

 
Changed:
<
<
Figure~\ref{figure:PJP_job_specification} depicts an example job group for the parallel job processor. In this case, it is a unit test for the system. Here, a job group of two jobs is defined. One of the jobs specifies a hello world function with a sleep time argument and the other specifies a multiplication function with multiplicand arguments. There is a timeout specified for each job.
>
>
The code below depicts an example job group for the parallel job processor. In this case, it is a unit test for the system. Here, a job group of two jobs is defined. One of the jobs specifies a hello world function with a sleep time argument and the other specifies a multiplication function with multiplicand arguments. There is a timeout specified for each job.
 
Changed:
<
<
\newpage
>
>
%CODE{"python"}% ## @brief unit test for working functions # @detail This method is a unit test of the parallel job processor # testing the processing of two simple, working functions. def test_working(self): msg.info("\n\n\n\nPARALLEL JOB PROCESSOR WORKING TEST") jobGroup1 = JobGroup( name = "working functions test", jobs = [ Job( name = "hello world function", workFunction = helloWorld, workFunctionKeywordArguments = {
'sleepTime'
1, }, workFunctionTimeout = 10 ), Job( name = "multiplication function", workFunction = multiply, workFunctionKeywordArguments = {
'multiplicand1'
2,
'multiplicand2'
3 }, workFunctionTimeout = 10 ) ] ) parallelJobProcessor1 = ParallelJobProcessor() parallelJobProcessor1.submit(jobSubmission = jobGroup1) results = parallelJobProcessor1.getResults() self.assertEquals(results, ['hello world', 6])
%ENDCODE%
 
Changed:
<
<
\noindent Figure~\ref{figure:PJP_terminal_output_1} illustrates example (abbreviated) parallel job processor terminal output (for the parallel validation of typical physics data).
>
>
The figure below illustrates example (abbreviated) parallel job processor terminal output (for the parallel validation of typical physics data).
  parallel job processor logging
Changed:
<
<
\section{challenges}
>
>
The video below illustrates some benefits of the use of the parallel job processor -- in this case, for the purpose of making a Monte Carlo estimation of pi.

in a pickle: the current difficulties of serial communications between processes

 
Changed:
<
<
The Job Transforms infrastructure was written before there were standardised ways of reliably parallelising processes in Python, so, naturally, there were challenges in adapting Job Transforms to parallisation. One problem encountered involved communications between processes breaking down in a confusing, seemingly inconsistent way. Pools use certain signals to receive termination commands etc.~and Job Transforms was using the same signals. The solution in this case was to render Job Transforms silent while the parallel processor is running. Another problem encountered was that certain objects in Job Transforms cannot be ``pickled'' (serialised in inter-process communications). Serialisation of objects for the purposes of communications is provided by the \href{http://docs.python.org/2/library/pickle.html}{\textcolor{black!100}{Python pickle module}}. Certain objects important to file validations cannot be pickled. Specifically, the legacy validation functions were not workable. The solution was to make new validation functions. While this extended the service task somewhat, it proved to be the most obviously reliable and validated approach -- and was successful.
>
>
Serialisation is the process of converting an object to a bytestream.
 
Changed:
<
<
\section{service task status}
>
>
Serialisation of objects for the purposes of communications is provided by the Python pickle module. Certain objects important to file validations cannot be pickled. Specifically, the legacy validation functions were not workable. The solution was to make new validation functions. While this extended the service task somewhat, it proved to be the most obviously reliable and validated approach -- and was successful.
 
Changed:
<
<
The main activities in the service task were as follows:
>
>
The serialisation limitations are likely to be addressed in due course in the shorter term with more advanced serialisation methods such as those offered by dill (https://pypi.python.org/pypi/dill) and in the longer term with improvements in Python parallelisation and GIL handling.
 
Changed:
<
<
\begin{itemize} %\small \item familiarisation with Job Transforms basic usage and code \item starting documentation for new Job Transforms \item general support (lots of coding tasks to support new development) \item systematic investigation of parallel processing methods in Python, principally methods associated with the Python multiprocessing module \item coded multiprocessing code library \item coded \href{https://svnweb.cern.ch/trac/atlasoff/browser/Tools/PyJobTransforms/trunk/python/trfUtils.py}{\textcolor{black!100}{parallel job processor}} \item coded \href{https://svnweb.cern.ch/trac/atlasoff/browser/Tools/PyJobTransforms/trunk/test/test_trfUtilsParallelJobProcessor.py}{\textcolor{black!100}{unit tests for parallel job processor}} (validation for parallel job processor) \item coded new validation functions and associated exception handling procedures \item high-level controls on how much of the CPU resources the parallel job processor may request \item validation of final unit tests (almost complete) \item paper \emph{ATLAS Job Transforms: A Data Driven Workflow Engine}~\cite{Stewart_1} \end{itemize} %\item[\XSolidBrush] bar %\item[] bar
>
>
Dill provides the user with the same interface as pickle and also includes additional features. Its primary use is to send Python objects across a network as a bytestream. It is quite flexible and allows arbitrary user-defined classes and functions to be serialised. Dill is part of pathos, a Python framework for heterogeneous computing. Dill is in the early development states.

In order to address the limitations of pickle, separate data validation functions, such as returnIntegrityOfPOOLFile, are provided by the module PyJobTransforms.trfFileValidationFunctions. The integrity functions defined in this package are associated with the arg classes for the various data types in the module PyJobTransforms.trfArgClasses. Specifically, the name of the validation function appropriate for a type of data are defined as data members of the class for the data type. In the case of POOL files, the specification is as follows:

%CODE{"python"}% class argPOOLFile(argAthenaFile):

integrityFunction = "returnIntegrityOfPOOLFile" %ENDCODE%

  \section{characterising impact}

Revision 22014-06-30 - WilliamBreadenMadden

Line: 1 to 1
 
META TOPICPARENT name="Main.WilliamBreadenMadden"

parallel file validation

Line: 125 to 125
  Here is described visually a non-exhaustive illustration of the parallel job processor and associated concepts.
Changed:
<
<
\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/1.jpg} \end{center} \caption{parallel job processor} \label{figure:PJP_step_0} \end{figure}
>
>
parallel job processor
  Figure~\ref{figure:PJP_step_0} illustrates the basic structure of the parallel job processor. Of particular note are the objects
Changed:
<
<
\begin{itemize} \item function, \item Job, \item JobGroup, \item ParallelJobProcessor and \item pool. \end{itemize}
>
>
  There are a number of steps involved in the typical usage and operation of the parallel job processor.
Line: 228 to 222
 \noindent Figure~\ref{figure:PJP_terminal_output_1} illustrates example (abbreviated) parallel job processor terminal output (for the parallel validation of typical physics data).
Changed:
<
<
\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/8.jpg} \end{center} \caption{parallel job processor: submission} \label{figure:PJP_terminal_output_1} \end{figure}
>
>
parallel job processor logging
  \section{challenges}
Line: 291 to 279
 \caption{parallel job processor: large efficiency improvement as a result of parallelisation} \label{figure:impact_3} \end{figure}
Added:
>
>

META FILEATTACHMENT attachment="2014-06-30T1358.gif" attr="" comment="parallel job processor" date="1404133307" name="2014-06-30T1358.gif" path="2014-06-30T1358.gif" size="862869" user="WilliamBreadenMadden" version="1"
META FILEATTACHMENT attachment="2014-06-30T1401.jpg" attr="" comment="parallel job processor logging" date="1404133353" name="2014-06-30T1401.jpg" path="2014-06-30T1401.jpg" size="1185878" user="WilliamBreadenMadden" version="1"

Revision 12014-06-25 - WilliamBreadenMadden

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="Main.WilliamBreadenMadden"

parallel file validation

threading and multiprocessing

  • thread: stream of instructions in a process
  • multithreading: multiple threads running within a process
  • multiprocessing: multiple processes instantiated by the operating system (an error in one process can not bring down another process)

  • multithreading versus multiprocessing: An error in one thread can bring down all the threads of a process while an error in a process can not (easily) bring down another process.

Python Global Interpreter Lock

The Python Global Interpreter Lock (GIL) is a mutual exclusion lock held by a programming language interpreter thread in order to avoid sharing code that is not thread-safe with other threads. Programs written in programming languages with a GIL can use separate processes to achieve parallelism.

The Python interpreter is not fully thread safe and, so, a GIL is implemented in order to support multithreaded Python programs. Without the GIL, even many simple operations are problematic (e.g. incrementing the reference count of some common object simultaneously in two threads could result in a single increment, rather than the intended two increments). As a rule in Python, Only the thread that has acquired the GIL may operate on Python objects or call API functions. The Python interpreter attempts to switch threads regularly (using set.checkinterval()) in order to emulate concurrency.

Python multiprocessing

The Python module "multiprocessing" is a process-based 'threading' interface. It

  • supports spawning multiple processes,
  • offers local and remote concurrency (subprocesses are used instead of threads in order to avoid the GIL),
  • good control of multiple processors of a computer and
  • acceptable control of multiple processors on multiple computers, for example, via SSH.

The multiprocessing module has many built-in options for building a parallel program, but the three most basic are the Process, Queue and Lock classes.

The Process is an abstraction that sets up another Python process, provides it code to run and a way for the parent program to control its execution. Processes are spawned using a Process object and then called using its start() method.

%CODE{"python"}% from multiprocessing import Process

def function1(text): print text

if name == '__main__': process1 = Process(target=function1, args=('hello world',)) process1.start() process1.join() %ENDCODE%

In this example, the Process class is imported, a function for the process to run is created and then a Process object is instantiated with the function it is destined to run. The process is started. It runs and then returns a result. The process is completed using the method join (without this, the child process becomes a zombie process).

multiprocessing communications

Objects may be exchanged between processes through two main types of communication:

  • queues
  • pipes

a taste of Python multiprocessing: queues

Queues are simple to implement objects that are essentially a thread/process safe first in, first out data structure. They can store any pickleable (serialisable) Python object and and are straightforward to use for sharing data between processes.

%CODE{"python"}% from multiprocessing import Queue

queue1 = Queue()

queue1.put('text') queue1.put(['data1', 1, {'data2': 2}])

queue1.get() # returns 'text' queue1.get() # returns ['data1', 1, {'data2': 2}] %ENDCODE%

pipes

The Pipe() function returns a pair of connection objects connected by a pipe allowing for duplex (two way) communications. The two connection objects returned by Pipe() represent the two 'ends' of the pipe. Each end has methods such as send() and recv().

%CODE{"python"}% from multiprocessing import Process, Pipe

def function1(connection1): connection1.send([1, 2, 3]) connection1.close()

if name == '__main__': parent_connection1, child_connection1 = Pipe() p = Process(target=function1, args=(child_connection1,)) p.start() print parent_connection1.recv() # prints "[1, 2, 3]" p.join() %ENDCODE%

locks

Locks are straightforward ways of allowing code to claim a lock, blocking other processes from executing similar code until the process is complete. The lock then may be released.

%CODE{"python"}% from multiprocessing import Lock

lock1 = Lock()

lock1.acquire() print 'I AM THE ONE!' lock1.release() %ENDCODE%

simple programs

Using Processs, Queues and Locks, simple programs may be implemented. A rough idea for a program may have a work queue (data to be processed in some way) and result queue (answers produced from the data) used as communications. Processes may be created, as required for performing the work, and then started. The processes may be controlled using a list or a Pool. After all processes have been started, processes may be stopped by telling the processes to join. Each process may store the result of its work in the result queue and this queue could be printed after the child processes have exited.

multiprocessing in Job Transforms

future: implementation of multiprocessing in Job Transforms (travelling to CERN soon to work more directly on project)

Job Transforms file validation

Job Transforms can allow most of the complexity of standard Athena jobs to be hidden and to be updated by experts as necessary. As part of managing the processing of the job, Job Transforms offers the ability to test and validate input and output files. This is because errors in file access can be difficult to diagnose in the online software and, once a file has been written, it is useful to check that it reads the file back off the storage system.

The transforms should offer the ability to do both fast (e.g., event count checks) and deep (full ROOT scans) of files. (As the value of such checks must be balanced against the cost of diong them, in practice ATLAS makes a fast check on input files and a deeper check of output files.)

The legacy method of validating files produced by Job Transforms is to complete file validations one by one, serially. Parallelisation of processes such as file validations offers a means of reducing running time and is useful for the purposes of AthenaMP.

Amdhal's law

Amdhal's law asserts that the speedup of a program using multiple processors in parallel computing is limited by the sequential fraction of the program.

parallel job processor

The parallel job processor is a central part of the service task. It is, as its name suggests, the system that manages the processing of jobs in parallel. Core to its functioning is the Python multiprocessing module. Multiprocessing is a Python module that supports the spawning of processes using an interface similar to the threading module. In effect, multiprocessing offers an approach to bypassing the Python Global Interpreter Lock by using subprocesses instead of threads.

Here is described visually a non-exhaustive illustration of the parallel job processor and associated concepts.

\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/1.jpg} \end{center} \caption{parallel job processor} \label{figure:PJP_step_0} \end{figure}

Figure~\ref{figure:PJP_step_0} illustrates the basic structure of the parallel job processor. Of particular note are the objects

\begin{itemize} \item function, \item Job, \item JobGroup, \item ParallelJobProcessor and \item pool. \end{itemize}

There are a number of steps involved in the typical usage and operation of the parallel job processor.

\subsection{Job and JobGroup definitions}

\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/2.jpg} \end{center} \caption{parallel job processor: Job and JobGroup definitions} \label{figure:PJP_step_1} \end{figure}

A Job for the parallel job processor consists of

\begin{itemize} \item a work function, \item its arguments and \item other information (such as its timeout specification). \end{itemize}

\subsection{submission}

\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/3.jpg} \end{center} \caption{parallel job processor: submission} \label{figure:PJP_step_2} \end{figure}

A Job or a JobGroup is submitted to the parallel job processor. The parallel job processor submits all jobs to the process pool it created on initialisation, recording the submission time.

\subsection{checking for results, timeouts and failures}

\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/4.jpg} \end{center} \caption{parallel job processor: checking for results, timeouts and failures} \label{figure:PJP_step_3} \end{figure}

Submissions are monitored for timeouts and failures, with exceptions being raised when appropriate. Timeouts are measured at the JobGroup level, rather than at the Job level.

\subsection{getting results}

\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/5.jpg} \end{center} \caption{parallel job processor: getting results} \label{figure:PJP_step_4} \end{figure}

Results of jobs returned by the process pool are propagated to the Jobs and to the JobGroup as they become available.

\subsection{return results}

\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/6.jpg} \end{center} \caption{parallel job processor: return results} \label{figure:PJP_step_5} \end{figure}

Results of all jobs are stored in order in the results list of the JobGroup. This list is returned by the parallel job processor.

\subsection{usage}

\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/7.jpg} \end{center} \caption{parallel job processor: job specification in Python} \label{figure:PJP_job_specification} \end{figure}

Figure~\ref{figure:PJP_job_specification} depicts an example job group for the parallel job processor. In this case, it is a unit test for the system. Here, a job group of two jobs is defined. One of the jobs specifies a hello world function with a sleep time argument and the other specifies a multiplication function with multiplicand arguments. There is a timeout specified for each job.

\newpage

\noindent Figure~\ref{figure:PJP_terminal_output_1} illustrates example (abbreviated) parallel job processor terminal output (for the parallel validation of typical physics data).

\begin{figure}[H] \begin{center} \includegraphics[width=\textwidth]{figures/8.jpg} \end{center} \caption{parallel job processor: submission} \label{figure:PJP_terminal_output_1} \end{figure}

\section{challenges}

The Job Transforms infrastructure was written before there were standardised ways of reliably parallelising processes in Python, so, naturally, there were challenges in adapting Job Transforms to parallisation. One problem encountered involved communications between processes breaking down in a confusing, seemingly inconsistent way. Pools use certain signals to receive termination commands etc.~and Job Transforms was using the same signals. The solution in this case was to render Job Transforms silent while the parallel processor is running. Another problem encountered was that certain objects in Job Transforms cannot be ``pickled'' (serialised in inter-process communications). Serialisation of objects for the purposes of communications is provided by the \href{http://docs.python.org/2/library/pickle.html}{\textcolor{black!100}{Python pickle module}}. Certain objects important to file validations cannot be pickled. Specifically, the legacy validation functions were not workable. The solution was to make new validation functions. While this extended the service task somewhat, it proved to be the most obviously reliable and validated approach -- and was successful.

\section{service task status}

The main activities in the service task were as follows:

\begin{itemize} %\small \item familiarisation with Job Transforms basic usage and code \item starting documentation for new Job Transforms \item general support (lots of coding tasks to support new development) \item systematic investigation of parallel processing methods in Python, principally methods associated with the Python multiprocessing module \item coded multiprocessing code library \item coded \href{https://svnweb.cern.ch/trac/atlasoff/browser/Tools/PyJobTransforms/trunk/python/trfUtils.py}{\textcolor{black!100}{parallel job processor}} \item coded \href{https://svnweb.cern.ch/trac/atlasoff/browser/Tools/PyJobTransforms/trunk/test/test_trfUtilsParallelJobProcessor.py}{\textcolor{black!100}{unit tests for parallel job processor}} (validation for parallel job processor) \item coded new validation functions and associated exception handling procedures \item high-level controls on how much of the CPU resources the parallel job processor may request \item validation of final unit tests (almost complete) \item paper \emph{ATLAS Job Transforms: A Data Driven Workflow Engine}~\cite{Stewart_1} \end{itemize} %\item[\XSolidBrush] bar %\item[] bar

\section{characterising impact}

Figures~\ref{figure:impact_1},~\ref{figure:impact_1} and \ref{figure:impact_1} depict tentative graphs of processing POOL files. Figure~\ref{figure:impact_1} illustrates the processing time against number of files validated in parallel while figure~\ref{figure:impact_2}illustrates the ratio of processing time to processing time for one file.

\begin{figure}[H] \begin{center} \includegraphics[width=\imageWidthASpecification]{figures/2014-04-10_1.eps} \end{center} \caption{parallel job processor: large efficiency improvement as a result of parallelisation} \label{figure:impact_1} \end{figure}

\begin{figure}[H] \begin{center} \includegraphics[width=\imageWidthASpecification]{figures/2014-04-10_2.eps} \end{center} \caption{parallel job processor: large efficiency improvement as a result of parallelisation} \label{figure:impact_2} \end{figure}

\noindent Figure~\ref{figure:impact_3} illustrates the performance for the unusual task of parallel validation of up to 40 POOL files.

\begin{figure}[H] \begin{center} \includegraphics[width=\imageWidthASpecification]{figures/2014-04-11_1.eps} \end{center} \caption{parallel job processor: large efficiency improvement as a result of parallelisation} \label{figure:impact_3} \end{figure}

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback