(r15) ProductionRota < NA62

NA62 Web>ProductionRota (revision 15) (raw view)~~EditAttach~~
---++ NA62 Monte Carlo Grid Production Shifts

This wiki contains information about and for the volunteers for the NA62 MC Grid production rota. Check this page regularly for updates.

%TOC%

---+++ Today's production plan 

We run decay type 1, Kch2pipi0 simulations with software version v7/r193 at IC, RAL and BIR. Revision 193 is 5 times faster so we will run 6000 events runs. More sites will be included in the production, as soon as the software is installed on them. Check [[https://ppes8.physics.gla.ac.uk/twiki/bin/view/NA62/NA62MonteCarlo#Site_resources][here]] for current status.

Production jobs from the Kch2pipi0-1 production round have decay type 1, description set to "Kch2pipi0-1 production round", 1500 events for v6 runs and 6000 events for v7 runs. Please pay attention when submitting jobs.

When the [[http://na62.gla.ac.uk/index.php?task=production&tag=Kch2pipi0-1][Kch2pipi0-1 production round]] is done, we will switch to the next round, which is Kch2munu-1 (decay type 30).

%X% This wiki section will be updated often, so please check carefully before your shift. If you find any discrepancies, please notify Dan and Janusz immediately. Read the logbook/blog for more up-to-date information: http://na62shifts.wordpress.com


---+++ Shifts sign up

As of October 15, 2012 we have the following names:  Antonio Cassese, Mark Slater, Philip Rubin, Paolo Massarotti, Monica Pepe, Spasimir Balev, Vito Palladino, Mario Vormstein , Karim Massri.

We have a Doodle poll entitled "NA62 Grid Production Shifts", where volunteers can  sign up for shifts. If you've volunteered here for shifts, please check your email for the poll address. 

---+++ Production plan

The production plan is located here:

   http://na62.gla.ac.uk/index.php?task=production

This displays a database-generated table that fills up along the way. New items will be added by the production coordinator as discussed at the Siena meeting in August 2012.


---+++ Grid Production Shifts

If all goes well, GP "shifts" will not require too much work. Shifts are defined as 9am to 9pm CERN time but it is enough if the person on shift checks the production status and submits jobs twice a day or so. Is problems arise, this might require another hour or so of your time, since you will not be expected to actually fix the problem but only to notify the admins.

---+++ What to do on shift

MC jobs should be submitted manually. The person overseeing the production (the "shift" taker) will have to check periodically if everything goes according to plan, and 

   * intervene if possible to fix the error(s)
   * notify the production coordinator if not possible to directly intervene to rectify the error
   * notify the [[NA62MonteCarlo][site admin]] if jobs systematically fail at a given site.  *Check the jobs history to detect any problems at sites (e.g. jobs consistently finishing early, or going to status CLEARED without registering any output)*. %X% If jobs fail at a site, exclude it from job submissions and notify the <a target='new' href='https://ppes8.physics.gla.ac.uk/twiki/bin/view/NA62/NA62MonteCarlo#Site_resources'>site admin</a>! 
   * notify Janusz if outputs fail to be replicated on CERN Castor (labelled LS instead of CC in the <a href="http://na62.gla.ac.uk/index.php?task=files">files table</a>)

---+++ Checklist

The person in charge must do the following:

   1. check current production status from the wiki and from our [[http://na62shifts.wordpress.com][logbook]]
   1. keep an eye on the [[http://na62.gla.ac.uk/index.php?task=jobs][jobs list]] and make sure we have *200 jobs RUNNING at all times and not more than 50 SCHEDULED.* %X% Please note that these numbers will change when new resources are added. Check this number at the beginning of each shift.
   1. if queue is low (less than 100 RUNNING), use online [[ProductionHowto][script creator]] to submit a new batch of jobs
   1. check [[http://na62.gla.ac.uk/index.php?task=production][production page]] to see if we've reached 100%. If yes, then start next production round on the list.
   1. re-validate results weekly (every Monday) with the physics group (email Tonino to check the outputs on CAstor)
   1. notify production coordinator if any major errors occur
   1. log everything in our logbook/blog: http://na62shifts.wordpress.com

---+++ Troubleshooting

In case you find an error produced by the online interface, please immediately notify Dan, Janusz and Tonino.
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	NA62_requirements_table.pdf	r1	manage	38.1 K	2012-09-21 - 13:51	DanProtopopescu	Production plan (August 2012)
Topic revision: r15 - 2012-10-23 - DanProtopopescu
NA62