TroubleshootingMC < NA62

NA62 Web>NA62MonteCarlo>TroubleshootingMC (2013-04-12, DanProtopopescu) (raw view)
---++ Troubleshooting MC

This page lists the most common problems encoutered with the NA62 MC code and the whole grid production setup, along with our solutions.

%TOC%

---+++ GGUS tickets

Some of the problems encountered are available as GGUS tickets:

   * 83133: CERN FTS channel problem - https://ggus.eu/ws/ticket_info.php?ticket=83133
   * 82327: NA62 VO not in the CIC portal - https://ggus.eu/ws/ticket_info.php?ticket=82327
   * 83159: Can't create directories on castor at RAL - https://ggus.eu/ws/ticket_info.php?ticket=83159 %Y%
   * 82901: Authz error when submitting na62 jobs to RAL - https://ggus.eu/ws/ticket_info.php?ticket=82901 %Y%
   * 81669: FTS channel for na62.vo.gridpp.ac.uk - https://ggus.eu/ws/ticket_info.php?ticket=81669 %Y%
   * 83768: Request for an FTS channel from LIV to RAL - https://ggus.eu/ws/ticket_info.php?ticket=83768 %Y%
   * 86349: Request for FTS channels from Birmingham and Liverpool to CERN - https://ggus.eu/ws/ticket_info.php?ticket=86349
   * 86351: Request for an FTS channel from RAL to CERN  - https://ggus.eu/ws/ticket_info.php?ticket=86351
   * 87068: Request to have NA62 in the VO drop down menu on the GGUS pages -https://ggus.eu/ws/ticket_info.php?ticket=87068 

---+++ Code compilation

Shared object portability issue. In =Generator/GNUmakefile=, define a new variable =CMCLIBNAME=, just above the existing =CMCLIB= definition:
<verbatim>
CMCLIBNAME := libcmc.so
CMCLIB := $(CMCDIR)/libcmc.so
</verbatim>
and then the target:
<verbatim>
Generator: $(CMCLIB)

$(CMCLIB): $(OBJF) $(OBJC) $(COMMONOBJ) $(OBJCC)
       $(FC) -shared -Wl,-soname,$(CMCLIBNAME) -o $(CMCLIB) $^
</verbatim>
This way the .so is linked w/o the full path, which ensures  portability of the executable. Same trick in =Beam/GNUmakefile=, but variable names =BEAMLIBNAME= and =BEAMLIB=.

---+++ Access permissions

Many errors came from read/write access permissions either for the local software area on sites, or for storage paths on various SEs. Using two certificates - one for jobs submission (and writing to the local storage) and the other for output replication to RAL and CERN complicates things somewhat.

---+++ Utilities

There are many utilities, mostly =bash= scripts for fixing various problems:
   * =recover_zombies.sh= - checks ZOMBIE jobs and updates the database if output is on CERN Castor (on svr020)
   * =cleanup.sh= - deletes old output files from local storage on sites (on svr020)
   * =verify-castor.sh= - checks if MC outputs are on CERN Castor and updates the DB (ppepc102)
   * =castor.sh= - produces  missing files lists for Janusz's controller
Topic revision: r9 - 2013-04-12 - DanProtopopescu
NA62