Troubleshooting MC

This page lists the most common problems encoutered with the NA62 MC code and the whole grid production setup, along with our solutions.

GGUS tickets

Some of the problems encountered are available as GGUS tickets:

• 83133: CERN FTS channel problem - https://ggus.eu/ws/ticket_info.php?ticket=83133
• 82327: NA62 VO not in the CIC portal - https://ggus.eu/ws/ticket_info.php?ticket=82327
• 83159: Can't create directories on castor at RAL - https://ggus.eu/ws/ticket_info.php?ticket=83159
• 82901: Authz error when submitting na62 jobs to RAL - https://ggus.eu/ws/ticket_info.php?ticket=82901
• 81669: FTS channel for na62.vo.gridpp.ac.uk - https://ggus.eu/ws/ticket_info.php?ticket=81669
• 83768: Request for an FTS channel from LIV to RAL - https://ggus.eu/ws/ticket_info.php?ticket=83768
• 86349: Request for FTS channels from Birmingham and Liverpool to CERN - https://ggus.eu/ws/ticket_info.php?ticket=86349
• 86351: Request for an FTS channel from RAL to CERN - https://ggus.eu/ws/ticket_info.php?ticket=86351
• 87068: Request to have NA62 in the VO drop down menu on the GGUS pages -https://ggus.eu/ws/ticket_info.php?ticket=87068

Code compilation

Shared object portability issue. In Generator/GNUmakefile, define a new variable CMCLIBNAME, just above the existing CMCLIB definition:

CMCLIBNAME := libcmc.so
CMCLIB := $(CMCDIR)/libcmc.so

Generator: $(CMCLIB)

$(CMCLIB): $(OBJF) $(OBJC) $(COMMONOBJ) $(OBJCC)
       $(FC) -shared -Wl,-soname,$(CMCLIBNAME) -o $(CMCLIB) $^

Access permissions

Many errors came from read/write access permissions either for the local software area on sites, or for storage paths on various SEs. Using two certificates - one for jobs submission (and writing to the local storage) and the other for output replication to RAL and CERN complicates things somewhat.

Utilities

There are many utilities, mostly bash scripts for fixing various problems:

• recover_zombies.sh - checks ZOMBIE jobs and updates the database if output is on CERN Castor (on svr020)
• cleanup.sh - deletes old output files from local storage on sites (on svr020)
• verify-castor.sh - checks if MC outputs are on CERN Castor and updates the DB (ppepc102)
• castor.sh - produces missing files lists for Janusz's controller