Many errors came from read/write access permissions either for the local software area on sites, or for storage paths on various SEs. Using two certificates - one for jobs submission (and writing to the local storage) and the other for output replication to RAL and CERN complicates things somewhat.
\ No newline at end of file
Added:
> >
Utilities
There are many utilities, mostly bash scripts for fixing various problems:
recover_zombies.sh - checks ZOMBIE jobs and updates the database if output is on CERN Castor (on svr020)
cleanup.sh - deletes old output files from local storage on sites (on svr020)
verify-castor.sh - checks if MC outputs are on CERN Castor and updates the DB (ppepc102)
castor.sh - produces missing files lists for Janusz's controller