CustomReconstruction < DetDev

DetDev Web>EUTelescopeAndIBLTestbeam>CustomReconstruction (2014-07-09, KennyWraight) (raw view)

---+!! %MAKETEXT{"Customising IBL Reconstruction" args="<nop>%WEB%"}%

Notes on customising reconstruction process for ITk (formerly IBL) testbeam data using EUTelescope software. <b>NB</b> this is a perilous process and, so far, not perfected.

---++ %MAKETEXT{"Overview"}%

To make a custom reconstruction you will need appropriate =[[#GearfileAnchor][gear]]=, =[[#ConfigAnchor][config]]=, and =[[#SteeringAnchor][steering]]= files. If possible, ask the author or someone who might know where to get these files as it is much easier to edit working code than to invent your own. *PS* [[#GitAnchor][git]] is a useful cool repository tool.

The =gear= file defines the positions and orientation of the devices present: Mimosas and devices under test (DUTs). The =config= file sets the essential variable values which the reconstruction code will need to find and write files and run the code successfully. The main part of the of the reconstruction comes in calling the appropriate sub-routines via the =steering= files. 

---++ %MAKETEXT{"=config= file layout"}%
#ConfigAnchor

The =config= file is written in <b>python</b>. It lists all the variables set by the user for the reconstruction. Essential parameters are things like the file structure the reconstruction will run in, e.g. the directories to find input files and write output files to, and the location of the =gear= and =steering= files. These files must exist (i.e. be set correctly) or the program will not run.

In addition to essential parameters other variables can be set which will overwrite the defaults in the code. Alternately these could be set on the command line using the <code>-o VARIABLE=VALUE</code> option, but this is cumbersome for several variables or longwinded values. The variables for each =steering= file are listed in the =config= file after the global values have been set, each set is introduced in by the task name in square brackets. Here is an example of the converter section: 

<code>
# Section for the converter step
[converter]

# Hot Pixel detection
<nop>FiringFrequency         = 0.1
</code>

Here, the =FiringFrequency= variable used by the =converter= steering file is set. Other parameters tasks are similarly defined. 

---++ %MAKETEXT{"=gear= file layout"}%
#GearfileAnchor

The =gear= file is written in <b>xml</b>. It lists the geometry and orientation of the Mimosas use for track formation and the DUTs. The essential parameters of each include the position, rotation, size and pitch of each detector. =gear= files should be obtained with 
=raw= data as they must correspond to the devices used to record data in the testbeam.

To edit or amend a file be sure to set the parameters according to the following conventions (regrettably, these may change between testbeams):

<latex [attr="value"]* > eta \equiv \eta \equiv ZY</latex>

<latex [attr="value"]* > phi \equiv \phi \equiv XZ</latex>

<latex [attr="value"]* > theta \equiv \theta \equiv XY</latex>

Rotations are specified by the individual rotation parameters: =rotationZY=,	=rotationZX= and =rotationXY=; or, in the XY case the rotation matrix. The XY-rotataion matrix has the form <latex [attr="value"]* > 
\begin{bmatrix}
  Cos\theta & -Sin\theta \\
  Sin\theta & Cos\theta \\
 \end{bmatrix}
</latex>
so the entries are usually 1,0,-1 for 90 degree rotations. 

Here is some lined of code for a standard (250x50) DUT from the =anemone= example:

<code>
	<!--APIX-Plane 1 - FE-I4 DBM -->

					<layer>

					<ladder 	ID="20" 

							positionX="0.00"	positionY="0.00"	positionZ="334.0" 

							rotationZY="180.0"	rotationZX="0.0"	rotationXY="0.0" 

							sizeX="20.0"		sizeY="16.8"		thickness="0.700" 

							radLength="65.000000"

					/>

					<sensitive 	ID="20" 

							positionX="0.00"	positionY="0.00"	positionZ="334.0" 

	                                                sizeX="20.0"            sizeY="16.8"            thickness="0.500" 

                                                        npixelX="80"            npixelY="336"

                                                        pitchX="0.250"          pitchY="0.05"           resolution="0.1000" 

							rotation1="0.0" 	rotation2="-1.0"

							rotation3="1.0" 	rotation4="0.0" 

							radLength="65.000000"

	                     		/>

			        	</layer>
</code>

---++ %MAKETEXT{"=steering= files"}%
#SteeringAnchor

<b>Common Features</b>

Steering files for each run or set of runs are produced on execution from the template files (found in the =steering-templates= directory). The general templates become specific when the variables in specified in the config files are substituted into the templates. To check the of specific executions are correct use the =jobsub -dry-run= option on execution. This will copy the steering files used for each run to the current directory.

The =steering= files are also written in <b>xml</b>. There is a file for each stage of the reconstruction. The program which is being steered is called <b>Marlin</b> and it is executed by =jobsub=. Information from the =config= file, i.e. variable values, can be referred to in the steering file using =@VARIABLENAME@= syntax.

Each =steering= file has several sections in common. To begin there is an =execute= section where the list of all processes to be run is defined. There is also a =global= section where the global (e.g. =gear= file location) from the =config= file are passed. The rest of the file is taken up with setting the variables of each processor defined in the =execute= section. The task specific parameters (e.g. <nop>FiringFrequency) are passed to the processors in these sections.

<b>Converter</b> _Read DUT to into EUTelescope framework_ 

EUDaq processor is used to record testbeam data in organised fashion, however this organisation is not standard, rather it is specific to categories of device (e.g. Atlas Pixel, "APIX").  Converters can be found in =$EUTELESCOPE/external/eudaq/tags/v01-00-00/main/src/= (NB slight differences in directory structure between versions). For current (!) testbeam data =APIX-CT-ConverterPlugin.cc= is the converter used. Any manipulation of data as it enters the EUTelescope software framework should take place here. As EUDAq is separate from EUTelescope, i.e. the converter is not an EUTelescope processor, you cannot read in variables through the variables through the steering file.

<b>Clustering</b> _Manipulating DUT pixels_

Clustering collects pixels into clusters according to some constraints (such as minimum and maximum cluster size). It also contains a correlation processor which can be used to calculate the orientation of the DUTs with respect to one another. *NB* Correlations at the clustering stage are done on a pixel-to-pixel basis. 

__Using the ROOT file__

Consistency checks of detector geometry and orientation can be performed by inspecting the histograms produced by the clustering root file. Inside (assuming relevant processors have been used) are hit maps for the mimosas and DUTs (from clustering processors), as well a correlation maps (from correlation processor). 

The hitmap axes should correspond to the number of pixels and rows defined in the gear file. If there is a cosmetic problem (i.e. the histogram is wrongly binned) then change the gear file as this is referenced to set the histogram axes. If the gear file is correct but the histogram is not correctly filled then there has been a problem in the converter stage. To remedy the converter step must be rerun with the converter file appropriately changed (and recompiled) to read the correct number of row and column pixels.


This is an example of a hitmap with correct number of pixels for a mimosa: 1152x576 pixels.
%ATTACHURL%/anemone_clu_m26_good.gif

This is an example of a hitmap with correct number of pixels for a standard DUT (note, the X-edge pixels are masked). Standard devices have 80 columns (20cm/250um) and 336 rows (16.8cm/50um).
%ATTACHURL%/anemone_clu_DUT20_good.gif

This is an example of a hitmap with incorrect number of pixels for the same standard (250x50um**2) DUT. The gear file specifies too many rows and too few columns. 
%ATTACHURL%/anemone_clu_DUT20_bad.gif

The correlation plots compare pixels based on their X or Y position. If pixels are correctly orientated with respect to one another then the correlations should be positive, i.e. greatest intensity from bottom left to top right: in the case of comparing devices with the same number of pixels, correctly orientated devices will produce a correlation running up the central diagonal; any offset in the devices will displace the diagonal. Devices which are flipped with respect to one another will produce an anti-correlation, i.e. greatest intensity from bottom left to top right: similar comments apply for case when devices have the same number of pixels. Devices which are rotated will lose correlation, such that for 90 rotation case no correlation will be seen. Changes to orientation are made in the [[#GearfileAnchor][gear file]], either by the XY-rotation matrix or individual rotation parameters.

This is an example of an X-correlation map between planes 0 and 1 in the telescope. The correlation can be seen in the increasing in the positive direction as the diagonal line between corresponding pixels. Here it runs for x=y as planes are well aligned and orientated and devices have same number of pixels. *NB* changes in orientation will not be seen in =clustering= tep as this only deals with pixels not global orientation of devices. Changes in orientation will have affect in =hitmaker=.

%ATTACHURL%/anemone_clu_p0p1_Xcor_good.gif

This is an example of an X-correlation map between planes the above DUT and plane 0 in the telescope. The correlation here is positive but offset (as is typical), there is also a mismatch in axes bins due to the difference in detector geometries.

%ATTACHURL%/anemone_clu_DUT20p0_Xcor_good.gif

The correlation ROOT file also contains plots pertaining to the offset of devices. These are used to quantify the misalignment of devices in the code (_in normal circumstances_ no manual input of offset should be necessary). There are also projection plots of the maps which peak at offset values. The projections come from offset plots w.r.t. plane 0.

This is an example of an X-correlation shift plot (and corresponding projection) between planes the above planes 0 and 1 in the telescope. 

%ATTACHURL%/anemone_clu_p0p1_XcorShift_good.gif
%ATTACHURL%/anemone_clu_p0p1_XcorShiftProj_good.gif

This is the corresponding X-correlation shift plot (and corresponding projection) for the above DUT and plane 0 in the telescope.

%ATTACHURL%/anemone_clu_DUT20p0_XcorShift_good.gif
%ATTACHURL%/anemone_clu_DUT20p0_XcorShiftProj_good.gif

The correlation shift plots are used to identify the offset peaks. Cutting around these peaks is used by the =prealign= processor in =hitmaker= to improve track fitting. The peak parameters are used in the  =PreAligner= and =Correlator= processors in the =ResidualsXMax=, =ResidualsXMin=,  =ResidualsYMax= and =ResidualsYMin= vectors. 

<b>Hitmaker</b> _Changing to global telescope geometry_

Hitmaker moves the detector planes and DUTs into the global geometry of the telescope so that alignment can be preformed. It also contains a correlation processor which can be used to check the orientation of the DUTs with respect to one another. *NB* Correlations at the hitmaker stage are done on a pitch-to-pitch basis i.e. the pixels are translated into the global coordinate system of the telescope. 

__Using the ROOT file__

The =hitmaker= root file has similar hitmaps for each mimosa plane and DUT as the clustering output but with axes translated into the global telescope geometry. Changes to correlations due to gear file gemometry parameters will be seen in the correlations and hitmaps.

These are the local an global hitmaps corresponding to the above DUT.

%ATTACHURL%/anemone_hit_DUT20_local.gif
%ATTACHURL%/anemone_hit_DUT20_global.gif

These are correlation and shift plots corresponding to the above DUT. Note the cuts around the peaks.

%ATTACHURL%/anemone_hit_DUT20p0_Xcor.gif
%ATTACHURL%/anemone_hit_DUT20p0_XcorShift.gif
%ATTACHURL%/anemone_hit_DUT20p0_XcorShiftProj.gif

There are also plots from the pre-alignment which show the same results as the shift plots. Here are plots for the plane 1 and DUT20 offset with respect to plane 0.

%ATTACHURL%/anemone_hit_p0p1_XhitCor.gif
%ATTACHURL%/anemone_hit_p0DUT20_XhitCor.gif


<b>Align</b> _fixing relations between planes_

<nop>DafFitter output histograms can be used to diagnose problems using the resolution plots, which should be single peaked usually with some offset. Any offset in resolution is automatically used by the code for fitting purposes and should not be manually applied. After pre-alignment residuals should not shift more than the pitch size. Residuals should be centred very well at 0, with +-few tens of um for mimosas and up to +- couple 100 um for DUTs.

---++ %MAKETEXT{"=steering= files comments"}%
(Integrate this section later.)

A few variables are used several times throughout steering files and so can be defined for ease in the config file e.g. residuals. 

<b>Tips</b>

If you editing the =steering= files you must be careful that the correct referencing is done between the values set in the =config= file and the values used in the =steering= files. Problems are often generated when directories or values defined in the =config= file are not properly referenced or simply hard-coded in the =steering file=. The code will fail if files, directories or values mentioned in the =steering= files do not exist. Always check the error messages if running =jobsub= fails.

Be sure to replace hard-coded references with values set in =config= file, e.g.:

<code>
someplacewithgeomfile </code> 

change to 

<code>
@GeometryPath@/@GearFile@" </code>

where both variables =GeometryPath= and =GearFile= are set in =config.cfg=:

<code>
# Geometry file
<nop>GearFile    		= my_gear_file.xml 

# Path to the geometry file
<nop>GeometryPath		= %(<nop>BasePath)s
</code>

In general, make sure any definition of the number of planes is correct: #plane = #mimosas + #DUTs.

In the steering files where number of planes is defined (e.g. in <nop>DafFitter processor in  =align-tmp=, =fitter-tmp=) by =TelescopePlanes=, the set-up can be defined in two ways. Firstly all planes (mimosas & DUTs) can be defined in one _ordered_ vector (gear file is not used for z-ordering here). This uses all planes for the fitting. Alternatively, only mimosa planes can be defined and DUTs can be referenced in the =DutPlanes=. If this is done the =FitDuts= parameter mus be set =true=. Either way the =RequireNTelPlanes= parameter should be appropriately set (e.g. to =6= if the mimosa planes alone are used). 

<b>Notes</b> (integrate in text fully later)


Filter data in later version of the reconstruction software (i.e. >= v01-17-05) cleans up hot pixels/clusters before hitmaker steps which should aid pre-alignment. 


---++ %MAKETEXT{"EUTelescope analysis"}%
#AnalysisAnchor

The source code of the EUTelescope analysis can be found in the <code> ../vXX-YY-ZZ/Eutelescope/vxx-yy-zz/ </code> directory. For example the converters are found in <code> ../external/eudaq/tags/v01-00-00/main/src/ </code>.

When these are edited, the code must be recompiled in the <code> ../external/eudaq/tags/v01-00-00/main </code> directory (using <code> make </code>) and in the <code> ../vXX-YY-ZZ/Eutelescope/vxx-yy-zz/build/ </code> directory (using <code> make install </code> , NB the extra.potion this time).This way all the relevant executables are updated. 


---++ %MAKETEXT{"Git repository (very) basics"}% (just added sketch, needs proper home)
#GitAnchor

To update a git repository with your customised code follow these instructions:

assumed cloned version...
* set git directory (=git init=)
* clone remote repository (=git clone <url>=)

make changes...
* edit appropriate branch
* add changes to staging area (=git add=)
* commit changes to branch on local repository (=git commit ... -m "COMMENT"=)

*set ssh keys (use this link for easy recipe)
https://help.github.com/articles/generating-ssh-keys
* check urls are =git= *not* =https=
then...
* push to remote repository (=git push <remote_rep> <local_branch>=)

upload for later...
* fetch from remote repository (=git fetch <remote_rep>=)
* merge new changes (=git merge <remote_rep_branch>=)


-- Main.KennyWraight - 2013-06-11

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
gif	anemone_clu_DUT20_bad.gif	r1	manage	10.7 K	2014-06-17 - 11:10	KennyWraight	Form example with bad gear file
gif	anemone_clu_DUT20_good.gif	r1	manage	18.8 K	2014-06-17 - 10:25	KennyWraight	Form example with good gear file
gif	anemone_clu_DUT20p0_XcorShiftProj_good.gif	r1	manage	5.3 K	2014-06-17 - 10:25	KennyWraight	Form example with good gear file
gif	anemone_clu_DUT20p0_XcorShift_good.gif	r1	manage	8.1 K	2014-06-17 - 10:25	KennyWraight	Form example with good gear file
gif	anemone_clu_DUT20p0_Xcor_good.gif	r1	manage	12.9 K	2014-06-17 - 11:09	KennyWraight	Form example with good gear file
gif	anemone_clu_DUT20p0_YcorShiftProj_good.gif	r1	manage	7.8 K	2014-06-17 - 10:24	KennyWraight	Form example with good gear file
gif	anemone_clu_DUT20p0_YcorShift_good.gif	r1	manage	8.6 K	2014-06-17 - 10:25	KennyWraight	Form example with good gear file
gif	anemone_clu_m26p0_good.gif	r1	manage	28.7 K	2014-06-17 - 10:25	KennyWraight	Form example with good gear file
gif	anemone_clu_p0p1_XcorShiftProj_good.gif	r1	manage	4.2 K	2014-06-17 - 11:15	KennyWraight	Form example with bad gear file
gif	anemone_clu_p0p1_XcorShift_good.gif	r1	manage	13.6 K	2014-06-17 - 11:15	KennyWraight	Form example with bad gear file
gif	anemone_clu_p0p1_Xcor_good.gif	r1	manage	20.3 K	2014-06-17 - 11:09	KennyWraight	Form example with good gear file
gif	anemone_hit_DUT20_global.gif	r1	manage	10.6 K	2014-06-18 - 10:19	KennyWraight	Form example with good gear file
gif	anemone_hit_DUT20_local.gif	r1	manage	19.6 K	2014-06-18 - 10:19	KennyWraight	Form example with good gear file
gif	anemone_hit_DUT20p0_Xcor.gif	r1	manage	8.3 K	2014-06-18 - 10:19	KennyWraight	Form example with good gear file
gif	anemone_hit_DUT20p0_XcorShift.gif	r1	manage	5.6 K	2014-06-18 - 10:19	KennyWraight	Form example with good gear file
gif	anemone_hit_DUT20p0_XcorShiftProj.gif	r1	manage	4.1 K	2014-06-18 - 10:19	KennyWraight	Form example with good gear file
gif	anemone_hit_p0DUT20_XhitCor.gif	r1	manage	3.7 K	2014-06-18 - 10:19	KennyWraight	Form example with good gear file
gif	anemone_hit_p0p1_XhitCor.gif	r1	manage	3.5 K	2014-06-18 - 10:19	KennyWraight	Form example with good gear file

Topic revision: r14 - 2014-07-09 - KennyWraight

DetDev