Default Pre-Processing Pipeline

The LOFAR Default Pre-Processing Pipeline (DPPP) software reads and writes radio-interferometric data in the form of Measurement Sets. It goes through visibilities in time order and contains standard operations like averaging, phase-shifting and flagging bad stations. Between the steps in a pipeline, the data is not written to disk, making this tool suitable for operations where I/O dominates.

As part of the ASTERICS/OBELICS project, gain calibration was implemented as a part of DPPP, using the StefCal algorithm. Furthermore, a move towards open standards for calibration tables was implemented, by adopting the HDF5 file format.

Specification of the data reduction is done by parameter sets (‘parsets’) which are used throughout the LOFAR system. An example data reduction could be to concatenate several subbands in frequency,  flag the data (using default parameters), average the data, calibrate it, and average some more. Intermediate data products are stored in this example, if this is not necessary the steps can be skipped by leaving them out of the ‘steps’ parameter.

msin  = [band1.MS, band2.MS] # Bands will be concatenated in frequency

steps = [aoflag, avg1, out1, cal, out2, avg2]

aoflag.type       = aoflagger

avg1.type         = average

avg1.timestep     = 4        # Average every 4 timeslots

out1.type         = out

out1.name         = allbands_avg_t4.MS # Save the averaged file

cal.type          = calibrate

cal.skymodel      = 3C196.skymodel

cal.applysolution   = true

cal.parmdb        = instrument.parmdb

out2.type         = out

out2.name         = allbands_avg_t4.MS # Add calibrated data to allbands_avg4_t4

out2.datacolumn     = CORRECTED_DATA # Save calibrated data in another column

avg2.type         = average

avg2.timestep     = 2 # average down some more

avg2.freqstep     = 16

msout             = allbands_avg_t8_f16.MS # Save final averaged data

The DPPP software is distributed as part of the LOFAR software suite, under the GPL v3 license. The source code can be obtained from https://svn.astron.nl/LOFAR/

Documentation for DPPP is found at https://www.astron.nl/lofarwiki/doku.php?id=public:user_software:documentation:ndppp