wfcam data handling
Raw Data Flow
The raw data generated by each of the 4 SDSU controllers is
transmitted over a dedicated fibre optic link to a PCI card in a data
acquisition Computer. This data acquisition computer runs a ukirt Data
Handling System (DHS) and records the data along with various header
data collected from various sources in a standard UKIRT NDF HDS
container. These files are written to a file system located on a RAID5
volume local to the aquisition computer. Thus, there are 4 seperate
raw data disks for WFCAM, each located on a seperate aquisition
Whilst observing is taking place, 4 other computers dedicated to
data reduction read the raw data from the aquisition machines via NFS
and reduce it using oracdr running the so called "summit pipeline" to
produce various data quality parameters along with reduced images and
In the morning following a night of WFCAM observing, the raw
data are written to LTO-1 tapes on drives located in the acquisition
computers. Thus, 4 seperate data tapes, one for each camera, are
written in parralell.
In addition, the 4 Data Reduction computers also write copies of
the raw data (now transformed into compressed FITS files) to LTO
tapes. These sets of tapes are mailed to CASU in the UK where the data
are processed and ingested into the WFCAM science archive.
Note. The LTO tape systems in use operate most efficiently with
a 64k blocksize. In Linux, this is achieved by running 'mt setblk 0'
to configure the drive for variable blocksize, then using 'tar -b128
...' to specify to tar that it should generate 64k (128 times
512-bytes) blocks. It is necessary to use the same parameters when
reading the tapes.
The WFCAM raw data is pipeline processed at UKIRT during
observing. This gives the observing team near-real-time feedback on
the quality of the data they are aquiring and allows them to assess
whether the current weather and conditions are appropriate for the
observations they are executing or planning to execute. To do this,
the summit pipeline carries out a reasonably complete data reduction
sequence, including flat fielding, microstep processing and mosaicing,
followed by object detection and catalogue creating, followed by
astrometric and photometric calibration.
In addition, the raw WFCAM data shipped to CASU in the UK is
processed by an offline pipeline to generate reduced data for ingest
into the WFCAM science archive. The offline pipeline has the advantage
that it does not have to deal with data in the order it was observed,
for example, calibration frames such as darks or flats can be built up
from data spanning the entire night, or even several nights.
It is expected that data will be dispatched to CASU within
approximately 7 days of being taken. Allowing a further 7 days for
transit and a further 7 days for processing / ingest, data should be
at the science archive within 3 weeks of being observed.
to WFCAM data
In general, wfcam users (including UKIDSS, PATT, UH, Japanese,
etc) will obtain their WFCAM data from the WFCAM science archive in
the UK. This will be reduced data processed by the offline (UK)
pipeline. In order to gain access to the archive, WFCAM PIs must
register with the WSA.
It is not expected that the majority of users will require
access to raw data. Those that do may request it from CASU.
Reduced data products from the summit pipeline will be
availiable to observers at UKIRT. This includes calibrated catalogues
and mosaiced images. In general, it will be possible for observers to
take away small amounts of this data (eg catalogs and a few mosaics)
if they wish. Likewise, small amounts can be supplied to remote
PIs. Both of these are on a best efforts basis, and will probably be
limited to a few gigabytes in practise.
Some projects may have a valid case for expedited access to bulk
data, where the ~6 week delay between observations being taken and data
becoming availiable in the science archive presents a serious
compromise to the sciencetific potential of the project. PIs who
believe that this applies to their project should raise the issue with
thier support scientist well in advance.
Japanese and University of Hawaii WFCAM runs
In addition, it is realised that retrieving bulk raw data from
the UK may be difficult and slow for people located a long way from
the UK (eg Hawaii). Strictly on a best
efforts basis, we may be able
to write data to either LTO-1 or LTO-2 tape or USB-2 interfaced hard
JAC. These tapes or disks must be supplied by the observer. If you wish
to take advantage of this, you must contact your
support scientist well in advance of your data being taken, typically
2-3 weeks before the run. Last minute requests will likely be
general, there will be a delay of 1-2 weeks after your data are taken
before the data are availiable on your tapes or disks - you will not
be able to take the data away at the end of your run.
Access to raw and/or reduced data at the summit
Raw and reduced data can be accessed directly at the summit. The
easiest way is to bring your laptop and transfer the data using the
internal network. Once your laptop is connected (using a cable, not
wireless) you will be able to use ssh and sftp to the computer called
Another possibility is to bring an external disk. There is only
one computer where you are allowed to plug an external disk: it's the
weather monitor computer, currently located under the desk in the
control room. Please ask your support astronomer or the TSS for
instructions. We do NOT allow external disks to
be plugged anywhere else.