Show document only
Contact Info
UKIRT Staff wiki
Web Cameras

Survey Observing
Other Observing
Observing Tool Manual
Applying For Time
Get non-UKIDSS Data
Data Reduction
Acknowledge Us
Technical Reference
UKIRT Publications
Graphical Weather Server
EAO Safety Manual
UKIRT Waiver Form
TSS Priority Page
CGS4 data files under ORAC

CGS4 data files under ORAC

Raw files in $ORAC_DATA_IN

The CGS4 raw data files are now stored as starlink HDS containers. Each file is equivalent to 1 observation, and as such contains a header component and 1 or more (actually NINT) integrations. Each integration is stored as an NDF component of the HDS file.

The filenames are thus: cYYYYMMDD_NNNNN.sdf, where YYYYMMDD is the numeric UT date (year, month and day) and NNNNN is the observation number, padded with leading zeros when necessary.

Reduced single frame files in $ORAC_DATA_OUT

First, some conventions:

The filename structure is: (PREFIX)(UTDATE)_(FRAME NUMBER)[_(EXTENSION)].sdf, where (THIS) is allways there and [THIS] is optional.

(PREFIX) is the letter 'c' if the file contains data from a single observation. It is 'gc' if the file contains data from a number of observations - ie a group.

_(EXTENSION) is used by individual primitives (think of a primitive as a single step within a recipe) for their output files. The pipeline its self keeps track of passing these files between primitives, though all the useful ones are left on the disk at the end so you can look at intermediate data products if you wish.

For example, c20000410_00123_ff.sdf would be data from a single observation, number 123, that has been flat fielded (the table later tells you the _ff means flat fielded).

So is that and HDS or an NDF?

Well, often, it depends; the pipeline passes files between primitives as either HDS or NDF, depending on which is most appropriate. For example, with 2x2 sampled data, the raw file will be an HDS container containg the 4 NDF data arrays from the 4 sample positions. If we're using a 1x1 sampled flat field, the flat field primitive will flat field each component, and write out an HDS container containg the four flat fielded data arrays as NDFs. On the otherhand, if the flat field is taken with the same sampling as the data, the pipeline will interleave the samples of both the data and the flat field before carrying out the flatfielding, thus the _ff file would be a single NDF.

Of course, some primitives will only write out single-NDF files; for example, the primitive that interleaves and/or coadds the samples into one larger image writes out _inc files, and these are allways single component NDFs.

An additional factor is that later on in the processing, the pipeline may go back to using HDS containers. This happens for example when we start extracting beams and spectra from the reduced group image - HDS containers are used, containing information for each beam - for example opt-extract profiles, and the spectra themselves, whilst they pass through the primitives that operate on the individual extracted spectra before these are combined to give the final spectrum - for example de-rippling.

File Extensions for single observation files
Extension HDS or NDF Description
_mraw either A Modifiable copy of the raw data.
_bp either Bad Pixel Mask has been applied
_rnv either Read Noise Variance added
_sbf either Subtracted Bias Frame
_acb either Added Chop Beams (used for Flats)
_scb either Subtracted Chop Beams
_pov either Includes Poisson Variance
_bgl either How BackGround Limited the integration is
_ipm either Interleave Prepared and Masked
_inc NDF Interleaved and Coadded
_ff either Data has had flat field applied
_nf either Data is a normallised flat field
_wce NDF Wavelength Calibrated by estimation. This is the equivalent of the old ro* file
_ss NDF Sky Subtracted

This figure illustrates the data flow between primitives for the reduction of a typical single frame.

Reduced group files in $ORAC_DATA_OUT

The pipeline adds the individual frames into the group as they are processed. The group number is usually the frame number of the first frame in the group. Group files are allways single NDFs.

File Extensions for group files
Extension HDS or NDF Description
No extension NDF This is simply the difference between all the main and offset beam images. These are the equivalents of the old rg* files.
_oep HDS The opt-extract profiles
_oer HDS The opt-extract profiling residuals
_oes HDS The opt-extracted spectra
_rif HDS The de-rippling flat fields
_dri HDS The de-rippled spectra
_ccs HDS Cross-Correlated and Shifted. All the beams are spectrally aligned to beam 1
_ccf HDS The Cross-Correlation Functions from forming the _ccs frames
_sp NDF Extracted Spectrum - the coaddition of all the beams
_aws NDF Aligned with Standard. Spectum is specrally aligned with the standard star
_scf NDF Standard cross-correlation function. The CCF from forming the _aws frames.
_dbs NDF Data has been divided by a standard star (including the standard star black body model).
_fc NDF Flux calibrated.

Contact: Tom Kerr. Updated: Wed Oct 6 11:54:13 HST 2004

Return to top ^