Show document only
Contact Info
UKIRT Staff wiki
Web Cameras

Survey home
WFCAM home
UKIDSS survey operational model


This document defines the UKIRT operational model for WFCAM observing periods. Some information is included on how PATT, UH and Japanese programmes are incorporated, but the main subject is UKIDSS completion and monitoring.

Statements in italics identify software or operational areas where assumptions may be being made which would benefit from further discussion.


UKIDSS comprises five major surveys, some of which themselves have significant subsurveys which are well-separated on the sky. We employ the following definitions of the various components of the UKIDSS survey.

  1. Survey - one entire UKIDSS survey, e.g. the LAS.
  2. Project - one part of a survey programme (for example, one DXS field or one RA slice of the LAS). Defined mainly for operational and scheduling convenience as described later, and roughly corresponding to a PATT programme in the way it is treated by the database and supported by UKIRT scientists


Monitoring of progress is done on three different timescales.

  1. day to day, including adjustments made in the course of an observing night. The summit observer is responsible for in-night adjustments (such as flagging Observations and MSBs as bad); the UKIRT scheduler and support astronomers are responsible for adjustments to the queue made on the basis of the night report from the previous night.
  2. short-term, defined roughly as the time taken to obtain data, ship to CASU, receive back data quality information (as required to implement repeats of given MSBs) and for the data to arrive in the science archive. Of order three weeks to one month.
  3. long-term, defined as equal to or longer than the time for significant progress to be made on a given survey (greater than one month).

OMP database structure

Surveys are divided into projects cconsistent with a monitoring timescale of order one WFCAM block. Each project is given an individual "project ID" - for example, DXS sub-fields entitled u/ukidss/dxs1, u/ukidss/dxs2 etc. Bigger surveys such as the LAS are divided up into N-S strips, each of which is effectively a project and which is also allocated a separate project ID (u/ukidss/las1, 2 etc.). Uploading of projects into the OMP database is done on a flexible timescale; all projects for a 6-month period may be uploaded but the local survey manager may keep most of them disabled until they become relevant. In part this is to keep the OMP database size manageable and database query time to a reasonable level; in part to allow for easy monitoring of completion on the short-term timescale. Table 1 shows typical parameters of the Projects.

Table 1. Subsurveys and examples of component "projects" in the database


Subsurvey component

Example component ID

Upload or Enable/Disable Frequency


One N-S Strip


monthly, approximately two hours of RA range


One Dec/RA strip


monthly, as for LAS


One DXS field (or its two-year plan subset), with a repeat count


One or two fields at a time


One cluster


one or more clusters at a time, RA depending


UDS, with a repeat count


whole survey always present in the OMP database; repeat count used

Each survey group produces its MSBs in advance and the uploads will be done from the UK, on agreed timescales. Uploads will be the responsibility of the survey head or their delegate.

Observing Blocks

The semester is divided into blocks, with UKIDSS time interspersed with approximately one-week blocks of PATT time. PATT time is typically staffed by one programme's PI per block. In the event that no UKIDSS MSBs appear in a given set of observing conditions during a UKIDSS run, the summit observer fails over to PATT-sponsored programmes (and vice versa).

UH and Japanese time is similarly blocked, and staffed by UH and Japanese observers. In these cases there is no requirement to undertake either PATT or UKIDSS observing.

Selection of MSBs at the summit

The UKIRT Query Tool is used to select MSBs to execute. Therefore no MSB is ever commenced in inappropriate conditions (i.e. not matched to the MSB's embedded requirements on photometric quality and seeing).

UKIDSS may specify lower seeing limits for surveys if required, and at the summit it is also possible to employ an hour-angle limit. Internal priorities are set by the individual Survey Heads according to their needs (e.g. top-down priorities in the case of a need for contiguity). Surveys operate in a partly manual, partly automatic manner with some input from the local survey manager and programme support astronomers. For example, given the large number of MSBs in some of the surveys, occasional direct intervention on the part of the survey manager is needed to instruct summit observers to prefer some particular survey at a given time. Such forcing is catered for in the Query Tool and OMP database.

The summit pipeline continually produces indexed information on observing conditions, including delivered seeing and photometric quality. This is used to set MSB constraints in the query tool and thus effectively filter the database for appropriate MSBs.

Quality control information and MSB repeats

Wherever possible, MSBs which are determined to have produced substandard data are repeated within the WFCAM block in which they were obtained. On the shorter timescales, repeat requirements are determined in two ways:

(i) Observer feedback entries and summit-pipeline DQ information are used to quickly schedule repeats as necessary. This is possible on the day-to-day schedule and can be accommodated entirely in-house, requiring no interaction with UKIDSS, WFCAU or CASU. In this case the relevant OMP database feedback entries and good/bad flags are set such that CASU will know in advance of processing that the data are compromised and not to be propagated into the science archive.

(ii) DQ feedback from CASU will be used to schedule repeats of MSBs on the short-term timescale. In this case, it has been agreed that there will need to be a feedback mechanism between CASU and the UKIRT scheduler, which will probably need to be no more complex than a weekly email unambiguously identifying the affected MSBs. The affected MSBs will then be flagged as available once more.

This remains in planning. A unique MSB instance identifier (the MSB transaction ID) is being developed. Similarly, we will need a small utility to reenable a range of MSBs using an ID list sent from CASU.

Survey monitoring

On the short-term timescale, overview of progress is the responsibility of the UKIRT survey manager, with the Head of Operations standing in in the event of absence or illness. Duties include ensuring that the expected progress rate is maintained in each of the surveys (and its components, if relevant). Particular requirements and day-to-day monitoring of the data quality from individual surveys are the responsibility of the associated support scientist (see Annex 1). The main tools in this are queries to the OMP database.

On the long-term timescale, the surveys are monitored by UKIDSS survey heads and their delegates, using facilities provided by the science archive. 

Observer instructions

Various issues relating to observer execution of UKIDSS projects are covered in the UKIDSS Observer Notes pages.

ANNEX 1: survey Support

Each survey is supported by a primary and secondary UKIRT support scientist, provisionally as follows:

GPS - C Davis, M Rawlings

GCS - M Rawlings, W Varricatt

DXS - T Kerr, A Adamson

LAS - W Varricatt, C Davis

UDS - A Adamson, T Kerr

The primary support scientist has the following responsibilities for their survey:

  1. Supporting UKIDSS in generating the MSBs
  2. MSB vetting (in as much as this is possible)
  3. Monitoring pipeline output and DQ parameters the day after a night's observing. It is not expected that this item requires much more than checking that data were taken in appropriate conditions (i.e. that the pipeline data quality output was adhered to by the observer). In practice, the support scientist for each UKIDSS run takes more of a role in this.
  4. Feedback to the PI (UKIDSS survey chair or their delegate)
  5. Detailed interaction with the UKIRT survey manager where necessary
  6. Training visiting UKIDSS observers
  7. Observing. With its current level of support, UKIRT accounts for some 30% of the survey observing.

The secondary support scientist stands in for the primary in the event of absence or illness, etc.


ANNEX 2: Communications between JAC and UKIDSS

All regular communication between JAC and UKIDSS must be done through the OMP feedback system. For each survey, UKIDSS nominates "P.I.s and Co-I.s" to whom each nights' worth of feedback entries are automatically emailed. These include the CSV, the Survey head, and the UKIDSS Survey Scientist. 

For information, the figure below shows an example of a project feedback page - in this case, the Co-Is are identified as not receiving every email the programme generates. There are two support astronomers - primary and secondary - and this is also the case for the UKIDSS surveys. For each UKIDSS survey, there is one such feedback page.

Figure 1. Section of a feedback page for a PATT project. Notes:

  1. In this example, only the PI and support scientists receive project email (the PI can alter this at any time);
  2. any number of Co-Is can be defined;
  3. data can be "retrieved" via the night links at the bottom of the page (though of course this cannot be used for the WFCAM data rates, the "eavesdrop" link may grant access to reduced-data "thumbnails" as there are for the other UKIRT instruments).

Nightly DQC parameters are also plotted on the night report and eavesdropping pages.

On the short-term timescale, plots are generated showing observation completion of subsurvey components. These are needed to make decisions about whether to carry over unobserved parts of a subsurvey component into the next block, etc. This information will be made available on a separate UKIDSS-related web page, maintained by the UKIRT survey manager and Associate Director.


ANNEX 3: Communications between JAC and UK data centres

Data are shipped to CASU on Ultrium LTO-II tapes, using a recognized express carrier. The data format has been separately agreed between JAC and CASU.

The JAC makes available all quality control and feedback information resulting from each night of observing to CASU. CASU are free to specify the database fields they wish to copy.

CASU will feed back repeat observation requests resulting from analysis of full nights and longer-term trends in the form of a weekly email.

It is not expected that there will be any direct feedback mechanism from the WFAU science archive and the Hawaii operation; on the long-term timescale, UKIDSS will use the science archive to make decisions about the global properties of the surveys and their requirements.

Contact: Watson P. Varricatt. Updated: Tue May 8 10:51:16 HST 2007

Return to top ^