UKIDSS survey operational model
|WFCAM/UKIDSS SURVEY OPERATIONS
This document defines the UKIRT operational model
for WFCAM observing periods. Some information is included on how PATT,
UH and Japanese programmes are incorporated, but the main subject
is UKIDSS completion and monitoring.
Statements in italics identify software or
operational areas where assumptions may be being made which would
benefit from further discussion.
UKIDSS comprises five major
surveys, some of which themselves have significant subsurveys which are
well-separated on the sky. We employ the following definitions of the
various components of the UKIDSS survey.
- Survey - one entire UKIDSS survey, e.g.
- Project - one part of a survey
programme (for example, one DXS field or one RA slice of the LAS). Defined mainly for operational and scheduling convenience
described later, and roughly corresponding to a PATT programme in the
way it is treated by the database and supported by UKIRT scientists
Monitoring of progress is
done on three different timescales.
- day to day, including
adjustments made in the course of an observing night. The summit
observer is responsible for in-night adjustments (such as flagging
Observations and MSBs as bad); the UKIRT scheduler and support
astronomers are responsible for adjustments to the queue made on the
basis of the night report from the previous night.
- short-term, defined roughly as
time taken to obtain data, ship to CASU, receive back data quality
information (as required to implement repeats of given MSBs) and for
the data to arrive in the science archive. Of order three weeks to one
- long-term, defined as
or longer than the time for significant progress to be made on a given
survey (greater than one month).
OMP database structure
Surveys are divided into
projects cconsistent with a monitoring timescale
of order one WFCAM block. Each project is given an individual
"project ID" - for example, DXS sub-fields entitled
u/ukidss/dxs1, u/ukidss/dxs2 etc. Bigger surveys such as the LAS are
divided up into N-S strips, each of which is effectively a project
and which is also allocated a separate project
ID (u/ukidss/las1, 2 etc.). Uploading of projects into
the OMP database is done on a flexible timescale; all projects for
a 6-month period may be uploaded but the local survey manager may keep
most of them disabled until they become relevant. In part this is to
the OMP database size manageable and database query time to a
level; in part to allow for easy monitoring of completion on the
short-term timescale. Table 1 shows typical parameters of the Projects.
Table 1. Subsurveys and
examples of component "projects" in the database
Example component ID
Upload or Enable/Disable
One N-S Strip
two hours of RA range
One Dec/RA strip
monthly, as for LAS
One DXS field (or its
two-year plan subset), with a repeat count
One or two fields at a time
one or more clusters at a
time, RA depending
UDS, with a repeat count
whole survey always present
in the OMP database; repeat count used
Each survey group produces its MSBs in
advance and the uploads will be done from the UK, on agreed
timescales. Uploads will be the responsibility of the survey head or
The semester is divided into blocks, with
UKIDSS time interspersed with approximately one-week blocks of PATT
time. PATT time is typically staffed by one
programme's PI per block. In the event that no UKIDSS MSBs appear in a
given set of observing conditions during a UKIDSS run, the summit
observer fails over to PATT-sponsored programmes (and vice versa).
UH and Japanese time is similarly blocked, and
staffed by UH and Japanese observers. In these cases there is no
requirement to undertake either PATT or UKIDSS observing.
Selection of MSBs at the summit
The UKIRT Query Tool is used
to select MSBs to execute. Therefore no MSB is ever commenced in
inappropriate conditions (i.e. not matched to the MSB's embedded
requirements on photometric quality and seeing).
UKIDSS may specify lower seeing
limits for surveys if required, and at the summit it is also
possible to employ an hour-angle limit. Internal priorities are set by
the individual Survey Heads according to their needs (e.g. top-down
priorities in the case of a need for contiguity). Surveys operate in a
partly automatic manner with some input from the local survey manager
programme support astronomers. For example, given the
large number of MSBs in some of the surveys, occasional direct
intervention on the part of the survey manager is needed to instruct
summit observers to prefer some particular survey at a given time.
Such forcing is catered for in the Query Tool and OMP database.
The summit pipeline
continually produces indexed information on observing conditions,
including delivered seeing and photometric quality. This is used
to set MSB constraints in the query tool and thus effectively filter
the database for appropriate MSBs.
Quality control information and MSB repeats
Wherever possible, MSBs which are determined to have
produced substandard data are repeated within the WFCAM block in
which they were obtained. On the shorter timescales, repeat
requirements are determined in two ways:
(i) Observer feedback entries and summit-pipeline DQ
information are used to quickly schedule repeats as necessary. This is
possible on the day-to-day schedule and can be
accommodated entirely in-house, requiring no interaction with UKIDSS,
WFCAU or CASU. In this case the relevant OMP database feedback entries
and good/bad flags are set such that CASU will know in advance of
processing that the data are compromised and not to be propagated into
the science archive.
(ii) DQ feedback from CASU will be used to schedule
repeats of MSBs on the short-term timescale. In this case, it has been
agreed that there will need to be a feedback mechanism between CASU and
the UKIRT scheduler, which will probably need to be no more complex
than a weekly email unambiguously identifying the affected MSBs. The
affected MSBs will then be flagged as available once more.
This remains in planning. A unique MSB instance
identifier (the MSB transaction ID) is being developed. Similarly, we
will need a small utility
to reenable a range of MSBs using an ID list sent from CASU.
On the short-term timescale, overview of progress is
the responsibility of the UKIRT survey manager, with the Head of
Operations standing in in the event of absence or illness. Duties
include ensuring that the expected progress rate is maintained in each
of the surveys (and its components, if relevant). Particular
requirements and day-to-day monitoring of the data quality from
individual surveys are the responsibility of the associated
support scientist (see Annex 1). The main tools in this are queries
to the OMP database.
On the long-term timescale, the surveys are
monitored by UKIDSS survey heads and their delegates, using facilities
provided by the science
Various issues relating to observer execution of
UKIDSS projects are covered in the UKIDSS Observer Notes pages.
ANNEX 1: survey Support
Each survey is supported by a
primary and secondary UKIRT support scientist, provisionally as follows:
GPS - C Davis, M Rawlings
GCS - M Rawlings, W Varricatt
DXS - T Kerr, A Adamson
LAS - W Varricatt, C Davis
UDS - A Adamson, T Kerr
The primary support scientist has
the following responsibilities for their survey:
- Supporting UKIDSS in generating the MSBs
- MSB vetting (in as much as this is possible)
- Monitoring pipeline output and DQ parameters
the day after a night's observing. It is not
this item requires much more than checking that data were
taken in appropriate conditions (i.e. that the pipeline data quality
output was adhered to by the observer). In practice, the support
scientist for each UKIDSS run takes more of a role in this.
- Feedback to the PI (UKIDSS survey chair or their delegate)
- Detailed interaction with the UKIRT survey manager where necessary
- Training visiting UKIDSS observers
- Observing. With its current level of support, UKIRT accounts for
30% of the survey observing.
The secondary support scientist
stands in for the primary in the event of absence or illness, etc.
ANNEX 2: Communications between JAC and UKIDSS
All regular communication between
JAC and UKIDSS must be done through the OMP feedback system. For each
survey, UKIDSS nominates "P.I.s and Co-I.s" to whom each nights'
worth of feedback entries are automatically emailed. These include the
CSV, the Survey head, and the UKIDSS Survey
For information, the figure below
shows an example of a project feedback page - in this case, the Co-Is
are identified as not receiving every email the programme generates.
There are two support astronomers - primary and secondary - and this is
also the case for the UKIDSS surveys. For each UKIDSS survey,
there is one such feedback page.
Figure 1. Section
of a feedback page for a PATT project. Notes:
- In this example, only the PI
and support scientists receive project email (the PI can alter this at
- any number of Co-Is can be
- data can be "retrieved" via the
night links at the bottom of the page (though of course this cannot be
used for the WFCAM data rates, the "eavesdrop" link may grant access
to reduced-data "thumbnails" as there are for the other UKIRT
parameters are also plotted on the night
report and eavesdropping pages.
On the short-term timescale, plots are generated
showing observation completion of subsurvey components. These are
needed to make decisions about whether to carry over unobserved parts
of a subsurvey component into the next block, etc. This information
will be made available on a separate UKIDSS-related web page,
maintained by the UKIRT survey manager and Associate Director.
ANNEX 3: Communications between JAC and UK data centres
Data are shipped to CASU on Ultrium LTO-II tapes,
using a recognized express carrier. The data format has been separately
agreed between JAC and CASU.
The JAC makes available all quality control and
feedback information resulting from each night of observing to CASU.
CASU are free to specify the database fields they wish to copy.
CASU will feed back repeat observation requests
resulting from analysis of full nights and longer-term trends in the
form of a weekly email.
It is not expected that there will be any direct
feedback mechanism from the WFAU science archive and the Hawaii
operation; on the long-term timescale, UKIDSS will use the science
archive to make decisions about the global properties of the surveys
and their requirements.