See e.g., 20061107.002
a stuck countdown fault or if you have difficulties switching filters,
quickly check the main WFCAM DM screen. If there is an interlock fault
the interlock status will be red on this screen. You must
follow the procedures in the
first response to 20061107.002
in order to get going again. Interlock faults occur when the WFCAM CCS
loses the filter paddle position and forces you to park the filters
then redatum in order to refind the positions and prevent the paddles
stuck/star looks like a doughnut:
Typical symptom of hexapod
stuck fault is a guide star that looks like a doughnut due to being out
of focus. First, toggle the telescope
focus between 0 and 1 to see if the hexapod responds. If it does, try
refocusing, if it doesn't, slew to zenith, reset the hexapod and retry.
fault for this is 20051214.010
and please respond to this fault if you have an occurrence of this but
no time is lost.
Light leak in dark frames:
Since the filter paddles do not completely mask the
arrays when in the blank (dark) position, there is a light leak which
affects dark frames when the ambient light is bright. The DR should
report a light leak. To avoid this, the beginning of night dark
sequence should be started no earlier than five minutes before sunset.
(If the darks done early, the dome should be closed and the dome lamps should
be turned off).
This fault was originally misunderstood (20050619.001
but the current fault with more details and an image can be found
Sometimes this can be caused by the array running at a lower (if the heater fails) or higher temperature
than the set values. See
pops up error after WFS:
The sequence console pops up the error "Failed to set configuration
(phase 1)". This is due to the focus mechanism hitting the limit switch
because the WFS focus offsets were left in and WFCAM is now datumming
the filters. While the filters are finding datum, open the wfcamMain.dl
DM screen and put 0.0 into the focus offset field and wait for the
datum process to finish.
Arrays fail to
enable/blank first frame:
The controller for camera 2 had a fault which resulted in camera
2 not enabling on the power on command sent by the TSS, however this is
now believed fixed (see 20050403.001
If you do see any of the arrays blank after being enabled (powered on),
then disable the arrays, remove inst. (or ocs_down), run down the
low level software, power cycle the controller for the problem array
and restart everything. Take a quick frame using the read noise
or flush sequence to check that the camera is now enabled.
Read noise is
The DR will report high read noise, and if the
read noise is so high that it's clear there is an electronic problem,
will exit. First thing to try is a flush sequence and then another read
noise sequence. If the noise is lower but not nominal, repeat. If the
noise is still very high, run down the observing software, power cycle
the controllers (or the controller for the problem array) and restart.
current is high (especially camera 4):
The DR reports dark current although it
will not exit if the current is high. Note that camera 4's DR always
reports large dark currents for long exposures (typically in the
hundreds). These numbers can be ignored. It is due to a read-out issue
which puts a bias level into the array. This is steady and is removed
in dark subtraction. See 20061116.004
Channel edge problem used to
appear in two flavours: (1) "hard" channel edge problem-
The edges of the channels appear in one bright and a next dark
row or column, where the counts are higher and lower respectively
from the rest of the rows or columns in the channel by a few hundreds
or thousands of ADUs. The frame is totally useless when it happens.
This was seen mostly in camera3 and sometimes in camera 4;
(2)"weak" channel edge problem- a few rows or columns
at the end of a channel taken with NDR waveforms will be brighter than
the rest by a few or a few tens of ADUs. The number of bright rows or
columns will increase with increase in the NDR exposure time.
This was seen most pronounced in camera2,
but was present, but weak, in other cameras.
With the improvements implemented in late 2009 and early 2010,
both these problems are removed. If you see any of these happening,
please report it.
The canonical fault is 20070118.002
Occasionally the arrays do not disable at the end of the
night or at the end of daytime checks. The obvious symptoms of this are
1) the arrays remain powered on
in the sequence console and 2) the TSS does not get the powered off
response in the wfcamControl log (please do watch for this message when
disabling the arrays). If this problem occurs, you should remove inst.
run down the low level software, restart the software, add inst. enable
and then disable the arrays. Sometimes this is not enough, in which
case a full ocs_down is needed.
turning on/off the SDSU controllers remotely:
This is to turn off or power cycle a WFCAM SDSU controller remotely
and should be done only by a person trained in operating WFCAM
like a Support Astronomer, TSS or engineering personnel.
WFCAM remote power switch
on the wiki troubleshooting page for details.
While operating WFCAM remotely, if you find that the array is in
a state and you cannot revive it using the usual methods (like running
WFCAM including the low-level software down/up; rebooting drama/rtai
if these are stopped), it can be revived or at least disabled
using the remote power switch.
Rebotting an SDSU controller remotely should be done only as a
last step - like
(a) the usual ways of recovery do not work and
(b) there is nobody at the summit who can do it and nobody
will be going up very soon, unless the weather is deteriorating and
it is not good to leave the array/s powered.
If you end up power cycling the SDSU controller remotely, make sure
to file a fault report (or reply to an existing fault report about it).
with fewer than four cameras:
See the link: http://www.ukirt.hawaii.edu/instruments/wfcam/troubleshooting/running_with_broken_cameras.html
Temperature limits for safely operating WFCAM:
See the link: WFCAM safe operations
freezes at zero: "nonfatal":
The canonical fault is 20100713.001
For essentially all zero countdown faults you must remove inst and run
down the low level software (it is unlikely you will be able to disable
the array). When the software is run down, check the camera status. Do
a nuke if necessary (and there is no harm in doing one anyway when the
software is run down). If everything is OK, restart the low level
software and again check status. If everything is running, add inst,
enable the arrays and restart observations.For easier reference, follow
freezes at zero due to machine failure: message involving "integer":
In this case the stuck countdown is a symptom of a wfacq
machine having died. You will need to reboot whichever one is involved.
For rebooting a wfacq machine, first try soft reboot via kvm switch as given below:
kvm (this gives an error message that you can ignore)
connect to irtkvm as "admin" with the usual !password
connect to the machine in question (double click) [see the note regarding machine names below]
then log in as root and type '/sbin/shutdown -r now'
Note: Once you log in to the kvm, the wfacq machines are
listed by their 'wfunit_hex address' listed below.
where wfunit_ea, wfunit_0e, wfunit_e4 and wfunit_52 are on irtkvm
and wfunit_fc and wfunit_b8 are on irtkvm2.
It is the 'wfunit_hex address' that you need to select in kvm for rebooting.
These addresses will correspond to the 6 wfacq machines (4 in use and 2 standby).
To know what `wfunit_hex address' correspond to which wfacq machine, run
the command "wfacqmachinetable" from a terminal on ohi or kauwa.
You will get a table with 6 lines, one line for each wfacq machine.
On each line, there will be a string similar to "00:E0:81:31:FF:EA", where "EA"
at the end is the 'hex address'. The corresponding "wfacq" number also
will be listed in the same line. So, for that machine, the entry in the
kvm will be "wfunit_ea".
If a soft boot doesn't bring it back (if you don't get a response
on the kvm switch) try a hard reboot:
"connect wfunit_hexaddress power" (eg: connect wfunit_ea power), (give the admin's password when prompted for)
and reboot using the command "/boot wfunit_hexaddress".
After the reboot you must remount the raw data disk using
"/local/bin/remount wfcamN". Do this from your own account using your
own password and make sure the DR for the relevant camera has been
stopped. If the remount sticks, hit ctrl-c and it should
continue. To make sure, remount the disk a couple of times.
This will remount the raw data disks on all Linux computers except
lemi. If you need the raw data disks remounted on lemi, log into
lemi from an account that is NOT 'observer' and unmount and mount
the disks manually. This will require sudo privileges.
Acquisition fails to write data file:
Occasionally WFCAM fails to write a raw data frame. This
usually goes unnoticed since we are using the -skip option in the DR,
but the DR might complain about the mosaic. Unfortunately there's
little that can realistically be done about this at the telescope. See 20060429.002.
mounted/ompobslog can't see data:
If the DR does not see the raw data and/or
ompobslog doesn't see the data then the relevant raw data disk needs
remounting (if ompobslog doesn't see data, then the problem disk is on
WFACQ1). Stop the DR, and from your own account on any summit machine
run "/local/bin/remount wfcamN" where N is the machine number. Use your
own password when prompted. If the remount sticks, hit ctrl-c to
continue. Redo the remount if necessary and then restart the DR and /or
Failed to set
configuration (phase 1):
See Sequence console
pops up error after WFS and fault 20080215.001.
DR exits on first
Typically due to an array or all of them being disabled. If the
arrays were enabled earlier but one is blank, the relevant controller
has to be power cycled (see Arrays fail to
enable/blank first frame).
the arrays were not enabled, then stop the sequence, enable the arrays,
restart both the sequence and the DR.
See Read noise is high.
flush the array and redo the read noise sequence. If this doesn't help,
the controllers may need to be power cycled.
the read noise is unusually high, the DR will exit as it's telling you
there is a problem that needs fixing. If this is the case, it's usually
best to call for help.
is high (especially camera 4):
See Dark current is high
(especially camera 4).
just about all cases this can be ignored, especially if it's in camera
4 as the DR provides an incorrect dark current value in the hundreds.
in dark frames:
The DR will complain if it detects
light leak during initial darks (see Light leak in
). If this occurs, wait until 5 minutes before sunset
before starting the darks sequence.
The DR will report channel edge
problems but in general will not exit. There is little that can be done
at the moment (a software fix is being looked at) apart from marking
the bad frames in the night log. Restart the DR from the start of the
No suitable dark:
The DR will exit if it cannot
find a suitable dark for the current observation. This is to warn you
that the data you are taking cannot be calibrated. You must make sure
that a correct dark is taken, in fact it is recommended to take at
least 5 of them. They must have the same exposure time and number of
coadds. The DR will exit for the same reason if the darks sequence was
not taken at the start of the night or in daytime checks. (Note: At
present, the DR at UKIRT will not crash if darks do not have the same
number of coadds as that of the science frames. Hence special care
must be taken to have darks obtained with the same number of coadds as the
science frames. CASU will use these darks when they reduce the data).
DR can't see
the raw data:
See Disk not mounted/ompobslog can't see data
the DR just sits there producing dots and not reducing the raw data
that you know have been taken, then it is likely the raw data disk is
not mounted. Stop the DR, remount the disk using "/local/bin/remount
wfcamN" and your own password. If the remounting sticks, use ctrl-c to
make it continue. Run the remount script twice if necessary and then
restart the DR.