ULTRACAM user manual

Vik Dhillon, version: 3 December 2010

1. Introduction
2. Powering up
3. Powering down
4. Afternoon activities
5. Software startup
6. Looking at data
7. Logging of data
8. Archiving data
9. Drift mode
10. Changing filters
11. Before you go to bed
12. Troubleshooting
13. Contacts

• ULTRACAM MNRAS paper. This paper summarises the design and performance of ULTRACAM.
• CCD application document. This document describes the various CCD controller applications that can be run with ULTRACAM in more detail than the above paper. You should regard the applications document, which is written by David Atkinson (UKATC), as the primary source of reference for the different readout modes and their associated parameters.
• ULTRACAM system maintenance document. This document describes ULTRACAM's camera head handling and system maintenance procedures. Written by Dave Atkinson (UKATC), it should only be of relevance to project personnel.
• ULTRACAM pipeline software manual. This document describes how to use Tom Marsh's pipeline data reduction software, which is the best way of viewing the data whilst observing.
• UCam User Manual. This is Stewart McLay's user manual for the UKATC UCam data acquisition system, which is used by ULTRACAM. Useful information can also be found in the corresponding UCam Installation Manual and the UCam Configuration File Reference Manual.
• GPS170PEX User Manual. This document contains the Technical Information and Operating Instructions for the Meinberg GPS170PEX, the new ULTRACAM GPS system installed in March 2010.
• ULTRACAM logistics page. Details the status of ULTRACAM computers, system backups and data archiving.
• Telescope operators guide. This document describes how to use ULTRACAM with the WHT Telescope Control System and is intended primarily for telescope operators at the ING.
• Paul Kerry's user guide to the ULTRACAM computers. This can be found in /home/star/ultracam_userguide/ultracam_user_guide.pdf on the data reduction PC and a paper copy resides in the blue ULTRACAM folder in the control room. I don't provide a link to it here as it contains all of the system passwords.
• The ULTRACAM google group.

1. In the control room, check that the ULTRACAM network switch housed in the blue NETGEAR box near the data reduction PC is powered on.
2. Turn on the data reduction PC, housed in the aluminium case, by flicking the switch at the rear of the case and then pressing the round button on the front panel.
3. In the dome, check that the green light on the mains socket strip at the top of the rear of the electronics rack is illuminated - this means that the rack is connected to the mains.
4. Flick the red switch at the rear of the remote power module (RPM), housed in a black box on the second shelf down in the electronics rack. This will cause the rack PC, located in the bottom of the electronics rack and on which the ULTRACAM data acquisition system runs, to automatically boot up. The imedia PC, located just below the second shelf in the rack and which is used to monitor the peltier temperatures and control the focal-plane slide, will also boot up automatically. Note that it is very easy to dislodge the power cables going into the rear of the RPM when turning it on, so ensure that they are securely pushed in.
5. If you want to see what is happening on the PC whilst it is booting, use the sliding LCD monitor and keyboard at the top of the rack; turn the silver knob clockwise to release the lid, slide out (it is stiff!) and press the top-most button on the left-hand side of the monitor to power it on. To switch between the rack and imedia PCs, click the white button mounted on the left of the second shelf in the rack PC. Remember to turn the monitor off again when finished. To stow away, release the two catches at the front left and right-hand sides simultaneously and push in.
6. Turn on the SDSU controller by flicking the switch "SDSU POWER SUPPLY" from 0 to 1. The switch can be found approximately half way up the rear of the rack near the centre of a white panel. If successful, two green LED's should illuminate on the SDSU controller and the red LED on the SDSU-PCI card in the rack PC should come on. (Need to check this). If nothing happens, make sure the switch on the mains input connector to the Peltier/SDSU power supply unit is also switched from 0 to 1. If this doesn't work, check the fuses (see the Troubleshooting section for details).
7. Don't flick any of the switches described below until you have read the whole of items 8 to 11. It is possible to damage both the peltiers (if there is no water flow) and the chips (if they are cooled too quickly) if the following procedure is not followed.
8. Make sure that the power to the peltier temperature controllers is switched off. The peltier temperature controllers are located in the two aluminium-fronted units of the electronics rack. To turn off the power to peltier 3, flick the switch marked "PELTIER 3" at the rear of the rack from 1 to 0. To turn off the power to peltiers 1 and 2, locate the label "PELTIER 2" at the rear of the rack and then flick the switch to the right of this (next to where the mains cable plugs in) from 1 to 0.
9. Turn on the water chiller by flicking the switch at the top of the rear of the unit and then pressing the right-hand button on the front panel with the "I/O" on it. The chiller will spring into life and you should immediately check both that the peltier temperature displays at the front of the rack remain unilluminated and that none of the pipe connections on the chiller, SDSU controller, flow sensor (attached to the underside of the mid-plate of ULTRACAM) are leaking. If you discover a leak, or if the peltier temperature displays illuminate, immediately power the water chiller off.
10. The current temperature of the water is displayed on the front panel of the chiller and it will begin to drop as the chiller starts cooling the water down to the set temperature. The set temperature should be 10 degrees, unless this is within approximately 5 degrees of the dew point (in which case the water temperature should be increased). The dew point can be determined from the observatory's meteorological system. A useful check on this is to run your hand along the base of the SDSU controller - if it feels even slightly clammy to the touch, increase the set temperature. To change the set temperature, press the left-hand button with the penannular arrow on it and then alter the set temperature with the up and down arrow buttons. Press the button with the penannular arrow on it again to complete the setting process.
11. Turn on the peltier temperature controllers, which are located in the top two units of the electronics rack. To turn on the power to peltier 3, flick the switch marked "PELTIER 3" at the rear of the rack from 0 to 1. To turn on the power to peltiers 1 and 2, locate the label "PELTIER 2" at the rear of the rack and then flick the switch to the right of this (next to where the mains cable plugs in) from 0 to 1. The peltier displays on the front of the rack will now illuminate. If nothing happens, make sure the switch on the mains input connector to the Peltier power supply units are also switched from 0 to 1.

    There is one display for each CCD head. The upper figure on the display shows the current CCD temperature and the lower figure shows the set temperature. The temperature controllers will immediately begin cooling the chips to the set temperature by running the peltiers at their maximum power of -55%. However, it is possible to damage the peltiers by allowing them to cool by more than 5 degrees per minute, so it is essential to immediately increase the set point to 5 degrees below the current chip temperatures using the up and down arrow buttons to the right of the temperature displays. Once attained, drop the set temperatures by approximately 5 degrees per minute until the desired temperature of -40 degrees is reached. Once the chip temperatures have stabilised at -40 degrees, check the peltier power levels by pressing the left-hand turquoise buttons on the temperature displays once. The power level should be approximately -30%. If the value is close to or equal to the maximum value of -55% then the heads may have lost their vacuum and need to be pumped down. Alternatively, the water chiller might be running at too high a temperature. You can return to the temperature display by pressing the black button in the middle.

1. In the dome, turn off the SDSU controller.
2. Turn off the water chiller. This will cut the water flow and hence will automatically turn the peltiers off (but check the peltier displays to make sure). With all cooling turned off, the chips will gradually warm up to ambient temperature at a safe rate.
3. Turn off the peltier temperature controllers.
4. Shut down the rack PC by typing "init 0" in an xterminal connected to the PC or by using the sliding monitor and keyboard in the electronics rack. This does not turn the PC power off, so when the LCD monitor says the system is halted you can press the power button on the front-panel of the rack PC. (Check).
5. Turn off power to the entire ULTRACAM electronics rack by switching it off at the mains.
6. Turn off the feed to the nitrogen gas flushing system (see Afternoon activities).
7. In the control room, shut down the data reduction PC by selecting the Shut Down... option from the System pull-down menu on the top bar. The PC will power down automatically on completion.
8. Turn off the network switch, data reduction PC speakers, monitor and any attached USB drives.

1. Check water temperature on the chiller display is set to 10 degrees unless this is within approximately 5 degrees of the dew point (in which case the water temperature should be increased). The dew point can be determined from the observatory's meteorological system. A useful check on this is to run your hand along the base of the SDSU controller. If it feels even slightly clammy to the touch, increase the water temperature slightly.
2. Check CCD temperatures on the rack-mounted temperature controllers are stable at -40 degrees.
3. Check peltier power usage on the rack-mounted temperature controllers. These should typically be around -30%. If the value is close to or equal to the maximum value of -55% then there is either a problem with the vacuum in the head or the CCD temperatures have been set too low (or the water temperature too high).
4. Check for water leaks by running your fingers along the pipe connectors on the water chiller, the SDSU controller, the flow sensor (attached to the underside of the mid-plate of ULTRACAM) and the individual CCD heads.
5. If the humidity in the dome is likely to be high that night, check that the nitrogen gas flushing system is operable by ensuring that the gauge feeding ULTRACAM is set to a flow rate of appoximately 0.5 litres/minute, which you can feel as a very gentle breeze if you hold the end of the pipe to your lips. Every once in a while, it is also worth checking that the blue pipework and syringe needles on ULTRACAM are connected, and that the small brass valves on ULTRACAM are fully open.
6. Ensure that the pipes and cables leading to the instrument via the twister arrangement are not overly taut, are neatly arranged, show no signs of damage and are in no danger of snagging on anything.
7. Check that the SDSU controller is powered on by removing one of the black velcro strips covering the status window and looking for lit green LEDs. If none are present, power on the SDSU (see Powering up).
8. Visually inspect the focal-plane mask and the brushes it runs through.
9. Check to see if you need to make a filter change for the start of the night. If you do, refer to Changing filters.
10. Ensure all lights are turned off in the dome.

In the control room

1. Check that the end_of_night_tasks data-archiving script (see Archiving data for details) successfully completed without error. Fill in the data archiving information sheet, ensuring you run the md5sum command on the USB disks as instructed.
2. Once you are happy that the previous night's data is safely archived, delete the contents of /data on the rack PC. This ensures that the first run of the coming night will be run001 (strongly recommended) and also ensures that there will be sufficient disk capacity for the entire night's observing.
3. Start the observing system (see Software startup), take some full frame bias frames in both slow and fast readout and check the readout noise and bias level using the python quality control script: /home/observer/qc/qc.py. This script writes the results to a database where you can compare your results with those obtained previously.
4. Take bias frames for all of the setups on the previous night.

1. Login as observer to the ultracam data reduction PC (known as ultracam; IP address 192.168.1.1). If you don't know the password, please contact one of the ULTRACAM team members.

2. Open an xterm on ultracam and login as observer to the rack PC (either ucam4 or ucam5 - see the ULTRACAM logistics page for which is currently in use). You can do this in a number of ways:
   1. By clicking on the desktop icon labelled rack.
   2. By typing rack in the xterm.
   3. By typing ssh 192.168.1.2 in the xterm.
   You shouldn't need a password to do this. If you need to know the password, please contact one of the ULTRACAM team members.

3. Ensuring that the SDSU controller is switched on, type the following in the xterm on the rack PC:

   start_ucam

   The following windows should then appear: The "Camera" window provides information on the commands used to control the EMCCD, which are sent to the SDSU controller. The "Filesave" window provides information on the commands used to define the quantity of data to be expected, which are sent to the SDSU-PCI card in the rack PC. The java-based GUI (known as udriver) sends the xml documents containing the camera and filesave parameters to the SDSU controller and PCI card via the http protocol. Note that if you have to shut the system down for some reason, you should always press ctrl-c in the filesave window before the camera window, otherwise you might need to reboot the rack PC to get the system running again.

4. Perform a software power-on of the CCD controller by clicking on the "Initialise" button on the left-hand side of the GUI. The filesave window should then report the creation of a new run file in the /data directory. You are now ready to take data.

ultracam8.1.32

You can then type any of the pipeline reduction commands, e.g. rtplot. In order to run the system whilst observing, it is necessary to access the data on the rack PC over the server. To do this, open an xterminal on the rack PC and type:

FileServer

AutoLogger is a c-shell script which produces a log of ULTRACAM observations on a web browser. AutoLogger also provides status information on the current run, outputting an audible and visual alarm if the CCD temperatures rise above -40 degrees, if the GPS stops working and if the file size goes over a user-defined limit. Please refer to the Troubleshooting section for advice on how to deal with the problems reported by AutoLogger.

The AutoLogger script runs in real time on the rack PC. The script works by polling the directory containing the data (usually /data) and extracting information from all the xml files it finds. For each ULTRACAM data file, it also determines the start/end times, file size, number of frames and exposure time. Comments on each run are input using an optional comments file, which must reside in the same directory in which AutoLogger is run. An example of the optional comments file can be found here - it is essential that you do not change the format of the file, i.e. the two header lines and the order of the columns. Any keyboard character can be used in this file except < and >.

To run AutoLogger whilst observing on 2010_04_24, for example, open an xterm on the rack pc, logged in as observer (see Software startup for how to do this). Then type:

cd AutoLogger
emacs 2010_04_24_log.dat &  -   and enter the run number and comments for any runs already taken
AutoLogger
  >   /data (the directory on the rack PC to which data is written)
  >   2010_04_24_log (the name of the log file - omit the .dat)
  >   6000 (the file size at which you wish a maximum filesize alarm to go off)
  >   y (to test the speaker volume level)
(AutoLogger   /data   2010_04_24_log   6000   y   typed on the command line will also work).

The script will run indefinitely, polling the data directory a few times every minute and looking for changes in either the data files or the comments file. If it finds a change, it will update the log displayed on the web browser. To exit AutoLogger, just type crtl-c, but only do this when AutoLogger says that it is safe to do so. The final log is written to a file in html format - in the example above the resulting file would be called 2010_04_24_log.html.

To view the logs from previous nights, open a web browser on the rack PC, avoiding the use of the konqueror browser, which would clash with AutoLogger if running. Then load the html log file, all of which are stored in /home/observer/AutoLogger.

At the end of the night, ensure that AutoLogger has been correctly shut down and that data taking has completely finished. Run the script end_of_night_tasks from the data reduction PC. This archives the original data, which is stored in /data on the rack PC, to two large-capacity USB disk drives in the control room. The script also makes a copy of the data on the archiving disk /data1 in the rack PC, and a subdirectory of /data. The script is self-explanatory to run, and is described in more detail in /home/star/archiving/ntt_archiving_apr_10_frontispiece.odt.

To take data in drift mode, it is recommended that the observing system is started (see Software startup) with the "-q" (for "quiet") option as follows:

start_ucam -q

The above command suppresses the frame number from being written to the filesave window, reducing the demand on the data reduction PC. It is recommended that you do not overload the rack PC, data reduction PC and ULTRACAM internal network with non-essential tasks when running in drift mode at the highest frame rates in order to minimise the chances of crashes.

If the sky is bright, you might notice that the top part of a window has a different background level compared to the bottom half. This occurs when it is impossible to fit an integer number of windows in the image area and hence part of each window exists on the chip (and hence accumulates sky) for slightly longer than the other part. To negate this effect, put the focal plane mask in the beam. The focal plane mask can also be used to prevent bright stars lying on the same column as your target star but on a higher row from corrupting your image. The slide is also useful if you want to minimise the light falling on the chips when taking bias frames and darks.

The focal plane mask is most easily moved from udriver by pressing the Focal Plane Mask button on the observing tab. The slide can also be moved by logging into the imedia PC. You can do this from the data reduction PC in a number of ways:

1. By clicking on the desktop icon labelled imedia1.
2. By typing ssh root@imedia1 in the xterm.
3. By typing ssh root@192.168.1.3 in the xterm.

slide

This will list the various options available to you on the command line.

The red and green filters are relatively straightforward to change: simply unfasten the two velcro strips holding the top edge of the foam baffle down, remove the foam piece and replace the filter. The blue filters are more tricky. Carefully slide out the cartridge containing the filter you wish to remove. Using the 15cm metal ruler found in the ULTRACAM tool box to prevent the blue foam ring from slipping, slowly slide the new filter cartridge into the filter slot (you may feel the ball-bearing retainer click into position when complete). Slide the metal ruler out, remembering to check that the two syringe needles for the dry-nitrogen flushing system are still attached to the blue foam ring. Don't worry if the blue foam ring is not centrally located over the CCD window - it would have to be seriously out of position to vignette the field. Make sure you keep the metal ruler parallel to the filter, CCD head and re-imaging camera at all times to ensure that you don't inadvertently scratch the outer surface of the re-imaging lens, CCD window or filter with it.

Always use the optics handling equipment (e.g. latex gloves, lens tissues, air spray) when changing filters - you can find these either in the blue briefcase or in the "optics handling equipment" box in one of the ULTRACAM packing crates. When blowing air across the filter, be sure to hold the can steady and upright, otherwise propellant may fall on the filter. It is also wise to make a few test blows into the air before spraying the filter. A head torch to assist with the filter change can be found in the tool box.

1. Make sure that you have written the night report on the ULTRACAM google group.
2. Make sure that the AutoLogger logs are up-to-date and complete for the night.
3. Shut down the observing system by typing ctrl-c in the filesave and camera windows (in that order) and closing udriver using "File" on the top bar.
4. Archive all data obtained during the night (see Archiving data).
5. If you suspect that the dew point might rise to within approximately 5 degrees of the water chiller set temperature whilst you are asleep, raise the set point accordingly.
6. Go to bed.

If you are unable to get the data reduction PC working again, you can switch to the spare data reduction PC ULTRACAM2, or the ULTRACAM laptop magnetar, which are both stored in the black crates. Please refer to Paul Kerry's user guide for details on how to do this.

Inexplicable crash whilst taking data

This hasn't happened with the new data acquisition system installed in early 2010. If it does recur, one reason could be because you have filled the /data disk. Another reason could be that you are pushing the system to its limits either in terms of data rate or frame rate. The former occurs, for example, when using drift mode at high-frame rates with the fast readout speed. If you experience a crash, try running the same application with the slow readout speed, in quiet mode, which should be more stable.

GPS problems

Chips suddenly output nothing but 32768 counts

Pickup noise

If you are determined to try to remove it, first try power cycling the SDSU controller and then allow it to settle down by taking some images for a while before measuring the readout noise again. If the pickup noise persists, try rearranging the cables at the back of the rack, between the rack and the instrument, and on the instrument itself. The aim here is to try to separate any cables carrying mains electricity from cables carrying data, although I'm not convinced how much this helps. It might also be worth avoiding tying cables together in loops, but again I'm not sure this makes much difference.

If you have no luck with this, you could try the various earthing cables in the cables box in the black crates. Connect the plug to a mains socket and then try earthing the instrument and/or electronics rack to this same point. Try to see if there are alternative routes to earth from the instrument, e.g. I once noticed the metal pipes carrying helium to the WHT Cassegrain focus were touching the top of the (isolated) electronics rack. If you have no luck with this, and you've noticed that the pickup noise is particularly bad on one of the chips, you might be able to isolate the problem by swapping the peltier power supplies around by exchanging the cables going into the power supplies at the rear of the electronics rack (and power cycling them in the process). If you have no luck with this, check the dome environment to see if anyone has turned on equipment which might be interfering with ULTRACAM and, if possible, ask them to turn it off. The latter I think is by far the most likely source of pickup noise.

SDSU won't turn on

Low/high red-channel bias

This is a rare, intermittent fault which occurs when a new run is started. The bias level on the red CCD suddenly jumps to a very high or low level, sometimes with a loss of sensitivity and an increase in the readout noise. Note that Tom Marsh has implemented a check in rtplot which issues a warning if it detects an abnormal bias level. It can usually be fixed by simply stopping the run and restarting it. If this does not work, try it again. If the problem persists, try a system reset by clicking on the System reset button when in expert mode on udriver, followed by an Initialise.

If this fails, try switching the SDSU controller on and off, which can be done remotely using the remote power module (see above).

CCD temperature problems

1. The water chiller has failed. There is a spare in the packing crates.
2. The water flow might have stopped. If the chiller is still running, this is almost certainly because one of the water pipe connectors on the CCD heads, SDSU controller, flow sensor (attached to the underside of the mid-plate of ULTRACAM) or water chiller have become disconnected. The connectors are self-sealing, so if you have inadvertently pressed the quick-release catch on the connector (perhaps when changing filters), the pipe will pop out by a few millimetres, close its seal, and cut off the water supply to the flow sensor (even though the chiller is still running), thereby cutting power to the peltiers. To determine if this has occurred, go round the instrument ensuring each pipe is securely connected to its socket. Remember that the moment you rectify the problem, the peltiers will power up. Therefore, it is essential that you immediately raise the set temperatures on the peltiers so that it is within 5 degrees of current chip temperature, and then slowly reduce the temperature as described in Powering up.
3. The grey box on the top shelf of the electronics rack marked "ULTRACAM PELTIER POWER SUPPLY CONTROL BOARD" might not be functioning. On one occasion, a chip in the box failed. In this case, you will need to bypass the grey box completely in order to get the peltiers to work. To do this, remove the cables from the grey box and attach all of them to the small metal box marked "FLOW CONTROL BYPASS" which you can find in the cardboard box marked "ULTRACAM ELECTRONICS TEST EQUIPMENT" in the packing crates. Remember that the moment you rectify the problem, the peltiers will power up. Therefore, it is essential that you immediately raise the set temperatures on the peltiers so that it is within 5 degrees of the current chip temperature, and then slowly reduce the temperature as described in Powering up. Remember also that the ULTRACAM CCDs are not now protected if there is a real disruption in the flow, e.g. if the water chiller fails, in which case the chips would overheat and be damaged. It is imperative that you fix the problem as soon as possible and in the meantime carefully monitor the water flow by placing your hand on the SDSU controller at frequent intervals: if it is cold, then the water is flowing.
4. A fuse has blown. There are three different fuses to check. The first is in the plug in the power cable from the back of the peltier rack unit to the socket strip at the top of the electronics enclosure. The second is in a small tray which slides out from beneath the socket at the back of the peltier rack unit. The third is inside the peltier rack unit itself, at the back of the temperature controller. To get at this, you will have to slide the rack out of the electronics enclosure and open the lid. There are spare fuses in the ULTRACAM tool box (usually found in either the control room or the packing crates). There is also a spare peltier unit in the packing crates.
5. The flow meter has failed. This is a small black box located on the underside of the central plate of ULTRACAM through which the water flows. When the sensor in this box detects a disruption of flow, or if the sensor fails, the peltiers will cut out. To rectify this problem, you will need to bypass the flow sensor. The easiest way of doing this is to disconnect the two water pipes leading into the flow sensor and connect them together. This will get you on-sky very quickly, but remember that the ULTRACAM CCDs are not now protected if there is a real disruption in the flow, e.g. if the water chiller fails. It is imperative you fix the problem as soon as possible and in the meantime carefully monitor the water flow by placing your hand on the SDSU controller at frequent intervals: if it is cold, then the water is flowing. Try cleaning the inside of the flow sensor with a cotton bud. If that fails, replace it with the spare, which is in the in the cardboard box marked "ULTRACAM ELECTRONICS TEST EQUIPMENT" in the packing crates.

Networking and computing problems

Please refer to Paul Kerry's user guide.