ULTRACAM user manual

Vik Dhillon, version: 19 May 2004

1. Introduction
2. Powering up
3. Powering down
4. Afternoon activities
5. Taking data
6. Looking at data
7. Archiving data
8. Changing CCD parameters
9. Setting up on an object
10. Drift mode
11. Calibration frames
12. AutoLogger
13. Changing filters
14. Before you go to bed
15. Troubleshooting
16. Contacts

• ULTRACAM system maintenance document. This document describes ULTRACAM's camera head handling and system maintenance procedures. It should only be of relevance to project personnel. Word document version available here.
• CCD application document. This document describes the various CCD controller applications that can be run with ULTRACAM. You should regard the applications document as the primary source of reference for the different readout modes and their associated parameters, although a summary of its contents is given below.
• ULTRACAM pipeline software manual. This document describes how to use Tom Marsh's pipeline data reduction software, which is the best way of viewing the data whilst observing.
• Telescope operators guide. This document describes how to use ULTRACAM with the WHT Telescope Control System and is intended primarily for telescope operators at the ING..
• UCam User Manual. This is Stewart McLay's user manual for the UKATC UCam data acquisition system, which is used by ULTRACAM. Useful information can also be found in the corresponding UCam Installation Manual and the UCam Configuration File Reference Manual.
• GPS170PEX. User Manual. This document contains the Technical Information and Operating Instructions for the Meinberg GPS170PEX, the new ULTRACAM GPS system installed in March 2010.

1. Turn off the SDSU controller.
2. Turn off the water chiller. This will cut the water flow and hence will automatically turn the peltiers off (but check the peltier displays to make sure). With all cooling turned off, the chips will gradually warm up to ambient temperature at a safe rate.
3. Turn off the peltier devices.
4. Shut down the rack PC by typing "init 0" in an xterminal connected to the PC. This does not turn the PC power off, so when the mini-LCD panel says "Power down" you should press the "Power" button on the front-panel of the rack PC.
5. Turn off the SCSI hard-disk array. Never turn off the the SCSI hard-disk array when the rack PC is powered on, as the disks may be corrupted.
6. Turn off power to the entire ULTRACAM rack by switching it off at the mains.
7. Turn off the feed to the nitrogen gas flushing system (see Afternoon activities.
8. Shut down the data reduction PC by pressing the ctrl-alt-del buttons at the login prompt. The PC will power down automatically on completion.
9. Turn off the network switch.

1. Check water temperature on the chiller display is set to 10 degrees, unless this is within approximately 5 degrees of the dew point (in which case the water temperature should be increased). The dew point can be determined from the observatory's meteorological system. A useful check on this is to run your hand along the base of the SDSU controller. If it feels even slightly clammy to the touch, increase the water temperature slightly.
2. Check CCD temperatures on the rack-mounted temperature controllers are stable at -40 degrees.
3. Check peltier power usage on the rack-mounted temperature controllers. These should typically be around -30%. If the value is close to or equal to the maximum value of -55% then there is either a problem with the vacuum in the head or the CCD temperatures have been set too low (or the water temperature too high).
4. Check for water leaks by running your fingers along the pipe connectors on the water chiller, the SDSU controller, the flow sensor (attached to the underside of the mid-plate of ULTRACAM) and the individual CCD heads.
5. Check the nitrogen gas flushing system is operable by ensuring that all the blue pipework and syringe needles on ULTRACAM are connected, that the small brass valves on ULTRACAM are fully open, and that the gauge feeding ULTRACAM is set to a flow rate of appoximately 0.5 litres/minute.
6. Ensure that the water pipes and GPS cable leading up to the instrument are neatly arranged beneath it and are in no danger of snagging on any equipment.
7. Ensure that the star-field tube is not mounted on the ULTRACAM collimator, that the focal-plane mask is fully retracted (when the upper scale is on the far right-hand side) and its foam baffle is in position.
8. Check which filters are installed in front of each CCD. The most commonly used filters have a little label visible on their cartridge saying what the filter it is. If you can't see this, you can determine which filters are mounted by either taking the cartridges out and inspecting the filters or by looking in the filter case to see which filters aren't mounted (see Changing filters).
9. Turn on the SDSU controller and reset the SDSU-PCI card (see Powering up).
10. Move the contents of /data on ucam2 to one of the five disks in the SCSI disk-array (see Archiving data for details). This ensures that the first run of the coming night will be run001 (strongly recommended) and also ensures that there will be sufficient disk capacity for the entire night's observing.
11. Prepare the hand-written log files for the coming night by printing out one copy of this title page template and multiple copies of these blank tables
12. Start the observing system (see Taking data), take some bias frames and check the readout noise and bias level (see Calibration frames and Looking at data).

1. Log into the ultracam data reduction PC (known as ultracam) or Vik's lap-top (known as magnetar). If you don't have a user account on either of these computers, please contact Paul Kerry.
2. Log into the rack PC (known as ucam2) as root using the command:

   ssh root@192.168.1.2

   If you don't know the root password for ucam2, please contact Paul Kerry. To ease this process, you can either type rack, which will automatically log you into ucam2 as root without having to type a password, or click on the ucam2 icon in the control panel on the right-hand side of the desktop. If these don't work for you, a setting is probably amiss - please contact Paul Kerry. Because you are logged in as root you need to be extra-cautious with your typing.
3. Ensuring that the SDSU controller is switched on, type the following in an xterminal on ucam2:

   start_ucam

   The following windows should then appear:

   

   The "Camera" window (top-left) provides information on the commands used to control the CCD cameras, which are sent to the SDSU controller. The "Filesave" window (top-right) provides information on the commands used to define the quantity of data to be expected, and are sent to the SDSU-PCI card in the rack PC. The "Netscape" window (or "WebGUI") sends the xml documents containing the camera and filesave parameters to the SDSU controller and PCI card via the http protocol.

4. Set the file headers by clicking on the following links in the "engineering" column of the netscape window:
   • setup wht (camera)
   • setup wht (filesave)
   • setup ultracam (camera)
   • setup ultracam (filesave)
   These links must be clicked in order from top to bottom. Each time you click on a link you must wait until you see the word "ok" in the status window at the bottom left of the web browser before clicking on the next link. You will also obtain a line in the history window at the bottom right of the web browser showing you which links have already been activated.
5. Power up the SDSU controller by clicking on the following links in the netscape window:
   • reset PCI
   • power on (camera)
   • power on (filesave)
   • GO
   The filesave window should then report the creation of a new run file in the /data directory and 6 green and 1 amber LED's should illuminate on the SDSU controller, On occasions, the above operation fails to create a new run file. The SDSU power-on sequence must then be performed again by following these steps:
   • Press ctrl-c in the camera window.
   • Press ctrl-c in the filesave window.
   It is essential that you do this in the order given above. There is no need to close the netscape window. Now restart the software using the command:

   start_ucam

   The camera and filesave windows will reappear and, after a few seconds, the netscape window will be respawned - make sure you wait for this to happen before clicking on any links. Set the file headers again by clicking on the following links in the "engineering" column of the netscape window, in the following order:
   • setup wht (camera)
   • setup wht (filesave)
   • setup ultracam (camera)
   • setup ultracam (filesave)
   Now perform a software reset of the SDSU controller and PCI card and power the controller on again:
   • reset PCI
   • reset SDSU
   • reset PCI
   • power on (camera)
   • power on (filesave)
   • GO
   Note how the software reset of the SDSU controller must be followed a PCI-card reset. The filesave window should then report the creation of a new run file in the /data directory and a full set of illuminated LED's should appear on the SDSU controller. If this procedure fails again, refer to the Troubleshooting section.
6. You are now ready to take data. You can select either one of the "generic" modes or one of the "user-defined" modes - the difference between these two options is described in the section on Changing CCD parameters. If, for example, you decide you want to take data in 2-windowed mode using the CCD parameters given in the generic xml file, you should click on the following links:
   • 2 windows (camera)
   • 2 windows (filesave)
   • GO
   • STOP
   It is always necessary to download first the camera application and then the filesave application. You must wait for each application to successfully download (indicated by the "ok" in the status window) before clicking on the next link. In the above example, the NO_EXPOSURES parameter is set to -1 in the xml file (see Changing CCD parameters), and hence data will be taken indefinitely until you click on the "STOP" link. When this occurs, you will obtain a message in the filesave window telling you how many exposures were taken and a message in the status window saying "ok".
7. Once you have started a run, type the following command in an xterminal logged into the rack PC:

   ucam_data_limit

   Hit return in response to all the prompts (unless you want to check the speaker volume or change the file size limit). This script checks the latest run and outputs an audible warning when it becomes dangerously close to the maximum file size limit of approximately 2 GBytes. When you hear the alarm, you must stop the current exposure. If the file size goes over the limit, the system will crash - see Troubleshooting for recovery information. The script also checks whether the GPS timestamping is working correctly and outputs an audible alarm if it detects a problem - refer to the Troubleshooting section for advice on how to get the GPS going again.
8. When you are ready to take your next exposure, you must reload the camera and filesave applications for your chosen CCD setup and then click "GO", even if you have not changed any of the parameters.

ultracam3.0.3

You can then type any of the pipeline reduction commands, e.g. rtplot. In order to run the system whilst observing, it is necessary to access the data on the rack PC over the server. To do this, open an xterminal on the rack PC and type:

FileServer

Please refer to Paul Kerry's guide to archiving ULTRACAM data.

cd /ultracam/config/xml (or just type xmldir)

• NO_EXPOSURES - the number of exposures. It is usual to set this to "-1", which means that exposures are obtained until the "STOP" link is clicked on the netscape window.
• GAIN_SPEED - the gain and digitization speed. It is usual to set this to "0xcdd", which is a slow readout speed (11.2 microseconds/pixel/second/channel) with a gain of 1.2 electrons/ADU and readout noise of ~4 e-. Setting to "0xfdd" gives a faster readout speed (3.4 microseconds/pixel/second/channel) at the expense of significantly worse readout noise (~9 e-, with 1.2 electrons/ADU gain).
• EXPOSE_TIME - this is the exposure delay time, which must be an integer in units of milliseconds. This value should be added to the irreducible minimum exposure time due to the time it takes to read the chip out to give the total exposure time. Do not enter a value of 0 (or less), as it will crash the system.
• X_BIN_FAC - the x-binning factor, which must be an integer multiple of the x-size.
• Y_BIN_FAC - the y-binning factor, which must be an integer multiple of the y-size.
• V_FT_CLK - the vclock speed. It is usual to set this to "0x8c0000", which gives a vertical clocking speed of 23.3 microseconds/row. If you lower this, you run the risk of lowering the count-level at which the "peppering effect" occurs (see the Troubleshooting section for details).
• X1L_START - the x-start position of the first window on the left-hand side of the chip.
• X1R_START - the x-start position of the first window on the right-hand side of the chip.
• Y1_START - the y-start position of the first pair of windows.
• X1_SIZE - the x-size of the first pair of windows.
• Y1_SIZE - the y-size of the first pair of windows.
• X2... ditto for the second pair of windows.
• X3... ditto for the third pair of windows.

Once you have set your desired parameters, save the file and then download the application by clicking on the appropriate links in the netscape window. For setups which you will reuse during the run, it is much easier to define your own "user-defined mode". To do this, save the xml file you have been editing as:

set_objectname.xml

cd /ultracam/src/ucam/WebGUI/vik
emacs control_250.html

<li> <a href="javascript:cam('set_objectname.xml')">
objectname (camera)</a> </li>
<li> <a href="javascript:fs('set_objectname.xml')">
objectname (filesave)</a> </li>

Change the "objectname" and "set_objectname.xml" strings to whatever you have decided to call your xml file, save the file, and then press the "reload" button on the netscape window. Your new setup should now appear as an option under the "user-defined modes" column. Be careful in this directory, as it is the main repository for the html files used for the WebGUI (i.e. the netscape window on which you run the observing system). If you do delete anything by accident, there is a copy of the directory on ucam2 in /ultracam/src/ucam/WebGUI/vik/webgui_backup_31_10_03.

Note the ULTRACAM is aligned in the lab using u', g' and r' filters. If you use a different filter in one of the channels, you might find it has a different optimum telescope focus position to the other channels. The problem should not be severe, however, as all of ULTRACAM's filters have been designed to have approximately equal optical thickness.

Target acquisition

The WHT points to better than a few arcseconds and the centres of the ULTRACAM chips are within a few arcseconds of the rotator centre (to which the WHT points). Hence, as long as your coordinates are accurate to a few arcseconds, you can be certain that when you slew to your target it will appear at the very centre of the ULTRACAM chips, making target identification very easy.

The scale of the ULTRACAM chips on the WHT is 5 arcminutes (0.3 arcseconds/pixel). The orientation is variable as it depends on the position of the Cassegrain rotator (or "ROT SKY PA"). With the rotator at a sky position angle of 0 degrees, north is at the top of the frame and east is to the left. It is therefore advisable to ask the telescope operator to use the command "ROT SKY 0" when slewing to a new target so that you can more easily identify the field.

The best way of positioning targets on the ULTRACAM detectors is to use the "acquisition" application under the "user-defined modes" column on the netscape window. This is just a full-frame/no-clear setup with fast readout and 4x4 binning. With a 1 millisecond exposure delay, this results in an exposure time of approximately 0.2 seconds, with negligible dead-time, although you can change the EXPOSE_TIME parameter in the set_acquisition.xml file to increase the exposure length. The rapid readout and large field of view results in an acquisition-TV-type mode, which greatly facilitates target acquisition.

Once you have acquired a target in a window and started to autoguide, don't forget to write down the telescope position (including any offsets the telescope operator might have made), the position angle of the Cassegrain rotator on the sky, the autoguider probe positions (autotheta, autoradial) and the x,y coordinates of the guide star on the autoguider CCD. This will enable you to rapidly reacquire a target onto exactly the same pixel on ULTRACAM as before.

Setting CCD windows

Exposure times

The ULTRACAM chips saturate at 65536 counts. However, the peppering effect only disappears at approximately 30000 counts in the u' and g' chips, and at approximately 50000 counts in the red chip. These latter two figures should therefore be used as the upper count limits for ULTRACAM exposures. Note, however, that the peppering is strongest in the central regions of the chip and it is possible to use 65536 as the upper count limit if one is using windows at the bottom of the chip (e.g. in drift mode). To be sure if a particular chip position is safe from peppering at a given count level, however, always check the flat fields.

The profile-fitting option in "rtplot" can be used to determine the maximum counts in an aperture centred on the star. If the exposure time needs to be lengthened, increase the "EXPOSE_TIME" parameter in the appropriate xml document (see Changing CCD parameters). If the exposure time needs to be decreased, and the EXPOSE_TIME parameter is already at its minimum value of "1", you will need to either decrease the size and/or number of windows being read out, bin the pixels (although this will make things worse if you are worried about saturation), change the "GAIN_SPEED" parameter, adjust the window positions to minimise the "DIFF_SHIFT" (see the CCD application document), use drift mode, or defocus the telescope.

To take data in drift mode it is necessary to follow exactly the same procedure as described in Taking data. If, however, hundreds of frames per second are required, it is recommended that the observing system is started with the "-q" (for "quiet") option as follows:

start_ucam -q

The above command suppresses most of the debugging information currently output in the filesave window, which can place extreme demands on the pc when running at the fastest frame rates, and even cause it to crash (when it fails to pick up time-stamps rapidly enough).

It is important to choose the Y1_SIZE parameter of the windows in drift mode very carefully, so as to minimise the "pipe delay", i.e. the delay added to all bar one of the windows in the pipeline to ensure that they all have the same exposure time. The CCD application document gives a full description of this term and provides a useful table (and plot) of the special values of Y1_SIZE that should be used in order to maximise the frame rate.

If the sky is bright, you might notice that the top part of a window has a different background level compared to the bottom half. This occurs when it is impossible to fit an integer number of windows in the image area and hence part of each window exists on the chip (and hence accumulates sky) for slightly longer than the other part. To negate this effect, put the focal plane mask in the beam, taking care to prevent the blue sponge light-baffle from falling inside the mounting collar. The vertical position should be set at 19.00mm on the micrometer and should never be adjusted. The horizontal position depends on the size and position of your windows and can be calculated using the scale printed on the edge of the sliding mask. Note that the focal plane mask can also be used to prevent bright stars lying on the same column as your target star but on a higher row from corrupting your image.

These are best selected from the list of SDSS standards in Smith et al. 2002 (AJ, 123, 2121). The easiest way of obtaining data on standards is to use the full-frame application set_ap3_250_fullframe.xml, keeping it unbinned and in slow readout mode. Try using stars which are fainter than about 12th magnitude, which enables you to use an exposure delay of approximately 1000 milliseconds. This is significantly longer than the 25 milliseconds it takes to vertically clock the image into the storage region, which means that streaking of the star-light due to frame transfer is negligible. When acquiring standard stars, they will usually appear at the centre of the chip, and you should therefore ask the telescope operator to move the star by an arcminute left or right to get it clear of the border between the two readout channels.

To run the AutoLogger for data on the rack PC whilst observing on 2003-05-03, type the following:

cd /ultracam/src/ucam/WebGUI/vik/AutoLogger (or just type autologger)
emacs 2003-05-03_log.dat    -    and enter the object name, filters and comments for each run
AutoLogger
  >   /data
  >   2003-05-03_log
(AutoLogger   /data   2003-05-03_log   typed on the command line will also work.)

The script will run indefinitely, polling the data directory a few times every minute and looking for changes in either the data files or the comments file. If it finds a change, it will update the log displayed on the web browser. To exit AutoLogger, just type crtl-c, but only do this when AutoLogger says that it is safe to. The final log is written to a file in html format - in the example above the resulting file would be called 2003-05-03_log.html. To print out the html log file, it is best to run the konqueror web browser on the data reduction PC, load the html log, and then print to a postscript file, setting the page size to A4 and the orientation to landscape.

AutoLogger can also be run off-line, either on the rack PC in exactly the same way as described above, or on the data reduction PC. In the latter case, AutoLogger resides in /home/vsd/AutoLogger and it can be used to poll any directory on the data reduction PC which contains ULTRACAM data.

As well as the automatic log, we also keep a hand-written log which details weather conditions, run numbers, filters and any comments. It is essential that this log is carefully kept as it provides a backup to the electronic version. Before you start observing on each night, print out one copy of this title page template and then multiple copies of these blank tables.

1. Make sure that the hand-written and automatic logs are up-to-date and complete for the night.
2. Shut down the observing system by typing ctrl-c in the camera and filesave windows and closing the netscape window (see Taking data).
3. Copy all data obtained during the night from the rack PC to the data reduction PC (see Archiving data). This is a quick way of ensuring that a copy of the data exists on a separate system. Proper archiving can then wait until the following afternoon (see Afternoon activities).
4. Turn off the SDSU controller (see Powering up).
5. If you suspect that the dew point might rise to within approximately 5 degrees of the water chiller set temperature whilst you are asleep, raise the set point accordingly.
6. Go to bed.

Unable to connect to the rack PC

Unable to take data

1. Have you clicked on a link out of order, e.g. by clicking on the "GO" link without having previously loaded an application? Fortunately, the system is much more robust to this kind of finger trouble than it was in the past, and it is almost always possible to rectify any mistakes by simply typing ctrl-c in the camera and filesave windows (in that order) and then retyping start_ucam, remembering to wait for the WebGUI to respawn before clicking on the links again.
2. Have you powered the SDSU controller on in the dome?
3. Have you reset the SDSU-PCI card in the dome?
4. Is the green LED on the SDSU-PCI card lit? If so, there is a problem. Power cycle the SDSU controller and rack PC. If this fails, check the fibres between the PCI card and the controller. If damaged, there is a spare in the packing crates.
5. Have you powered on the SDSU by clicking on the "power on" links?
6. As a last resort, power down the rack PC and then remove mains power to it by unplugging it. This cuts the power to the SDSU-PCI card and should completely reset it if all else fails.

Data reduction PC hangs whilst exposing

There has been one occasion when the data reduction keyboard stopped responding, probably due to an illegal combination of keystrokes. This was fixed by killing the windows one by one until the offending window had been killed and the keyboard started responding again.

WebGUI problems

Inexplicable crashes whilst taking data

Another reason could be because you are pushing the system to its limits either in terms of data rate or frame rate. The former occurs, for example, when using exposure delays of less than 100 in the full-frame/no-clear mode. The latter occurs, for example, when using "0xfdd" in fast drift-mode applications. If you experience a crash, try running a different application or using different parameters in the same application.

Another, very subtle, reason for crashes is if you prevent terminal output from being written to the filesave window. This occurs, for example, when moving windows around on the fvwm window manager. It also occurs when opening up new windows. In the latter case, it is essential that you do not use ActivePlacement or SmartPlacement of windows, but rather use RandomPlacement (which prevents any user interaction in the placement of new windows) - check your .fvwm2rc file to determine which of these are in use. By freezing terminal output, GPS timestamps are no longer read from the FIFO, which eventually overflows and crashes the system. The FIFO buffer is quite large, so this is only a problem when running drift mode, where there is a great deal of terminal output. This is why it is recommended that the observing system is started with the "-q" option, as described in the drift mode section.

GPS problems

High red-channel bias/loss of sensitivity in red channel

Chips suddenly output nothing but 32768 counts

Pickup noise

If you are determined to try to remove it, first try power cycling the SDSU controller and then allow it to settle down by taking some images for a while before measuring the readout noise again. If the pickup noise persists, try rearranging the cables at the back of the rack, between the rack and the instrument, and on the instrument itself. The key thing here is to separate any cables carrying mains electricity from cables carrying data. It might also be worth avoiding tying cables together in loops, although I'm not sure this makes much difference. In addition, make sure the metal pipes carrying helium to the WHT Cassegrain focus are not touching the top of the electronics rack, as this provides an alternative route to earth. If you have no luck with this, and you've noticed that the pickup noise is particularly bad on one of the chips, you might be able to isolate the problem by swapping the peltier power supplies around by exchanging the cables going into the power supplies at the rear of the electronics rack. If you have no luck with this, check the dome environment to see if anyone has turned on equipment which might be interfering with ULTRACAM (and, if possible, ask them to turn it off).

Two tier background level in drift mode

If the sky is bright and you are using drift mode, you might notice that the top part of the windows have a different background level compared to the bottom half. This occurs when it is impossible to fit an integer number of windows in the image area and hence part of each window exists on the chip (and hence accumulates sky and, to a lesser extent, dark current) for slightly longer than the other part. To negate this effect, put the focal plane mask in the beam, taking care to prevent the blue sponge light-baffle from falling inside the mounting collar. The vertical position should be set at 19.00mm on the micrometer and should never be adjusted. The horizontal position depends on the size and position of your windows and can be calculated using the scale printed on the edge of the sliding mask.

Temperature displays blank

The water flow might have stopped. If the chiller is still running, this is almost certainly because one of the water pipe connectors on the CCD heads, SDSU controller, flow sensor (attached to the underside of the mid-plate of ULTRACAM) or water chiller have become disconnected. The connectors are self-sealing, so if you have inadvertently pressed the quick-release catch on the connector (perhaps when changing filters), the pipe will pop out by a few millimetres, close its seal, and cut off the water supply to the flow sensor (even though the chiller is still running), thereby cutting power to the peltiers. To determine if this has occurred, go round the instrument ensuring each pipe is securely connected to its socket. Remember that the moment you rectify the problem, the peltiers will power up. Therefore, it is essential that you immediately raise the set temperatures on the peltiers so that it is within 5 degrees of current chip temperature, and then slowly reduce the temperature as described in Powering up.

The grey box on the top shelf of the electronics rack marked "ULTRACAM PELTIER POWER SUPPLY CONTROL BOARD" might not be functioning. This could be because the cables leading to it are not properly connected, so this is the first thing that should be checked. On one occasion, a chip in the box failed. In this case, you will need to bypass the grey box completely in order to get the peltiers to work. To do this, remove the cables from the grey box and attach all bar the grey cable from the flow sensor (which is now redundant) to the small metal box marked "FLOW CONTROL BYPASS" which you can find in the cardboard box marked "ULTRACAM ELECTRONICS TEST EQUIPMENT" in the packing crates. Remember that the moment you rectify the problem, the peltiers will power up. Therefore, it is essential that you immediately raise the set temperatures on the peltiers so that it is within 5 degrees of the current chip temperature, and then slowly reduce the temperature as described in Powering up.

Networking and computing problems

It is impossible to list everything that might go wrong here. Best to contact Paul Kerry (see below).