There is one display for each CCD head. The upper figure on the display shows the current CCD temperature and the lower figure shows the set temperature. The temperature controllers will immediately begin cooling the chips to the set temperature by running the peltiers at their maximum power of -55%. However, it is possible to damage the peltiers by allowing them to cool by more than 5 degrees per minute, so it is essential to immediately increase the set point to 5 degrees below the current chip temperatures using the up and down arrow buttons to the right of the temperature displays. Once attained, drop the set temperatures by approximately 5 degrees per minute until the desired temperature of -40 degrees is reached. Once the chip temperatures have stabilised at -40 degrees, check the peltier power levels by pressing the left-hand turquoise buttons on the temperature displays once. The power level should be approximately -30% to -35%. If the value is close to or equal to the maximum value of -55% then the heads may have lost their vacuum and need to be pumped down. Alternatively, the water chiller might be running at too high a temperature. You can return to the temperature display by pressing the black button in the middle.
As of March 2022, the above cool-down procedure is now obselete, as Peter Sinclaire (ESO) implemented the ramp function in the peltier controllers. This automatically cools (and warms) the CCDs at the safe rate of 5 degrees per minute. So you can simply set the temperature to the required value and leave the controller to cool/warm to the set temperature. I'm going to leave the above procedure in the user manual for now though, as a backup.
start_ucam
The following windows should then appear: The "Camera" window provides information on the commands used to control the CCD, which are sent to the SDSU controller. The "Filesave" window provides information on the commands used to define the quantity of data to be expected, which are sent to the SDSU-PCI card in the rack PC. The java-based GUI (known as udriver) sends the xml documents containing the camera and filesave parameters to the SDSU controller and PCI card via the http protocol. Note that if you have to shut the system down for some reason, you should try to remember to kill udriver first so that the current settings are saved.
ultracam9.16
You can then type any of the pipeline reduction commands, e.g. rtplot. In order to run the system whilst observing, it is necessary to access the data on the rack PC over the server. To do this, open an xterminal on the rack PC and type:
FileServer
As of February 2019, you can also use the python-based HiPERCAM pipeline reduction software. This is now the recommended method of looking at ULTRACAM data. It requires the same FileServer to be running, but no initialisation command is required to start the pipeline. Please ensure that the real-time data reduction performed on each night is stored in a directory of the form /home/observer/reduce/yyyy_mm_dd on the DRPC. Please also ensure that all output files from the pipeline are named according to the run number being reduced. So the standard reduction sequence using the HiPERCAM pipeline would be:AutoLogger is a c-shell script which produces a log of ULTRACAM observations on a web browser. AutoLogger also provides status information on the current run, outputting an audible and visual alarm if the CCD temperatures rise above -40 degrees, if the GPS stops working and if the file size goes over a user-defined limit. Please refer to the Troubleshooting section for advice on how to deal with the problems reported by AutoLogger.
The AutoLogger script runs in real time on the rack PC. The script works by polling the directory containing the data (usually /data) and extracting information from all the xml files it finds. For each ULTRACAM data file, it also determines the start/end times, file size, number of frames and exposure time. Comments on each run are input using an optional comments file, which must reside in the same directory in which AutoLogger is run. An example of the optional comments file can be found here - it is essential that you do not change the format of the file, i.e. the two header lines and the order of the columns. Any keyboard character can be used in this file except < and >.
To run AutoLogger whilst observing on 2010_04_24, for example, open an xterm on the rack pc, logged in
as observer (see Software startup for how to do this). Then type:
cd AutoLogger
emacs 2010_04_24_log.dat & - and enter the run number and comments for any runs already taken
AutoLogger
> /data (the directory on the rack PC to which data is written)
> 2010_04_24_log (the name of the log file - omit the .dat)
> 6000 (the file size at which you wish a maximum filesize alarm to go off)
> y (to test the speaker volume level)
(AutoLogger /data 2010_04_24_log 6000 y typed on the command line will also work).
The script will run indefinitely, polling the data directory a few times every minute and looking for changes in either the data files or the comments file. If it finds a change, it will update the log displayed on the web browser. To exit AutoLogger, just type crtl-c, but only do this when AutoLogger says that it is safe to do so. The final log is written to a file in html format - in the example above the resulting file would be called 2010_04_24_log.html.
To view the logs from previous nights, open a web browser on the rack PC, avoiding the use of the konqueror browser, which would clash with AutoLogger if running. Then load the html log file, all of which are stored in /home/observer/AutoLogger.
Tom Marsh has written an excellent observing check list which is well worth running through each night to ensure you've not forgotten anything. It is also full of useful tips to get the best quality data as efficiently as possible. Some additional useful notes are given below.
Target acquisition: If you are going to repeatedly revisit a target, it is usually desirable to be able to reacquire it onto the same ULTRACAM pixel with no time-consuming tweaking of telescope position. This can be done on the WHT, but not the NTT. (Need to check this.) To do this, it is imperative that you note down the following information once you've acquired a target for the first time:
When you wish to reacquire a target, give the above information to the TO. He/she should then slew the telescope to the given RA, declination and rotator angle. Once there, he/she should apply the offsets. Then, he/she should move the autoguider probe (i.e. the x-y or radial-azimuthal stage that the autoguider CCD is mounted on) to the given position. The guide star should then appear on the autoguider CCD. The TO should then position the guide star onto the desired pixel position on the autoguider CCD by moving the telescope (not the guide probes, which is what the TO would normally do). The above operation can place a target back on the same pixel of the ULTRACAM CCDs from one night to the next with no additional tweaking.
Twilight flats: When taking twilight-sky flat fields, try to observe a blank field near the zenith and also dither the telescope in a spiral pattern. Scripts are available at the WHT and NTT to do this automatically. On the NTT, make sure that you ask the telescope operator to use the spiral parameters 50, 6, 10, which corresponds to the number of steps, the time in seconds between steps, and the size of a step in arcseconds, respectively.
Catalogue files: To save time and minimise errors during acquisition at the NTT, it is useful to send the catalogue file of target positions created by Tom's phase II web pages directly to the telescope operator. To do this, navigate to the directory on the DRPC that the catalogue is saved in. Assuming the catalogue name is ultracam.cat, enter the following commands on the DRPC:
At the TO's end, they follow the same steps but use get ultracam.cat instead of put ultracam.cat. Then, on the TO's PC, they should move the catalogue file to /diska/vltdata/pointing/catalogs, and select it using the telescope control system.
Telescope rotator: Since September 2019 when ULTRACAM was mounted on the cube, a Sky PA of 0 on the TCS places N up and E left on the ULTRACAM CCDs. So no rotator offsets are required. For the record, when the cube is not in use, the sequence of events for dealing with Sky PA when acquiring a new target is as follows:
Autoguiding:
When using the autoguider, it is important to be aware of the "Zone of
Avoidance" for the autoguider probe. If it enters this, you will
vignette the ULTRACAM beam. The original email I sent around about
this on the google group in 2011 can be found here: https://groups.google.com/forum/#!searchin/vikcam/executive$20summary/vikcam/FO3k2uN9hR8/lRb28xi3K_gJ
There is a printout showing the zone of avoidance in the blue ULTRACAM
folder in the control room.
Active optics calibration: You should ask the TO to calibrate the active optics in twilight at the start of each night, which takes ~10 mins. However, don't bother doing it again later in the night. Even with the active optics calibrated, you will often need to refocus after each slew to a new target. It is imperative that you do this with reduce, looking at all three chips simultaneously and ensuring that you're plotting the target aperture FWHM (not the comparison). Doing this you should always be able to find a telescope focus that approximately minimises the target FWHM on all three chips, although this telescope focus would be different depending on whether your target is in the centre or the edge of the field (as we have a curved focal plane).
At the end of the night, ensure that AutoLogger has been correctly shut down and that data taking has completely finished. Run the script end_of_night_tasks from the data reduction PC. This archives the original data, which is stored in /data on the rack PC, to two large-capacity USB disk drives in the control room. The script also makes a copy of the data on the archiving disk /data1 in the rack PC, and a subdirectory of /data. The script also now makes a copy of the /home/observer/reduce/yyyy_mm_dd directory on the DRPC containing the pipeline-reduced data obtained that night. The script is self-explanatory to run, and is described in more detail in /home/star/archiving.
To take data in drift mode, it is recommended that the observing system is started (see Software startup) with the "-q" (for "quiet") option as follows:
start_ucam -q
The above command suppresses the frame number from being written to the filesave window, reducing the demand on the data reduction PC. It is recommended that you do not overload the rack PC, data reduction PC and ULTRACAM internal network with non-essential tasks when running in drift mode at the highest frame rates in order to minimise the chances of crashes.
If the sky is bright, you might notice that the top part of a window has a different background level compared to the bottom half. This occurs when it is impossible to fit an integer number of windows in the image area and hence part of each window exists on the chip (and hence accumulates sky) for slightly longer than the other part. To negate this effect, put the focal plane mask in the beam. The focal plane mask can also be used to prevent bright stars lying on the same column as your target star but on a higher row from corrupting your image. The slide is also useful if you want to minimise the light falling on the chips when taking bias frames and darks.
The focal plane mask is most easily moved from udriver by pressing the Focal Plane Mask button on the observing tab. The slide can also be moved by logging into the imedia PC. You can do this from the data reduction PC in a number of ways:
cd /home/httpd/slide ./slide
This will list the various options available to you on the command line.
Since September 2019, ULTRACAM has been mounted on the cube. Changing filters is probably best done with the instrument horizontal. You can either ask the TO to rotate it to this position from the TCS in the control room, or you can do it manually yourself in the dome. To do this, press the red emergency stop button on the end of the cable twister. This prevents anyone moving the rotator whilst you are changing filters. Press the hand-held rotator brake button, which will make a hissing sound as the brake releases. Keeping this button pressed, carefully rotator the rotator into the required position and release the button to engage the brake again. After you have changed the filter, remember to release the emergency stop button by twisting it either left or right, after which it will pop out.
The red and green filters are relatively straightforward to change: simply unfasten the two velcro strips holding the top edge of the foam baffle down, remove the foam piece and replace the filter. The blue filters are more tricky. Carefully slide out the cartridge containing the filter you wish to remove. Using the 15cm metal ruler found in the ULTRACAM tool box to prevent the blue foam ring from slipping, slowly slide the new filter cartridge into the filter slot (you may feel the ball-bearing retainer click into position when complete). Slide the metal ruler out, remembering to check that the two syringe needles for the dry-nitrogen flushing system are still attached to the blue foam ring. Don't worry if the blue foam ring is not centrally located over the CCD window - it would have to be seriously out of position to vignette the field. Make sure you keep the metal ruler parallel to the filter, CCD head and re-imaging camera at all times to ensure that you don't inadvertently scratch the outer surface of the re-imaging lens, CCD window or filter with it.
Always use the optics handling equipment (e.g. latex gloves, lens tissues, air spray) when changing filters - you can find these either in the blue briefcase or in the "optics handling equipment" box in one of the ULTRACAM packing crates. When blowing air across the filter, be sure to hold the can steady and upright, otherwise propellant may fall on the filter. It is also wise to make a few test blows into the air before spraying the filter. A head torch to assist with the filter change can be found in the tool box.
At the NTT, we have to use the longer 7.5m peltier power cables for ULTRACAM, which sometimes causes an "Err" to be displayed on the green peltier power supply. This usually happens when first powering on the system from cold at the start of a run. In the past we have tried tricks like repeatedly power cycling the unit, touching the green peltier connector at the rear of the rack to earth it, or even temporarily swapping the green and red peltier power cables at the rear of the unit to start cooling the green peltier (as once it has started cooling, the error seems to disappear). These approaches have had mixed success in the past and we have realised recently that the best course of action seems to be to just leave the green peltier turned on in its error state for about 10 minutes and, as its internal electronics warm up, the error state seems to clear of its own accord. Of course, you need to be careful to spot when the error finally clears, as if the set point is at -40 degrees and the CCD is warm, the set point must immediately be raised to the current CCD temperature. Since the new ramp cooling function has been set up in the peltier controllers, we have noticed that the ramp is sometimes disabled when the "Err" state eventually clears. If the ramp is functioning, the letters "rP" will flash intermittently on the display, so if this is not appearing, set the green CCD temperature to its current temperature, allow it to settle, and then set it to -40degC again.
The correct response to such a problem depends on whether the hard disk has failed or some other piece of hardware, e.g. the PC motherboard, has failed. In the former case, it is possible to recover quite easily using the clone disks. In the latter case, it will probably be necessary to install the spare rack PC or spare DRPC from the ULTRACAM crates. For full details on how to respond to each such problem, please refer to Paul Kerry's user guide - don't try anything before you've read that document!
Unless you can spot a broken or disconnected network cable, this usually happens because the rack PC has crashed. Go into the dome, turn on the LCD monitor in the rack and hit return on the keyboard. If the normal login prompt does not appear, then the rack PC has probably crashed. If you are unable to bring it back up, switch to the spare rack PC, which is located in one of the black crates, and contact one of the ULTRACAM team.
There has been one occasion when the data reduction keyboard stopped responding, probably due to an illegal combination of keystrokes. This was fixed by killing the windows one by one until the offending window had been killed and the keyboard started responding again.
If you are unable to get the data reduction PC working again, you can switch to the spare data reduction PC ULTRACAM2, or the ULTRACAM laptop magnetar, which are both stored in the black crates. Please refer to Paul Kerry's user guide for details on how to do this.
This hasn't happened with the new data acquisition system installed in early 2010. If it does recur, one reason could be because you have filled the /data disk. Another reason could be that you are pushing the system to its limits either in terms of data rate or frame rate. The former occurs, for example, when using drift mode at high-frame rates with the fast readout speed. If you experience a crash, try running the same application with the slow readout speed, in quiet mode, which should be more stable.
If you are determined to try to remove it, first try power cycling the SDSU controller and then allow it to settle down by taking some images for a while before measuring the readout noise again. If the pickup noise persists, try rearranging the cables. Experience has shown that by far the most important factor seems to be the arrangement of the data cables running between the SDSU controller and the CCD heads. Without disconnecting them, try altering the position and angle of these cables with respect to each other using cable ties to hold the data cables in place. If this fails, adjust the position of the other cables on the instrument so that none of them passes close to the data cables and their connectors. The aim here is to try to separate any power-carrying cables from cables carrying data, although I'm not convinced how much this helps. It might also be worth avoiding tying cables together in loops, but again I'm not sure this makes much difference.
If the above fails, ensure that the chiller is not too close to the electronics rack, as I've noticed pickup from this before on the NTT.
If you still have no luck, you could try the various earthing cables in the cables box in the black crates. Connect the plug to a mains socket and then try earthing the instrument and/or electronics rack to this same point. Try to see if there are alternative routes to earth from the instrument, e.g. I once noticed the metal pipes carrying helium to the WHT Cassegrain focus were touching the top of the (isolated) electronics rack. If you have no luck with this, and you've noticed that the pickup noise is particularly bad on one of the chips, you might be able to isolate the problem by swapping the peltier power supplies around by exchanging the cables going into the power supplies at the rear of the electronics rack (and power cycling them in the process). If you have no luck with this, check the dome environment to see if anyone has turned on equipment which might be interfering with ULTRACAM and, if possible, ask them to turn it off.
This is a rare, intermittent fault which occurs when a new run is started. The bias level on the red CCD suddenly jumps to a very high or low level, sometimes with a loss of sensitivity and an increase in the readout noise. Note that Tom Marsh has implemented a check in rtplot which issues a warning if it detects an abnormal bias level. It can usually be fixed by simply stopping the run and restarting it. If this does not work, try it again. If the problem persists, try a system reset by clicking on the System reset button when in expert mode on udriver, followed by an Initialise.
If this fails, try switching the SDSU controller on and off, which can be done remotely using the remote power module (see above).
The chiller water temperature monitor is designed to notify the user, via a status display and alarm in the AutoLogger window, of the water temperature in the chiller reservoir falling close to freezing. If ice forms, it can destroy the chiller, and this can occur if the ambient temperature falls to within a few degrees above zero. It has happened once before on the NTT. If the alarm goes off on AutoLogger, go up to the Nasmyth room and open the chiller water reservoir cap and check for ice. If any ice is present, turn the chiller off immediately and allow it to melt (you may have to move it to a warm room to do this). To carry on observing, you will have to use the spare chiller located in one of the ULTRACAM crates. If no ice is present, which should be the case as the alarm goes off well above the temperature where this should happen, you can keep the chiller running, but you should try to insulate the area around the chiller with foam, without blocking the air flow around it completely. This should allow the chiller's internally generated heat to warm the water reservoir.
If the chiller water temperature monitor is not working, it is likely that it needs to be power cycled. The easiest way to do this is to use the RPM, to which there is a link bookmarked on the web browser running on the ULTRACAM data reduction PC. Just select the chiller water temperature monitor and turn it off and then on again. If this still doesn't work, check that the chiller water-temperature probe is immersed in the water in the chiller reservoir by passing it through the black filler cap, and that its connector (a headphone jack and socket) is fully pushed together - it is easy to leave this connector only partially connected. Check also that the other end of the cable is connected to the control box in the shelf in the rack.
If rtplot or reduce have been working nicely all night and then suddenly fails with the following error:
Traceback (most recent call last): File "/usr/local/bin/rtplot", line 8, inThis is probably because you've inadvertently overwritten the defaults file, e.g. with a badly timed ctrl-c, and it is using the ULTRACAM server, say, rather than the HiPERCAM one. The fix is simple - run rtplot or reduce again with the prompt parameter on the command line, and check the value of the source parameter.sys.exit(rtplot()) File "/usr/local/lib/python3.7/dist-packages/hipercam/scripts/rtplot.py", line 583, in rtplot with spooler.data_source(source, resource, first, full=False) as spool: File "/usr/local/lib/python3.7/dist-packages/hipercam/spooler.py", line 405, in data_source return HcamServSpool(resource, first) File "/usr/local/lib/python3.7/dist-packages/hipercam/spooler.py", line 319, in __init__ self._iter = hcam.Rdata(run, first, True) File "/usr/local/lib/python3.7/dist-packages/hipercam/hcam.py", line 796, in __init__ Rhead.__init__(self, fname, server, full) File "/usr/local/lib/python3.7/dist-packages/hipercam/hcam.py", line 214, in __init__ self._ws = websocket.create_connection(URL + fname) File "/usr/local/lib/python3.7/dist-packages/websocket/_core.py", line 592, in create_connection websock.connect(url, **options) File "/usr/local/lib/python3.7/dist-packages/websocket/_core.py", line 252, in connect self.handshake_response = handshake(self.sock, *addrs, **options) File "/usr/local/lib/python3.7/dist-packages/websocket/_handshake.py", line 79, in handshake status, resp = _get_resp_headers(sock) File "/usr/local/lib/python3.7/dist-packages/websocket/_handshake.py", line 164, in _get_resp_headers raise WebSocketBadStatusException("Handshake status %d %s", status, status_message, resp_headers) websocket._exceptions.WebSocketBadStatusException: Handshake status 200 OK
Please refer to Paul Kerry's user guide.
