Pumping and cooling
This section describes how to pump and cool HiPERCAM’s CCDs. It is assumed that the instrument has already been powered up. Click on the photos below for larger images.
You will need to access the Elevation (Folded Cassegrain) Platform in order to pump and cool HiPERCAM. Only HiPERCAM team members who have completed the relevant safety courses and GTC paperwork, and have GTC permission to do so, are allowed to access this area. Before going up to the platform, put on your personal protection equipment, including: helmet with chin strap, safety shoes/boots, safety harness with lanyard and scaffold hooks (located in the HiPERCAM crates), and safety gloves (if necessary). Make sure that you have a radio, set to channel 8, and that you have told the GTC Jefe de Turno (during the day) or Telescope Operator (during the night) if it is ok for you to go up to the platform. He or she will have to slew the telescope to vertical, insert the elevation axis locking pins and turn on the dome lights.
On your way up to the platform, do not carry anything in your hands that would prevent you from safely using the stair rails on your way up - any such equipment should be transported to the platform using the cradle and crane. When descending the platform, always place one hand on the stair rail behind you and one on the rail in front of you, as shown in the photo below.
Once on the platform, carefully pivot the access stairs if they are not already down. It has been known for the telescope elevation to be incorrectly positioned, forcing the access stairs to rest on a flimsy piece of metal conduit rather than the red railing. In this case, ask the Jefe de Turno or Telescope Operator to adjust the telescope elevation accordingly.
The next step is to vacuum pump the CCD heads. The principle here is that cooling the CCDs when the heads do not contain a good vacuum risks condensation forming on the detectors, which are the coldest part of the CCD head. So you must only ever cool down the CCDs whilst pumping.
Prior to June 2019, we used the OSIRIS vacuum pump for the HiPERCAM CCD heads. For instructions on how to use this pump, refer to OSIRIS pump.
The HiPERCAM pump is mounted on the collar, which is mounted on the rotator. The pump is permanently connected to the vacuum manifold on HiPERCAM via a stainless steel pipe. Power to the pump is provided by a custom mains cable that is permanently plugged into the patch panel on the rotator. The vacuum pump is mounted such that the rotation axis of the turbo is aligned with the rotation axis of the rotator, so it should not be a problem if the rotator is moved whilst the HiPERCAM CCD heads are being pumped. Moving the telescope elevation axis whilst pumping, however, could damage the turbo, so should be avoided if possible.
When pumping, it is recommended that you position HiPERCAM in the orientation shown in the pictures below, with CCD2 pointing upwards. This ensures that there are no CCD heads pointing downwards, which reduces the risk of a person damaging one of the CCD heads as they pass underneath the instrument.
If the instrument is not at the correct orientation, you will have to move the rotator. To do this, you will need to switch control of the rotator from Remote to Local by turning the key on the control panel mounted on the cabinet door to the right of HiPERCAM. Make sure that you tell the Jefe de Turno or Telescope Operator before you do this. Even if the rotator is already in the correct location for pumping, you should still switch to Local control, as it prevents anyone from moving the rotator remotely by accident whilst pumping.
Once in Local mode, you will need to release the brake to move the rotator to the correct orientation. To do this, take the key hanging from the shelf at the rear of the HiPERCAM electronics cabinet, and open the rotator brake box. (Be careful not to lose the little loop of black velcro used to attach the key to the cabinet shelf.)
If the switch inside the box is pointing to AUTO, the rotator brake is on and you will need to switch it to MANUAL to move the rotator. When you switch it to manual, you should hear air escaping continuously from the pneumatics near the switch. If you don’t hear this noise, which is quite loud, this means that the rotator is probably interlocked by the telescope control system and you have forgotten to switch to Local mode. If you still don’t hear the noise, it is likely that the air supply to the brake has been turned off, in which case you will have to ask the Jefe de Turno or Telescope Operator to turn it on for you. With the brake released, rotate HiPERCAM to the correct orientation - you may need two people to do this. Leave the switch in the MANUAL position whilst pumping.
If the cable wrap blocks access to the box and it is impossible to move the rotator away from this position, you can detach the two air hoses going into the bottom of the box, as shown in the final photo below. This will release the rotator.
This pumping procedure assumes that the vacuum manifold is already under a reasonable vacuum. If it is not, e.g. because it has not been pumped for a few weeks or more, refer to item 7 below.
To start the pump, simply press the red power button. This assumes the power cable to the pump is permanently attached to the power connector on the rotator patch panel - if it isn’t, you will have to manually attach the power cable from the nearby GTC electronics cabinet. In this case, great care must be taken not to move the rotator whilst pumping.
When switched on, the power button on the pump will light up red and a blue LED will turn on indicating that the pump is operational. You will also hear the backing pump turn on immediately, and the pressure displayed on the front panel (in mbars) will start falling. Within a minute you will hear the turbo whining as it automatically turns on and speeds up, with the whining increasing in frequency as the pressure drops to the low e-5 mbar level. At this pressure, you should hear the frequency of the turbo whining stabilise, which indicates that it is up to speed.
When the turbo is up to speed and the pressure on the pump gauge falls to the low e-5 mbar range, open the valve to the HiPERCAM vacuum manifold. You should see the pressure on the gauge rise slightly and then quickly drop again.
If the HiPERCAM vacuum manifold is at atmospheric pressure, follow the above procedure, but open the valve to the vacuum manifold before you power on the vacuum pump. If you don’t do this, and you open the manifold valve when the turbo is up to speed, you will damage the pump with the sudden rush of air.
Before you can open the vacuum valves on the CCD heads, you need to inspect the head pressures. To do this, open the front of the HiPERCAM electronics cabinet and view the vacuum gauge controllers at the top of the cabinet. The pressures they display are in mbars.
You are now ready to open the vacuum valves to the CCD heads. For the safety of the CCDs, this should be a two-person job, with one person opening the valves and the other watching the vacuum gauges. The person opening the valves must ensure they are hooked onto the railings for safety.
Choosing the CCD head with the highest pressure, and ensuring that the pump pressure is below the CCD head pressure, slowly open the valve on the head, but do not open it fully yet. If the person watching the gauge sees the pressure rise, shout out to the other person to close the valve immediately and then check the pump and vacuum pipe connections for problems. If, on the other hand, the pressure drops, then shout OK and continue to open the valve until you feel it go loose, at which point you should start closing it until you feel resistance and then stop. Then move onto the CCD head with the next highest pressure. Only open the valve to this head if the pressure in the first CCD head is approximately equal to or lower than the head to be opened. In this way, one can avoid reverse pumping from one CCD head to the other. Repeat the procedure until all five CCD heads are open - you can identify which CCD head is which using the sticker on each head or the colour-coded anodised PCB boxes on the rear of each head.
u
CCD1
purple
g
CCD2
blue
r
CCD3
orange
i
CCD4
red
z
CCD5
dark red
If the CCD heads have not been pumped for weeks/months, their pressures are likely to be in the e-2 mbar range. When the CCD head valves are opened, the pressures should drop steadily, getting into the e-3 mbar range within a matter of minutes and then into the e-4 mbar range after a few hours. When the CCDs reach the low e-3 mbar range, it is safe to begin cooling them.
The CCD heads incorporate a getter which acts as an internal pump. This getter becomes saturated with time and hence ineffective. To reactivate the getter, connect the getter PSU to the getter connector on the CCD head using the getter cable. Prior to switching on the getter PSU, ensure that the vacuum pump is running, the CCD is warm, the CCD head pressure is in the e-3 mbar range or lower, and that the valves between the pump and CCD head are open (but none of the other CCD head valves should be open, to prevent contamination). Switch on the getter PSU, which will automatically deliver a current of 6A to the getter, heating it to +400degC and ejecting the gas molecules stored within it, which are then taken away by the pump. You should see the pressure in the CCD head rise rapidly, peak, and then decline rapidly. After 10 minutes, switch off the PSU - you should then see the pressure drop rapidly to the low e-5 mbar range.
The HiPERCAM CCDs are cooled by pelter coolers (or Thermo-Electric Coolers, TECs) that are themselves cooled by the GTC +5degC water/glycol chiller system. The CCD heads are cooled in parallel, fed by two 6-way manifolds mounted on the instrument. The cooling pipes are colour coded, with blue markings indicating the cold water flow and red markings the hot water return. Each of the 6 parallel arms in the cooling circuit is equipped with a flow sensor that cuts power to the associated peltier if the flow falls below 0.3 litres/minute. As a backup, the power is also cut to the peltiers if their hot side goes above a temperature of +50degC.
The small propeller inside the flow sensor has magnets that accrete small magnetic particles in the coolant over time. Eventually, these particles prevent the propeller from spinning, causing the sensor to fail. So it is recommended that at the start of an observing run, prior to cooling the CCDs, each of the flow sensors is opened up and the propellers are cleaned. For instructions, please contact Vik Dhillon.
On first connecting HiPERCAM to the GTC cooling circuit, it is essential to ensure that no water bubbles pass through the HiPERCAM flow sensors, as this could damage them. So, disconnect all 6 of the flow sensors from the manifold and attach a single hose to the manifold to make a circuit. (You may actually find it easier to remove the hoses from the flow sensors rather than disconnect the flow sensors.) Then turn on the coolant flow by turning the yellow lever marked HiPERCAM CCDs underneath the rotator by 90 degrees so that it is pointing down to the ground, ensuring that the red valve next to the yellow lever is fully opened (at which point it reads 4.9 or 5.0). You can tell that the coolant is flowing by inspecting the red propeller mounted above it.
Watch the red propeller for a few minutes, waiting for all of the air bubbles to pass through the system. Then close the yellow valve lever, reconnect the 6 flow sensors to the manifold and then re-open the yellow valve lever.
You can check the flow rate through each CCD head of HiPERCAM, as well as the NGC, on the Honeywell data recorder at the front of the electronics cabinet. The values should all be approximately 1 litre/min. If no flow is indicated in any of the CCD heads, and you have verified that the red propeller is spinning, check that the lap-top style power supply at the rear of the electronics cabinet is plugged in. If there is no flow in just one of the CCD heads, it could be that the flow sensor has failed. Try swapping the offending flow sensor with the spare stored in the HiPERCAM crates.
Once you have flow through all 5 CCD heads, you are ready to cool the detectors using the Meerstetter peltier controllers. To do this, open the front door of the cabinet and press the blue round OK button on the right-hand side of the front panel to display the CCD temperatures. If the coolant is not flowing (or is below a flow rate of 0.3 litres/min), the relays will be tripped in the Meerstetters and a circular power symbol will be indicated on both sides of the CCD temperatures. This means that cooling will not be possible. On the other hand, if the flow is above a rate of 0.3 litres/min, a tick appears on the left-hand side of each CCD temperature, and either an hour-glass symbol or a tick appears on the right-hand side of each CCD temperature, depending on whether or not the CCD has reached its target temperature.
Now set the CCD target temperatures, which are -90degC for CCDs 1, 4 and 5, and -89degC for CCD2 and -88degC for CCD3. You can set the temperatures locally using the Meerstetters, or remotely using
hdriver. Note that CCDs 2 and 3 will not be able to get to these temperatures once the CCDs have been powered on inhdriverunless the GTC have decreased the coolant temperature to +5degC and increased the flow rate to over 1 litre/min by increasing the coolant pressure to 6 bar or more.To set the CCD temperatures locally:
Press the left arrow on the blue control panel as many times as necessary to get to the top-level LDD/TEC screen.
Press the middle blue OK button to display all 3 CCD temperatures.
Press the right arrow to get to the LTR settings screen.
Press the down arrow to get to the Terminal 1 page, i.e. CCD1.
Press the right arrow to get to the temperatures menu.
Press the down arrow 5 times to get to the Target object temperature setting.
Press the middle button to enable the temperature to be changed.
Press the up and down arrows to change the value of the digit, the left and right buttons to change the digit, and the middle button to set the value.
Press the left arrow to return to the Terminal 1 page and then the down arrow to enter the Terminal 2 page, i.e. CCD2. Repeat the above to change its Target object temperature, and then repeat again for Terminal 3, i.e. CCD3.
To set them remotely:
On
hdriver, click on the Settings menu at the top and select Expert Level 1. An additional button will appear towards the top-left hand side of the GUI labelled CCD TECs.Click on the CCD TECs button to enter the temperatures page. Enter the target temperature setting for CCD1, for example, remembering to press carriage return after you have entered the value. Repeat this operation for the other CCDs.
The Meerstetter peltier controller has a ramp function that has been set to cool down (or warm up) the CCDs at a rate of 0.1deg/s. Therefore, it takes approximately 15 minutes to cool the CCDs down. The power output of the Meerstetters has been limited to 85% of the maximum power that can be safely applied to the peltiers. This is the reason why the set temperatures of CCDs 2 and 3 are only -89/-88degC: if set to -90degC, these CCDs would struggle to maintain this temperature and permanently draw the maximum power available, potentially shortening the lifetime of the system.
Once the CCDs have achieved their target temperature, or are within a few degC of it, close the valve on each CCD head. The valves should be closed as tightly as it is possible to achieve using only the tips of ones thumb and two fingers - this minimises the risk of overtightening (which could break the weld between the valve and the CCD head) yet ensures the valve is properly closed. Then close the valve on the manifold. It is then safe to turn off the vacuum pump, simply by switching off the red power button on the pump. The red button light and blue LED display will turn off, and you will hear the backing pump stop immediately and the turbo gradually spinning down. If you used the pump power cable from the nearby GTC cabinet, remove it (you do not need to wait for the turbo to spin down) and attach it to its storage tray using the two velcro strips.
To prevent condensation forming on the exterior surfaces of the CCD head windows, HiPERCAM employs a nitrogen-gas flushing system, fed by the observatory supply via a 5-way gas manifold mounted on the large central plate on the instrument. To check that the gas is flowing, remove one of the black gas pipes going into one of the CCD heads by simultaneously pulling the pipe and pushing the shoulder of the self-sealing connector. Place the pipe near your lips - you should be able to feel a very gentle breeze. If you can’t, check with the GTC staff to see if the gas supply has been turned on. If it hasn’t, make sure you disconnect the main gas pipe feeding the manifold prior to turning the supply on, as otherwise you may send high pressure gas into the manifold, popping the pipes out of their connectors.
Once you have finished on the elevation platform, switch the rotator brake back to AUTO, lock the brake box with the key, and attach the key to the cabinet shelf using the velcro loop. Then switch the rotator control back to Remote. It is essential you clear all loose material from the elevation platform to prevent damage to the mirror if the telescope is moved. Hence you must remove any equipment, tools and notebooks, ensure that both doors of the electronics cabinet are closed, and swing the access stairs back out of the way. Then inform the Jefe de Turno or Telescope Operator that you have finished working on the platform and that the telescope can be moved.
To log the pressures and temperatures during the pumping and cooling procedures, go to the control room and use the DRPC to open a terminal on the rack PC and, in the
/home/insuserdirectory, typehwserver &. (Note thathwserveris also started as part ofstart_hicam- see Software startup - assuming it is not already running.) Then, open a terminal on the DRPC and typehwlogger &, or you can simply double-click on thehwloggerdesktop icon on the DRPC.hwloggersends the CCD pressures, temperatures and flow rates to an InfluxDB database running on pulsar - the desktop PC sitting in Vik’s office in Sheffield. The same PC also hosts a Grafana server, which you can access via the url http://pulsar.shef.ac.uk:3000 to view graphs of the data, as shown in the screenshot below. Please ask one of the HiPERCAM team for the username and password required. The Grafana server also sends automated alerts to the #ops Slack channel for HiPERCAM. It is very important to leaveHWServerandhwloggerrunning at all times, otherwise the Grafana server showing the CCD pressures, temperatures and flow rates will not receive any data.The Grafana server will issue a Slack alert and turn red or amber under the following conditions:
Flow Rate: 0.8 l/min (amber/warning), 0.5 l/min (red/error).
Pressure: 5e-3 mbar (amber/warning), 8e-3 mbar (red/error).
Temperatures (amber warning/red error):
CCD1: -89 / -86 degC
CCD2: -88 / -86 degC
CCD3: -87 / -86 degC
CCD4: -89 / -86 degC
CCD5: -89 / -86 degC
