HiPERCAM/ULTRACAM/ULTRASPEC logistics

Introduction

HiPERCAM is operated at the WHT and GTC, ULTRACAM is operated at the WHT, VLT and NTT, and ULTRASPEC is operated at the ESO 3.6m, NTT and TNT. All three instruments also spend time in the labs at Sheffield and the UKATC. We therefore have equipment in the UK, Chile, La Palma and Thailand, making logistics a nightmare. This web page is an attempt at improving our documentation and record-keeping in this respect.

Where are HiPERCAM, ULTRACAM and ULTRASPEC at the moment?

• HiPERCAM is at the 10.4m GTC on La Palma. Since Q2 2023 it has been permanently mounted on its new rotator at the Folded Cassegrain G focus. It is now in continuous use.
• ULTRACAM is permanently mounted on the Nasmyth cube of the 3.5m NTT on La Silla in Chile and is in continuous use.
• ULTRASPEC is permanently mounted on the 2.4m Thai National Observatory and is in continuous use.

Backups

The following backups of HiPERCAM/ULTRACAM/ULTRASPEC's various PCs exist. Note that ULTRACAM refers to the ULTRACAM data reduction PC. MAGNETAR is Vik's old (ULTRACAM) lap-top, which has long since been decommissioned. Information on the HiPERCAM backups will appear here shortly.

rsyncs

For ULTRACAM and ULTRASPEC, we have two identical rack PCs, known as UCAM4 and UCAM5 (which replaced the older rack PCs UCAM2 and UCAM3 in March 2010). They contain all of the ULTRACAM and ULTRASPEC system software. The idea is that if one fails whilst observing (which has happened in the past), a user needs simply to replace one with the other and everything should work without any other software/hardware changes.

In practice, of course, we continually make changes to the system software on the rack PC, and hence we need to regularly rsync the two PCs to keep them up to date with each other. The way we used to do this with the old rack PCs was to regularly rsync the PC currently in use onto /data4 in the SCSI disk array. When we decided to synchronise the two PCs, all we then had to do was to connect the other PC to the disk array and rsync it to /data4. This method, as opposed to the alternative of connecting the two PCs together and rsyncing them directly, had the advantage that we always had an up-to-date copy of the latest system on /data4. With the advent of UCAM4 and UCAM5 we kept the same system, but we decommissioned the SCSI disk array and replaced it with a hot-swappable 2TB SATA disk (known as /data1) for archiving data and a hot-swappable 500GB SATA disk for automatic rsyncing every day whilst observing.

In March 2010 we also commissioned a spare IMEDIA PC (known as IMEDIA2) and a spare (Viglen) data reduction PC (known as ULTRACAM2: from now on, I'll refer to the old (aluminium) ULTRACAM data reduction PC as ULTRACAM1). Confusingly, this means there are now three IMEDIA PCs: IMEDIA1-VLT, which is currently stored at the VLT, IMEDIA1-WHT (more commonly referred to as IMEDIA1 on this web page), which is the usual inhabitant of the electronics rack, and IMEDIA2 (the spare, an exact copy of IMEDIA1-WHT).

In 2012-2013, we upgraded ULTRASPEC for use on the 2.4m Thai National Telescope. This involved procuring a completely new rack and data reduction PC (and spares), as we now want to operate ULTRACAM and ULTRASPEC at the same time at different telescopes. The new ULTRASPEC rack PCs are called UCAM6 and UCAM7, and are more or less identical to UCAM4 and UCAM5. The new data reduction PCs are ULTRASPEC1 (aka DRPC3) and ULTRASPEC2 (aka DRPC4).

The table below lists which rack PC, data reduction PC and IMEDIA PC was being used for each ULTRACAM/ULTRASPEC observing run. Information on the HiPERCAM PCs in use will appear here shortly.

Note that the change UCAM2 -> UCAM3 in the middle of the August 2008 run was made to eliminate the SDSU-PCI card as the possible cause of the SDSU crashes we were experiencing. The change UCAM3 -> UCAM2 in June 2009 was due to the PSU failing in UCAM3. UCAM4 and UCAM5 replaced UCAM2 and UCAM3 in March 2010. The change IMEDIA1 -> IMEDIA2 in March 2014 was due to the disk crash we experienced; we had to buy a new motherboard for IMEDIA1, which is now acting as the spare.

The table below records when each PC was last rsynced/cloned and lists any changes made to the system which necessitated the rsync. The direction UCAM2 -> UCAM3, for example, means copying files from UCAM2 to UCAM3, so that UCAM3 is made identical to UCAM2. Information on the HiPERCAM rsyncs will appear here shortly.

Hot-swappable disks

We now have hot-swappable bays installed in all of the ULTRACAM PCs. In this section we record details of the various disks in each of the bays.

• Main (ULTRACAM1) DRPC disk:
  Western Digital RE3 750GB
  Model Number: WDC WD7502ABYS-02A6B0
  Serial Number: WD-WMATV1727404
  
• Main DRPC (ULTRACAM1) clone disk (cloned daily at 14:00UT):
  Western Digital RE 1TB
  Model Number: WDC WD1003FBYZ-010FB0
  Serial Number: WD-WCAW36ZS2CH2
  
• Spare DRPC (ULTRACAM2) disk (occasionally cloned from main drpc disk: last cloned 28/02/2019):
  Western Digital RE4GP 2TB
  Model Number: WDC WD2002FYPS-01U1B1
  Serial Number: WD-WCAVY1833511
  

During Spring 2024, we will be replacing the above ULTRACAM DRPC and its spare. They will have the following disks:

• o/s:
  Model Number: Samsung SSD 870 EVO 500GB Serial Number: S6PYNM0W951758J
• o/s clone (cloned daily at 14:00UT):
  Model Number: Samsung SSD 870 EVO 500GB Serial Number: S6PYNM0W951902T
• o/s spare (last cloned on 04/03/2024):
  Model Number: Samsung SSD 870 EVO 500GB Serial Number: S6PYNM0W951904W
• home:
  Model Number: Samsung SSD 870 EVO 1TB Serial Number: S75CNX0WB28245N
• home clone (cloned daily at 14:00UT):
  Model Number: Samsung SSD 870 EVO 1TB Serial Number: S75CNX0WB31969M
• home spare (last cloned on 04/03/2024):
  Model Number: Samsung SSD 870 EVO 1TB Serial Number: S75CNX0WB31972X

We also have hot-swappable bays installed in the two ULTRASPEC data-reduction PCs:

• Main (ULTRASPEC1) DRPC disk:
  Western Digital Re 2TB
  Model Number: WD2004FBYZ-01YCBB1
  Serial Number: WMC6N0DA6KHA
  
• Main DRPC (ULTRASPEC1) clone disk (cloned daily at 14:00UT):
  Western Digital RE4 2TB
  Model Number: WD2003FYYS-02W0B1
  Serial Number: WMAY04084056
  
• Spare DRPC (ULTRASPEC2) disk (occasionally cloned from main drpc disk: last cloned 13/01/2017):
  Western Digital Re 2TB
  Model Number: WD2004FBYZ-01YCBB1
  Serial Number: WMC6N0E5VVRU
  

We also have hot-swappable bays installed in the two HiPERCAM data-reduction PCs:

• Main (HiPERCAM1) DRPC disk:
  Samsung SSD 850 PRO 1TB
  Model Number: MZ7KE1T0
  Serial Number: S252NXAGA22975P
  
• Main DRPC (HiPERCAM1) clone disk (cloned daily at 14:00UT):
  Samsung SSD 850 PRO 1TB
  Model Number: MZ7KE1T0
  Serial Number: S252NXAGA22974J
  
• Spare DRPC (HiPERCAM2) disk (occasionally cloned from main drpc disk: last cloned 19/07/2017):
  Samsung SSD 850 PRO 1TB
  Model Number: MZ7KE1T0
  Serial Number: S252NXAGA22958E
  

Data

Below is a record of each HiPERCAM/ULTRACAM/ULTRASPEC run and where/how the copies of the data from each run are stored. Note that all of the "Media" are currently stored in the HiPERCAM/ULTRACAM/ULTRASPEC lab at Sheffield . "USB" refers to USB external hard disks. "/ucam" refers to the hard disk storage array located in Vik's office and "SAN" refers to Warwick's storage area network. "RACK" refers to the hard disk storage associated with the HiPERCAM/ULTRACAM/ULTRASPEC rack PCs.

Info on other disks:

1. PT5M_USB1 contains all the qsi data obtained with pt5m between 2012-09-13 and 2013-06-26 and is now stored in the Sheffield USB-disk archive.
2. PT5M_USB2 contains all the qsi data obtained with pt5m between 2011-09-06 and 2012-09-10 and is now stored in the Sheffield USB-disk archive.
3. USB28 (1.5TB, tiny Toshiba) contains all the qsi data obtained with pt5m between 2013-06-26 and 2015-01-14, the date we switched to the Officina Stellare telescope. A backup of all of the python scripts and the log files are also on this disk, taken on 2015-01-14. The disk is now stored in the Sheffield USB-disk archive.
4. PT5M_USB3 (2.0TB, Western Digital, USB3) contains all the qsi data obtained with pt5m between 2015-09-16 and 2021-06-30 and is now stored in the Sheffield USB-disk archive.
5. USB3 is in Vik's office and is used for ULTRACAM backups and storing reduced ULTRACAM data.
6. USB10 died during the October 2007 WHT run - Maxtor replaced it and this is now in the pt5m crates (I think!)
7. USB11 (750GB) is in the ULTRACAM crates and acting as a spare.
8. USB15 is lost! It was returned to Sheffield from Warwick in March 2009 and seems to have disappeared.
9. USB29 (1.5TB, tiny Toshiba) was in the pt5m crate acting as a spare, but I can't find it now! Seems to be lost.
10. USB33 (1.5TB, tiny Toshiba) also contains some backups - silver_drpc_backup/ contains an rsync of the ULTRACAM1 (silver DRPC) with the following directories: admin, cc, observer, rhickman, root, sl, star, trm, ukatc, ultracam_data, vsd. Note that the ultracam_data/ directory contains old backups from IMEDIA1, UCAM2 and UCAM3, plus another folder (ultracam_data/root) with older backups of ULTRACAM1. This disk is now stored in the Sheffield USB-disk archive.
11. USB36 is currently in the ULTRASPEC crates in Thailand, ready to be re-used for the ULTRASPEC+TNT runs in the 2018/19+ seasons. (It currently contains GTO+NARIT data from 13/10/2015 to 01/11/2017, all of which can be safely deleted.)
12. USB38 (1.0TB, tiny Toshiba) is in the pt5m crate on La Palma and awaiting connection to ron to use as the main archiving disk.
13. USB44 (2.0TB, tiny Western Digital) is currently connected to the ULTRASPEC DRPC and contains the NARIT+GTO data from February 2019 onwards.
14. USB45 (2.0TB, tiny Western Digital) is currently connected to the ULTRASPEC DRPC and contains the NARIT+GTO data from May 2023 onwards.
15. USB46 (2.0TB, tiny Western Digital) is with Tim (Somsawat) (or is it now with Stu?) for temporarily storing his MOSCAM calibration data. This will be reused in a future MOSCAM science run.
16. USB60 and USB61 (2.0TB, tiny Western Digital) are currently mounted on the ULTRACAM DRPC and are in use for the period 113 and 114 NTT runs in 2024-25.
17. HIPERCAM_USB1 (2.0TB, Western Digital Elements, USB3) is now permanently stored in the Sheffield lab and contains data from the second HiPERCAM run on the GTC in April 2018 (14th-19th, but no data on 19th due to poor weather). USB1 also contains a copy of the data from the first HiPERCAM run on the WHT in October 2017 (17th-22nd) - these data have been processed by Stu's script to correct the exposure times in the FITS headers. The original, unaltered files have been kept on this disk as well, in the directory orig_data/, but a bug in Stu's script means a few of the exposure time headers are wrong: if you want the uncorrupted orig_data/, use the copy on HIPERCAM_USB2.
18. HIPERCAM_USB2 (2.0TB, Western Digital Elements, USB3) is now permanently stored in the Sheffield lab and contains data from the first HiPERCAM run on the WHT in October 2017 (17th-22nd). The data on USB2 have been processed by Stu's script to correct the exposure times in the FITS headers. The original, unaltered files have been kept on this disk as well, in the directory orig_data/.
19. HiPERCAM_USB11 and HiPERCAM_USB12 (4.0TB, Seagate, USB3) are currently connected to the HiPERCAM DRPC and in use from 2024 onwards.
20. HiPERCAM_USB13 to HiPERCAM_USB20 (5.0TB, WD Elements, USB3) are currently in the HiPERCAM crates at the GTC awaiting use from 2025 onwards.
21. USB62-64 (2.0TB, Western Digital Elements, USB3) are currently being stored in the ULTRACAM crates at La Silla and awaiting use in 2024 onwards.
22. USB65 (2.0TB, Western Digital Elements, USB3) is currently connected to the ULTRASPEC DRPC and contains the NARIT+GTO data from Feb 2024 onwards.
23. USB66 (2.0TB, Western Digital Elements, USB3) is in the ULTRASPEC crates, awaiting use in 2024 onwards.
24. The /data1 disk in the ULTRACAM rack PC (UCAM4) died at the end of March 2022 and was replaced with the spare /data1 disk, which was originally used in the spare ULTRACAM rack PC (UCAM5). A replacement spare has been shipped to La Silla and is currently stored in the ULTRACAM crates at the NTT.