Archiving data

At the end of each night, the following tasks should be performed to archive the data:

  1. Ensure that data taking and data reduction have completely finished.

  2. Ensure that hlogger has been correctly shut down.

  3. Tidy up the contents of the relevant /home/observer/reduce/yyyy_mm_dd directory on the DRPC, deleting any superfluous files. All data reduction files of the form run0***.ape, run0***.hcm, run0***.red and run0***.log in this directory will be archived.

  4. Open an xterm on the DRPC in an empty virtual desktop and run the script end_of_night_tasks_hipercam. The script is self-explanatory - it only requires you to confirm the start date of the night. The script then:

    • Checks how much data has been generated on that night, and the current capacities of the various archiving disks.

    • Checks that no existing directories with the same night start date already exist.

    • Creates an MD5SUM file of the raw data files to ensure data integrity.

    • Copies the MD5SUM file, the raw data files, the hlogger files, and the reduce files to three locations: the GTC disk rack:/gtcnfs, two portable USB disk drives connected to the DRPC in the control room, and a subdirectory of the /data disk of the form /data/yyyy_mm_dd.

    • Uses the MD5SUM files to verify that the data were copied without error to the USB drives.

    • Removes the raw data from the /data directory on the rack PC in preparation for the next night of observation.

If the rack:/gtcnfs disk is not mounted on the rack PC, the end_of_night_tasks_hipercam script will skip this step of the archiving process. If this happens, do not attempt to run the end_of_night_tasks_hipercam again. Instead, do the following:

  1. NFS mount the /gtcnfs disk on the rack PC. To do this, first type su in an xterm on the rack PC. You will then need to enter the root password - if you don’t know the password, please contact one of the HiPERCAM team members. Then type mount /gtcnfs.

  2. In an xterm on the DRPC, run the /home/observer/scripts/archive_to_gtc_disk script to copy the data to the GTC disk.