Archiving data

Paul Kerry, version: 2004-05-04

Disks available

Even though the hard disks are large within the ultracam computer systems, you must remember that they are not infinite voids of empty space.
You must regularly monitor disk usage (especially on ucam2 which can create 6GB/hr) by using the df -h command. If disk capacity is above 90%, then I/O is impaired. If for example, the capacity of /data on ucam2 hits 100% while data is being written, then the system will crash. You should either move, delete or compress files before this situation arises.

ultracam disks

(information taken from df -h)

The /ultracam_data disk is for copying data from ucam2 so backups can be created.

The /archive disk is for long term storage of data files. You should read the further instructions on the use of /archive.

The /win disk is the Windows XP Professional disk. Files can be copied from this disk but not to it.

ucam2 disks

(information taken from df -h)

The /ultracam disk contains the control software.

The /data disk is where the instrument writes data to (the ultracam software actually writes data to /ultracam/data - do not remove the data symbolic link in /ultracam).
Ultracam raw data is created in two files, a data file (.dat) and a xml header file (.xml). Both files are saved with the prefix "run", followed by a number from 001 to 999.

/data1, /data2, /data3, /data4 and /data5 make up the SCSI data array which is housed in the lowest crate in the rack unit. The SCSI data array is not a RAID system. These disks are for saving recent data from the /data disk. The filesystem in use on these disks is reiserfs (the other disks are ext3 filesystems).

Copying data from ucam2 to ultracam

At the end of the night you should

• (from ultracam) copy the run files and the AutoLogger html file from the rack PC to the data reduction PC so that you have a supplemental backup. This can then be used to create tapes or DVDs.

on ucam2

• du -hsc /data/run*
  
  noting the range of run files and the size reported

on ultracam

• df -h /ultracam_data
  
  noting the space available. Make sure that the capacity will not exceed 90% when the files are copied. You may use your home directory (use df -h / to check capacity) if space is limited. Do not copy files directly to the /archive disk.
  
  

Copying the files may take several hours as the network speed is around 10MB/sec.
DO NOT USE scp -r AS THIS COULD CRASH THE COMPUTER BY OVERLOADING THE FILESYSTEM BUFFERS.
You can automate copying the files by using a script that is provided as a template.

on ultracam

• mkdir ~/bin
• cp /home/star/ultracam_scripts/scp_files_from_ucam2_data ~/bin
• using your favourite editor, edit ~/bin/scp_files_from_ucam2_data making changes as specified in the file.
• rehash
• scp_files_from_ucam2_data

General notes about writing data files

DDS tapes, DVD-R and CD-R disks are kept in one of the ULTRACAM packing crates.

It is likely that you will need to use multiple DVD-Rs or tapes to write an entire observing run.
If you are observing on multiple nights, then it is likely you will end up with files of the same name, ie run001.dat and run001.xml from 1st Jan and run001.dat and run001.xml from 2nd Jan. Be careful that you do not copy files/create links with the same name onto your media. It is a good idea to create separate subdirectories using the date of observation for the directoryname.

Tape Drive

The tape drive within ultracam is a Sony SDT-10000 DDS4 20GB (native) device. This also supports all lower DDS versions.
If the status light is on and then flashes off every 4 seconds, then the drive needs cleaning. There is a cleaning cartridge within the computing box file. Simply insert the cleaning tape and wait a few moments - the lights will flash while cleaning is in progress and then the tape will eject automatically. All the lights should now be off. Please write the date on the label so there is a record of how many times the tape has been used.

The tape drive names are

• /dev/st0
  
  for the rewind after writing device (if used incorrectly it is possible to rewrite over data just written).
  
  
• /dev/nst0
  
  for the non-rewinding after writing device (you probably want this one and it is set to the TAPE environment variable).
  
  

Tapes are written in the following style

• start of tape marker, datafile1, marker, datafile2, marker, datafile3, marker, etc

When using multiple tar files on a tape, a marker is automatically inserted when you start the next write.

Useful tape drive commands (man mt for more information)

• mt status
  
  gives drive status and other information.
  
  
• mt fsf
  
  fast forwards the tape to the beginning of the next marker.
  
  
• mt rewind
  
  rewind the tape back to the beginning.
  
  
• mt offline
  
  rewind and eject the tape - make sure that the case door is open first!
  
  

Writing files to tape

tar is an easy way of writing tapes.
At the time of writing this guide, the version in use is GNU tar 1.13.25-2 from the debian distribution.

• tar cvf /dev/nst0 directoryname
  
  write a standard tar file to tape.
  
  
• tar czvf /dev/nst0 directoryname
  
  write a gzip compressed tar file to tape on the fly.
  
  
• tar czvfh /dev/nst0 directoryname
  
  follow symbolic links and include the file rather than just including the link.
  Use the same method that is under DVD creation
  
  

other useful tar options can be found via man tar or info tar but selected ones are

• --label=ARCHIVE-LABEL (ie tar czvf /dev/nst0 /home/star --label=home_star_backup)
  
  
• --exclude filename (ie tar czvf /dev/nst0 --exclude /home/star/massivefile.ps --exclude /home/star/trashdirectoryname /home/star)
  
  

GNU tar uses relative archives by default. This means that you can extract a tarfile into any location. Other tar programs usually use absolute archives which require root access on a machine to create the complete path from "/".
It is a good idea to write on the tape which method you used to write the tape in the first instance - in the future you may not remember how you did it and you will no doubt be pestering your system manager!

Verifying tapes

• tar tvf /dev/nst0
  
  verify a standard tar tape.
  
  
• tar tzvf /dev/nst0
  
  verify a gzip compressed tar tape.

Extracting tapes

• Be sure to use df -h /ultracam_data to check for available capacity before running this command
• cd /ultracam_data/username
• tar xvf /dev/nst0
  
  to extract a standard tar tape.
  
  
• tar xzvf /dev/nst0
  
  to extract a gzip compressed tar tape.
  
  

DVD-R Drive

The DVD writer is an IDE interfaced Sony DRU-500AX which supports CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-R, DVD-RW, DVD+R and DVD+RW.
At the time of writing, Linux software writes DVD-R and DVD-RW formats which are based on the standards from the DVD Forum. The other formats are available from Windows if required.

Currently supported maximum write speeds are...

Take note of the maximum write speed of the media you are using. DVD-R disks must be specified as being 4x capable to achieve the maximum speed otherwise the drive defaults to a lower writing speed automatically.

Take note that the capacity of media may not be a true representation of the filesystem equivalent size. 4.7GB DVD-R disks for example, hold around 4700000000 bytes whereas the true value of 4.7GB is 5046586572 bytes. This can cause confusion if you are using commands like ls -h, df -h or du -h. When creating iso images, filesystem information is also written onto the disk which takes a small amount of capacity.

If you do use the -h options, make sure that the files are 4.4GB or under.

The Yamaha SCSI CD-R drive is now disconnected and is retained as a spare drive in case of equipment failure.

Creating a CD-R

• Use df -h to check capacity before continuing.
• mkisofs -l -r -o /home/username/cd.iso path
  
  Note that the -r option is now preferred over -R. Please review the manual pages for further information.
  You can use symbolic links with the -f option, using the same method that is under DVD creation.
  
  
• cdrecord -v driveropts=burnproof dev=1,0,0 speed=16 /home/username/cd.iso
  
  speed=16 is for University of Sheffield Intenso disks - the speed may need adjustment for different media.
  It takes approximately 10 minutes to write a full CD-R.

Creating a DVD-R

All data runs obtained in a night should be written uncompressed to DVD-R. This is the master copy and is kept at the University of Sheffield.

Always label the DVD BEFORE WRITING in case you scratch the surface layer. You should have the following details on both the disk and jewel case...

• ULTRACAM DATA ARCHIVE
• YYYY-MM-DD
• DISK ??? OF ???
• RUN??? - RUN???
• ??? GB
• AUTOLOGGER LOGS

It takes approximately 15 minutes to write a full DVD-R at 4x speed.

Observers can make their own DVDs containing just the data they require.

Make sure that you verify the DVD after burning.

Instead of copying large amounts of data around the system, symbolic links can be created. These are simply a pointer to a file in another directory.

A symbolic link is created by the following command...

• cd to a directory where you want the link to be created
• ln -s /directory/filename
  
  this will create a link with the same name as the original file.
  for more options see man ln or info ln
  

If you use rm on a symbolic link, then the link is removed and not the actual file.

Examples of symbolic links can be found in the /home/star/dvd/2004-04-29/sym_links/disk1 directory on ultracam.

You can run full archive backups while the system is being used by observers if you use the nice command, examples of which are...

• nice mkisofs -f -l -r -o /ultracam_data/star/2004-04-29_disk1.iso /home/star/dvd/2004-04-29/sym_links/disk1
• cdrecord-prodvd-2.01a24-i686-pc-linux-gnu -v -dao -driveropts=burnfree dev=1,0,0 speed=4 /ultracam_data/star/2004-04-29_disk1.iso
• rm /ultracam_data/star/2004-04-29_disk1.iso
  
  

Do not use the other DVD drive or the /archive disk while writing a DVD as you may cause a buffer underrun (this is caused by simultaneously accessing IDE devices on the system and reducing data throughput).

Do not put nice before the cdrecord-prodvd command.

If you are unsure as to the speed capability of the disk, you may still specify speed=4 as the software automatically detects the speed by probing the disk when inserted in the drive.

Note that -r option is now preferred over -R. Please review the manual pages for further information.

By using the -v option, the current status of the input and output buffers can be viewed as well as the writing speed and MBytes written. Do not worry if it appears that cdrecord-prodvd has hung at the start - it takes at least sixty seconds before the status information is echoed to the terminal. If the input or output buffers go to zero percent, then the disk will be corrupted. You can usually increase the buffer values by stopping other processes running on the system.

A typical successful writing session's last few lines of output should look like this...

• Starting new track at sector: 0
  Track 01: 3400 of 3400 MB written (fifo 100%) [buf 89%] 4.6x.
  Track 01: Total bytes read/written: 3565748224/3565748224 (1741088 sectors).
  Writing time: 724.278s
  Average write speed 3.6x.
  Min drive buffer fill was 86%
  Fixating...
  Fixating time: 47.536s
  cdrecord-prodvd-2.01a24-i686-pc-linux-gnu: fifo had 56165 puts and 56165 gets.
  cdrecord-prodvd-2.01a24-i686-pc-linux-gnu: fifo was 0 times empty and 9655 times full, min fill was 84%.

If the "Min drive buffer fill" hits 0%, then the DVD is corrupt and you will have been probably running too many processes on the system. Try again with a new disk, making sure the system is in a quiescent state.

If you get a large amount of SCSI errors instead of the output above, then the media itself is most likely to have a fault on the writing layer. Try again with a new disk.

Walk-through

You will more than likely require multiple disks so you need to work out which files will go onto which DVD.

• cd /ultracam_data/username/yyyy-mm-dd
  
  Look closely at the directory contents - there should be two log files and two files for each run - if not, then you may have to scp files from ucam2 again.
  
  
• du -shc yyyy-mm-dd_log.dat yyyy-mm-dd_log.html run*
  
  This shows all the files that need to be written to DVD and their total size.
  
  
  You should then decide on the run numbers that you will write to each disk along with the log files, making sure that the total filesize is 4.4GB or under. Examples are...
  
  
• du -shc yyyy-mm-dd_log.dat yyyy-mm-dd_log.html run00*
• du -shc yyyy-mm-dd_log.dat yyyy-mm-dd_log.html run00[1-5]*
• du -shc yyyy-mm-dd_log.dat yyyy-mm-dd_log.html run00[6-9]* run01[0-3]*
• du -shc yyyy-mm-dd_log.dat yyyy-mm-dd_log.html run01[4-9]* run02*
  
  It is a good idea to paste the output into a text file so you can tell which files are been written to which disk. An example of such a file is /home/star/dvd/2004-04-29/2004-04-29_disk_info on ultracam.
  
  Always write the data files in numerical order - do not put small runs out of sequence onto disks to save space.
  
  Always write both log files to each disk.
  
  If the size is under 650MB, then use a CD-R instead of a DVD-R. Follow the "Creating a CD-R" section, remembering to add the -f option.
  
  

In a different directory on ultracam, you are going to create a range of symbolic links to the files instead of copying large files around the system.

• mkdir ~/dvd/yyyy-mm-dd
  
  
• mkdir ~/dvd/yyyy-mm-dd/disk1
  
  You will probably need to create more directories for each DVD required calling them disk2, disk3, disk4, etc.
  
  
• cd ~/dvd/yyyy-mm-dd/disk1
• ln -s /ultracam_data/username/yyyy-mm-dd/yyyy-mm-dd_log.dat
• ln -s /ultracam_data/username/yyyy-mm-dd/yyyy-mm-dd_log.html
• ln -s /ultracam_data/username/yyyy-mm-dd/run???.dat
• ln -s /ultracam_data/username/yyyy-mm-dd/run???.xml
  
  Instead of doing this for each file, you can create a range of symbolic links at the same time by using a perl command (simply substitute whatever numbers you like for the run numbers you want in the open brackets part)
  
  for run001 to run009...
  perl -e 'foreach $a (1..9) {`ln -s /ultracam_data/username/yyyy-mm-dd/run00$a.dat`}'
  perl -e 'foreach $a (1..9) {`ln -s /ultracam_data/username/yyyy-mm-dd/run00$a.xml`}'
  
  for run010 to run099...
  perl -e 'foreach $a (10..99) {`ln -s /ultracam_data/username/yyyy-mm-dd/run0$a.dat`}'
  perl -e 'foreach $a (10..99) {`ln -s /ultracam_data/username/yyyy-mm-dd/run0$a.xml`}'
  
  for run100 to run999...
  perl -e 'foreach $a (100..999) {`ln -s /ultracam_data/username/yyyy-mm-dd/run$a.dat`}'
  perl -e 'foreach $a (100..999) {`ln -s /ultracam_data/username/yyyy-mm-dd/run$a.xml`}'
  
  
  After you have finished creating the links, you must make sure that they are correct by typing ls -la. There should be two links for the log files and two links (one dat, one xml) for each run.
  
  
  
• df -h /home /ultracam_data
  
  This will show you what the disk capacity is - make sure that this will not exceed 90% when the files are created. You may use the /home directory if there is insufficient space on /ultracam_data.
  
  
• nice mkisofs -f -l -r -o /ultracam_data/username/yyyy-mm-dd_disk1.iso ~/dvd/yyyy-mm-dd/disk1
  or
  nice mkisofs -f -l -r -o /home/username/yyyy-mm-dd_disk1.iso ~/dvd/yyyy-mm-dd/disk1
  
  This command will create the image following the symbolic links that you have created. You can leave off the nice if the system is not being utilised.
  
  
• cdrecord-prodvd-2.01a24-i686-pc-linux-gnu -v -dao -driveropts=burnfree dev=1,0,0 speed=4 /ultracam_data/username/yyyy-mm-dd_disk1.iso
  
  This command will write the disk. If the image is too large, the software will abort before writing so the disk will not be corrupted. The output should be like the earlier example.
  
  
  
  Repeat the procedure for the set of disks required. You can now go onto the verification stage.
  
  
  

Verify the DVD

Do not verify a DVD while writing another DVD as you could cause a buffer underrun to the disk being written.

You should have written all of the information on the label before verification.

Eject the DVD from the writer and put it in the DVD-ROM drive which has a much faster access time.

• mount /mnt/cdrom
• nice diff -r /mnt/cdrom /ultracam_data/username/yyyy-mm-dd
  
  You can leave off the nice if the system is not being utilised.
  
  If you were able to get all the files onto a single disk, then you should have no further output unless the disk is corrupt.
  If not, you will get output telling you that certain files are missing. This is fine as you have written a subset of the entire data set for the night. If however, any files differ, then the DVD is corrupt.
  
  In either case of corruption, you should remake the image files and try again.
  
  
• umount /mnt/cdrom

Eject the DVD and you are finished!

The /archive disk

The 187GB /archive disk should be used when /ultracam_data (or /home) is getting full.
There is a performance decrease compared with the SCSI disks in use within the ultracam system as /archive is an IDE disk. It is possible to overload the filesystem buffers if you try to copy too much data at the same time. You should therefore copy files individually or use a small group of files. It is a good idea to run diff to check the files afterwards.