Remote observing with ULTRASPEC

Vik Dhillon, version: 28 January 2022

Introduction
Remote observing checklist
Remote observing from Sheffield
Remote observing from Warwick
Remote observing from Chiang Mai
Troubleshooting
Contacts

Introduction

This manual describes how to observe remotely with ULTRASPEC at the TNT. It is possible to do this from three different locations - the NARIT headquarters in Chiang Mai, Sheffield and Warwick. This document is an attempt to bring the following information into a single place:

Vik/Dave's guide to Remote observing from Chiang Mai.
Tom's ULTRASPEC remote observing checklist.
Tom's Remote observing on TNT+ULTRASPEC from Warwick.
The various emails that have been flying around between members of the ULTRACAM team.

This document does not give you checklists on what to do in the afternoon/start-of-night/end-of-night, nor does it tell you how to take bias frames, focus the telescope, take flat fields, or what to do when something goes wrong with the instrument. Please refer to the ULTRASPEC user manual for such information. Lots of useful links, e.g. to the blog, logs, filters, pipeline manual, phase II, etc, are given on the left-hand side of the ULTRASPEC homepage. A few additional links that may be useful whilst remote observing:

Useful contact information for the observatory.
Weather at the TNT.
ING object visibility (using the Thai National Observatory option). This gives times of sunset/rise and twilight in UTC and LST.

Remote observing checklist

The section provides a checklist of things to remember when remote observing with ULTRASPEC on the TNT, whether that is from Sheffield, Warwick or Chiang Mai. For general checklists on what to do in the afternoon, start of night and end of night, please refer to the ULTRASPEC user manual.

The checklist below starts from a point where you have successfully followed your site-specific instructions (for Sheffield, Warwick or Chiang Mai) and you now see the DRPC desktop on your own local desktop:

Before starting

Make sure that the status monitor is not running, as this will slow things down. If it is running, type killall gkrellm to stop it.
Make sure no one else is doing anything over the network that will slow you down (e.g. check with top and ps aux for ssh processes). If you need to send a message to everyone of the system, wall as root is a possibility: Type "su -" in a DRPC terminal followed by the DRPC root password, type "wall", then your message ("Dear observer, please log out of the ULTRASPEC drpc as we are about to start a remote observing session") then ctrl-D.
Open the NARIT display showing the TCS, weather, dome webcams, etc. To do this, open a new terminal on your local PC (not the DRPC, unless you are desperate, as this runs over the ssh tunnel and really slows things down) and type anydesk. The AnyDesk window will pop up. If you've run it before, the machine number and password should already be stored in the application and all you need to do is click on the downwards arrow at the bottom right of the black "NARIT THE NEXT STEP" box at the bottom of the AnyDesk window and select "Connect". If this is the first time you've run anydesk, you'll need to enter the machine number and password, which you can find in this file on the ULTRASPEC DRPC: cat /home/star/anydesk_info. Once you've connected, you will see a copy of (part of) the TOs desktop. You can switch between the following screens by pressing the numbered buttons at the top of the AnyDesk window: 0 = web browser (often showing a day/night world map); 1 = NARIT weather graphs and webcam; 2 = internal dome webcams; 3 = TCS display. If nothing appears after you select "Connect", it is likely that there is another anydesk process running, so you'll have to kill all such processes (using ps aux and kill -9) and try again. When you close the AnyDesk window, make sure you do it cleanly by going to the orange AnyDesk icon in the system notification area of the main taskbar (typically located at the bottom of the screen on the right), right-clicking on it and selecting "Quit". If you just close the anydesk window, some background processes keep running.
Check the sunrise / sunset times, e.g. using the ING object visibility page (selecting the Thai National Observatory option). Check also that if you have the full night or not. If you do not, you'll need to be careful with things like vnc, end_of_night_tasks, etc.
Obtain an 'eplanner' plot for the night to work out what you will observe. The easiest way of doing this is to use the generic ones given on the main phase II page under the "eplanner plots" link. Remember greyed out sections indicate mast obscuration which affect the NE direction. You will normally want to avoid this unless the programme can tolerate dodgy photometry (e.g. "does this star have eclipses?"). If you need to create you own eplanner plots, and you've not got eplanner on your laptop, then those observing from Sheffield can run eplanner on europa. Download the "positions and ephemerides", "phase ranges", and "prefixes" files from the main phase II page and then run the command:
eplanner.py targets.lis TNT targets.prg 2020-02-09 -p targets.pfx
Note the filters that will be needed during the night. At the start of the run, it is worth checking that the filters displayed on the ULTRASPEC GUI are the same as those actually in the filter wheel on the telescope. Just ask the TO to check the whiteboard in the TNT control room where the current filter contents are displayed. The TOs have been trained to change ULTRASPEC filters, so you can ask them to do this for you. Just tell them which filter you want to add and which you want to take out. Note that there is a second wheel which can be swapped in and out, so installing your filter in this might be preferable to prevent disturbing the first wheel.
Get some food and drink in place. It might be a long session!

Start of night

Before starting the observing system, open a terminal on the rack (double click the rack icon). cd to "/data", check for and run###.dat or run###.xml files. There shouldn't be any, unless the previous night's observer forgot to run the end-of-night archiving script. If there are, check the files have been backed up (open terminal on drpc, type "df -h" to find usb drives, navigate to where they are and look for corresponding date). If backed up, delete the run*.dat and .xml files in /data. If they are not, create a sub-directory of /data and move the files into it before proceeding. Then send an email to vikcam about this to let people know.
Especially if this is the first night of a run, check for any observing-related processes on the rack (camera, filesave, slogger, FileServer, Lakeshore, usdriver). These need to be closed down before starting the observing system. You will have to kill them ("kill -9 "). Take care when killing processes not to kill important ones!
On a rack terminal, start the observing system and slogger, as described in the ULTRASPEC user manual.
Open skype to "ControlRoom TNO". ("TNO Observer" is an old alternative which might be out of date.) If you don't have skype available on your laptop, Sheffield users can run it on europa by typing "skype", but you will need your own personal skype account to log in. Once logged in, send a "hello" message to introduce yourself to the TOs. If you have a problem getting in touch with the TO, try email: operations@narit.or.th. If you want to see if there is anyone in the control room, try running skype with the video turned on, as that accesses a camera pointing towards the desk where the TO normally sits.
Before the dome is opened, take biases with the following setup:
1. Dome lights off
2. Dome aircon off
3. primary mirror covers closed
4. M4 mirror moved to the 4k camera position
5. lowest throughput filter in, e.g. u
6. focal plane slide to "block"
If you forget any of the above settings, make a note in slogger. Always take full-frame 1x1 and 2x2 biases, and in any specific windowed formats from the preceding night. Remember if you get bad weather in the night, that can always be a good time for biases. Make median biases from these with "makebias".
Check weather at the TNT
Remove FP slide, point M4 at ULTRASPEC, open mirror. If the sky is clear, tell the TO to point at a blank field near the zenith (they should have a list). If more than one filter is in use, start with the lowest throughput filters (e.g. u and z), end with the highest (e.g. KG5). Check with the TO that the telescope is spiralling to allow stars to be removed when taking medians. Once you are done, make the mean flat with "makeflat".
After the sky flat, you might want to take a flux standard, but only if the sky is absolutely clear.
If you can't take biases and/or flats, you might want to try to find some old ones to use during the night.
Move to the first science target.

During the night

For each target, give the TO the telescope RA, Dec and PA from the usfinder-generated chart (not the coordinates listed in the phase II table!). An easy way of doing this is via a skype message. If there is no usfinder chart, you will likely have to move straight to the target RA, Dec coordinates given on the phase II table with PA=0, unless you can quickly run 'usfinder'. If you have a lot of targets in a night, it might be best to make a target list, so you can send it to the TO. Format:
WD0145+234 , 01:48:03.74 , +23:39:16.4
NLTT-11748 , 03:45:26.61 , +17:48:34.5
although you might want to keep your own version with the position angles as well.
Once you are on the target, get an exposure going, set it to have type "acquisition" in usdriver. Check positions with "rtplot". You might want to set the exposure time to be less than you will use during the science run just to get frequent updates.
You need to run rtplot with a file of defects to avoid the more obvious ones. Vik created one during the Dec 2019 run - it is in /home/observer/reduce/DEFECT_hicam.dft. Also, if possible, use a bias and flat in rtplot because it helps the contrast. Set the xlo, xhi, etc region to avoid the extreme edges to cut off any bias strip as it makes percentile image scaling better. Check also that the obvious comparison stars are free of defects. If you do need to move, there is a useful utility on the DRPC called tweakPointingNew, which you can run by double clicking its icon (or right-clicking and selecting Execute) on the left-hand side of the desktop. This allows you to type in the x,y offset you want in pixels along with the PA and returns the required shift in arcseconds. It then asks you if you want to execute these shifts - if you type y the telescope will be moved. Or you can type n and ask the TO to make the offset instead (which will be something like "3 arcseconds West" and "2 arcseconds North"). These offsets are best communicated via a skype message.
Check exposure levels; make necessary adjustments to usdriver settings.
If using drift mode, put the focal plane mask in; if you previously used drift mode, take the mask out!
Switch the data type in usdriver from acquisition to data, start a science run, but don't relax just yet because you want to get reducing asap to enable focussing.
The very first time you run "genred" you might want to run it with "prompt" on the command line to get the full set of prompts. Make sure you do all data reduction in a directory named according to the date of the start of the night in the format /home/observer/reduce/YYYY_MM_DD.
Get reduce going. First thing to do is look at the FWHM seeing to see if you can improve the focus. Note the current focus (should appear on usdriver window). Ask the TO to change. Typical useful increment is +/-0.03, and you can see the current focal position on usdriver. I normally like to change it enough that it is obviously getting worse and to do so on both sides of the focus. So you tell the TO to go to -1.90 or whatever. Do not be tempted to leave off the sign as they will take you literally! Always remember to let it settle so you are getting a good idea of the true FWHM at a given focus position. Always difficult in very variable seeing. You should check the focus on every target, and it might help to plot the seeing of both the target and the comparison, esepcially if the target is faint.
Now you can relax, sort of. Things to be alert for are focus and position drifts. The latter seem to occur particularly around the meridian. You may need to ask the TO to make adjustments, again via skype messages. Try not to let the target drift more than 4 pixels in x and y before correcting it.
Also stay awake for: rotator limit and the need to move the shutter. Both of these will cause reduce to fail and you are best advised to stop and re-start.
Remember to keep a blog of the night: problems, weather etc. These are a good way to communicate what has happened to everyone. However don't use them as an observing log. It's always best to write comments about seeing and clouds into the run-by-run slogger comments file. Remember you can write comments lines as long as you want; just don't insert any line breaks. Good run comments are specific and factual, e.g. "Science. Primary eclipse. Seeing 1.3-2 arcsec, clear." or "Potential pulsator. No obvious pulsations. Seeing 2.3 arcsec, clouds affected 25 mins of the run".

End of night

Possibly another chance for sky flats, although you and the TOs may be wanting to go to bed.
Tell the TO that you are finished and thank them.
Finish the night blog.
Get Paul Kerry's archiving script going on the DRPC by typing end_of_night_tasks.

Remote observing from Sheffield

This section contains specific information for remote observing from Sheffield.

Prior to remote observing from the Department, please make sure you have the following: up-to-date fire, out-of-hours and Covid-19 training (if applicable); out-of-hours U-card swipe permission for the Hicks Building, E corridor and office E13B. Please also make sure that you have read and signed the Physics Health and Safety Policy document distributed via email by Richard Davis. Please also make sure that you have read the Remote observing with ULTRASPEC risk assessment, which was distributed by email. Finally, make sure that you have a mobile phone on you at all times, even when going to the toilet or tea room.

The Sheffield control room is in the Astrophysics postdocs' room, E13B, on the first floor of the Hicks building. You will need your UCard to access the office out of hours - make sure you have out of hours access to the building! Also, make sure you sign in if you're working out of hours, including when staying late on a week day.

If observing during a pandemic, use the cleaning materials provided in E13B to clean your hands and wipe down the work area prior to use, and do the same again when you're finished (including the office entrance door handles). Since you will almost certainly be observing on your own during a pandemic, please follow the lone-worker procedure detailed in the risk assessment. In particular, ensure you email vikcam when you enter the building to start observing, quoting your mobile phone number and the E13B phone number (0114 222 4541), and email again when you've finished observing and are about to leave the building. As a fail-safe, you should also set up a pre-scheduled email, available in gmail using the Schedule send option. This email should be timed to be sent to vikcam around sunrise (Thai time) and informs the team that someone should try phoning you (and, if necessary, University security) to check that all is ok. You must remember to cancel this scheduled email before you leave the building to prevent any false alarms. Of course, if you live with someone, you could instead tell them where you will be, when you are expected home, and ask them to phone University security on 01142224444 if you don't come home at the expected time.

Please now refer to the ULTRASPEC blog for the most up to date information on how to observe remotely from Sheffield.

Remote observing from Warwick

For the most up to date information on how to observe remotely from Warwick, please refer to the ULTRASPEC blog.

Remote observing from Chiang Mai

Where do we observe from in Chiang Mai?
A remote control-room has been set up in the new NARIT headquarters building on the AstroPark. A new, dedicated 4MB/s link has been installed between the TNT and the NARIT HQ.
Where should I stay in Chiang Mai?
There is no accommodation at the AstroPark so you will need to stay at a hotel in Chiang Mai. Apichat usually makes the hotel bookings for us, and he tries to get hotels on the same side of the city as AstroPark to make travelling there easier. Vik recommends the Kantary Hills hotel on Nimman Road, which has a lovely pool. Dave recommends the IBIS Styles hotel on Cholprathan Road. Both are only a short walk from the Maya shopping centre where you can find many noodle bars for breakfast, and do shopping at the Rimping supermarket. Other hotel options are the Eastin Tan, Yesterday and Chiang Mai Orchid (in order of decreasing desirability).
How do I get to the AstroPark?
The easiest way is to use a "Grab" taxi. Grab is the Thai equivalent of Uber. To avoid ridiculous call charges by your UK mobile phone provider, you will need a Thai SIM card. These can be easily obtained at either Bangkok or Chiang Mai airports, and most 7-Eleven shops sell them too, although some of the shops are only allowed to sell them during specific periods of the day. So if you can, get one at the airport. Make sure you make a note of your new Thai mobile phone number, because you will need to input it into the Grab app. Download the Grab app if you don't already have it on your phone. (Do this when on wifi at the hotel to avoid using up your data allowance!). Using the Grab app is easy, you can specify your destination either by typing in the name or on a map. I did have a couple of problems when the GPS location of the pick up point was slightly different to what the driver was seeing on his screen. However, the app shows you a map of where the taxi is, so I simply told the driver to wait where he was and I walked to him. The driver will not give you a receipt. You have to download them from the Grab website.
How do I get into the control room AstroPark office?
Ideally you should arrange for someone to meet you on your first afternoon to show you how to get to the control room (hereafter ACR) and where all the facilities are. Ask Apichat to nominate someone and arrange with them exactly where on site you should meet them. There is a security guard at the entrance to the park, just tell him you are going to the NARIT offices. You will be given a key to the door to the control room. On the day of your departure, this key has to be dropped into the 'Key-drop' box in the stairwell on the way down to the car park. Make sure they show you where the box is.
What facilities are there at the ACR and where do I eat?
In the ACR, there is a kitchen with a fridge, a water dispenser and microwave. The kitchen has cutlery and crockery. You will need to bring your own tea/coffee and milk/sugar if you want hot drinks. And you will need to take your own food. If you are staying near the Maya shopping centre, then I suggest using the Rimping supermarket (see below) and the many noodle bars nearby. If you forget, and the hour is still reasonable you can get food delivered from https://www.foodpanda.co.th/city/chiang-mai.

If you're staying near the Nimman Road area, some suggested places to eat before/after you go to AstroPark are:
- Mix (http://www.mixrestaurantandbar.com): Award-winning Thai restaurant, about 5 minutes walk from the Maya shopping centre. Does fun things like drinks and desserts prepared with liquid nitrogen!
- Maya (http://www.mayashoppingcenter.com): Massive shopping centre on the corner of Nimman Road and Huay Kaew Road. There are loads of fancy restaurants on the fourth floor, and trendy bars on the roof (which has great views). Try out the Lanna Food area on the 4th floor if you're looking for good-value local food. The basement also has some good-value places to eat. There is also a supermarket in the basement, called Rimping, which is open from 6am-midnight and useful for snacks and drinks to take to the NARIT HQ with you.
- Tom n Toms coffee (http://en.tomntoms.com): 24-hour coffee shop on the corner of Nimman Road and Huay Kaew Road - very useful for drinks and food in the early hours of the morning when everything else is closed.
How I connect to the DRPC and Rack PC at the TNT?
The ACR has a dedicated link to the TNT on Doi Inthanon. A photo of the room is shown below. There is the Ubuntu PC on the right, with two monitors, and a Windows PC on the left, which has two monitors on the desk and four mounted on the wall. Use the Ubuntu PC to connect to the DRPC and Rack. You can either SSH to the DRPC (ssh -X observer@192.168.20.144) or you can connect using VNC. Connection using VNC is better if possible, since the graphical display will be much quicker. To connect using vncviewer, start it using "~/vncviewer" and enter the same IP address as above.

Contact a member of the Ultracam team if you have forgotten the passwords.

The windows PC can be used to display the interior and exterior webcams at the TNT, the TCS monitor, and a day-night map of the world which also has UT and LST showing. There are various icons on the primary monitor which open up these displays. They have to be manually dragged and dropped to their respective monitors. Normally the TO will remotely log in to this machine and set these displays up for you, as well as open up a skype connection to the telescope.
How do I use the Ubuntu PC?
The Ubuntu PC has two monitors but also allows four workspaces so that gives a total of eight available screens, within which applications can be dragged and dropped. To enable multiple workspaces, you may need to click on 'Settings', 'Behaviour' and then select 'Enable Workspaces'. You can move between workspaces using ctrl + alt + arrow keys. As of Feb. 2019, we have started using VNC to remotely use the DRPC desktop. To start this, open a new terminal, and run ./vncviewer from the observer home directory. This way, inevitably kicking the plug from its socket under the table won't kill all your processes!
How do I liaise with the Telescope operators at the TNT?
The best way of doing this is via skype on the windows PC in the RCR. First, get a skype account if you haven't got one. Then open skype by clicking the icon on the bottom bar of the desktop and log in. Then call 'TNO observer', ensuring that the speakers in the ACR are turned on. Another option is to use the telephone in the ACR. To call the TNT control room, press the speaker button and then type '611'. As a last resort, use a mobile phone (perhaps via skype to keep costs down). Make sure you get a list of the TO mobile phone numbers and email addresses when you arrive from a member of NARIT staff.
How can I print things?
There is a printer in the ACR but it wasn't working when I was there. This made it more awkward to use finding charts and the eplanner. However, the finders can be emailed to the TO so it is not essential that the printer is working. I suggest you check it when you first get there whilst someone from NARIT is around.
How do I connect my laptop?
The wifi in the ACR is good and the usual passwords work fine. There is also an ethernet connection. Beware of the extension lead under the desk. It is used to power all the PCs but the connector is not very secure and whilst I was unplugging my laptop charger it became dislodged and rebooted the PCs, right in the middle of the end-of-night tasks! So I 'borrowed' the extension lead from the TV room and plugged my laptop and phone charger into that.
What if I want to go up to the telescope?
This can be arranged with NARIT - it takes 1.5 hours to drive to the summit if there is no traffic. If you don't want to remotely observe, you can ask NARIT to book you a room in the Lodge at Doi Inthanon instead.
Where are all the printed finding charts, object catalogues and user guides?
These are currently in the ULTRASPEC folders in the control room at the telescope. If you want them, you'll have to ask someone to bring them down from the mountain for you the following day.
How do I get the weather information up?
- TNT weather information can be found at: http://1.179.136.234/weather/ or http://192.168.20.51/weather/
- Weather forecasts can be found at: http://www.meteoearth.com/#/,98.47,18.24,100.75,8,1,1,0,0,0,time=1449598582 or http://www.mountain-forecast.com/peaks/Doi-Inthanon/forecasts/2565
- Satellite images can be found at: http://www.sattmet.tmd.go.th/satmet/mergesat.html
How do I set up the wall-mounted screens? (Added by MJG)
The Windows PC controls the 4 wall-mounted screens, which display information on the current status of the telescope and have a default setup indicated by stickers attached to them. If it crashes you'll need to log in to the TCS to restore the displays; you may need to ask the TO for the password.
- Weather and Time are under the 'widgets' section on the TCS
- Telescope status is under the 'Telescope status' section on the TCS - choose 'Desktop Display'
- Dome cameras: These can be seen using a separate program called Video Viewer. (Viewing the dome cameras via the TCS didn't work when I tried, but this might only be a temporary problem.)
What should I do if it's cloudy?
If you can't work for whatever reason, and if it isn't too late in the night, try the cinema on the 5th floor of the Maya shopping centre opposite the NARIT HQ. The films in English usually have Thai sub-titles rather than being dubbed, apparently. You could also try the famous Night Bazaar (http://www.visitchiangmai.com.au/night_bazaar.html) or the Sunday Market (http://www.visitchiangmai.com.au/sunday_market.html), which both run until 11pm-ish.
Finally, a few top tips:
- Try to avoid arriving on a weekend or a public holiday - there may be no-one around to help you get set up for remote observing.
- If you are observing alone, make sure you have the mobile phone numbers for Apichat and someone else from NARIT, in case of difficulties.
- Chiang Mai is hot, and can be wet, so bring appropriate clothing. But also be prepared with warm clothing in case you have to go up to the cold mountain top if the link fails or if there are problems with the instrument.
- Make sure you have a head torch with you, as the ACR can be dark in places.

Troubleshooting

For problems specific to remote observing, please check the list below. For more general instrument problems, please refer to the troubleshooting section of the ULTRASPEC user manual.

ssh tunnel does not work

The DRPC may not be booted or the external network could be down. You need to ask the TO to check the latter by logging in as the observer user and using firefox to browse to a site at Sheffield (or Warwick, if observing from there), e.g. https://vikdhillon.staff.shef.ac.uk. If this page opens, then the link from Sheffield/Warwick should be fine. Note that the tunnel software is constantly monitoring the connection and if it drops it will automatically try again forever without any user intervention - rebooting the DRPC just delays this happening. Hence only reboot the DRPC as a last report if nothing else works.

vnc doesn't work

You may find that someone else is vnc'ed into the DRPC at the same time as you, e.g. from the control room in Chiang Mai, so that the two of you can both move the mouse. In this case, either ask the TO to find out who it is and ask them to kill their vnc session, or open a text editor on the DRPC that everyone would be able to see and type, for example, Vik Dhillon here. I am using ULTRASPEC now. Please could you disconnect your vnc session? Thanks.

If the DRPC physical keyboard has numlock or caps-lock enabled, vnc will also use this, so ask the TO to press the relevant button on the DRPC to turn it off.

Finally, it is possible someone has switched desktops by accidentally hitting ctrl-alt-f2, for example. This would make vnc appear as a black screen with a login prompt at the top. To fix this, ask the TO to type ctrl-alt-f7 on the DRPC in the control room, which will hopefully bring the desktop back.

everything is running really slowly, or freezes completely

The status monitor can slow things down, so make sure that it is not running by typing killall gkrellm.

If the link freezes completely, it could be because the vnc connection has gone down. Find the terminal in which you made the ssh tunnel (on the Warwick PC this can be found by clicking "activities" in the top-left of the Warwick PC, and raising the original terminal where you typed "tunnel"). The terminal may show that the connection has dropped. If so, try to re-establish the connection by re-opening the ssh tunnel. It may say "connection to port 40002 (or 50002 at Warwick) refused", or similar, a few times, but persist, because it can be a little slow to re-establish the link. Any run you have going will keep running, and you may have to ask the TO to stop the run directly on the control-room PC if you can't get things going again. Note that the software on the DRPC will automatically attempt to re-open the incoming tunnel to your local machine.

reduce light-curve plot corrupted

This could be due to a change to ignore bad times for hipercam in the pipeline, which has a bad effect for ULTRASPEC. You may have to start with first=5 to get past riduculous times in the past. Tom hopes to fix this at some point.

Really high readout noise

Sometimes the readout noise can be extremely high, so high in fact that a diagonal pattern can even be seen in the flat fields. This is possibly because the TO has forgotten to turn off the air conditioning in the dome. Ask them to do so, even if they do complain that the humidity is high (as there is little point in air conditioning the dome when it is open!).

Contacts

If you experience problems with remote observing with ULTRASPEC, please email/phone one of the contacts.