Observers Can Search The Skies At Night!
A system tour by Paul Collins, Electrical Supervisor at Gemini South
Paul Collins is the Electrical Supervisor at Gemini South, an optical/infrared observatory at the top of mount Cerro Pachón in Chile. For many years, Collins was frustrated by his inability to remotely monitor critical systems when the team was unable to travel up the mountain due to inclement weather. After trying many SCADA products, he discovered VTScada software by Trihedral, a comprehensive platform that could communicate with their diverse range of hardware using built-in, direct device drivers like Modbus and SNMP. Collins created a sophisticated application that allows remote observers to see critical information such as dome and shutter positions as well as alerts for rain, earthquakes, and loss of communications. The application even monitors vital systems such as UPS units, fire alarms, and the telescope’s complex cooling systems.
Below, Paul Collins takes us on a tour of the SCADA system he developed using VTScada software.
Some days you just can’t climb the mountain
“The Gemini Observatory is made up of two telescopes. Each one has an 8.1-meter mirror that weighs about 28 tons. Gemini North is on Mauna Kea volcano, on Hawaii’s Big Island and Gemini South is here in Chile. We can cover pretty much the entire North and South hemispheres.”
“I’m currently on Cerro Pachón in Chile. We are up in the foothills of the Andes at 2,700 meters. In the early stages of the telescope, we would have snowstorms and bad weather, where we would lose power or connection and we couldn’t come up for days on end. People would ask me, “What’s going on in the telescope?” I’d have to say, “We don’t know. I can’t tell you.”
“So, I started putting together my own system – a few parts here, a few parts there. So, I could say, yes, we do have power, and yes, this is running, though no, we don’t have that. Over the years, that became a huge package of web-based stuff, stuff I could connect to inside the firewall, and other stuff I couldn’t.”
Finding a unified SCADA software
“For years, I searched and tried all kinds of things to try to join everything together until I came upon VTScada by Trihedral. I downloaded VTScadaLIGHT, their free 50-I/O industrial license. After a couple of days, I put a call into Trihedral and talked to Dave Spencer from their sales and marketing team. He showed me how to do the first couple tags. I just figured it out from there. Later, I took the basic VTScada course at VTScadaFest in Orlando. After testing the free version, we purchased VTScada’s 1,000 I/O, Dual Server Premium package which includes two configuration/runtime licenses, redundant Alarm Notifications, unlimited thin client connections, and training credits.”
“The great thing about this system for me is that I can connect to all my PLCs, but I also have the Modbus and SNMP connections. That brought everything together tremendously for us. I’m quite pleased with the way this system turned out.”
Base facility operations
“We don’t have observers up here on the telescope at night. We used to, but now we do everything remotely. That’s what we mean by Base Facility Operations (BFO). This is the screen I set up for our nighttime observers. When they check this in the early the evening, before they start observing, they know immediately that we don’t have any rain on the summit, and we haven’t had any earthquakes during the day.”
“If we’re up here working at the telescope, we put the system in ‘Summit Mode’. That means there’s no remote control from the base facility. When we leave at the end of the day, we put it in ‘Standby’. Then when the observers come in the evening, they take over and it goes into ‘Base Mode’. This prevents someone in the base facility from controlling any systems while we are working up here on the site.”
The cooling systems
“The main thing for a telescope is cooling. If we lose coolant, it shuts everything down and if you shut down an instrument due to temperature, it could take days to weeks to return it to normal operation. Our instruments operate on a helium system that is cooled by our chillers. This is our cooling system screen. With one view I can see that we have normal power. I see the generator status. My UPSs are online. My chillers are running. Here are the temperature control valves. I see the amperage of my coolant pumps and if my air handling units are off or on.”
The helium compressor system
“Here’s our helium compressor system. We have five separate lines of helium that go throughout our instrument labs and onto the telescope. This screen shows our pressures, coolant temperatures with our coolant flow, and things like that. Creating a screen like this is very quick for me. If I just slapped on the dials, I could throw this together in 20 minutes. But I like to tweak stuff until it looks nice.”
Fire alarm system
“We recently replaced our fire alarm system up here. The alarm company wanted to sell us monitoring software for thousands of dollars. So, we went out and bought a $250 Modbus to TCP/IP converter and there you go. There’s the whole system right there. These are our construction drawings that I converted to PNG and dragged onto the screen in the VTScada Idea Studio.”
“That was really nice because prior to that, if my PLC sent out an alarm that said we had a fire, we had no idea where because it was just a dry contact telling me that a fire alarm was activated. With this I can pinpoint the exact location and see the device number.”
Built-in alarms management
“I think the last time I checked, I had about 280 items in my alarm list, but I only send email notifications on the critical ones. The others I can live with. I set up my notifications and my daily checks so I can click on them to log on my iPads.”
“This is also where I have my maintenance schedules. For example, I’m running a timer on my air handling equipment. I know after so many hours or days, I want to replace my fan belts. So, I get a notice that in two weeks I need to replace the belt on air handler one.”
Plant room pages
“What I’m tying together here is not only my PLC programs but also my fire alarm panel, which is not part of the PLC, air handlers for the data center computer room, and Modbus connections. In the bottom right you can see the UPS units which use SNMP connections.”
Power monitoring
“Here I have my hourly kilowatt consumption, my daily kilowatt consumption, and my monthly kilowatt consumption. I have my coolant pumps and air handlers and their amperages. If the air handler is running but at a lower amperage than expected, it gives me a pop-up alarm to check my fanbelt. The motor is running but it’s not blowing the air and I can tell that by the amperage. At the bottom is the fuel tank for the dual generators on the summit. Next to that are the chiller temperatures, nocturnal temperatures, night low temperatures, and things like that.”
“This is the ECS Enclosure Control System page for our dome. It shows that our shutters are opened up and our vent gates are opened up. Right now, I can tell they’re moving the dome because the amperages are changing. If we have a fault during the night, I can pull up the motor amperage trend and then look at the power trend and put them both on the same screen so I can say, okay, we had a power glitch when this fault occurred.”
“The dome has top and bottom shutters that open so the telescope can look out. Then we have the vent gates which open on the sides to allow the fresh air to blow across the telescope. If we had a top shutter fault, I could drill down to each piece of equipment that drives that shutter to see exactly what happened. You can also see the voltages that are supplying power to the dome itself. We can click that to see if we had any power glitches in the last day or so.”
Expansion to other NOIRLab observatories
“What we’ve done with NOIRLab is, we’ve taken numerous telescopes and grouped them together into one pool of resources. There is no sense in having an engineering group on this telescope and another on that one. We have one engineering group that covers all the sites. It is not only engineering, but all types of personal required to operate a wide variety of telescopes. What I would like to do is take the SCADA program that we’ve proven to be extremely helpful for us and possibly distribute it for use in our other situations. Not necessarily telescopes. We could use it at our offices. I use it in the UPS in our facility.”
https://noirlab.edu/public
A complete SCADA platform
“VTScada is great software. I really love it. I have barely touched the surface of what I can do with it. What I learned about VTScada is how complete of a package it is. There are so many things that Trihedral has already thought about, so I don’t have to. That was huge.”
Learn more about the Gemini Observatory
Try VTScadaLIGHT
VTScadaLIGHT is the free version of Trihedral’s award winning SCADA software, perfect for small industrial and personal applications with up to 50 I/O. Pros and hobbyists alike will love its built-in features and easy learning curve. Minimal PC requirements make it perfect for desktops or Panel PCs connected to small systems like micro breweries or for home automation. Watch the Quick Start Tutorial to get up and running in minutes. When you need more than 50 I/O, its easy. https://www.vtscada.com/light
Images used with permission of The Gemini Observatory.