Southwest Florida Water Management District turns to VTScada for critical flood control system
Over 4.5 million people depend on the Southwest Florida Water Management District for water conservation and flood protection, especially during the six-month hurricane season. Water conservation and flood control involves carefully holding and channelling surface water using a system of gates, canals, and reservoirs.
When Mike Bartlett became the SCADA Systems Administrator, the utility decided that the current SCADA system was not performing adequately. A search began for a single supervisory control and data acquisition (SCADA) software package that could communicate with their current hardware and would be flexible enough for the team to build upon as the district continued to grow. Additionally, the software vendor would need to be able to provide 24/7 access to technical support for six months of the year. In this article Mike Bartlett describes some of the strategies he and the team used to get the exact SCADA system they needed.
Southwest Florida Water Management District (SWFWMD)
SWFWMD is one of five water management districts created by the Florida Department of Environmental Protection to manage water quality for sixteen counties covering over 10,000 square miles. One of the District’s many responsibilities is to provide flood protection to low-lying areas through the operation of flood control structures. The largest of these is the ‘Tampa Bypass Canal System’ (TBC) which provides primary flood control for the cities of Tampa and Temple Terrace.
Mike Bartlett is currently the SCADA Control Analyst with the District. “I have worked for SWFWMD for eighteen years; four as the senior electrician, five as a network/telecommunications technician, four as the SCADA Systems Administrator, and five as the Structure Controls Analyst.” In 2004, he led the team that upgraded the Supervisory Control & Data Acquisition (SCADA) system that provided monitoring & control for the district’s water conservation and flood control infrastructure.
The Legacy SCADA System
“We started off in 2004 with eighteen or nineteen sites that required remote control. As far as data collection was concerned, we had over 250 sites and approximately sixteen hundred data collection sensors recording rainfall and water levels.” This data was collected by a network of data loggers and transmitted back to the server via a proprietary VHF network and telephone lines. Remote control of the water conservation structures was handled by Programmable Logic Controllers (PLCs) communicating via telephone lines to the operator’s laptop which was running a separate HMI application.
At that time the operators required two different software applications to perform even limited monitoring and control of the system. “We were remotely controlling the structures through a ladder logic program independent of the SCADA system that was really just a Data Acquisition (DA) system. The supervisory and control portion of the application never worked to our expectations. We were not able to see our data and operate our structures at the same time.”
The Goals for the new SCADA system
- “When I came aboard as the new SCADA Systems Administrator, my goal was to find a SCADA application that would meet our needs and more importantly take us into the future on a single software platform. We wanted one application to do everything.”
- “Our plan was to have remote control capability at all of our structures first. The next phase will be to add automation.”
- “The other thing on our wish list was the ability to keep our current hardware and communications network components.”
- Bartlett also wanted a single dedicated point of contact for technical support that they could rely on 24/7 during hurricane season.
The Selection Process
“The Hydrologic Data Section Manager, Granville Kinsman and I took an extended period to write a detailed ‘Request for Bid’ that laid out what we wanted in the system. The RFB we wrote was approximately sixty-five pages long.” One of the most important selection criteria requested in the RFB was a working ‘proof of concept’ application that incorporated all the key elements of the finished system. After carefully evaluating each of the submissions; the team selected the bid from Trihedral Engineering Limited. This decision was based on several factors. Trihedral’s was the only bid that addressed each item in the 65-page RFB
“Not only did they address everything, they had the most streamlined and efficient system in the selection process.” For example, the Trihedral design required the fewest servers. “When we sat down with [one of the other bidders] we were told we needed six servers just to do what one VTScada server could do.” The district would be spared the expense of buying all new PLCs, radio transmitters, and modems. With VTScada’s open architecture, the District could also begin to incorporate a variety of equipment manufacturers as they moved forward with their automation plans.
Granville Kinsman had seen Trihedral and VTScada at a trade show. On Trihedral’s recommendation the team decided to visit a utility that was already using the software and had an in-depth knowledge of its use in a working environment. “We went to [the city of] Ocala without Trihedral to look at VTScada and talk with the operators and developers. They were using it for lift stations. We just sat there in awe. It was the just what we were looking for.” What most impressed Bartlett and his team was Trihedral’s proof of concept.
“They went way above and beyond. They asked us to send them examples of all of our equipment. They set up mini networks and wrote all the drivers. When Trihedral demonstrated their VTScada application for us, it worked perfectly the first time. It was collecting data and controlling our structures both across our telephone network and across our proprietary VHF network.”
Implementation
The thoroughness of Trihedral’s proof of concept meant that the major elements of the design phase were already complete. Once the bid was accepted, Lawrence MacNeil, Engineering Manager at Trihedral Engineering sat down with the staff to coordinate the addition of the remaining structures.
“Once the District was ready to install VTScada we were collecting data and remotely operating our structures in less than 48 hours. I’d say within 30 days we had the system tweaked and running at one hundred percent. If there were challenges [for Trihedral] in implementing the system, they were transparent to us.”
The Transition
Ordinarily, when a mission critical SCADA system is replaced, the old and new applications can run side-by-side for a period to ‘shake down’ the new system and allow operators time to adjust. However, this was not possible in this situation. “We did not have the ability to run two systems at once due to the propriety VHF system which was fifty percent of our data collection. So, when we crossed over it was all or nothing. That was the importance of proof of concept. It had to work the first time.” The transition to VTScada was accomplished without incident. “We put Trihedral in a very difficult spot. It seems like Trihedral does its best work under pressure.”
The New System
The new application is shared by staff from both the Hydrologic Data and Structure Operations sections. “We wanted both these sections to have full control of their own worlds within VTScada.” To meet this mandate, the application provides different views of system data for each. The Structure Operations staff can see graphical displays that allow them to zoom in and out of a district map and select the sites they want to see. Hydrologic Data staffs can view data in a tabular format. “There was quite a bit of flexibility built into the system. It’s easy to use and it’s easy to walk other people through.”
The SCADA system runs on three servers. “We have a primary and a secondary server in Brooksville. They are hardware redundant and there is automatic failover between them. We also have backup servers in Tampa and Bartow in case we must move due to a hurricane. All the Tampa server does is keep its database updated until it is needed.” Bartlett and MacNeil also discovered a way to use VTScada to capitalize on underused server resources. “Each one of those three servers has a bank of eight telephone modems. VTScada’s Alarm Dialer allows us to increase our telephone network threefold by pooling those resources into a single modem pool. So, instead of having eight modems in our system we can have anywhere from 16 to 24, all collecting data. This decreases our polling time drastically.”
Remote system access
SWFWMD staff members have a variety of remote access options. “We have the ability to access the primary system through a VPN connection from anyplace with Internet access. In the field I use a broadband Sprint card and at home I use a cable ready modem connected to a broadband router.”
- Ordinarily, personnel access the application through a web browser using VTScada Internet Clients running on the district Intranet. A Citrix server allows authorized users to access these thin clients from outside of the network. “The nice thing about accessing VTScada through Citrix is that it’s lightning fast.”
- “As a last resort we also have the ability to access the system through a landline. So anywhere I have an Internet connection or a phone line, I’m in. The flexibility for remote use is tremendous.”
Advantages
- Operators and developers now can monitor water levels and control conservation and flood control structures from a single software application. In addition to improving workflow, this significantly reduces the amount of training needed for new staff.
- A single software package also means a single point of contact for technical support.
- SWMD did not need to spend money on new field equipment. “One of the great things about Trihedral is their ability to create drivers for our proprietary equipment; specifically, our phone line and VHF communication devices and our PLCs.”
- “The new system is very intuitive from an operation and viewing point of view. It’s logical and strait forward. The way the system was designed for us there was no need to constantly move from page to page to page.”
In-house Development
SWFWMD application developers attended a week-long training course provided by Trihedral Engineering. “Like any piece of software, once you have figured it out it became very easy to use. We develop at a low level. We are not writing scripts or designing new tags. We depend on Trihedral for that. What is amazing is the ability to move and copy existing tags around to create our own pages where we can look at data tags and multiple control points for systems spanning two or more counties all on one screen.”
“We were able to put the whole Tampa Bypass System on one page on a nineteen-inch monitor. From that screen we have full control of the system and can view all the required data needed to make operational decisions from one page. We can see all the information needed for a system that’s approximately sixteen miles long. We were able to build that page and manipulate the tags ourselves.”
Public access
Florida’s ‘Government-in-the-Sunshine’ policy ensures public access to information that effects the decisions of state law makers. This includes data pertaining to water quality and flood control. For this reason, one of the requirements of the new system was a way for the public to access information from the SCADA system online. Trihedral accomplished this by setting up VTScada Internet Client access with read-only permissions. By following a link on the SWFWMD website, the public can view the same real-time information used internally by operators. “This is how easy and intuitive it is to use. We just put [the link] out there and people just started using it.”
The Siren System
In 2006, the Operations team began identifying highly populated areas that could be affected by a water control structure failure. These ‘hot spots’ were outfitted with early warning sirens identical to those used in many of Florida’s nuclear power plants. “Our water conservation and flood control structures are regularly maintained and are in excellent condition. The sirens are an enhancement to our Emergency Action Plans for those structures. We ran into difficulty with the vendor providing us information to integrate the sirens into VTScada so Trihedral opted to control the sirens via a PLC.”
Technical Support
“I cannot over emphasize how incredible Trihedral’s support is. It is the best in the industry as far as I’m concerned.” During hurricane season (June 1st to November 30th) SWFWMD upgrades its Trihedral support package from SupportPlus to Premium Support Service™ which guarantees telephone access to a system expert 24 hours a day, 7 days a week. “Trihedral actually dedicates a specific person who knows us and is familiar with our system to be available 24/7 for six months in case we have an issue during a weather event. We have never had an issue where we were not able to get support.”
The Future
“We currently have 30 remotely controlled water conservation and flood control structures with plans for an additional eight more before we begin adding automation.” In the meanwhile, the groundwork is already being laid. “Right now we are switching our current PLCs to Allen-Bradley because of their strength to take us into the future when we get into automation.” Unlike our current PLCs, Allen-Bradley units allow data logging within the PLC and will allow for the automated switching between a primary and secondary communications network. This will ensure that data is not lost if the communication network is cut off and that we have a redundant path for control of our structures.”
“Currently we are getting ready to add redundancy to our system by taking all of our control systems and data collection systems that are relative to control and putting them on a proprietary VHF radio network. We are keeping the phone system as a secondary backup. Trihedral is writing a new driver to allow us to do this. That is a very cool new option. We could have kept our current PLCs and added third party devices. But through research done by Trihedral we found a multiport Allen-Bradley that could log data and switch between communications different communications networks. So now we are out of the third-party hardware business again. Again, we get the single platform, we move to the Allen-Bradley’s which will take us into the future, we get the transparent redundancy of switching back and forth between the VHF radio and land line and we get data collection from our PLCs.”
The District continues to work closely with Trihedral to add new features to the system and achieve more with the ones they already have. “VTScada is dynamic. It is always growing for us. It’s always changing into what it is that we require. Trihedral never says ‘No’ they just say, ‘Yeah. We can do it’. Plus, Trihedral’s support and engineering costs are some of the lowest in the industry. For us to turn to Trihedral for engineering or design support in the grand scheme of things is relatively inexpensive. With Trihedral we can have a single application that does whatever we need it to do. Show me another software package that you could run on a single server in a non-server operating system that’s fully integrated with data monitoring and full control?”
Visit SWFWMD
If you would like to see this system for yourself, Mike Bartlett is glad to give you a tour. “We would be more than happy to show you our system and answer any of your questions.“ Contact Mike Bartlett directly by calling 352-796-7211 (ext 4515) or writing to mike.bartlett@swfwmd.state.fl.us.
Images reproduced with permission.