Some Things You Might Not Have Considered
By Dan Naughton
I’d like to take a few minutes to talk about the Linux® operating system, particularly how it applies to SCADA systems. The reason we’re talking about this is that Linux comes up quite a bit in conversations about security. It’s common to hear people say things like, “I heard Linux is more secure than Windows. Maybe we should look at that for our SCADA system, because securing our SCADA systems is extremely important.”
So, what I’d like to talk about is Linux in general, the Linux security model, and how the two of them compare. For starters, let’s look at the top-level operating systems. Of course, you have Linux, Windows®, and Mac OS®. Assuming they are configured according to best practices, they’re firewalled, they use the correct authentication procedures, and the applications running on them are configured correctly, those operating systems are secure.
Who will support it?
With that in mind, when you deploy an operating system at a town, one thing you need to consider is the skill sets of the people who will maintain it. By and large, when you look at municipalities, their IT support staff are predominantly trained to support Windows desktops and servers. That’s important to consider. Do you have Linux expertise, either on staff or on contract? A deployed Linux server without support might as well be a black box, which is never a good way to deploy anything. Usually, this only works for the person who provided the box who would have a lock on your town or facility for a support contract, because there’s nobody internal to maintain it. Maintenance is extremely important with these systems.
Which Linux are we talking about?
When you start discussing Linux, an important thing to consider is, “Which Linux?” Linux, unlike Windows, normally comes in a form of what’s called a distribution. There’s probably 30 or so distributions that you might find on the internet. New ones are formed every month and others get discontinued. When you’re evaluating a distribution, one thing to consider is whether it is supported. Is there a company that offers a support contract, so you’d be able to get help if you run into issues with software running on that distribution?
In the United States, the two biggest distributions you’re going to find are Red Hat Enterprise Linux®, RHEL for short, and Ubuntu®. These are the two biggest ones that offer a support contract. Support contracts for RHEL are roughly $800 per server per year and Ubuntu, last I checked, was about $1,500 per server per year. So that’s an important consideration. When choosing a Linux distribution, find a supported one.
A couple other things worth mentioning. There are distributions out there that are called whatever the name is with an LTS at the end (examplename.lts). While LTS stands for long term support, this simply means that this distribution has not been abandoned. It does not mean that there’s somebody you can call if you have problems. If you do need support, there might be an online users’ forum or something but if you’re running a town and you’re in a crisis, that’s not the best plan.
There was an operating system based on Red Hat Enterprise Linux, which was called CentOS®, which people used for years because it ran essentially the same software as RHEL. However, that’s been discontinued and the replacement from Red Hat, called CentOS Stream®, is the opposite. It’s pre-released beta software, which is not something you’d want to use to run your SCADA system.
So, to review if someone mentions Linux, ask which one. Make sure you pick one that provides a support contract. The two you’d want to consider are Red Hat Enterprise Linux and Ubuntu. Those are my recommendations.
The Linux Security Model
We started discussing Linux because of security so it’s worth taking a moment to go through the Linux Security Model, at least at a high level. First, let’s talk about a Linux feature called Security Enhanced Linux, or SELinux. It’s a kernel module, and it’s normally implemented by all the major distributions. SELinux, typically runs in one of three modes, Enforcing, which is usually the default, Permissive, which is essentially just warnings, and Disabled. If your software application isn’t installed or set up correctly, you could run into problems with the default SELinux distribution policies. Often in those cases, people will recommend simply disabling it. Don’t disable it. It’s worthwhile going through all the policies for that application to make sure that they’re installed correctly and run correctly. Or we will be bypassing one of the primary security features of Linux.
File Capabilities and Permissions
The next part of the Linux Security Model is File Capabilities and Permissions. These are very analogous in Windows and Mac OS where each file has its own Read and Write capabilities and executable privileges. Those are pretty straightforward. The only thing worth noting is on Linux, a process cannot open a port or listen on a port under port 1024 unless it’s explicitly given permissions, or it’s running as Administrator or Root, which is never advisable. So, part of setting up Linux File Capabilities is that you need to explicitly allow an executable to open a port below 1024. That’s extremely important. One of the takeaways is when you run an application, make sure that it has those permissions set. Make sure you ask, “Is this SCADA process running with all the permissions set correctly or is it just simply running as Root or Administrator?” In general, all processes should not be running as Administrator. You should go through each one of the permissions it needs and specifically allocate it to that application. Running it as Administrator defeats a half of the security model. When discussing how a process runs, the question to ask is, “Is this process running as root?” If the answer is yes, then somebody has cut some corners on your security model.
Single Sign-On Recommendations
Next, part of the recommendations from DHS and from AWWA when accomplishing single sign on and single sign off, is to use Microsoft® Active Directory Services® for most towns and factories. When discussing your Linux distribution, it’s worth going through the exercise of deciding how to do the integration between Linux into an Active Directory server for authentication. For those people who do this all the time, it’s not terrible, but it’s by no means trivial. It’s worth asking what the plan is for integrating this Linux distribution into the town’s current Active Directory infrastructure.
Firewalling
The last thing I want to discuss as part of the Linux security model is firewalling. Linux uses either IP Tables or NF Tables to do network packet filtering. FirewallD® is the front end for setting up all the various rules. Just to security reminder, the firewall should remain up when it comes to SCADA systems which do an awful lot of network listening, and polling. Firewalls are a constant source of headaches. So, whoever’s managing your Linux distribution, it’s important for them to know the ins and outs of that FirewallD utility, to make sure that all the correct ports are open. Often, the first approach when people run into troubleshooting issues with a firewall is to disable it. Do not disable the firewall. Know how to use the configuration utilities and how to do the debugging.
A Word About Java
The last thing I want to talk about is Java, which is not exactly Linux, but in the world of SCADA, Java and Linux are a popular combination. All those things you would ask when talking about Linux, it’s probably worth having similar discussions about Java, particularly when it comes to support. Java has been around for a long time; a lot of people use it. In the United States, there are two popular organizations that will support Java: Oracle® and Azul®. Both offer a supported Java. There’s a number of free Java’s that you can find on the internet. The Oracle Java is about $25 per month per core. So, an average 10 core processor would be roughly $3,000 USD, a year for support from Oracle. The Azul platform core Java is roughly five grand a year. Make sure you ask which Java will be bundled with your application. Is it one of the two that are supported or is it bundled with one of the freeware ones? You’d want to know that because Java is constantly being updated. It is worthwhile to keep the software on your SCADA system current. So having support for your Java is as important as having it for the operating system underneath.
Let’s Recap
When it comes to Linux, some things to remember are, any one of the major operating systems can be secure. You want your operating system to be properly maintained, patched, and updated behind a firewall with the firewall software enabled. When it comes to Linux, you want to make sure that SELinux is set to Enforcing and you have all your file systems and permissions set correctly. Do not run any processes as Root. Make sure it’s connected to an Active Directory Server for single sign on. Ensure the firewall is enabled and you know how to debug it and add new rules when you put new pieces of equipment online. When it comes to Java, make sure you’re using one of the supported versions like Oracle and Azul and not a freeware one. If you are considering using Linux in your critical system, these are some good starting points. Good luck!
About Dan Naughton
Dan is a mechanical engineer with years of experience selling industrial computers to monitoring and control customers in a variety of industries. Dan is responsible for VTScada sales across the Midwest United States.