Is there a way to monitor the way the SCADA systems behave 'during pen-testing or security audit', and find out the implications of just port scans and/or monitor the state of activity while sending a payload ? Is it possible for a device to get faulty because of a SCADA Security Audit ?
SCADA devices are prone to the same common vulnerabilities—such as enumeration, password cracking, network eavesdropping, and denial of service—that are found in any other types of network devices.
I think it is pretty common to deploy health monitoring systems for your systems so you can pick up on unusual things happening on your SCADA network.
Your SCADA systems can deffinitly fail and break during a pentest. The outcomes of a pentest can be so many, and even if you do crash a PLC or other equipment that is also infact a security issue (denial of service). What happens if a long fuzz string hits your PLC and it breaks down?
If I were you I would treat your SCADA network like any other network. With this comes the worrying of taking something down, breaking data, and so on.
This video shows a staged SCADA attack on a power grid: http://www.youtube.com/watch?v=fJyWngDco3g
A SCADA system is just a large distributed network that is usually over IP. SCADA systems are very reliant on a database, and SQL is a common choice. The same rules apply for any pen test against a network. A SCADA system needs a firewall to block incoming connections. An IDS such a snort is still helpful in maintaining a secure network. SCADA systems can have web interfaces and a Web Application Firewall such as mod_secuirty (as a reverse proxy) is helpful.
The only gotcha with SCADA is that they are usually LARGE. As in computer systems for moving electricity across an entire country. So for physical penetration tests you need to look at gaining physical access to the network by looking at individual nodes in the SCADA system. Break out your hacksaw and a wire crimper :D.
The single most important rule when testing production SCADA systems is never test production SCADA systems if there is a risk of taking down any critical service, or spilling an oil pipeline etc. Always use a test environment.
If however you have to test in live, the best rule to adhere to is to ask for confirmation of every step - written confirmation, from the system engineer. This does make a proper test very cumbersome and expensive,but dramatically reduces the risk of you doing something fatal.
And if it is expensive enough, perhaps for next year's test you might get a test environment...
Pentesters / "Auditors" should keep a full packet capture of everything their tools put on the wire and this should be available to you as a client. Use this if something does happen to crash, is identified incorrectly, etc. to pinpoint the (mis)behavior of the systems under test.
Also, start basic and manual before assaulting the system with all kinds of traffic. (things can go bad quickly and you'll have no idea what caused it until you dig through the captures) This means passively identify ports in use, then start by sending simple connects/SYNS rather than a full on nmap fingerprint/service scan, etc. (tools are unlikely to have useful signatures to identify the equipment anyway)
Consider getting the data by other channels as well. For example, many devices support a standard SNMP MIB which will yield you information about model, tcp/udp ports, route table, etc. If you can recover this information with standard SNMP queries then you don't need to scan all 65000+ ports.
Disregard Rook's reply to your comment. Though many software components of control systems do run on Windows and *nix systems now, there is still plenty of risk to crashing/corrupting the applications on these systems, not to mention the embedded devices that make up the field of the control system which are even more notorious for not taking kindly to scanning or testing. Plenty of examples of both to back this up, even when running on modern operating systems.
TL;DR: There are always more passive / hands-off testing approaches you can fall back to in order to accomplish your testing task and get coverage of the control system. Be creative and cooperate with the client or test team.