Friday, July 10, 2015

What’s a High-Impact PACS like You Doing in a Low-Impact Substation like This?

I very recently engaged in a good email discussion that I’d like to share with you. I think it’s worth reading because:

  1. It can be important to certain entities that might be in a similar situation;
  2. The discussion illustrates how this kind of CIP-002 question can be addressed; and
  3. As an afterthought, a good – and much less expensive – solution to the problem came up. It’s important always to be looking for these.
The discussion began when a consultant that I have known for a long time – and engaged in a number of CIP discussions with, some of which have made it into this blog – emailed me about my recent post on the question of “far-end” relays. He stated that he had a client who has a PACS system in a Low impact substation, which is controlled by the PACS server that also protects their High impact Control Center. 

He had determined that, even though the PACS was in a Low substation, the fact that it was associated with the High impact PACS meant it was, itself, High impact. The client at first resisted this finding (understandably, given the higher compliance costs it would entail), but when he pointed out that an attacker could use the substation PACS workstation to alter the configuration of the PACS server in the Control Center, the client came around. The consultant wondered whether I agreed with his position.[i]

My answer was a very safe “It depends”.[ii] If the question were whether a BES Cyber System at a substation could be High impact because it’s associated with a BCS at a High Control Center, my answer would be “Absolutely not”. High BCS are limited to those “used by and located at” High assets (which are all Control Centers, of course). Since the device in question isn’t located at the Control Center, it clearly can’t be a High BCS.

But since we’re talking about a PACS system, not a BCS, the question is different. “Used by and located at” doesn’t apply in this case. What does apply is the word “associated”. Normally, when I talk about this word – which is a lot – I’m referring to the preamble to Section 2 of CIP-002-5.1 Attachment 1, which says that Medium BCS are those that are “associated with” the assets or Facilities that are the subject of the 2.X criteria. 

However, “associated” is used in another location in v5 (actually, many locations): that is the Applicability column of all of the CIP v5 standards (except for CIP-002 and -003). Just about every requirement applies to some variation of Medium and/or High impact BCS and “their associated…PACS”. The question now becomes, “Is the PACS workstation in the Low substation associated with the BCS in the High Control Center?” If so, then many (but not all) of the High impact requirements will apply to it.

Note the question isn’t whether the workstation is associated with the High impact PACS server itself; since it uses the server, the answer to that is clearly yes. But is the workstation in the Low impact substation really associated with the different BES Cyber Systems – ICCP servers, etc. – in the Control Center?

Two things are clear: First, the fact that the PACS workstation in the Low substation could be used to attack the PACS server in the Control Center is completely irrelevant here. Any computer in the world that’s connected to the Internet could, in theory, be used to attack the PACS server. Obviously, you want to take good precautions to protect the PACS workstation in any case, but it won’t be “High impact” for this reason.

Second, the “high water mark” concept has nothing to do with this situation. The fact that the workstation in the substation forms part of the overall PACS system – which includes the “High impact” PACS server – doesn’t mean that all of the components of the PACS system suddenly become “High impact”. Unless, of course, they’re all on a single network in a single ESP. But I really don’t recommend having an ESP that spans locations.[iii]

An implicit assumption so far is that the PACS server in the High Control Center is protecting the High BCS in the Control Center. If the server were really controlling just substations and/or generating stations, but was simply located at the Control Center for convenience, then I contend it would be a) Medium impact if one or more of the assets for which it controlled access were Medium, or b) Low if it controlled only Low assets.[iv]

At this point in the discussion (actually earlier – I’m condensing some), I brought in an auditor friend, with whom I often have conversations about different CIP v5 issues. He said he thought the PACS workstation in the Low impact substation was actually “associated” with the BCS in the High Control Center, and therefore would itself be subject to many of the High requirements - by the Applicability references in the v5 requirements that apply to High BCS. I believe he bases this interpretation on the idea that the workstation could actually change the security settings of the High BCS. This is because the workstation presumably would have access to the entire database on the server, not just the settings for the users with access to the substation it’s located at.[v] This means it would be associated with High BCS.

The auditor elaborated by saying:

“There are multiple components to a PACS and they all have different roles and types of access.  Starting at the top, you have the PACS server that contains the database of access rights for all access points being controlled by the PACS.  So, if you have multiple sites being controlled, the PACS server is associated with each and every one of them, and by extension, the BCS at those sites for which the PACS is controlling access.

“Then you have the door control panels, which is where the local site control takes place.  The PACS server pushes the access rights for the access points controlled by the door control panel to the panel itself.  The panel then operates autonomously, opening (or not) the doors and logging all transactions.  It will ship its logged transactions to server if the server is available, waiting if necessary until communications with the server is reestablished.  But, it can and will operate in the absence of the server using its latest download.  So, the door control panel is associated with whatever is behind the door(s) it is controlling.

“And, of course, you have the badge readers, door strikes/mag locks, and door open/closed sensors located at each access point.  They interface with the door controller.  The badge reader sends the information from the badge just presented (along with the scanned finger print, entered PIN, or whatever in a multi-factor access) to the door controller and the door controller sends the signal to unlock the door if access is to be granted.  These PACS components are explicitly excluded from the CIP standards.

“The odd man out is the workstation.  The workstation is how you interface with the servers to see the logs and/or configure/revoke a badge’s access rights.  A number of entities have tried to argue that the workstation is not part of the PACS because the workstation is not “controlling” access.  The Regions have argued that the workstation is part of the PACS because granting access is more than reading a badge and opening the door.  Granting access includes assigning access rights and that is done with the workstation.[vi]  So, the workstation is associated with every access point for which it can manage access rights; usually the entire PACS database.  And, by extension, it is therefore associated with the BCS sitting behind the door(s).  If you have multiple impact levels, then the workstation takes on the highest impact level of what is behind the doors the workstation can affect.[vii]

“Most, but not all, entities segregate their PACS network from their operational networks.  The workstations will either be on the PACS network, or they will be on the corporate network, with the server bridging the workstation and door controller networks.  Some entities have put their PACS inside their ESP; we usually recommend they rethink that decision.”

I’m stopping here. This particular auditor says the PACS workstation in the Low substation is associated with High impact BCS, in the case presented to him. I don’t agree with him, because I believe this is stretching the meaning of the word “associated”. It would be nice if NERC were to develop a Lesson Learned on this topic, but I doubt that will happen any time before the compliance date next April, and perhaps not even after that. I’m afraid this has to go on top of the already big pile of questions for which the entity – that finds itself in this situation – needs to consider all available guidance, then “roll their own” approach.

I also want to call your attention to footnote v, which lays out a technical control that might by itself prevent this situation from happening. It will probably be much easier to simply implement that control than have to apply a bunch of High impact requirements to the workstation in the Low substation.

The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte Advisory.

[i] Of course, this question is tied to the far-end relay question by the fact that both involve the issue of a High or Medium impact cyber asset being found in a Low impact substation – which can potentially be expensive, from the compliance point of view.

[ii] It was also followed by my standard disclaimer that the ultimate authority on any question of CIP v5 for any NERC entity is the applicable Regional Entity. In the end, it matters naught what I say, what NERC says, or what the other regions say. None of us will audit the entity.

[iii] An auditor friend – the same one quoted below in this post – pointed out that if an ESP encompasses multiple assets (locations), the communications equipment that connects the two assets will no longer be exempt from CIP v5, per Section of each of the Standards. This is because that exemption applies to devices that are between ESPs; if there’s just one ESP, all the devices are potentially BES Cyber Assets.

[iv] The PACS server wouldn’t be subject to High requirements because in this case it wouldn’t be “associated with” any High BCS. If it controlled access at any Medium substations or plants, it would be subject to some Medium requirements, because then it would be “associated” with Medium BCS. If it only controlled access at Low assets, it would only be subject to CIP-003-6 R2.

[v] As I wrote these words, it occurred to me that there could be a way to restrict any user of the substation workstation from changing any security settings on the PACS server, other than those that apply to that particular substation. I discussed this with the auditor, who confirmed that, if an account on the server were linked to only that workstation – through say a digital certificate, as well as the usual user name/password – and if that account were restricted to only being able to change the settings for access to the Low substation, then that could possibly mean the workstation would no longer be considered “associated with” High BCS. Thus, it wouldn’t have to comply with any High impact requirements. This really seems to be the best solution to this problem, rather than making the PACS in the Low substation subject to a number of the High requirements.

Another assumption we made is that the PACS server isn’t networked with the High BCS in the Control Center. If it were, that might conceivably strengthen the case that the workstation in the Low substation was “associated” with the High BCS. But I don’t think having the PACS server on the same network as the BCS is a good practice in any case – for one thing, it increases the compliance burden since the PACS server now also becomes a Protected Cyber Asset.

[vi] The auditor made this note:

“(Here is) the definition of PACS from the NERC Glossary:

Physical Access Control Systems (PACS): Cyber Assets that control, alert, or log access to the Physical Security Perimeter(s), exclusive of locally mounted hardware or devices at the Physical Security Perimeter such as motion sensors, electronic lock control mechanisms, and badge readers.

“It could be a bit more clear for sure, but the workstation is needed for the control function (the act of granting and revoking access rights) and the alerting function (alarm display).  That is why we have, since Day 1, viewed the workstation as part of the PACS.”

[vii] I think this is a good principle for entities to follow. In practice, it means that, when there are Medium and Low BCS at a single substation or generating station, the PACS will be Medium impact. This follows pretty clearly from the requirements.


  1. At the risk of being accused by many of those you are striving for CIP v6 of oversimplifying the debate and discussion about the implementation challenges of CIP v6, naively perhaps I say - it is plainly evident to many that NERC and the entities need to take things back to the basics and keep them there to resolve the issues with the new standard.

    To take that first step backwards can we all agree there are no more than six broad questions in implementing an acceptable, but not ideal, level of measurable security improvement?

    Broadly speaking the six questions CIP has to pose and answer could be phrased as follows:

    1) What IT systems do I have in my enterprise?

    2) What vulnerabilities do I need to worry about?

    3) What vulnerabilities BASED ON PROVEN EXPLOITS ONLY so I need to worry about RIGHT NOW.

    4) How can I configure my systems and their support systems more securely?

    5) How do I define a policy of secure configurations for my IT assets and how do I securely manage changes in the configurations ?

    6) How can I be sure my systems conform to the standards and I can determine where to invest resources to increase my conformance or remediate my CRITICAL, PRIORITIZED non-conformances.

  2. PACS/BCS/BES - Associated, Connected, or Controlling?

    The discussion in this post raises a very interesting and important point: where does the auditor set the boundary for categorization (High/Moderate/Low) on a PACS endpoint; and how is setting that boundary justified under CIP v6?

    I don't have an answer, but one possible way to think through the factors involved in deciding how to categorize and set the 'watermark' and to explain/back up the reasoning behind the decision would be to use the Swiss Cheese Model of Hazard Analysis


    I first came across this model when I was doing some research into NASA's fault analysis methods for my CISSP. The gist of the model is a fairly simple and easy to illustrate 'cause and effect' traceability chain. The model was created by James Reason. He developed the "Swiss cheese model" as a 'mental device" to illustrate how unrelated and independent weaknesses 'lined up' to cause major accidents and catastrophic systems failures. Mr. Reason put forth the Swiss cheese idea to show that 'big' failures are the result of the unexpected alignment of multiple, smaller failures.

    These smaller failures are embedded, like the holes in Swiss cheese, in the layers of subsystems that comprise the larger system. Each slice of cheese represents a 'protection system' acting as a layered defense against the occurrence of a particular hazard. Since no single layer is entirely foolproof, each one has weaknesses hat can be exploited.

    But when the 'holes' or weaknesses in these layers align themselves in a certain way, the hazard or accident then occurs, by literally 'passing through' all the layers. While a weakness in a single layer of protection by itself is not responsible for the accident, the model illustrates the common mistake of overestimating the strength of the 'defense in depth' strategy used to protect or prevent systems failures.The model highlights these alignments and raises the question: How likely are these alignments ?

    In the case of a compromised PACS server affecting the security of a BES, through a remote, connected PACS workstation operating within either a single or multiple ESPs model suggests the categorization (or inclusion for auditing purposes) could be made by deciding if when an alignment occurred, how deeply could the hazard penetrate through the layers and what the impact would be for each alignment.

    Keep in mind these 'layers' are not just hardware, software, and network connectivity. A robust Swiss cheese model would also include the human factors, and other non-IT 'coincidences' such as a leaving a port open after a maintenance operation, or not removing temporary elevated access permissions when a person assumes the duties of an absent co-worker.

    The simplicity of the model as a way to visualize multiple scenarios quickly should not be underestimated. The layers in the model should be independent of each other, but keep in mind, even if the layer's appear isolated, the 'holes' may have linkages that are not readily apparent.

    1. This comment has been removed by the author.

    2. This comment has been removed by the author.

    3. The other benefit of the Swiss Cheese Model is once the weaknesses in each of the layers can be estimated numerically, you can use either Bayesian Inference Models or Markov Chain Analysis to come up with a defensible likelihood estimate of the alignment's occurrence.

      I prefer the Markovian Analysis because it models the layers as 'stateless' and disconnected - meaning each layer possess the required property of being "memoryless" so the probability distribution of the next state (the next layer) depends only on the current state (of that layer) and not on the sequence of events that preceded it

  3. If I may offer a slightly more elegant solution:

    First, remove the workstation from the realm of being a PACS component altogether... uninstall the PACS client software and convert it into a tradition 'generic' client on the network.

    Second, either enable RDP directly on the PACS server, or - where vendor limitations dictate that the client software must be used to communicate with the PACS server - establish an RDP server alongside the PACS server (A simple workstation VM hosting the client-side software would work fine in this case) to which the now-generic workstation can create an RDP session to.

    Season with PACS firewall IP address-based rules to minimize source RDP sessions as appropriate.

    In this way, the VM and localized RDP session from the virtual workstation is what is part of the PACS and communicating with the server, and from which one can now fire off the PACS client-side software. On top of that, all of the CIP-scope PACS cyber assets are neatly housed at the PACS server location (if not on the PACS metal itself, depending on your approach).

    In your example the workstation is clearly the problem. The first question should not be how the workstations falls into scope, but whether a.) it is functionally necessary in the first place, and b.) it's functionality as a PACS cyber asset can be removed altogether.