Thursday, June 22, 2017

Remember “Programmable”?


Anyone who was involved with NERC CIP during the period from say mid-2014 through mid-2016 (i.e. the two years leading up to the CIP v5 compliance date) will remember that for a while it seemed that the fate of the universe hinged on the word “programmable”. This is because the v5 definition of Cyber Asset was “Programmable electronic device…”, and programmable wasn’t defined. Yet it was essential to properly identify Cyber Assets, since BES Cyber Assets, hence BES Cyber Systems, could never be properly identified unless the entity had properly identified their Cyber Assets.

It has always been clear what an “electronic device” is, but there was never a definition of “programmable” – and certainly not from a major dictionary – that NERC entities felt would properly identify Cyber Assets at a generating plant or substation. The problem at these assets is that there can be all sorts of weird devices that don’t run Windows, Linux, or any other operating system you’ve ever heard of – and often don’t run any operating system at all. Yet they still could conceivably be programmable, depending on how you define it.

In 2014-2016, billions, and probably quintillions, of electrons were utilized by the various online discussions of the meaning of programmable (I myself made a modest contribution to the discussion with this post and this one, as well as mentions in other posts that I’ve forgotten). More importantly, NERC made several concerted efforts to try to explain what the word really meant. The first was a well-received Lesson Learned in January 2015, which many NERC entities thought provided a very reasonable definition.

However, this LL was both withdrawn from the website (although I still have a copy if you want to email me for it. I may have to deliver it to you by hand in the dark of night in a city far away from Atlanta or Washington, DC, but I promise I’ll get it to you!) and also directly repudiated in one of the six infamous Memoranda that NERC issued in April of 2015. That Memorandum (this of course was itself later withdrawn from the website, along with the other Memoranda. I also still have a copy of this. Since it is even more sensitive, I may have to arrange to meet you in a foreign country to exchange it with you, and I’ll probably have to wear a wig and fake mustache, and talk with an impenetrable accent) took a very harsh line, insisting that even devices that could only be “programmed” by physically changing DIP switches were programmable. Moreover, like the other five Memoranda, it claimed some sort of absolute authority – i.e. it wasn’t subject to any industry review or correction. I discussed the controversy while it was still going on, in the second post linked in the paragraph above this one.

This and the other five Memoranda produced a huge firestorm in the NERC community, which culminated in what must have been a very interesting meeting on July 1, 2015. In that meeting, it seems there was a big revolt staged by the industry trade associations. NERC not only withdrew the Memoranda, they all but admitted there would never be any definitive guidance on the subjects of those Memoranda, including of course the definition of “Programmable”.

Finally, at the beginning of 2016 NERC turned this question, as well as a number of others, over to the drafting team that was formed to address FERC’s directive in Order 822, the Order where FERC approved CIP v6 but ordered further changes to CIP. Throughout all of this process, entities couldn’t wait for the question of the meaning of programmable to be settled (and they would have been disappointed had they waited!). I recommended that they “roll their own” definition, and many did that. I also recommended that entities follow the same approach for any other area of ambiguity in the CIP standards, of which there are many. My guess is all of the larger entities did that (not because I said it, just because it was really the only logical thing to do in this situation), although I also think the smaller entities (with Medium and/or High assets) simply muddled through without doing anything in particular.

I admit I haven’t heard (or thought) very much about this question until this week, when I attended a meeting of the Order 822 drafting team (known as “Modifications to CIP Standards) in Montreal. Yet on the first day of the meeting, there was a discussion of “Programmable”! I think the topic came up because this team is now discussing how virtualization can be incorporated into the CIP standards (one of the items on their very ambitious agenda – I will have a post on that topic shortly), and the definition of Cyber Asset will have to be changed for that.

The reason the Cyber Asset definition needs to be changed to accommodate virtualization is the word “device”. There is no getting around the fact that this word means something hard, not something that is just software (my operational definition is that if you drop a device on your foot, it will hurt. If you drop a virtual device on your foot, it won’t hurt – if you can even figure out how to do anything physical with a virtual device!).

The drafting team will definitely propose a change to allow virtual devices, but it seemed pretty obvious that they don’t have the stomach to deal with the Programmable issue. One person mentioned that issuing a definition of Programmable at this point would probably require all NERC entities with Medium and/or High impact assets to go back and re-run their entire asset identification methodologies. This would almost certainly require a huge expenditure of time and money on the part of NERC entities, and I can understand why this isn’t exactly right at the top of everyone’s wish list. They did mention possibly including a discussion of Programmable in the Guidance (it isn’t there in the current Guidance and Technical Basis for CIP-002-5.1, where it would logically be found), but the issue of standards guidance in general – which was extensively discussed – brings up a whole host of issues, which I hope to discuss in one of my next posts.

Meanwhile, I was reminded that a Lesson Learned approved in 2015 included a discussion of this issue (pages 3-4, under “Capability”). I just reread it, and found it a pretty good effort. Of course, it doesn’t say what NERC will consider to be programmable vs. not, but it does list some devices that entities in the CIP v5 Transition Study considered to be non-programmable, and the fact that this isn’t disputed obviously means at least some staff at NERC and the regions consider this OK. Obviously, this isn’t the sort of evidence that would stand up in a court of law, but in the world of CIP v5 this is about as good as it gets!

So I need to tell you that I don’t think there will ever be any more certainty as to the meaning of “Programmable” in the Cyber Asset definition than there is now. While this certainly isn’t a great situation, it’s nowhere near as bad as some of the other areas of ambiguity, missing definitions, or outright contradiction in CIP v5 (and if you want a discussion of some of these, just read almost any of my posts from the end of April 2013 through say the end of 2016. I did compile a partial list of these areas in this post).

And, as I intend to discuss in one of my next posts, the prospects of getting final answers to any of these CIP v5 interpretation questions are fast approaching zero, if they haven’t already passed zero and gone into negative territory. Have a nice day!



The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.

Tuesday, June 13, 2017

Debate on Enforceability of CIP-013


The day after I published my post questioning whether CIP-013 was enforceable (based on comments by a friend of mine who works for one of the NERC regions), an auditor (from another region) emailed me his dissenting opinion. He said:

“The requirement is essentially that the Registered Entity ask for the elements of the required process(es), not that the vendor agree to them.  Therefore, the Standard is correct in that the language of the resulting contract is not in scope.  (The procurement steps include) the Request for Quotation or Bid, Request for Contract Amendment, and/or any other Registered Entity-initiated correspondence that sets forth the expectations to be placed upon the vendor.  As long as I see the expectations in the procurement documents issued by the Registered Entity, they have satisfied the requirement.

“So, while I would readily argue that the Standard is weak as water, it is not unenforceable.”


I showed these comments to my friend (I should say other friend, since the auditor is also a good friend), and he had this (friendly) rejoinder:

“The ultimate goal of CIP-013 is to modify the terms of acquisition contracts used by the Responsible Entity:

“Quoting FERC Order 829 Page 59 (Note: this is the Order that NERC develop a supply chain security standard): ‘The new or modified Reliability Standard must address the provision and verification of relevant security concepts in future contracts for industrial control system hardware, software, and computing and networking services associated with bulk electric system operations.’

(My friend continues) “In keeping contracts out of scope for audits, CIP-013 does not fulfill the underlying purpose of the Standard. There may be some things that can be audited, but the auditors will be handicapped in reviewing evidence. They will not be able to audit that ICS contracts contain provisions which satisfy the security controls of R1, and they will not be able to verify that the entity enforces these controls.

“Ultimately, this version of CIP-013 does not fulfill the definition of a Risk-Based Requirement: “[D]efine actions by one or more entities that reduce a stated risk to the reliability of the Bulk Power System and can be measured by evaluating a particular product or outcome resulting from the required actions.” [NERC Rules of Procedure, Appendix 3A Section 2.4] If the outcome cannot be measured, then the Requirement fails as a Risk-based Requirement.”

I’m not going to take a side in this debate. These two people know a lot more about auditing than I will ever know. If you have an opinion on this question, I’d like to see it. Of course, this wouldn’t be the first time that two Regions have disagreed on some aspect of CIP!



The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.

Monday, June 12, 2017

FERC Asks Two Questions


I have heard from an industry source that FERC recently asked at least one of the trade associations to answer two questions (they in turn circulated these to their members):

Q1: Which, if any, of the existing CIP requirements would you get rid of and why? Do any of these requirements harm/hinder security efforts?
Q2: What would a performance or results-based approach to the CIP Standards look like? (e.g., a CIP-014 type approach) Could it help to improve compliance efficiency and certainty as well as the security of the BES? How?

FERC didn’t ask for my opinion on these questions, but since I always try to be helpful, and since I have answered these multiple times in various posts but not in one place, here is how I would answer them if asked.

Regarding Q1, my answer is very simple: Just getting rid of particular CIP requirements and leaving others makes no sense (no more sense than the “2 for 1” rule – in which two regulations must be eliminated for every new rule – makes). The problem isn’t that some of the specific things that CIP makes entities do are unneeded while others are needed. Everything that CIP requires is needed, just not all to the same degree. Plus there’s a lot more that CIP doesn’t require – controls on phishing, ransomware protections, etc. – that is also very much needed.

And that’s the point (which also brings me to my answer to Q2): By requiring that NERC entities allocate large amounts of resources to particular areas like CIP-007 R2 patch management compliance and not requiring any resources be spent on areas like phishing[i], CIP forces a misallocation of an entity’s cyber dollars.

In an ideal standard, the entity would a) come up with a list of current threats to its control of BES processes (and this list should ideally be generated externally by some group like the E-ISAC[ii]); b) get an assessment of its current control posture vis-à-vis these threats; c) from the list of vulnerabilities developed by that assessment, prioritize them by their degree of impact on control of the BES; d) develop a mitigation plan that allocates resources to each vulnerability based on its degree of impact; e) have their NERC Region review the plan and order changes if needed[iii]; f) execute the plan; g) be audited by their Region on how well the entity had executed the plan[iv]; and h) rinse and repeat – execute the whole cycle every two or three years (for larger and riskier entities) or longer (for smaller and less-risky entities. Of course, I’m talking about risk to the BES, not financial risk or something like that) – and the new vulnerability assessment must be based on the latest version of the “threat list”[v].

At one point I thought that CIP-014 was a good model for what I’m proposing, but I now think that it should have a lot more structure (like the above) if we’re talking about replacing CIP-002 through CIP-011. CIP-014 essentially tells the entity (that has assets in scope) to get an assessment and fix the problems, period. These are both steps in what I’m advocating, but they’re far from being the whole story.

I recently put together a list of the six principles (subject to later additions, of course. I don’t believe six is a magic number) that would govern the security regime I am advocating[vi]. They are:

  1. The process being protected needs to be clearly defined (the Bulk Electric System, the interstate gas transmission system, a safe water supply for City X, etc).
  2. The compliance regime must be threat-based, meaning there needs to be a list of threats that each entity should address (or else demonstrate why a particular threat doesn’t apply to it).
  3. The list of threats needs to be regularly updated, as new threats emerge and perhaps some old ones become less important.
  4. While no particular steps will be prescribed to mitigate any threat, the entity will need to be able to show that what they have done to mitigate each threat has been effective[vii].
  5. Mitigations need to apply to all assets and cyber assets in the entity’s control, although the degree of mitigation required will depend on the risk that misuse or loss of the particular asset or cyber asset poses to the process being protected.
  6. It should be up to the entity to determine how it will prioritize its expenditures (expenditures of money and of time) on these threats, although it will need to document how it determined its prioritization.

I have just answered the first part of Q2: “What would a performance or results-based approach to the CIP Standards look like?” My answer is that it would look like what I’ve just described. This is a results-based approach because it requires the entity to achieve a particular result (mitigation of the vulnerabilities identified in the assessment, ranked by their impact), without prescribing how that is to be achieved (of course, there will be lots of guidance on how to mitigate various types of vulnerabilities, and more importantly that guidance will be regularly updated, perhaps by the same group that updates the threat list).

The second part of Q2 reads: “Could it help to improve compliance efficiency and certainty as well as the security of the BES? How?” I will break this down into its parts.

  1. Does what I am proposing improve compliance efficiency? I guess it does, but compliance efficiency isn’t the goal – efficiency of improving cyber security should be the goal. As I said at the outset, NERC CIP now forces a mis-allocation of resources toward tasks that – while certainly producing some security benefit – probably don’t produce a benefit that justifies the expenditure required; while at the same time leaving less resources for areas like phishing that aren’t part of the CIP requirements at all and are probably more deserving of resources.[vii] What I am proposing would a) reduce (but not eliminate) the paperwork burden, but more importantly b) produce a great deal more cyber security for every dollar spent, vs. the current prescriptive CIP compliance regime.[viii]
  2. Does what I’m proposing improve compliance certainty? I’m not sure whether it improves it, but it certainly doesn’t hurt it. It makes compliance certainty fade as an issue, since there are no longer prescriptive requirements, whose interpretation one way or the other will make entities face million-dollar-a-day fines. An entity will still be subject to fines, but those will be for not fulfilling the promises it made when FERC approved its mitigation plan. The entity could be fined if it didn’t follow through in some way, and if it also didn’t address the corrections that the region ordered.[ix]
  3. Does this help the security of the BES? Absolutely. As I said, the biggest benefit is that it effectively increases the amount of money going to BES security without in itself requiring an overall increase in cyber security expenditures (although don’t kid yourself – cyber expenditures are going to have to go up every year for the foreseeable future. That’s the world we live in).

So here are my respectful answers to FERC’s questions. If anyone has comments or wants to discuss these, email me at talrich@deloitte.com.


The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.


[i] In practice, I’m sure most utilities are now spending a lot of money to address the phishing threat. But it’s almost axiomatic that, since the CIP requirements carry the possibility of big fines whereas anti-phishing measures don’t avoid any fines, entities will spend much more on other less serious threats if they’re part of the CIP requirements.

[ii] This list should be updated regularly, at least once a year and maybe every six months. The entity could always add a threat to its own list, if its own situation dictates it should be on the list. Conversely, they could remove a threat from the list if it doesn’t apply to them or if they’d already sufficiently mitigated this. They would have to document their justification for doing this, though.

[iii] Of course, they couldn’t order changes just because some auditor felt like it. There would be guidelines for what constitutes an acceptable plan.

[iv] Normally, if the Region feels an entity has not properly addressed a particular vulnerability, they could order it to re-do or augment the steps it took to address that vulnerability. And if the Region feels the entity has really screwed up the whole effort, they could order it be redone. Again, there would be guidance on what does and doesn’t constitute successful mitigation of a vulnerability.

[v] One of the big problems with CIP now is that it is close to impossible to address a new threat. For example, virtualization has brought lots of benefits but also new potential threats. Virtualization was widely used in 2011 when the CSO 706 (CIP) SDT turned its attention to developing v5, yet I don’t think there was any real thought of addressing virtualization in v5. Now NERC really has to address it, and the drafting team is doing very good work on addressing virtualization on a conceptual level, as well as starting to develop new (or revised) requirements and definitions. However, because of the sheer number of the changes that will be required, I am skeptical they will ever be able to get all of this approved by the ballot body, at least within the lifetime of the universe as we know it. I will attend the drafting team’s meeting in Montreal next week and hope to get a better take on this effort, since I admit I have not paid enough attention to virtualization lately.

[vi] While that post was specifically about natural gas pipeline regulation, I wrote the principles so that they would apply equally well to any process-based critical infrastructure. In the next post, I discussed them in the context of the electrical power industry, which is of course where they are needed first.

[vii] And don’t tell me that the solution to this problem is to write a SAR for a new CIP phishing standard! I don’t want the current CIP standards to be expanded beyond what they currently cover. Instead, I want a regime where all threats are considered and prioritized. I am sure phishing will be at the top of a lot of entities’ lists of threats to address, although that might not be true in five years (I’m sure there will be new threats we haven’t thought of yet by that time).

[viii] I’ve noted a couple times that I asked some entities how much of every dollar they spent on implementing CIP v5 went to cyber security vs. just compliance – the average answer I got was 50%. I doubt my plan will reduce any utility’s total spending on cyber security (including CIP compliance). However, I do believe it will increase that percentage to say at least 75 or 80%. If say two billion dollars a year is now being spent on CIP compliance by North American utilities and IPPs (and I don’t know what the full number is, although I’m sure it’s well over $1 Bn), increasing the percentage to 75% would be an effective increase of $500 million going to cyber security. That’s much more than I earn in a year.

[ix] I won’t pretend there isn’t a potential problem with rogue auditors who like to torment utilities by unreasonably saying they haven’t fulfilled their mitigation plan. There will need to be controls to prevent this, the most important being a much more extensive auditor training program and higher requirements for cyber security expertise.

Sunday, June 11, 2017

The Good News is this Post isn’t about CIP. The Bad News…Well, Read the Post


On June 9, the Opinion page of the Wall Street Journal published an article by Henry F. Cooper titled “North Korea Dreams of Turning Out the Lights”. It discusses the possibility of an EMP (electro-magnetic pulse) attack by North Korea, and makes the point that this is a lot more likely than we might think/hope. Since the WSJ’s online version is behind a pay wall, I can’t link the article here, but I’ll summarize some key points and quote others.

  • North Korea now probably has the ability to detonate a nuclear weapon 40 miles above Seoul – in fact, “a recent North Korean medium-range missile test that was widely reported to have exploded midflight could in fact have been deliberately detonated at an altitude of 40 miles.”
  • Such an attack on Seoul would “inflict catastrophic damage on South Korea’s electric power grid, leading to a prolonged blackout that could have deadly consequences.” Moreover, since the US has about 29,000 military personnel stationed in South Korea, plus more at sea nearby, such an attack would make it very hard for the US to respond to North Korean aggression (say, if North Korea were to invade the South after an EMP attack – note this is my inference. It isn’t stated directly by the author).
  • In 2001, Congress established a commission to study this danger. The chairman of the commission, William R. Graham, “noted that several Russian generals told the commissioners in 2004 that the designs for a ‘super EMP nuclear weapon’ had been transferred to North Korea.”
  • An EMP attack using a nuclear warhead with a yield of “only” 10 to 20 kilotons could “inflict catastrophic damage to unhardened electronics across hundreds of miles of surface territory.” This is the range of yields that North Korea has already successfully tested. So there would be no need for them to wait until they had produced a much more powerful device.
  • North Korea wouldn’t need to develop a long-range ballistic missile in order to attack the US. For one thing, there is no need for great accuracy – detonation almost anywhere over the US would produce devastating effects. For another, there would be no “need to worry about developing a reliable re-entry vehicle for their ballistic missiles” (which would be required for a conventional nuclear strike).
  • And there might be an even easier way to strike the US: According to William Graham in a recent blog post, North Korea could launch a “short-range missile off a freighter or submarine or by lofting a warhead to 30 kilometers burst height by balloon. Even a balloon-lofted warhead detonated at 30 kilometers altitude could blackout the Eastern Grid that supports most of the population and generates 75 percent of US electricity. Moreover, an EMP attack could be made by a North Korean satellite.”

The article concludes “The US and South Korea should ensure their ballistic-missile defenses are effective and harden their electric power grids against EMP effects as soon as possible. The day of reckoning could come sooner than anyone in either country thinks.”

Have a nice day!



The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.

Friday, June 9, 2017

Is CIP-013 Unenforceable?


I heard an interesting comment this week from a friend on the staff of one of the Regional Entities, who is very knowledgeable about CIP and NERC enforcement in general. His opinion is that CIP-013 is unenforceable, as it stands in the second draft – the one now being balloted. Why does he think this? Here’s the smoking gun: the note below CIP-013 R2 (and since it’s a note, not part of the blue boxes, it does need to be considered as a binding modification of the requirement. I’ll have more to say on those blue boxes in one of my next posts – there is more going on with them than meets the eye) reads

“Note: Implementation of the plan does not require the Responsible Entity to renegotiate or abrogate existing contracts (including amendments to master agreements and purchase orders). Additionally, the following issues are beyond the scope of Requirement R2: (1) the actual terms and conditions of a procurement contract; and (2) vendor performance and adherence to a contract.”

My friend points out that he has no problem with the first sentence. In fact, this thought was part of FERC’s Order 829, which ordered NERC to develop the standard; it was also in the first draft of CIP-013. His problem is with the second sentence, which was in neither Order 829 nor the first draft. And of the two parts of the second sentence, his problem is with the first part.

What is the problem? Let’s look at the requirement it’s attached to. R1.2 requires development of a supply chain cyber security risk management plan that includes (but isn’t limited to):

1.     One or more process(es) used in procuring BES Cyber Systems that address the following, as applicable:
a.      Notification by the vendor of vendor-identified incidents related to the products or services provided to the Responsible Entity that pose cyber security risk to the Responsible Entity;
b.     Coordination of responses to vendor-identified incidents related to the products or services provided to the Responsible Entity that pose cyber security risk to the Responsible Entity;
c.      Notification by vendors when remote or onsite access should no longer be granted to vendor representatives;
d.      Disclosure by vendors of known vulnerabilities;
e.      Verification of software integrity and authenticity of all software and patches provided by the vendor for use in the BES Cyber System; and
f.       Coordination of controls for (i) vendor-initiated Interactive Remote Access, and (ii) system-to-system remote access with a vendor(s).

R2 requires implementation of that plan. In an audit, the Responsible Entity is going to have to demonstrate that they have taken steps to get their vendors to provide all of these things. There are various ways to do this, and they can vary according to the risk posed by the vendor and/or the risk posed by the product or service being purchased. It is certain that many entities will attempt to achieve these goals by using legal contract language, at least in the case of some of their vendors.

My friend’s concern is that the first part of the second sentence of the note essentially says that the actual terms and conditions of a procurement contract are out of scope for the audit. This means that the auditor can’t ask the entity to show him or her a vendor contract, to demonstrate their assertion that they have used contract language to address one or more of the items listed in R1.2 for a particular vendor.

So let’s say the entity shows their auditors the section of their plan that describes how they will address each of the six items in R1.2, and the plan says that, for very important vendors like their EMS vendor, they will address all of the items using contract language; it looks at first glance like they’ve complied with R1.2, since they have a proper plan for achieving the objectives listed above.

Now the auditors move to R2, which requires the entity to implement the plan. They get to the section of the plan that corresponds to R1.2, and tell the entity that they would like to see evidence that they have complied with this part of the requirement for each of their top ten vendors, starting with the EMS vendor. And since the plan says that the entity will use contract language to address the six items in R1.2, at least for this vendor, the auditors request to see that contract to verify the language is actually there – or more likely, that the language is acceptable as evidence of compliance. At that point, the entity points them to the second sentence of the note and (politely) repeats that the “actual terms and conditions of a procurement contract” are out of scope. Therefore, they don’t need to produce the contract.

So how are the auditors going to verify the entity’s assertion that they have complied with R1.2 for their EMS vendor using contract language, if they aren’t allowed to see the contract? My friend’s answer is that the auditors can’t verify this assertion without being allowed to see the evidence. And what if the entity asserts they have used contract language to address R1.2 for all of their vendors? Then the auditors aren’t going to verify compliance – at least for the items in R1.2 - at all for this entity. Do you see any issue with this?

But other than the fact that it’s not enforceable, my friend doesn’t have any big problems with CIP-013.



Any opinions expressed in this post are those of the author, not necessarily those of Deloitte.

Sunday, June 4, 2017

NERC, FERC, LERC, ERC


My most recent post dealt (in part) with Felek Abbas’ response to the question whether a “jungle mux” should have an ESP around it. The day after I posted it, I realized there was an error in my footnote, which read “Of course, this isn’t to say that the serial devices that connect to the mux don’t have external routable connectivity; they still do, since the external routable communications stream crosses the asset boundary. If the mux controls electronic access, as does the device in Reference Model 5, then no further measures should be required for the entity to comply with the Low impact electronic access control requirement part in CIP-003 R2. If it doesn’t, then some further step will be needed, like a firewall or data diode.”

I replaced the footnote with the one quoted below, but the email feed had already gone out to the 500+ feed subscribers (whose names I can never see, in case you wondered). I thought it would be interesting to see how many of them caught the error, so I didn’t otherwise acknowledge it. I waited and waited and…guess what? Nobody seems to have caught the error. This means one of two things:

1.       None of my readers realize that a device that can be in an ESP is Medium or High impact, not Low (and therefore what I wrote in the footnote about ERC crossing the asset boundary and Reference Model 5 is irrelevant, since these concepts only apply to Lows); or
2.       Nobody reads my footnotes.

Since I know that I have a lot of readers who would spot this error very quickly if they read it, I have to conclude that number 2 is the case. That’s OK – I’m not complaining because nobody reads footnotes. It just means I need to avoid putting important information in footnotes (and I’ve already started doing that. I used to have posts with 8-10 footnotes, but lately I’ve hardly ever gone above two. I’ll try to bring that down to zero).

So here is the new footnote that I inserted in place of the erroneous one. I’m making all of this into a separate post because a lot of people have kind of forgotten the difference between LERC and ERC, and – while a lot of people know that “LERC” has been retired and the definition is now built into the requirement part in CIP-003-7 R2 – probably most (including me) have forgotten what the issues are with ERC, and where things stand in fixing that definition.

“Of course, this isn’t to say that the serial devices that connect to the mux don’t have external routable connectivity. Whether they do or don’t depends on what happens at the jungle mux. The definition of ERC is “The ability to access a BES Cyber System from a Cyber Asset that is outside of its associated Electronic Security Perimeter via a bi-directional routable protocol connection.” If all the jungle mux does is convert the routable communications stream to serial and send it to the proper device, then there is ERC. If the mux performs some sort of authentication, then this probably removes that ability, and there is no ERC.

“However, I’ll admit this is still an ambiguous area, and that points to an interesting irony. That is because FERC’s qualms about the ERC definition (discussed above in this post) led to a change; but the change was in the LERC definition, not ERC. When FERC ordered NERC to clarify the meaning of “Direct” in the LERC definition – in Order 822 approving CIP v6 in January 2016 – they weren’t reacting to problems with LERC since it wasn’t in effect yet. They were really responding to the discussions about ERC in 2014 and 2015 that I described above (I linked above to just one of the posts I wrote on this issue. Others include this one, this one, and this one (in the order I wrote them). I believe FERC was especially concerned about the idea that mere protocol conversion “breaks” ERC.

“If you want to plow through these three posts, you will see that, by the third one, I had concluded (in conjunction with Morgan King, a WECC auditor who has weighed in a lot on technical issues like this) that mere protocol conversion didn’t break ERC, but requiring re-authentication would. I thought when I wrote the third post that this was the last word on the subject. However, in this post, which was written a few days after the third one, I reported that another auditor said this wasn’t enough to break ERC – that there had to also be ‘reformulation of the user’s commands and the acting as a proxy for the user’.

“At this point, I threw up my hands and decided ERC was a black hole – I could write 100 posts and never get to the bottom of the problem, which was that the definition needs to be rewritten. This is one of the tasks on the CIP Modifications SDT’s agenda. I think they should build on the very good work they did with the LERC issue last year and base their new definition of ERC on use cases similar to the Reference Models they developed for CIP-003-7 (although some of those models won’t apply because the new Section 3.1 “defines” what used to be LERC in terms of the asset’s boundary, in order to avoid requiring a list of BCS at Lows). I think any attempt to write a dictionary-style definition will fail. They just need to say ‘In these particular cases, there is ERC. In these particular cases, there isn’t ERC.’”


The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.

Tuesday, May 30, 2017

The News from RF, Part V: Felek Tackles Jungle Muxes


I’ve already mentioned that the highlight of RF’s CIP workshop in April was the joint presentation by Felek Abbas of NERC and Lew Folkerth of RF, which addressed various current topics in CIP. Felek gave a very good presentation on CIP-003-7, which includes the revised requirement for electronic access control at Low impact assets that have external routable connectivity (CIP-003-7 was approved by the NERC ballot body and Board of Trustees, and is now awaiting FERC approval).

Felek spent a lot of time on the Reference Models that provide examples of different ways to comply with this new requirement; these are found in the Guidance and Technical Basis for CIP-003-7. When he discussed Reference Model 5 (found on page 39), he pointed out - and I agree with him - that this model addresses FERC’s concern that led them to order NERC to clarify the meaning of the word “Direct” in the LERC definition in CIP v6 (as you probably know, the current CIP Modifications drafting team decided to eliminate the defined terms LERC and LEAP altogether, instead incorporating them into the requirement itself. For a description of what they did, read this post, although if that isn’t punishment enough, you could read this one). I’d like to clarify why Reference Model 5 does actually address what FERC asked for, although this isn’t the point of this post. You can skip the next two paragraphs if this historical item isn’t of interest to you.

Reference Model 5 shows a routable communication stream entering a Low asset, but then being converted to a non-routable protocol for connection to a Low impact BES Cyber System. In 2014 and 2015, as NERC entities realized the clock was ticking on coming into compliance with CIP v5, there was a lot of discussion about whether or not just converting a routable to a serial connection (e.g. with a protocol converter) was enough to “break” External Routable Connectivity for Medium BCS. The regions and NERC made it fairly clear that merely converting the protocol wouldn’t break ERC, but it was also clear that the ERC definition itself didn’t provide an answer to this question.

Of course, at the time there was no requirement regarding electronic access control at Low assets (and also no LERC), since the CIP v6 drafting team was still working on this. CIP v6 was approved in January 2016, and by then FERC was worried enough about this issue that they explicitly included it in their order approving v6; that is, they required NERC to clarify the meaning of “Direct” in the LERC definition. Reference Model 5 shows (using the green dotted line) that the protocol conversion by itself doesn’t change the fact that there is external routable connectivity to a Low BCS. However, the fact that the “non-BES Cyber Asset” in Reference Model 5 also performs electronic access control is deemed sufficient mitigation of the risk caused by the presence of external routable connectivity to one or more Low BCS – which is the objective of the revised requirement. So Reference Model 5 shows one way to comply with the revised requirement in CIP v7 (indeed, the other Reference Models all show alternative ways to comply).

At this point, somebody in the audience raised the question of “jungle muxes”. These are devices – often found in substations – that accept a routable external connection, but then convert that into a number of serial connections to a set of serial devices (often relays). The question was whether, since the mux does communicate routably, it has to have an Electronic Security Perimeter around it.

Felek answered this question by going back to a Lesson Learned that dealt with communications between two Medium and/or High assets, at least one of which doesn’t have an ESP (i.e. it doesn’t have any internal routable network). The Lesson Learned was occasioned by the fact that Section 4.2.3.2 of all the CIP v5 and v6 standards (it’s also found in v7, at least in CIP-003-7) says that cyber assets involved in communications between “discrete” ESPs are exempt from CIP; at the same time, under v5 and v6 (unlike in v3) there can be assets subject to CIP that don’t contain any routably connected devices. The Lesson Learned addressed the question whether the cyber assets that are associated with the communications network between two Medium and/or High assets, one or both of which doesn’t have an ESP, are also exempt from CIP, and if so where the “inter-asset” communications stream begins in an asset without an ESP.

That Lesson Learned introduced the idea of the Demarcation Point (which is an old idea from the days when dinosaurs roamed the Earth and data communications was mostly over phone lines), as the device where the external communications carrier “takes over” from the building’s internal communications network. The LL said that an entity complying for an asset without an ESP should designate the device that is best described as the demarc point. All cyber assets that are “inside” of this device are potentially in scope for CIP; all cyber assets that are “outside” of this device are not.

Felek said that, even though the jungle mux does have a routable connection, the fact that this connection is used purely for external communications – and that there are no other devices at the Low asset that communicate routably – means the mux itself can be considered the demarcation point. Thus, it doesn’t need an ESP.[i]

I thought Felek’s answer was very good, but I was even more impressed by the fact that he was willing to state it in a meeting with 200 or so representatives from NERC entities in attendance. There are some who might consider what he said to be an “interpretation” of the wording of a standard, rather than simply implementation guidance. NERC and regional employees are permitted to provide the latter but not the former; in practice, this has meant that a lot of questions about CIP v5 and v6 have only been answered in private conversations (primarily in the private Small Group Advisory Sessions that were conducted in 2015 and 2016) or not at all. Felek wasn’t afraid to publicly state his opinion on a disputed issue like this; I wish it would happen more often.


The views and opinions expressed here are my own and don’t necessarily represent the views or opinions of Deloitte.


[i] Of course, this isn’t to say that the serial devices that connect to the mux don’t have external routable connectivity. Whether they do or don’t depends on what happens at the jungle mux. The definition of ERC is “The ability to access a BES Cyber System from a Cyber Asset that is outside of its associated Electronic Security Perimeter via a bi-directional routable protocol connection.” If all the jungle mux does is convert the routable communications stream to serial and send it to the proper device, then there is ERC. If the mux performs some sort of authentication, then this probably removes that ability, and there is no ERC.

However, I’ll admit this is still an ambiguous area, and that points to an interesting irony. That is because FERC’s qualms about the ERC definition (discussed above in this post) led to a change; but the change was in the LERC definition, not ERC. When FERC ordered NERC to clarify the meaning of “Direct” in the LERC definition – in the order approving CIP v6 in Order 822 in January 2016 – they weren’t reacting to problems with LERC since it wasn’t in effect yet. They were really responding to the discussions about ERC in 2014 and 2015 that I described above (I linked above to just one of the posts I wrote on this issue. Others include this one, this one, this one (in the order I wrote them). I believe FERC was especially concerned about the idea that mere protocol conversion “breaks” ERC.

If you want to plow through these three posts, you will see that, by the third one, I had concluded (in conjunction with Morgan King, a WECC auditor who has weighed in a lot on technical issues like this) that mere protocol conversion didn’t break ERC, but requiring re-authentication would. I thought when I wrote the third post that this was the last word on the subject. However, in this post, which was written a few days after the third one, I reported that another auditor said this wasn’t enough to break ERC – that there had to also be “reformulation of the user’s commands and the acting as a proxy for the user”.

At this point, I threw up my hands and decided ERC was a black hole – I could write 100 posts and never get to the bottom of the problem, which was that the definition needed to be rewritten. I believe this is one of the tasks on the CIP Modifications SDT’s agenda, and I hope they do this. I think they should build on the very good work they did with the LERC issue last year, and base their new definition of ERC on use cases similar to the ones they developed for CIP-003-7 (although some of those cases won’t apply because the new Section 3.1 “defines” what used to be LERC in terms of the asset’s boundary, in order to avoid requiring a list of BCS at Lows). I think any attempt to write a dictionary-style definition will fail. They just need to say “In these cases, there is ERC. In these cases, there isn’t ERC.”