Tuesday, January 16, 2018

It’s Time to start Planning for CIP-013!

In October, I wrote a post pointing out that, even though the likely implementation date for CIP-013, the new supply chain security management standard, was more than two years away, there were good reasons to at least start the compliance planning process. The main reason why I made this assertion was that vendor contracts come up for renewal all the time. If your NERC entity knows what cyber security language you should request for CIP-013 purposes and can get it incorporated in new contracts, you will be saving yourselves many times more effort when CIP-013 comes into effect, since it is always harder to get vendors’ undivided attention when there isn’t a new contract on the horizon.

That argument is still valid, but there are a couple more that demand even more attention. First, I heard this morning that FERC has CIP-013 on their agenda for their meeting this Thursday. They will almost certainly do one of two things: a) Issue a Notice of Proposed Rulemaking (NOPR) stating their intention to approve CIP-013 and asking for comments; or b) Issue an Order approving CIP-013. In either of these cases, they could also make clear their intention to order changes to the standard, which would then have to be drafted and voted on as CIP-013-2. But either way, CIP-013-1 will be on the path to implementation.

The difference between these two cases, as far as the implementation timeline goes, is that an Order would start the clock ticking on the 18-month implementation plan for CIP-013, meaning compliance will be due about 18 months after this Thursday (the due date would probably be October 1, 2019). However, if they issue a NOPR (and I believe this is the more likely course) and allow 3-4 months for comment before issuing their Order, the compliance date will be either January 1, 2020 or April 1, 2020; my guess is it will be the latter.

So does April 1, 2020 sound like it’s a long time away? If you are a small organization (with one or more Medium or High impact assets), this might in fact be a long time. But if you’re a medium-to-large organization, you can’t wait much longer to at least begin your planning process for coming into compliance with CIP-013. I have been discussing what CIP-013 compliance requires with some NERC entities in the past few months, and I can assure you it’s probably a lot more than you thought. In fact, I will soon start a series of posts on what is needed for CIP-013 compliance, so you can understand why I say this.

However, there’s another reason why it’s important to start CIP-013 compliance soon, that I realized when I wrote this post last week. The gist of the post is that plan-based requirements (like those in CIP-013) need to be treated differently by the NERC Regional Entities than prescriptive requirements (like many of those in most of the other CIP standards). When an entity is required to develop and implement a plan, as in the case of CIP-013 R1 and R2, there really needs to be some mechanism for the Region itself to be able to review the plan before it is implemented. The post describes such a mechanism, which was suggested to me by an auditor; most importantly, it’s a mechanism that’s already in effect in one Region and could be replicated in others.

So, while I can’t promise anything, I think it’s a good assumption that by maybe a year and a half from now, most if not all of the Regions will be able to review your CIP-013 supply chain cyber security risk management plan and offer you comments on it. The comments won’t touch on whether the plan is “compliant” or not, but will touch on how what you are proposing in the plan compares with best practices. My guess is most NERC entities will welcome being able to have this review, to avoid the problems that were discussed in relation to CIP-014 (another plan-based CIP standard) in this post and this one.

So let’s say your entity waits a few months, then starts leisurely thinking about what CIP-013 requires. Meanwhile, FERC issues their Order approving the standard and the compliance date is now set for April 1, 2020. You realize that you now have a little more than 18 months to become fully compliant. You accelerate the compliance planning process, and as soon as possible start to implement compliance (remember, you will have to be compliant on the effective date of the standard). You make a Herculean effort, and you are finished – including having a fully developed plan – by say February 2020.

You might feel pretty good about this, but let’s say you then decide to ask your Region to review your plan. They say they’ll be glad to do this, but since a number of other entities have just asked the Region to review their plans, it will be more than say six months before they can review yours and report back to you on it (say they’ll get back by August 2020).

This means you will have to start implementing your plan in April, without having the benefit of any feedback from your Region. The main reason you asked for the review was to be able to hear and act on the results before you started implementing the plan; while it will still be good to have those results, it would obviously have been much better to have them at least a few months before April 1. You will have to start implementing the plan without knowing what your Region thinks of it.

Ideally, it would have been better if you could have finished your plan say by October 1, six months before the CIP-013 implementation date. That would have given your Region time to review and comment on the plan, as well as given you time to change the plan to reflect those comments – all before the April 1, 2020 compliance date. But obviously, this would have required starting the CIP-013 process earlier, like say around January 2018!

The moral of this story is of course that you should really start thinking now about the different structures required for CIP-013 compliance, and how you will implement them at your organization. And now here’s the sales pitch: Tom Alrich Consulting is prepared to help you do this thinking! The first step might be a set of workshops over say three days to a week, including the different groups that will be involved with CIP-013 compliance – and unlike the previous CIP standards, CIP-013 will require substantial involvement from Supply Chain and Legal, as well as Cyber Security, IT and NERC Compliance. With the experience of those workshops, I can work with you to develop a roadmap for your CIP-013 compliance implementation – and leave enough time for review by your Region! Like more information on this? Drop me an email at tom@tomalrich.com

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com

Sunday, January 14, 2018

An Interesting CIP Compliance Question

I have known Monta Elkins of Foxguard Solutions for three or four years, and have a lot of respect for him. If you’ve never seen his demonstration of hacking into a power drill and making it play the Darth Vader theme, you’ve missed something! So I was intrigued when he contacted me last week and said he wanted to discuss an interesting CIP compliance question he’d come across, which relates to the new Spectre and Meltdown vulnerabilities in Intel (as well as other) chips, and to Microsoft’s efforts to patch them. I’ll tell you up front that I’m not sure about the answer to this question. But here are the facts[i]:

  1. Two weeks ago, Microsoft released a software update to address these two vulnerabilities (as well as other unrelated issues). However, they warned that, once the update was installed, some antivirus software packages would cause the “Blue Screen of Death” and refuse to boot the PC.
  2. To show that their software is safe to use with the new update, Microsoft is requiring that A/V vendors set a particular Windows registry key, although it is also possible for the user, or an organization, to set it apart from the A/V vendor.
  3. But here’s the kicker: If the user tries to install the update without the registry key being set, none of the patches will be installed. Moreover, since Microsoft’s monthly software updates are now cumulative, no further patches will be installed in subsequent months, until Microsoft changes the requirement for the registry key.
  4. While the majority of A/V vendors do now seem to support the update and have updated the key, some have not. And this leads to our NERC CIP question.
  5. CIP-007 R2 requires the NERC entity to designate a patch source; it will often be the vendor of application software running on a BES Cyber System –and the BCS in question here will often be an HMI (human-machine interface). That vendor will approve Microsoft patches for installation with their product; if they don’t approve them, you won’t install them.
  6. Often, the vendor will also require your organization to use a particular brand of A/V software. There are three cases to consider:
  7. First, let’s say your HMI vendor requires you to use an A/V vendor that has set the registry key. In this case, you don’t have to worry – your Windows patches will continue uninterrupted, as long as you have applied the most recent A/V update before you apply the Microsoft update. There will obviously be no change in your CIP-007 R2 compliance status.
  8. Second, let’s say the vendor requires you to use an A/V vendor that has not set the registry key, and you (or your organization) don’t want to set it yourself (I know I would certainly have issues with doing that on my own!). In this case, since Windows patches are likely to cause serious problems, the software vendor won’t approve any patches for release, until either a) Microsoft relaxes their policy or b) the HMI vendor decides to find a different A/V vendor, one that will set the registry key. Since the vendor is your patch source and they’re not releasing any Windows patches, you don’t have any obligation to install Windows patches (or even consider them as applicable) under CIP-007 R2.
  9. Finally, let’s say your HMI vendor doesn’t require that you use any particular A/V software, and you chose an A/V vendor that has not now set the registry key. This is where the compliance question comes in. The HMI vendor will presumably continue to approve Windows patches and send them out, but you won’t be able to install them, unless you change your A/V vendor. If you can’t change your A/V vendor for some reason, what do you need to do to comply with CIP-007 R2?

I must admit that, when I started writing this post, it seemed to me this was a question that didn’t really have an answer. But now that I’ve written it, the answer seems fairly clear: Under CIP-007 R2.2, you would likely determine that this patch is applicable (after all, it was released by the vendor of your HMI). You of course can’t install it, so you need to develop and implement a mitigation plan for whatever vulnerabilities were addressed by the patch, per R2.3.

If you have any other ideas about this, I’d like to hear them. 

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

[i] A lot of this discussion is based on this article from Computerworld.

Wednesday, January 10, 2018

When and how can you receive Advice from your Region on your Plan?

I recently wrote a series of posts about “plan-based” requirements (e.g. CIP-010 R4 and CIP-013 R1) and raised two main questions about them. The first was whether they could be strictly audited using the standard NERC auditing framework (which is embodied in the Compliance Monitoring and Enforcement Plan, or CMEP). My answer to that question was that they are auditable to various degrees (depending on how they are written), but none of them are auditable in the strict sense that the prescriptive CIP requirements (like CIP-007 R2) or the Operations and Planning requirements (like FAC-003 R1) are auditable.

The second question was more important; I only delved into this question in the last post  in the series. This question is effectively[i] “Given that the real goal of auditing is promoting the reliability and security of the Bulk Electric System, is it possible that trying to audit plan-based requirements (which, as I’ve pointed out several times, are the wave of the future for NERC. In fact, all of the important new CIP requirements developed since CIP v5 have been plan-based) using the standard NERC auditing framework will actually hinder this goal?”

And my answer to that question was yes. In that post, I referenced a previous post  I had written on CIP-014 enforcement. CIP-014 was the first plan-based standard to be approved by NERC, and the NERC Regional Entities are already auditing it. In this post, I recounted a conversation I’d had with a CIP physical security compliance person at a very large utility, who had been rebuffed by his regional auditors when he asked them a question about whether a particular technology – that this entity proposed to implement in their substations subject to CIP-014 – would likely be determined to be appropriate to include in the Physical Security Plan required by CIP-014 R5.

He was flatly turned down when he asked this question, on the grounds that answering it would constitute a violation of the principle of auditor independence: If the auditors answered it for him now, when they came to audit him later (perhaps years later), they would in effect simply be auditing themselves. On the face of it, this seemed to be the only possible answer that the auditors could have given. But the unfortunate result of this was that the utility he worked for would most likely cancel their plans to implement this technology (which would cost $80 million to deploy to all of their CIP-014 substations).

In my post, I pointed out that this clearly seemed to be a case where the standard NERC audit framework was actually working against the goal of enhancing the physical security of the BES. I further hinted that, given the choice between maintaining that standard framework and securing the BES, I would choose the latter any day. I was thus preparing to suggest that the standard NERC auditing framework – which works very well for the NERC Operations and Planning standards, but not for the CIP standards, and especially the plan-based ones – be replaced with a new auditing program just for the CIP standards.

However, as has happened before, an auditor had written in to me about this issue, and in an email dialogue he pointed out that not only is it not necessary that there be a new auditing framework, but that the elements required to deal with this problem are already in place in at least two of the NERC Regions, and could be implemented in the other Regions as well. In the rest of this post, I won’t usually quote the auditor directly, but I will include his ideas, as well as my interpretation of them, without necessarily saying at every point whether they are his or my ideas. After I initially wrote this post, I sent it to the auditor to review for any mis-interpretations on my part, and he corrected those.

Before I discuss this further, I want to make sure we all understand what the big problem is. It is not that plan-based requirements are not very auditable (if they are auditable at all) under NERC’s CMEP; I’ve already stated that I consider auditability to be a distant second to the main concern. The main concern is the security of the BES, and the problem is that, as exemplified in the case I just discussed, that goal will not be aided if, for plan-based requirements, NERC entities can’t get their NERC Region to review their plan before they implement it. Additionally, when it comes to implementing the plan, the entities would be greatly helped if they could ask their Region to review the implementation while it’s in progress and point out potential problems. The case just discussed is an example of how auditing concerns can prevent NERC entities from getting the advice they need on complying with plan-based requirements. As we have just seen, this problem has already appeared for CIP-014, and it will appear in spades when FERC approves CIP-013 and entities start working seriously on their supply chain cyber security risk management plans.[ii]

In my opinion, here is what is needed to address this problem:

  1. NERC entities, when faced with plan-based CIP requirements, will of course first have to develop the plan mandated by the requirement (the Physical Security Plan mandated by CIP-014 R5, the supply chain security plan mandated by CIP-013 R1, the Transient Cyber Asset/Removable Media plan mandated by CIP-010 R4, etc). In the process of developing the plan, they need to be able to ask their Region questions about what should be in the plan, what are best practices for mitigating the threats addressed by the plan, etc.[iii]
  2. Once they have developed their plan, the entity needs to be able to take it to their Region and ask them to review it. The review won’t tell the entity whether the plan is “compliant” or not; rather, the reviewer will point out whether the plan doesn’t address any threats that should be addressed in the plan, and whether the mitigations proposed follow best practices as the Region understands them. If the entity can’t get this review, they will have to take a deep breath, hope the plan they’ve developed is one their region will think is good, and then implement the plan. The danger is that they may go a long way down the road to implementation (or even finish it) before their next CIP audit, and that the auditors will then tell them the plan had a lot of problems and needs to be redone. Of course, that could possibly lead to a lot, or even most, of the work the entity has done implementing the plan needing to be re-done as well.
  3. If the Region does the review and sees problems with the plan, they will point those out to the entity. At that point, the entity could elect to re-work the plan to fix the problems, or else to dismiss the Region’s advice if they think it isn’t well-founded for some reason.
  4. If the entity did re-work their plan, they would be well-advised to ask for a new review from the Region, to make sure they have addressed whatever objections the Region brought up.[iv]
  5. Once the entity was satisfied that it had a good plan, it would start implementing it. However, at any point during the implementation, the entity could request of their Region that they review the entity’s implementation work so far, and let them know of any developing problems they see (e.g. that the entity isn’t implementing everything in the plan, or that they are implementing parts of it badly).
  6. If the Region does point out problems with the implementation, the entity has the choice either to try to address these problems or to dismiss them if they don’t think they actually are problems that need to be fixed – just as in the case of the plan review.

In reading this, you may have already thought of the objection that first came into my mind when I realized these steps would be required in order for the entity to be sure they had a good plan (and that they were implementing it correctly): How would it be possible for the Region to provide these services to the entity, then turn around and audit them later without having their auditor independence completely compromised?

The key to resolving this question is that the Region will need to have what is known as an Entity Development program in place; currently, one Region does have such a program, and I was told another Region is now putting one in place. The point of this program is for the Region to have a formal way of providing advice, like the above, to entities outside of an actual audit[v]. In general the staff members for this program will be separate from the auditors, although the auditor pointed out to me that it isn’t impossible for the CIP auditors to also provide Entity Development services, assuming the entities trust them not to mix the two functions.

One absolute requirement for the Entity Development staff is that they be knowledgeable in the subject matter of the plans they are providing advice on – for example, if they are providing advice on the Physical Security Plan in CIP-014, they need to understand physical security for substations. Of course, this requirement isn’t something to be taken lightly, since such people – with electric utility experience – may be hard to find. So putting together this staff may be a multi-year process.

While the Entity Development staff will in theory just be providing best practices-type information to the entity, it is likely that they will sometimes, in the process of reviewing an entity’s plan, discover some element of non-compliance. When this happens, they will point this out to the entity, but they won’t be able to provide advice on what the entity should do to remediate this non-compliance; that would probably compromise auditor independence.

Of course, the Entity Development staff wouldn’t report these instances of possible non-compliance to the auditors, and the report wouldn’t become part of the record for the entity’s next audit. However, when receiving a report like this from Entity Development, the entity that requested the plan review will need to decide whether to self-report a violation; and if they do self-report, they need to also “discover the scope and extent of the non-compliance” (to use the auditor’s words), as well as mitigate the non-compliance[vi].

If the entity does self-report, and the issue that was the subject of the report is discovered as part of the next audit, the entity won’t be subject to a Potential Non-Compliance (PNC) finding for the same issue, covering the same time period that was self-reported. Of course, if the entity doesn’t self-report and the potential violation is discovered, the entity would be subject to a PNC, which of course will be more serious since it was discovered at audit.

The auditor did also point out that there is a way that the Region can provide advice on whether a plan is likely to be found compliant or not, but they can only do this during the period before a standard becomes enforceable. If the entity develops their plan and specifically asks the Region for a “readiness assessment” of the plan, then, depending on available time and resources, Regional staff (either auditors themselves or Entity Development staff) can perform, to quote the auditor, “a non-binding, no risk, no consequence ‘audit’” of the requirement for developing the plan.[vii]

For example, suppose your entity wishes to have your Region review your CIP-013 supply chain cyber security risk management plan before the CIP-013 compliance date (which I am currently expecting to be toward the end of 2019, assuming FERC approves CIP-013 this spring). You would of course have to first develop the plan (and this would have to be done well before the compliance date), then request a readiness assessment of your compliance with CIP-013 R1.

Of course, developing your CIP-013 plan in the first place will require having some idea of what should be in the plan. There is the 13-page Implementation Guidance produced by the CIP-013 drafting team; this is a very good document as far as it goes, but it is nowhere near a comprehensive guide to developing the plan. There will also be more guidance coming from other sources (including specifically the North American Transmission Forum, although I’m not sure that will be available to non-members), and NERC is now considering a CIP-013 “Transition Study” similar to the one for CIP v5. In this study, a small number of early adopters will share their experience with NERC, to help them develop Lessons Learned (remember those?) and other guidance.

And there will be another source of guidance: I have been thinking about what should be in the CIP-013 plan and discussing this with an auditor, and I plan to write a series of posts (probably not consecutive posts, of course) about this question. Unlike the NATF, I’m not on NERC’s list of approved guidance providers, so you’ll of course have to take whatever I say with a grain of salt (but in fact the same applies to the approved providers like NATF, since NERC specifically says that no guidance these approved providers turn out they will itself be “approved” by NERC). But I hope you’ll find my posts on this subject to be helpful, and I’ll welcome any feedback you have on them.

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

[i] I say “effectively” because I now have a better way of wording the question than I did when I wrote that post.

[ii] This is because the requirements in CIP-013 provide very little “guidance” on what should be in a good supply chain security plan, even less so than the guidance that CIP-014 R5 provides to guide entities in developing their physical security plans.

[iii] Of course there is, and will be, guidance provided on these questions by various industry organizations. However, the guidance NERC provides will be very limited due to NERC’s limited interpretation of what guidance they are allowed to provide. NERC entities will need to weigh all the guidance they find, but in the end what will count most for them is what their Region says. This is the way it has been since CIP version 1, although this trend intensified with CIP v5.

[iv] It bears repeating that the Region’s review of the plan won’t be for the purpose of saying whether it is compliant with the requirement or not, but only a) whether all of the threats that should be mitigated in the plan are in fact addressed; and b) whether the proposed threat mitigations actually follow best practices. It is possible that whoever reviews the plan will notice something in the plan that is non-compliant and will point this out to the entity; it would then be up to the entity to decide whether they want to revise their plan to address this concern, or whether they think the observation is mistaken for some reason. In any case, the observation made by the Region wouldn’t become part of the audit record, and wouldn’t be passed to the auditors when the entity was next audited.

[v] The auditors have always been able to point out Areas of Concern covering cyber security practices that aren’t within the scope of CIP, when they notice something regarding the entity’s practices during the course of an audit. But there has never been an official way for them to provide such advice outside of an audit – and, as I pointed out earlier, an audit will often come a year or two after the entity has started implementing their plan. It will be much better if the plan can be reviewed as soon as it is developed.

[vi] The auditor did explicitly warn against what might be a temptation: agreeing with the opinion that your plan was non-compliant in some way and fixing whatever the problem was, but then still not self-reporting the issue. While it is true that the friendly advice of possible non-compliance that you receive from Entity Development will not in any way be reported to the auditors (and even if it were, they would ignore it), it is still very likely the auditors will discover that at a certain point your documentation changed, from reflecting the old non-compliant wording in the plan to reflecting the new compliant wording. Of course, the penalty for non-compliance discovered in an audit is likely to be much more severe than for a self-report.

[vii] Of course, the readiness assessments are nothing new. The Regions conducted a number of these during the run-up to the CIP v5 enforcement date. They were very helpful, both to the entities that received them and to the auditors that conducted them. However, the auditor did caution that there is no way that the readiness assessment will be able to issue an opinion that the plan seems to be compliant. The team will point out gaps in compliance and recommend steps for remediation; after that, the entity is on its own to determine what it should do with the information.

Monday, January 1, 2018

An (Impressionistic) History of NERC CIP

A friend – who probably knows a lot more than I do about NERC CIP, but came into the game later than I did - asked me recently to write a post on the history of CIP. I’m finally doing that, although I’ll say at the outset that this isn’t meant to be your standard Wikipedia article. It’s focused on what I know personally, and you have to be very careful on what you take away from this post as facts vs. opinions. That being said, I hope you’ll find it interesting. I was originally planning on calling this a “brief” history of CIP, but as you can see the brief part fell by the wayside (I've never been very good at brevity, anyway). You hereby have my permission to read this post in two or three sessions. I probably won’t be able to write another post this week, anyway.

The origins of CIP were long before I became involved with it (2008). I’m told there were some sort of market access standards (from FERC) that required cyber security measures around 1999 or 2000, but I’ve been unable to verify that in my extensive five-minute search of the internet. I have to assume this is actually some sort of creation myth dreamed up by pre-(cyber-)literate peoples as they chewed on mastodon bones while sitting around campfires.

NERC cyber security standards first enter the historical record around 2003 with NERC Urgent Action 1200, aka NERC Cyber Security Standards (CSS); like all NERC standards before passage of the Electric Power Act of 2005, compliance with them was voluntary. The standards applied only to control centers. They were later replaced by UA 1300, which now applied to generation as well as control centers.

After passage of EPAct, the UA 1300 standards were submitted to FERC as CIP-002 through CIP-009 in August 2006. After issuing a NOPR (Notice of Proposed Rulemaking) in July 2007, FERC approved these standards as CIP version 1 in January of 2008 in their landmark Order 706.

You’re free to read all 221 pages of Order 706, but the highlights of the order are:

  1. FERC approved the CIP v1 standards.
  2. At the same time, FERC ordered NERC to make “revisions” to these standards. Since NERC’s Rules of Procedure don’t allow any standard to be revised once approved by the NERC Board of Trustees, and since all standards are approved by the BoT before even being submitted to FERC for their approval, these changes would have to be in a new version of the standards, version 2. Accordingly, NERC constituted a new drafting team, dubbed CSO 706 for Cyber Security Order 706; the team first met in the fall of 2008.
  3. IMHO, the primary changes FERC requested in Order 706 were:
    1. The CIP v1 drafting team had used the phrase “reasonable business judgment” in a number of places in the standards, to indicate that certain questions were left up to the entity to decide. FERC didn’t like this language, and ordered it removed.
    2. The team had also used the phrase “acceptance of risk” in the standards. This is a common phrase in cyber security standards and guidelines in general. It means that the entity, in cases where mitigating a particular cyber risk may be too difficult or costly, agrees to instead accept the risk and move on. FERC pointed out – correctly, in my opinion – that the risk involved in the CIP standards, as well as all NERC standards, is risk to the Bulk Electric System, not to the organization itself. No organization can decide to accept risk on behalf of the whole BES.
    3. The drafting team had also used the words “where technically feasible” in multiple places, meaning that, if a particular requirement couldn’t be complied with for a particular type of device in scope (for example, a switch that couldn’t comply with the antivirus software requirement because there was no way for the end user to load such software on the switch), the entity would only need to point out that fact at audit. However, FERC ordered NERC to “develop specific conditions that a responsible entity must satisfy to invoke the ‘technical feasibility’ exception.” This led to NERC’s later development of the TFE process, which probably shortened the lives of a number of CIP compliance professionals by 5-10 years.[i]
    4. The Commission directed NERC to “provide additional guidance regarding the development of a risk-based assessment methodology for the identification of critical assets pursuant to CIP-002-1.” They were already skeptical that NERC entities’ use of the RBAM, as specified in CIP-002-1 R1, would end up identifying many Critical Assets, and they were right. Other than Control Centers, very few generating stations were identified as Critical Assets under CIP v1-3. There were a number of substations that were so designated, but that designation was usually obviated by the fact that, in CIP v1-v3, if there was no external routable connectivity at an asset, it wouldn’t have any Critical Cyber Assets – meaning no systems would be in scope at the substation. Because the great majority of substations at the time were not routably connected to the outside world, there were very few substations with CCA’s. This meant that there were very few substations that were effectively in scope for CIP, even though they might have been designated Critical Assets.
    5. FERC also ordered “external review of critical asset lists…” FERC didn’t directly tell NERC what they were asking for in this regard, but NERC did develop the “bright line criteria”, which first appeared in CIP v4. The criteria did away with the RBAM altogether and simply specified what types of assets would become Critical Assets, based on various technical conditions. When the CSO 706 drafting team started work on CIP v5, they modified the criteria so that they dropped the reference to Critical Assets, replacing it with High, Medium and Low impact levels.[ii]

When the CSO 706 SDT started meeting in 2008, they decided to triage the directives from Order 706. They would first address the “low-hanging fruit” in CIP v2, then address the rest of the directives in v3. So they immeidately developed CIP v2, which eliminated both the “reasonable business judgment” and “acceptance of risk” language from the v1 requirements. V2 was quickly balloted and passed, approved by the NERC Board, submitted to FERC and finally approved by FERC in September 2009.

When FERC approved CIP v2, they brought up a new requirement – for procedures for escorted physical access into Critical Assets by persons not authorized for direct access to Critical Cyber Assets - that they wanted to see added quickly; in fact they required that the new CIP version be approved and submitted to them in 90 days. NERC accomplished this feat, and FERC approved CIP v3 in March of 2010.[iii] It came into effect that October. 

Once the CSO 706 SDT had drafted CIP v3, they turned their attention to CIP v4. They had a radically different idea for v4, though: They would just have two standards. CIP-010 would replace CIP-002, and it would provide a new way of identifying assets in scope for CIP: through bright-line criteria. Also, those assets would no longer be classified as Critical Assets or not in scope; they would be classified High, Medium and Low impact. Furthermore, the systems in scope would be called by a new term, BES Cyber System. All of the requirements in standards CIP-003 through -009 would be combined into CIP-011, and there were many changes proposed for these requirements. The first drafts of these two proposed standards were developed in early 2010; the SDT expected to be able to get these two new standards approved by NERC by the summer of 2010.

This radical proposal led to a firestorm of suspicion among NERC entities, and 600 people signed up to attend as observers – in person or by phone – for the SDT’s next meeting, in Dallas in April 2010. At that meeting these observers weren’t exactly shy about registering their profound concern.[iv] It was clear that the SDT's proposal would never be approved in anything like its current form.

After this meeting, the SDT retreated to their phone meetings to decide what to do; they clearly couldn’t go ahead with the CIP-010/-011 proposal now. One of the big objections to the proposal might seem pretty mundane, but really wasn’t: By changing the numbering of the standards so drastically, NERC entities were going to have to make some huge changes in computer systems as well as other things like training programs, which were all based on the CIP-002 through CIP-009 scheme.

In the aftermath of the Dallas meeting, some CEOs of large utilities became convinced that, if NERC didn’t develop some new CIP version by the end of 2010, FERC would completely take away from NERC the task of developing new CIP standards. So somebody developed the idea of just replacing CIP-002-3 with a new CIP-002 that would implement bright-line criteria for identifying Critical Assets, but leave everything else in v3 (including identification of Critical Cyber Assets in CIP-002 as well as CIP-003 through -009 in their entirety) unchanged. This would address FERC’s primary concern, which was that too few assets had been designated Critical Assets, due to the looseness of the criteria for developing a risk-based assessment methodology in CIP-002-3 R1.

And this is what happened. The SDT got to work[v] adopting the bright-line criteria to CIP-002 (and making some changes to them from the original CIP-010 draft), but didn’t touch any of the other standards (except to change their version numbers, so all of the new draft standards were version 4). The final ballot approving CIP v4 occurred at the end of 2010, and v4 was submitted to FERC immediately.

With v4 behind them, the drafting team started work in January 2011 (at a meeting located at the headquarters of a large utility, during which meeting the power went out in most of the building for a couple hours - Oops!) on CIP v5, which was now going to be the version that finally provided most of the changes that FERC had asked for in Order 706 (plus some more changes that NERC proposed, like the concept of BES Cyber System. By the way, that term originated in a “concept paper” in June 2009. This paper introduced the BCS idea, as well as the idea of “BES Reliability Functions”, which later became the BES Reliability Operating Services).[vi]

And a funny thing happened: Even though what the drafting team was proposing was quite different from CIP v3, as the year 2011 wore on, popular support seemed to be growing for the new version; in fact, some SDT members began to regret that they’d been rushed into developing v4, when clearly v5 would do everything v4 does and a lot more. The hope was that FERC would never approve v4, and v5 would become the next version.

That’s why it was a shock when, in September 2011, FERC issued a NOPR saying they intended to approve CIP v4. At the time, I thought[vii] FERC wasn’t all that serious about this – that they were trying to get all of NERC to support v5. Why would it be in anyone’s interest to make a big transition to v4, then a year or two later make an even bigger transition to v5?

If FERC was trying to pressure the NERC ballot body to get behind v5, the strategy didn’t work. The first draft of v5 was posted in November 2011, and was roundly voted down in the ballot in December (the highest percentage of the vote that any of the standards achieved was around 25%). But the SDT wasn’t fazed, and at their next meeting, at ERCOT’s headquarters in January 2012[viii], they made a number of changes to the standards. Probably the most important change was eliminating any requirement for Low assets that might require identifying individual Low BCS; in fact, the SDT went further and inserted wording in two or three places in the standards to the effect that an inventory of Low BCS wasn’t required. Those assurances remain in place today, of course.

But FERC had another surprise up their sleeve. In April 2012, they approved CIP v4. My opinion remained[ix] that they weren’t serious about wanting v4 to come into effect (which was now scheduled for April 1, 2014). Instead, they were sending NERC entities another message: “You’d better get behind CIP v5 now. If you don’t, you’ll definitely have to comply with v4, plus you’ll still have to comply with v5 after that, because it will ultimately pass. Plus, the NERC Board has the power to write the standard and send it to us for approval without balloting, in case you’d prefer to do it that way.”

Whether the NERC membership was taking FERC seriously, or whether the further changes made to v5 over the course of 2012 convinced them that it was now a workable CIP version (I suspect the latter), the NERC membership did get behind CIP v5. It finally passed the fourth ballot late in the year, and v5 was submitted to FERC at the end of January 2013.

However, even though some people at NERC were expressing confidence that FERC would quickly approve v5 and at the same time say that v4 would never come into effect, I had become convinced that, since v4 was the law of the land (which it was after FERC approved it), NERC entities had no choice but to go full speed ahead in getting ready for v4. After all, as of January 2013 (when I started writing this blog) there were only 14 months left before the v4 compliance date. I even wrote a post in February 2013, saying that it was highly unlikely FERC would approve v5 anytime soon (in that post, I also reiterated a call I’d previously made for the compliance date for v4 to be moved back). I now call this my “Dewey Beats Truman” post, since two months later FERC issued a NOPR saying they intended to approve CIP v5, and that v4 would never come into effect.

FERC approved CIP v5 in Order 791 on November 22, 2013[x]. As with all their previous CIP orders, they also ordered some important changes, which of course became “CIP v6”. NERC put in place a new drafting team to implement these changes, having decided that, after four or five years of continual meetings, the CSO 706 team deserved a break (although two or three of the members, being gluttons for punishment, continued on the new team). Significantly, this new team was called the CIP Version 5 Revisions SDT. This wasn’t an accident; with all of the back-and-forth and “start-no, stop-no, start again!” of the CIP v4/v5 debacle, the industry was hardly in the mood for a nice new version to sink their teeth into. So the words CIP version 6 were evidently verboten in NERC circles; I didn’t hear or see the term used until Scott Mix of NERC used it at a meeting on the new version in Atlanta in early 2015.

Meanwhile, FERC had another surprise up their sleeve. In March 2014, FERC issued an emergency order in response to the Metcalf substation attack in 2013 (which had recently caused a lot of alarm in Congress), mandating that NERC develop a new standard for physical security of critical substations – and that they approve it and submit it to FERC in 90 days. Needless to say, this was a big task, but it was accomplished, and CIP-014 was approved by FERC that fall.

The “CIP v6” SDT worked throughout 2014 on drafting and then balloting their revised standards. There were some twists and turns along that way, but the "v6" standards were finally approved by the NERC ballot body at the end of 2014 and submitted to FERC in February 2015. In July of 2015, FERC issued a NOPR saying they intended to approve CIP v6. But FERC went beyond that to discuss new or modified standards they were considering ordering NERC to develop, probably the most important of which was a standard for supply chain security.

However, this wasn’t all that was going on in CIP World. In fact, the major story was that throughout 2014 and 2015 there were many discussions and battles in the industry about the meaning of the CIP v5 standards[xi]. NERC issued various documents – FAQs, Lessons Learned and Memoranda – many of which ended up being rescinded. NERC finally decided that only a new drafting team, developing even newer versions of some of the CIP standards, could clear up these issues.

As it turns out, the new SDT would have more on its plate than just trying to fix problems from CIP v5. In January 2016, FERC issued Order 822, approving CIP v6 and – as usual – also mandating some more changes in CIP. These changes were added to the new drafting team’s SAR (and the new drafting team was called the “Modifications to CIP Standards” SDT – note there’s no mention of any version numbers at all in that name). This team started meeting in the spring of 2016, and immediately focused on two items mandated by FERC, one of which had a one-year deadline: the clarification of the meaning of LERC and a requirement for Transient Cyber Assets used at Low impact assets. These changes were balloted and approved in 2016, and submitted to FERC in early 2017. In October, FERC issued a NOPR saying they intend to approve these changes and – of course – asking for comments on some further changes they may order regarding Lows. Assuming FERC orders those changes when they approve the two changes that were submitted to them early this year, these will most likely be added to the current SDT’s agenda.

Meanwhile, the other big development of 2016 (regarding CIP, of course. I think there was also some election in November 2016 that some people thought was significant. I, of course, was too caught up in NERC CIP issues to pay much attention to that) was that FERC ordered that NERC develop a supply chain security standard in July. This standard was indeed developed, approved and submitted to FERC at the end of September. It now awaits FERC’s approval.

The only other significant CIP development I haven’t already mentioned was the startling announcement that I relayed in my post dated April 1, 2017, that the US had decided to deploy NERC CIP to Mexico….And on that note, I’ll sign off. It’s been fun writing this summary of what’s gone on in CIP since 2008. I’ll plan on doing this every ten years from now on. Mark your 2028 calendar now!

Happy New Year!

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

[i] At the September NERC CIP meeting, Tobias Whitney of NERC presented a slide that showed the number of TFE’s submitted annually had fallen from in the thousands under CIP v3 to under 200 in 2016; of course, this was because CIP v5 was designed to be “TFE-friendly”. Instead of requiring particular technologies in all cases, CIP v5 deliberately makes use of terms like “per device capability”, so that the entity can choose different ways of mitigating a particular cyber risk, rather than have to implement just one particular technology. The best example of this is CIP-007-6 R3, where antivirus software is no longer specifically required. Instead, the objective of the requirement is malicious code prevention, and the entity can choose whatever method is best for any particular device in scope.

[ii] Of course, the drafting team made a further modification to the criteria, in which they tweaked the wording to indicate that the criteria applied to BES Cyber Systems, not actually to the assets themselves; however, the criteria themselves still apply to the asset, not the systems. As anyone who was following this blog in 2013-2015 knows, I thought this was a very bad decision that was going to cause endless confusion going forward. But guess what? I was totally wrong about this. Even though there is officially no such thing as a High impact, Medium impact or Low impact asset, I doubt that ten percent of NERC compliance professionals or auditors even know that this is the case. But it really doesn’t matter, since exactly 0% of compliance professionals and auditors are actually acting as if it isn’t the assets that are classified, but instead the BCS. And I’m fine with that; as long as there is complete agreement that this is what CIP-002 R1 and Attachment 1 say, there’s no point in standing on a rooftop and yelling that this isn’t the case.

I used to think that the chickens would come home to roost on this issue the first time a NERC entity was fined for not properly classifying their BCS at a particular asset – say, they classified BCS at a particular asset as Low impact when NERC thought they should have classified them Medium impact.  The entity would then take this dispute to an Administrative Law Judge (which NERC entities are always allowed to do, since all NERC standards, like all FERC-approved standards, are regulatory law). He or she would spend 15 minutes trying to make sense of CIP-002 R1 and Attachment 1 – as I did in my first-ever post on CIP v5 in 2003, although I spent a lot more than 15 minutes writing it! - and proclaim “What is this (stuff)?” He/she would throw out the penalty, but might even rule that any dispute over classification of BCS was undecidable because of the ambiguity and outright contradiction in CIP-002. I no longer think this is likely to happen, though, mainly because the ultimate enforceability of CIP v5/v6 has been undermined in other ways as well. In any case, no case against NERC for a CIP penalty has ever been fully adjudicated, and I doubt one ever will be. Too much depends on trust between NERC, the regions, and the entities, for an entity to want to break that trust by filing a lawsuit.

[iii] The single change that FERC ordered only applied to CIP-006, so NERC might technically have just revised that one standard to version 2, leaving the other standards at the v1 level. However, the drafting team decided to change the numbering embedded in all of the CIP standards to v3, even though that meant having to submit all of the standards for ballot. When the next drafting team – charged with implementing the changes that FERC ordered in Order 791, which approved CIP v5 in 2003 – faced a similar decision, they decided only to revise the standards where there had been a substantive change and leave the others at the v5 level, which is why NERC entities now have to comply with both CIP v5 and v6 standards. In the next couple of years, there will be CIP-003-7 (due to the LERC changes) and CIP-005-7 (to implement changes that accompany CIP-013); plus within 3-4 years CIP-003-8 will come along. And of course, CIP-010 and CIP-011 have their own version numbers; they’re both on v2, although CIP-010 will go to v3 once CIP-013 is approved by FERC. So, as I’ve pointed out recently, we all need to get rid of the idea that there are “CIP versions”. Instead, the CIP standards should all be considered as having their own version numbers; it is up to the entity to make sure they are always dealing with the current effective version of each standard.

[iv] I didn’t attend that meeting in person, but I did by phone. The tension in the room was readily apparent even 1,000 miles away.

[v] I attended the August 2010 SDT meeting in Chicago – the first one I’d ever attended – and wrote about it in an “open letter” published under the Matrikon name (Matrikon was the company I joined one month before they were acquired by Honeywell in June 2010, but for a year we continued to operate as Matrikon). I later published some posts in a Honeywell cyber security blog, but started my own blog in January 2013. If anyone wants to see this open letter, email me at tom@tomalrich.com. It also tries to explain the big change from the original CIP v4 proposal (i.e. the one rejected in Dallas) to what ultimately became CIP v4 – although of course that was never implemented.

[vi] I also attended this meeting and wrote about it in an open letter. You can email me if you’d like to see it.

[vii] And wrote about it in a post on a now-nonexistent Honeywell blog. I hopefully have the text of that post on a USB stick at home, but I won’t be there for a week. If you would like to see that, email me and if I’ve found it I’ll send it to you.

[viii] I also wrote about this meeting on the Honeywell blog. Again, I’ll know next week if I still have a copy of that post. Email me if you’d like to see it.

[ix] Again, I hope I still have the text of the post I wrote about this. Email me if you’d like to see it.

[x] 50 years almost to the hour after the assassination of John F. Kennedy. Coincidence, you say? Well, I don’t know…

[xi] I must have written 50-80 posts on various controversies and developments regarding CIP v5 in those two years. You’re welcome to go back and read all of them. And if you do that, please tell me what they all said! I myself have no stomach for doing that.

Friday, December 29, 2017

Two Holiday Presents for you – Present Number Two

This is the second of two posts that contain unadulterated good news for entities subject to the NERC CIP requirements. You make think my motivation for doing these posts is that I was visited by three ghosts on Christmas Eve who told me to mend my ways. It isn’t that, but rather the realization that I always have a big queue of things I want to write about, and I should try to prioritize the good things to post around holiday season.

In November, I wrote this post about the NOPR that FERC issued in October, stating their intention to approve CIP-003 version 7. This is the version of CIP-003 that was much debated and approved last year (i.e. in 2016), which eliminated the definitions of LERC and LEAP and incorporated those concepts into the requirement itself (and if you’re hazy about what this debate was all about, this post and this one may refresh your memory. My memory certainly needed refreshment!). Of course, this whole discussion has to do with Low impact assets (mostly substations) that have some sort of routable connection to the outside world.

As discussed in my November post, what I found most surprising in the NOPR was that FERC clearly stated their intention to push NERC to go well beyond what is in CIP-003 v7. That standard, which will come into effect 18 months after FERC issues the Order they committed to issuing in their NOPR, requires that an entity owning a Low impact asset, for which there is at least one routable communications stream with a cyber asset outside of the boundary of the asset itself, have in place some means of mitigating the risk posed by that communications. It could be a device like a firewall (formerly known as a LEAP), but it could be something else like network separation, a unidirectional gateway, etc.

In the NOPR, FERC said they will approve v7 as written. However, they also asked for comments on going beyond what is in v7 to require further steps for Low assets. Specifically, they pointed to authentication and password complexity as two of the four items[i] they would now like included in the requirement for securing BES Cyber Systems located at Low impact assets that contain some form of external routable connectivity. They might also ask for more than those four items. Of course, since NERC’s Rules of Procedure don’t allow changes to be made to a standard once it has been approved by the NERC Board of Trustees (and all new standards are approved by the BoT before they’re even submitted to FERC for their approval), these changes will be in a new version of CIP-003, which will be version 8 (and that version won’t be effective for 3-4 years from when FERC orders it).

In my November post, I pointed out at the end that I thought it was clear that the changes FERC is proposing will doom what has been a bedrock principle of NERC’s Low impact compliance program since CIP v5: that no inventory of Low impact BES Cyber Systems will be required. My reason for saying this (not stated in the post) was that I simply didn’t see any way that this principle could be preserved if CIP-003 is going to require authentication and password complexity for Low impact BCS. It seemed to me that there would be no way these requirements could be audited, absent an inventory of Low impact BCS.

However, a couple weeks after that post, an auditor wrote in to me with critiques of several of the points I had made in the post, including the one I just cited. I will quote in full what he said on this topic:

“(Auditing the new requirements that FERC is considering can be done without requiring the entity have) an enumerated list of Low Impact BCS or their component Cyber Assets.  Now, as an auditor, I may ask the entity to demonstrate that the controls have been implemented, so at that time I may ask for a list of the relays in a sampled substation.  Or, knowing that there will be a breaker relay in the substation for a circuit breaker on a Transmission Line or bus, I might ask for the Station 1-Line diagram or SCADA substation display, point to a breaker drawn on that diagram, and ask for evidence associated with that breaker’s relay.  I might even visit a randomly selected substation, point to a SEL-421 distance protection relay and inquire how it is managed.  Remember that relaying engineers typically do not do anything without a work order and all sorts of authorizations, so being able to come up with the work order to change a password on that very device might not be an insurmountable challenge requiring additional records keeping not already being done.  And if they are not managing the passwords on the relays, then shame on them.  Then again, if they implemented the SEL-3620 or equivalent, I don’t need to look at any controls on the relays because access is well managed at the gateway.  The point is that there is still no mandate that the entity identify every Cyber Asset in a Low Impact environment, identify the subset of those Cyber Assets that are Low Impact BCAs, and group them into a documented list of Low Impact BCS, and I can envision how the new requirements can be implemented and audited without requiring a list of Low Impact BCS.”

To paraphrase this quotation, the auditor points out at the end why many NERC entities are so worried about possibly needing to maintain an inventory of Low BCS. To do this will require doing pretty much everything that needs to be done to identify BCS at Medium and High impact assets now: identify every Cyber Asset at the Low impact asset, use the BES Cyber Asset definition to determine which of these are BCAs, and finally group BCAs (and possibly other Cyber Assets) into BES Cyber Systems. Then repeat this on a regular basis, so as to include new Cyber Assets that may have been added.

But he makes it clear that he thinks I’m wrong in (implicitly) stating that requiring authentication and password complexity on Low BCS will inevitably require an inventory. I assumed this was inevitable because there would be no way to audit these requirements without an inventory, but he thinks I’m simply wrong in this assumption. He points out several ways he could audit these requirements without demanding that the entity produce an inventory of all of their Low impact BCS.

And now that I read his words again, I realize I was making a false analogy from Medium and High impact BCS. While these BCS are also required to have authentication and password complexity, these and other such requirements aren’t the reason why an inventory of Medium and High impact BCS is required under CIP v5/v6. The reason an inventory is required is because CIP-002 R1 requires it, period. And CIP-002 R1 not only doesn’t require an inventory of Low BCS, it explicitly states that such an inventory won’t be required.

All of this isn’t to say that, once CIP-003 version 8 comes into effect, NERC entities won’t be under even more pressure from their regions to maintain an inventory of Low impact BCS. Some of the regions have already stated that they would like to see their entities do this, and they will be able to make that statement with even more justification once CIP-003-8 comes into effect. But until the statement that an inventory of Low BCS isn’t needed is actually removed from the CIP standards – and now it’s found in CIP-003-6 and CIP-003-7, as well as in CIP-002 R1 and Attachment 1 – I think NERC entities can rest assured that nothing fundamental has changed in this regard, no matter what requirements end up in CIP-003-8. 

The views and opinions expressed here are my own, and do not reflect those of any organization I work with. If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com.

[i][i] For a list of the four items FERC is proposing to require, see bullet point 4 toward the end of the November post.