Thursday, March 15, 2018

NERC’s Supply Chain Webinar: Good, but I still have Questions

I eagerly awaited NERC’s Supply Chain webinar yesterday. I found it was a good webinar on supply chain security, although the CIP-013 discussion raised more questions in my mind than it answered.

To start with the good news, I thought two of the presentations, by my friends Corey Sellers of Southern Companies and Joe Doetzl of ABB, were excellent – and I’m looking forward to seeing the slides for their presentations, as well as those of the other speakers, including Tobias Whitney of NERC. Corey, who is head of the team that drafted CIP-013 and guided it – with help from others, of course - through the shoals of four NERC ballots to deliver it to FERC in time to meet their one-year deadline, spoke mainly about the CIP-013 guidance document that the North American Transmission Forum (NATF) is putting together. This sounds like an excellent document, although Corey pointed out that it’s mainly focused on best practices for supply chain security, not CIP-013 compliance per se.

Joe Doetzl, who was with KCPL for many years and was an important part of the CSO706 drafting team as they drafted CIP versions 2, 3, 4 and the beginning of v5, gave a really excellent presentation on what you (i.e. a NERC entity) should expect from your software vendors (his slides had much more information than he could get through in his presentation, so I’m especially looking forward to seeing those).

Now to the not-so-good news (although I’ll say upfront that this isn’t bad news): I still have a few fundamental questions about how entities should comply with CIP-013, and these weren’t answered. Or if they were answered, the answers seem at odds with what is written in the requirements. Here’s what I mean:

  1. My biggest question is what should be in the supply chain cyber security risk management plan, which is of course required by CIP-013 R1. In the presentations by Tobias and Corey, they focused exclusively (as far as I could see) on the six items that are required by R1.2. The implication is that these are the only things that must be in the plan. 
  2. As I have pointed out in several posts (at least), CIP-013 R1.1 requires the entity to “identify and assess[i] cyber security risks to the Bulk Electric System” resulting from a) procuring vendor equipment and software, b) installing vendor equipment and software, and c) transitions from one vendor to another. The six items in R1.2 all have to do with a), not with b) or c). And even for a), there are clearly many more risks to be addressed than the risks underlying the six items in R1.2. For example, how about the risk that a vendor won’t follow good security practices for their own network, and will experience a breach where information on your software and how it is configured is stolen? This has happened with at least one SCADA vendor in North America in the past four or five years.
  3. So is the entity required to identify and assess other risks than the risks behind the items in R1.2? I’m sure Tobias and Corey would answer this question in the affirmative, but then the question becomes “Given that there are a huge number of supply chain security risks – some with extremely small impact or probability – that could be identified, how many does the entity have to identify? Is it a certain number like 10 or 20? Or is it something like “all risks with high impact and probability”? And if that’s the case, who will determine which risks meet that description? The auditor? The entity? Someone else?
  4. CIP-013 R1 is one of a category of what I call “plan-based requirements” which seem to have found favor with the CIP drafting teams in the last few years, as the only type of CIP requirement they should consider drafting. I am all for that, since I think – as do a lot of others – that the alternative - prescriptive requirements like CIP-007 R2 - just doesn’t work for cyber security, period. But, as I pointed out in this post (and a few subsequent ones), the only way I can see a plan-based requirement being successfully audited under NERC’s auditing procedures is to have criteria for what should be in the plan in the requirement itself. My best example of this is CIP-010 R4, where Attachment 1 (which is called out in R4, and is thus part of the requirement) provides a very detailed list of items that should be addressed in the plan for Transient Cyber Assets and Removable Media – e.g. software vulnerability mitigation, malicious code mitigation, etc. When reviewing an entity’s CIP-010 R4 plan, the auditor can fairly easily go through this list and determine whether the entity has addressed (and meaningfully addressed) each item on the list. Unfortunately, no such list is to be found in CIP-013-1 R1.
  5. My second question is about the meaning of “mitigation” of risks (as I pointed out in the first end note below, this word was left out of R1, but clearly is meant to be in there). As far as I could see, both Tobias and Corey were only considering vendor contract language as a mitigation means for the six risks implied in R1.2. Yet, as I’ve pointed out in two recent posts (this one and this one), contract language is only one means of risk mitigation, and IMHO it isn’t anywhere near the best one.
  6. But suppose you follow Tobias and Corey (and I’ll readily admit that both Order 829 and the NERC Implementation Guidance for CIP-013 take the same position as these two gentlemen) and first try to get your vendors to accept contract language for the six items in R1.2. And suppose a few of them absolutely refuse to insert the language you ask them to include. Furthermore, suppose these few vendors are absolutely critical to your organization, and it would be very disruptive and costly to try to replace their product with a competitor’s. What do you do then? Do you not have to do anything more to mitigate[ii] the risk?
  7. To answer this question, look at the title of CIP-013: “Cyber Security – Supply Chain Risk Management” (my emphasis). You are supposed to manage each risk, not simply try one particular mitigation strategy and then give up if that fails. In this post, an auditor described an alternative strategy – and of course it’s one of many alternatives – that an entity could use if the vendor refuses to provide prompt notice of employees who either leave the company or no longer need access. In my opinion (and the auditor agrees with me), you need to be prepared to try multiple approaches to mitigating each risk that you identify.

However, I’m not waving a red flag yet. CIP-013 hasn’t even been approved by FERC, so there is still some time to address these questions[iii]. But – again, in my opinion – these two questions need to be addressed as soon as possible, since the entities will need to know the answers to both of them before they start developing their plans.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

[i] As I pointed out in this post, it seems an important word was left out here: mitigate. Your plan must describe how you will identify and assess risks, but the plan doesn’t have to say a word about what you will do about those risks! And since R2 just requires implementation of the plan, mitigation doesn’t automatically come in there, either. Of course, all of the other documents related to CIP-013, including FERC’s Order 829 and the NERC Implementation Guidance, make it abundantly clear that mitigation of these risks is the purpose of the standard – so I don’t recommend that you decide to stonewall it and only include identification and assessment of risks in your plan! But this does need to be fixed at some point. Assuming FERC orders some changes to CIP-013-1, as their NOPR strongly hinted they would, this should be included in the SAR for CIP-013-2.

[ii] Tobias seemed to strongly indicate this was the case, when he talked about what to do if a vendor won’t provide a means to authenticate identity and integrity of vendor-supplied software and patches. He didn’t talk about anything else you could do to mitigate the risk of a bad actor inserting malware into a patch, when there are clearly other things you could do (ask to download the patch from a secure FTP site, ask the vendor to burn the patch on a CD and overnight it to you, etc). BTW, I wasn’t able to ask a question about this because it was announced at the beginning of the webinar that there wouldn’t be time for questions from the online webinar audience.

[iii] Although the fact that FERC wants to shorten the implementation time for CIP-013 from 18 to 12 months, and that I believe they’ll approve the standard in Q2, means there isn’t much time remaining.

Sunday, March 11, 2018

Complying with CIP-013 Part 6: A (near-fatal) Flaw in the Standard

I have come to realize that CIP-013 suffers from a very serious flaw. It would be a fatal one, were it not that the human spirit is infinitely resourceful, and I think the NERC Regions will rise to the occasion and develop a work-around for this flaw. While I have kinda sorta known of this flaw for a while, it’s only recently that I’ve been able to articulate it, and also realized what the solution is.

I want to emphasize that I still stand by what I said in a post last September: CIP-013 is the closest to my idea of the ideal NERC CIP standard of all the CIP standards, both those currently in effect and those that have “retired”. However, it nevertheless does suffer from the serious flaw, which I will describe in this post.

To understand the flaw, I need to go back to my series of posts late last year that discussed “plan-based” requirements (which includes the requirements in CIP-013, of course). In those posts, I came to the conclusion that a requirement to develop a plan can’t simply tell the NERC entity to develop a plan to mitigate a certain class of threats (in the case of CIP-013, these are supply chain threats), then leave it up to the entity to determine what threats they should address in their plan. The requirement to develop the plan needs to include a list of threats (although I called them “criteria” in one or two of the posts on plan-based requirements) that should be addressed in the plan. This should be a comprehensive list of all the threats that the drafting team felt should be included. Of course, the entity is always welcome to add to it, but the drafting team needs to assume that a threat that isn’t on their list usually won’t be addressed in the plans.

So does CIP-013 R1, which mandates that the entity develop a supply chain cyber security risk management plan, provide a list of threats that need to be addressed in the plan? As I pointed out in this post, R1.2 does list six types of mitigation (ordered by FERC) that need to be included in the plan – and these mitigations correspond to six particular supply chain threats. However, R1.1 says that the entity must “identify and assess” risks resulting from “(i) procuring and installing vendor equipment and software; and (ii) transitions from one vendor(s) to another vendor(s).” And I believe the word “all” needs to be assumed after “risks”, since otherwise it wouldn’t make sense (if you’re not going to address all risks, what are you going to address? Just those that begin with the letter A?)

I rephrase this list as (i) risks from procuring vendor equipment and software; (ii) risks from installing vendor equipment and software; and (iii) risks from transitions between vendors. The six mitigations in R1.2 all fall under the heading of procuring vendor software, and even then they hardly exhaust all the possible risks just in that one category; they don’t do anything to address risks in the other two categories.

So the serious flaw in CIP-013 R1 is that it requires development of a plan to mitigate supply chain threats (the requirement uses the word risks, but I prefer “threat” for several reasons) but doesn’t provide a list – beyond the six items in R1.2 – of threats that should be included in the plan. This means that someone auditing compliance with R1 only has two choices:

a)      Make up their own criteria for what should be in a plan and audit against that; or
b)      Restrict the audit to only the six items in R1.2. If these items are all sufficiently addressed in the plan, the entity doesn’t get a PNC. If they aren’t all sufficiently addressed, the entity is likely to get a PNC.

To be honest, neither of these is an acceptable choice. Clearly, for the auditor to make up their own criteria is completely unacceptable, meaning a) is off the table. But b) is also unacceptable, since R1.1 wants the plans to address a lot more threats than just the six threats that are implied by R1.2. Moreover, FERC said the same thing in Order 829, and NERC said it in the Implementation Guidance for CIP-013.

Option b) might be acceptable if it were likely that NERC entities would bend over backwards to identify supply chain risks that go well beyond the six items (threats) in R1.2. In that case, the auditor still couldn’t give the entity a PNC for not including a particular threat in the list, but they could certainly ding them if they listed a threat in their plan developed for R1 but didn’t take any steps to mitigate[i] that threat as they implemented the plan in R2.

However, I have two pieces of bad news for anyone who thinks this will happen:

  1. There is no Easter Bunny; and
  2. NERC entities aren’t going to bend over backwards to identify supply chain threats beyond the six threats referenced in R1.2. While they may identify particular threats that their Region told them they should identify (or that might be included in a future “NERC-approved” guidance document, say from the North American Transmission Forum), they simply aren’t going to search high and low for every threat they can think of and include it in their list. Even if they are already addressing a threat outside of the compliance process, just including it in the list will entail new paperwork, as well as compliance risk if their auditor decides their implementation of mitigations of that threat in R2 is inadequate.

So neither of the above options is acceptable. And it is very unlikely that the FERC commissioners will all read this post and immediately order that CIP-013 be rewritten to include a list of supply chain threats that must be addressed in the plan, since this would delay implementation for probably 2-3 years.  What is likely to happen? I (optimistically) think the Regions will develop a CIP-013 process roughly like what I outlined in this post at the end of last year – namely, that an entity will be allowed to have their Region review and suggest changes to their CIP-013 plan prior to starting implementation, and that the entity will also be able to request that their Region review and advise on their implementation of the plan, after they have started implementing it.

As part of the review of the entity’s plan, the Region will be able to suggest that there are threats (aka risks) the entity should address in their plan, which they haven’t addressed. It’s a good bet that, if an entity’s Region suggests they should add threats X, Y and Z to their plan, they will add them![ii] That’s why I think this is an acceptable solution to the problem posed by this serious flaw in CIP-013.

Let’s cut to the chase: Is what I’m suggesting completely “legal” by the NERC Rules of Procedure? I’ll bet it isn’t. But as far as I can see, the only other alternatives are the a) and b) options listed above. So there’s a choice between the somewhat illegal and the completely unacceptable. Which will it be?

If you would like to comment on what you have read here, I would love to hear from you. Please email me at Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

[i] There’s a slight problem here, which I only discovered while writing this post: R1.1 requires the entity to develop a plan to “identify and assess” supply chain risks, but it doesn’t require the plan to mitigate them! So in theory, the entity could develop a plan that simply listed a bunch of risks but didn’t propose to do anything about them. Of course, it’s clear that CIP-013 was ordered and developed in order to mitigate supply chain risks, so I can’t see this omission shutting down the implementation of CIP-013. But this omission does need to be fixed by amending the standard, assuming FERC orders other changes (in a second version, of course) when they approve CIP-013-1.

[ii] This arrangement will only work if the Region develops a standard set of threats that it wants included in CIP-013 plans, so that individual auditors can’t develop their own. If I were new to this business, I would also suggest that each region publish a list of threats that need to be included in the plans, but that would go way beyond what NERC would allow them to do. So the advice will need to be provided on a one-on-one verbal basis with individual entities. The compliance advice the Regions currently provide for other CIP standards is delivered in the same way, e.g. in the SGAS.

Sunday, March 4, 2018

Complying with CIP-013, Part 5: An Auditor sets me Straight on Contract Language

My most recent post discussed the important question of the role of procurement contract language in CIP-013. My general point in the post was that I think it is a big mistake for a NERC entity to focus their CIP-013 compliance efforts solely, or even primarily, on getting the vendor to agree to incorporate particular language in their contract. After that post came out, an auditor wrote in to suggest some of my statements needed clarification. We exchanged a number of emails, and I’ve identified four important points that he raised.

I.                    Tales from Beyond the Scope
Early in the post, I quoted from a note that appears after CIP-013 R2: “the following issues are beyond the scope of Requirement R2: (1) the actual terms and conditions of a procurement contract; and (2) vendor performance and adherence to a contract.” Regarding (1), I had this to say: “The first part of the second sentence…says that the terms and conditions of a procurement contract are ‘beyond the scope’ of R2. What does this phrase mean? It specifically means that contract language is not the subject of R2 – and if an auditor asks to see an entity’s vendor contract, they can be turned down because it isn’t in scope.”

It turns out the truth is more nuanced. To quote from the auditor’s email: “For compliance finding purposes (i.e., determination of a PNC) the contract is out of scope.  However, if the entity states that the vendor will (or has promised to) do something, such as advise its customers within x days of identification of a vulnerability in its products, I want to see something from the vendor that demonstrates that, not just a verbal statement of assurance from the audited entity.  In many cases, the MSA, SLA, or contract T&C will address that expectation and it is appropriate for the entity to provide the written vendor assurances.”

What does this mean? It means that, if you assert that a vendor has promised to do X in a contract, the auditor can (and most likely will) ask for evidence of this. If you refuse to show him the contact and point to phrase (1) above, the auditor will probably say “OK, Mr. Smart Guy. Show me some other evidence they’ve made this promise.” At this point, you’ll wish you had a letter or email from the vendor making the same promise; presumably, you won’t feel you’re violating some legal principle by not showing him one of these.

But what if your only evidence is the contract – and you’ve been told by your lawyers never to show the contract to the auditors, even if they jump up and down and promise divine retribution for this refusal? In this case, the auditor technically can’t issue a PNC (potential non-compliance) finding, but they can certainly list in their report that you were unable to substantiate your assertion regarding the promise by vendor Y to do X.

So if you are going to be at all constrained in your ability to show a contract to the auditors, why even take the contract route in the first place? Get a letter from your vendor. This constitutes obtaining a vendor promise just as much as contract language does, and it will make your audit go much more smoothly. As the auditor pointed out above, other documents from the procurement process, such as an RFP stating terms the vendor agreed to, a Master Services Agreement or a Service Level Agreement, might also constitute evidence of the vendor’s promise.

II.                  Trying is Everything
What happens if you beg your vendor to promise to do something and they refuse – yet dropping them as a vendor is out of the question because of the cost of making the change, the unavailability of close substitutes, etc? Will you receive a PNC because of this? The auditor further states in the same email: “The key point in the contractual language exclusion is that the auditor cannot hold the entity at fault with a PNC if the entity was not successful in getting the vendor to agree to a CIP-013 requirement expectation.  And, failure of the vendor to comply with its contractual obligations or promises does not constitute a compliance failure on the part of the audited entity.  But the entity needs to demonstrate how it addressed the expectation during its procurement process.  The Plan should address how the entity will go about asking and the pre-contract negotiations and perhaps elements of the contract itself will demonstrate the Plan was followed.”

In other words, your supply chain cyber security risk management plan (which R1 requires you to develop, of course) must describe how you will go about requesting that your vendor promise to do something, as well as the different vehicles available for the vendor to document that promise. For each vendor, you should document how you tried to get them to make the promise. If they agreed to make it, then you need some document to verify that. If they didn’t agree, you should document that fact as well. To quote the auditor again: “The entity does need to keep records of any communication with the vendor; preferably date, time, with whom, and a summary of the discussion or agreement.  I like emails better than phone logs for the obvious reason; emails are better evidence than one-sided hand-written records.”

The auditor also points out that you don’t have to simply throw up your hands if the vendor tells you “no” the first time you ask for something, even though you know that finding another vendor for the same product isn’t an option, for whatever reason. How about negotiating, using tools that you (as a compliance or cyber security person, not a supply chain or legal person) have access to? The auditor also says “…if I cannot get, for example, prompt notifications of vendor staff terminations, then maybe I should reconsider my willingness to allow electronic or unescorted physical access to my BES Cyber Assets.  That is called a negotiation point.” Good idea!

Before I leave this topic, I want to point out something else: Your supply chain cyber security risk management plan must list risks and how you intend to mitigate them. Your first step to mitigate a particular vendor risk may well be getting the vendor to promise to take certain steps that will mitigate the risk. But if the vendor absolutely refuses to do this, are you now off the hook for that particular risk? Will your mitigation be “Well, we tried our best, but that vendor is as stubborn as a senior citizen mule!”?

Unfortunately, this isn’t mitigation – this is making an excuse for non-mitigation. There should usually be some step you can take to mitigate the risk in question, which doesn’t require the vendor to do anything at all. Let’s go back to the example the auditor just used, the risk that a vendor won’t provide prompt notification when an employee leaves who had access to your systems (this is the risk addressed by R1.2.3). The auditor suggested that, if the vendor refused to promise to give you this notification, you might then refuse to give their employees electronic access and/or unescorted physical access to their systems at your facilities. This isn’t just a negotiation tactic. It’s also a legitimate mitigation for this risk, since an employee who had just been fired and wanted to take out his anger on your systems would have a hard time doing so without electronic or unescorted physical access. He/she would have to come onsite and be escorted to the systems!

III.                Just Do It!
In my last post, I made this assertion: “I don’t think the entity needs to show any procurement documents at all, because I don’t think the entity needs to prove that they ‘initiated correspondence’ with the vendor to implement certain controls, if they can show other evidence of compliance. And what other evidence can the entity show? The best evidence will demonstrate that the vendor is actually doing what they were supposed to do!  After all, isn’t the goal of CIP-013 to protect the BES against supply chain risks, not simply document that you’ve asked your vendors to do certain things?”

In this statement, it seems I was flat out wrong. While it is important that you gather some evidence to show the vendor is actually keeping a particular promise[i], that evidence alone isn’t sufficient. In his email, the auditor further stated “I do have a problem with a “handshake” agreement.  I cannot evaluate that – it is essentially an attestation in the absence of supporting evidence.  Yes, the entity can show me that they received a notification of a vendor staff termination.  But, what prompted the notification?  Was there a contractual expectation or a vendor declaration that such (notification) would be sent by the vendor consistently and in a timely manner?  How do I know that the vendor will inform me of every applicable termination or reassignment?  Similarly, how do I know that the vendor will apprise me of every security vulnerability in their products?”

In other words, if a vendor makes a promise, it needs to be made in writing in some way (although contract language is just one of those ways). You also need to document either a) that the vendor is keeping that promise (using emails, etc); b) that you tried to get the vendor to promise something but you were turned down; or c) the vendor promised to do something but didn’t keep the promise. If either b) or c) is the case, you also need to document some other mitigation steps you took to address the risk in question – assuming there is a way you could mitigate the risk without the vendor’s cooperation (obviously, as required in R1.2.4, if the vendor absolutely refuses to tell customers about vulnerabilities it knows about but hasn’t yet made public, you will not be able to take steps to address those vulnerabilities - except perhaps by simply tightening up the normal security measures that would be applied to that system).

IV.                The Big Guys
Finally, the auditor also made a good point about how you mitigate vendor risks when it comes to big vendors like Microsoft, who obviously aren’t going to change their contract language for a single electric utility, no matter how large it might be. He says “…the plan can certainly address how procurement from the huge international players like Microsoft, Red Hat, Cisco, and Oracle will be handled.  I would prefer to see you get your networking gear from a Cisco channel partner, not eBay or some unknown used equipment reseller.  I would expect you to download software from a known, reputable source, not Sammy’s Software Shack.  Hopefully the vendor provides a digital signature (MD5 or SHA(2)-256 hash) of its software for verification purposes.  It is all about risk management.  You cannot eliminate all risk, but you can certainly reasonably mitigate it.”

If you would like to comment on what you have read here, I would love to hear from you. Please email me at Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

[i] Although you definitely don’t have to document every instance where the vendor kept the particular promise in question, as would be the case if we were talking about a prescriptive requirement like CIP-007 R2.

Friday, February 23, 2018

Complying with CIP-013, Part 4: The Contract Language Bogeyman

Regular readers of this blog (all three of you!) may have figured out that I’m writing two series of posts on CIP-013: one for NERC entities and one for vendors. The important thing to remember is that both series should be of interest to everybody. The two sets of concerns are almost completely the same, because a) vendors have all sorts of incentive to make sure their customers don’t get into trouble due to something they did or didn’t do; and b) most vendors have their own supply chain, which has to be secured in almost the same way as NERC entities will be securing theirs.

If you ask any NERC CIP compliance professional (or say a supply chain or legal professional at a NERC entity subject to CIP-013) what is the main focus of compliance with CIP-013, the upcoming supply chain security risk management standard, they will most likely tell you it’s contract language. This isn’t terribly surprising, since there was endless debate during the drafting/balloting periods on the role contract language would play in compliance. And I’ve heard that some utilities are already requesting security contract language from their vendors.

I must admit that I was very focused on contract language in the first few posts I wrote about CIP-013 last fall. But as I started my current posts on the standard this year, and focused more on the standard and the Implementation Guidance, I realized that it’s a mistake to think that contract language is in some way the ultimate goal of CIP-013. Far from it.

First, here’s a little quiz: Why was CIP-013 written? The possible answers are: a) to secure NERC entities’ supply chains; or b) to beef up language in vendor contracts. You’re right! The answer is a). Security is the object of CIP-013, not contract language. To see this, you first need to look at the requirements of CIP-013. Do the actual requirements ever refer to contract language? No, they don’t. The only time contract language is mentioned is in this note found after R2:

Implementation of the plan does not require the Responsible Entity to renegotiate or abrogate existing contracts (including amendments to master agreements and purchase orders). Additionally, the following issues are beyond the scope of Requirement R2: (1) the actual terms and conditions of a procurement contract; and (2) vendor performance and adherence to a contract. 

Does either of these sentences say anything about contract language being an objective of CIP-013? The first sentence says that renegotiation of existing contracts isn’t required. I’ll admit this does somewhat imply that, when a new or existing contract is negotiated, there needs to be a discussion of contract language. But that isn’t the same thing as requiring that contract language be the primary means for compliance with the standard, even though so many people apparently believe that is the case.

The first part of the second sentence is even more interesting, since it says that the terms and conditions of a procurement contract are “beyond the scope” of R2. What does this phrase mean? It specifically means that contract language is not the subject of R2 – and if an auditor asks to see an entity’s vendor contract, they can be turned down because it isn’t in scope.

I’ve covered this ground before, but in a different context. In a series of three posts (starting with this one, following up with this one and concluding with this one), I addressed the idea that CIP-013 was unenforceable because of the above provision about contract language being out of scope. A staff member (and former auditor) from one region had made this assertion to me. He argued that NERC entities are required[i] to get their more-risky (i.e. important) vendors to agree to contract language for at least the six items listed in R1.2. However, since the NERC entity isn’t required to show the actual contract to the auditors, this means the auditors don’t have any way of verifying the entity’s assertion that they obtained the contract language.

An auditor (from another region) wrote in after that post to say that NERC entities aren’t required to get the vendor to agree to particular contract language in order to obtain the six items in R1.2 (I wrote about this in the second of the above posts). To quote him, “The requirement is essentially that the Registered Entity ask for the elements of the required process(es), not that the vendor agree to them.  Therefore, the Standard is correct in that the language of the resulting contract is not in scope.  (The procurement steps include) the Request for Quotation or Bid, Request for Contract Amendment, and/or any other Registered Entity-initiated correspondence that sets forth the expectations to be placed upon the vendor.  As long as I see the expectations in the procurement documents issued by the Registered Entity, they have satisfied the requirement.”

I agree with the auditor that succeeding in getting language placed in a contract isn’t required, and that it should be sufficient that the Registered Entity “initiated correspondence that sets forth the expectations to be placed upon the vendor”. But I would go even further. I don’t think the entity needs to show any procurement documents at all, because I don’t think the entity needs to prove that they “initiated correspondence” with the vendor to implement certain controls, if they can show other evidence of compliance. And what other evidence can the entity show? The best evidence will demonstrate that the vendor is actually doing what they were supposed to do![ii] After all, isn’t the goal of CIP-013 to protect the BES against supply chain risks, not simply document that you’ve asked your vendors to do certain things?

Suppose you want to show your auditor that your vendor is providing “Notification…when remote or onsite access should no longer be granted to vendor representatives..” as required by R1.2.3. Let’s say the vendor has had several people leave their company in the last year who had electronic or physical access to the vendor’s systems at say one of your substations, and each time this has happened they’ve sent you an email asking you to remove the person’s access. All you should have to do is show those emails to the auditor. What possible further benefit would there be from also showing them procurement documents, let alone a contract? Of course, if you also have these, that’s fine. But if the vendor verbally agreed to do something, or just sent you a letter outside of the procurement process, but you can show they’re actually doing what they said they’d do, that should be sufficient evidence.

Let’s turn from the requirements of CIP-013 to the Implementation Guidance. This document has language that implies that contract language is a goal of the standard (although certainly not the only one), while at the same time it also says just the opposite. However, I can see how people who have read the document have come to believe that contract language is the primary control they should be focusing on.

First off, let me point out that there is nothing mandatory about the Implementation Guidance. The auditors aren’t bound to follow it in their audits, and the entities aren’t bound to follow it in their implementation of CIP-013. That being said, you can be sure most entities and all auditors will have read the document carefully and will usually try to follow it as often as they can. So it is important to at least understand what it says. On page 1, you’ll find this paragraph:

First, in developing their supply chain cyber security risk management plan(s), Responsible entities should consider how to leverage the various components and phases of their processes (e.g. defined requirements, request for proposal, bid evaluation, external vendor assessment tools and data, third party certifications and audit reports, etc.) to help them meet the objective of Requirement R1 and give them flexibility to negotiate contracts with vendors to efficiently mitigate risks. Focusing solely on the negotiation of specific contract terms could have unintended consequences, including significant and unexpected cost increases for the product or service or vendors refusing to enter into contracts.
Here, the Implementation Guidance is specifically warning the entity against relying solely on contract language. However, the message is very different when you get to pages 5-8, where the six specific items required under R1.2 are addressed. For each of these items, there are one or more processes listed. These are described as “Examples of ways that a Responsible Entity could, through process(es) for procuring BES Cyber Systems required by Part 1.2, comply with Parts 1.2.1 through 1.2.6.” (my emphasis) It seems that, at least for the six risks addressed in R1.2, the members of the Standards Drafting Team that wrote the Implementation Guidance were primarily considering controls that occur in the procurement phase of the vendor relationship – including contract language.[iii]

When you look at the 20 actual processes suggested under the parts of R1.2, you will notice that all but three of these begin with one of two phrases: “In an RFP or during contract negotiations, request that the vendor include in the contract provisions…” or “During procurement”. Of course, both of these phrases simply confirm the fact that whoever wrote this section of the Implementation Guidance was primarily looking at the process of obtaining contract language – or other processes focused entirely on the procurement phase of the vendor relationship – to achieve the mitigation of risk required in each of these six cases.

But in my opinion, the big emphasis on the procurement process in this section of the Implementation Guidance is a mistake. Essentially, what is being implied in this section is that the only possible way to coordinate security controls with vendors is during the procurement process, when obviously the customer has the most leverage over the vendor. This might be a good model for a purely commodity market, where you have a bunch of vendors competing viciously with each other and they all receive a very low margin (on presumably a high volume of sales). In this case, the customer clearly has to obtain everything they want from the vendor up front, and they need to negotiate simultaneously with multiple vendors and play them off against each other for concessions.

I don’t know about you, but I don’t think this is how it works in the market for control systems that run the grid. I am betting that almost all vendors in this market will do almost anything they can to make their customers happy, since they are counting on having a long-lasting relationship with them. And different vendors’ products are very different, so it’s impossible to play vendors off against each other, as in a commodity-type market.

Clearly, a customer’s leverage isn’t limited to the procurement process. I am sure that, in most cases, a field technician (backed up by their VP) with a serious complaint about a vendor, midway through the contract lifetime, will usually have as much leverage as a purchasing agent on the eve of contract renewal.[iv] Even though it is notoriously difficult to change vendors of control systems in the power industry, I really don’t believe that any vendor would last very long if they didn’t do everything they could to provide what their customers need.[v]

Again, I’m saying that I think the emphasis on the procurement process in this section of the CIP-013 Implementation Guidance is a mistake. I really doubt there are auditors who will insist that NERC entities need to wait for the procurement (or contract renewal) process in order to request any meaningful changes to vendor security controls. And it follows almost certainly that auditors won’t require that contract language be the primary vehicle for obtaining agreement on controls – especially since that’s the one type of evidence that the auditors won’t be able to see!

If I’m so sure that the emphasis of this whole section of the Implementation Guidance is wrong, how did it end up there in the first place? To answer that, you need to know that at least half of the members of the drafting team for this standard were supply chain people – and I’m speculating that this particular section was drafted by some of them.

For supply chain people, the procurement process is kind of the whole show, since that is when they have by far the most interaction with vendors, as well as when what they do and say will have the most impact. But they don’t get involved much with the day-to-day interaction with vendors that goes on in the field. So supply chain people are naturally going to consider the procurement process – and especially contract language – as the primary vehicle for influencing what vendors do. They don’t see how vendors often easily agree to requests that come through completely different channels.

The moral of this story? It would be a big mistake for NERC entities to focus on the procurement process – and especially on contract language – as the sole, or perhaps even the predominant vehicle for getting vendors to agree to implement security controls. There are lots of paths to achieve this goal, including simple phone calls from a field technician to their friend Joe at the vendor. However, what will need to change with CIP-013 compliance is that those calls or emails – when related to vendor controls the entity has decided need to be in place in order for it to comply with CIP-013 – will need to be logged and archived so they can be evidence of compliance.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

[i] The staff member based this on the fact that FERC seemed to have made this a requirement in their Order 829, in which they ordered NERC to draft this standard. The auditor replied that FERC’s statements have no bearing on how a standard should be interpreted, once they have approved it. You can read their back-and-forth in the second of the three posts I linked above.

[ii] And keep in mind that, if the vendor absolutely refuses to do something that the entity requested, the entity isn’t in violation for that reason. However – and this is important – they are still under obligation to if possible mitigate the risk in question, presumably using a different method that doesn’t require the vendor to do something for them. Of course, there are certainly some risks that it won’t be possible to mitigate without the cooperation of the vendor, such as the risk of someone who had access to your systems leaving the vendor’s employment, and your organization not being notified that this has happened (this is, of course, the risk addressed in R1.2.3). In these cases, if the vendor simply won’t cooperate, the entity isn’t obligated to change vendors; but the entity should document how they tried to obtain cooperation.

[iii] Of course, you need to keep in mind that the six items listed in R1.2 are there because FERC ordered these six items be specifically included in the standard. They aren’t in any way intended to be the only items that need to be included in the entity’s supply chain cyber security risk management plan. The plan needs to address each of the three risk areas listed in R1.1: a) risks from procuring vendor equipment and software, b) risks from installing vendor equipment and software, and c) risks from transitions from one vendor to another. The six items in R1.2 all correspond to risks that fall into category a). However, there are certainly many other risks that fall into that category, which aren’t included in these six. And of course, the six items don’t address any risks that fall into categories b) and c).

[iv] Of course, I’m not talking here about huge vendors like Microsoft and Adobe, for whom the electric power industry accounts for a very small part of their revenue. They will of course provide product support, but if XYZ Power calls up and demands that Microsoft insert a new feature into the current Windows version, I rate the likelihood they will succeed in this demand as very low.

[v] Of course, if anyone has a different view on this, I’d love to hear it. Send me an email or comment on this post.

Tuesday, February 20, 2018

What about Resiliency?

Yesterday I was interviewed by the always astute Blake Sobczak of Energy and Environment News[i] about a cyber security report issued by the White House last week, and specifically about what it had to say about the cyber security of the power grid. The article appeared today.

The discussion of the grid in the report was certainly good and not terribly inflammatory. My feelings about this report are very similar to my feelings about Ted Koppel’s book Lights Out, which I discussed in this post in early 2016: What Koppel was writing about had very little to do with cyber security. It had everything to do with the amount of devastation that any widespread and prolonged grid event could wreak (and we’re talking about a much more serious event than even the 2003 Northeast blackout), whether caused by a huge weather event (even bigger than Superstorm Sandy), solar storm, EMP event, physical attack, or yes a cyber attack. Even more importantly, the book documented in horrifying detail the country’s almost total lack of preparedness for this. Unfortunately, whoever wrote the book jacket decided the book would sell more copies if it were made to appear as a book about a big cyber attack on the grid, without doubt to leverage the popular movies that had depicted such an attack. So that is how the book is known, but it isn’t what’s actually between the covers.

In the same way, the section of the White House report that I commented on quotes the 2015 study by Lloyd’s and the University of Cambridge that estimated the total cost of a worst-case cyber event at $1 trillion. I totally agree that a worst-case cyber event could cost that much. But there are two considerations: 1) The cost would be the same whether the cause of the event were weather, solar storm, or anything else; and 2) I’d say a worst-case cyber event is probably the least-likely cause of such a huge grid outage – with a probability somewhere around 1 in 10 to the 10th or the 20th power. I think a solar storm is far more likely to cause such an event (and the 1859 Carrington Event probably would have done it, had it occurred today. We’d better hope these aren’t every-200-year events!).

Although the report didn’t advocate any particular policy actions (it was actually quite good as a broad overview of the risks posed by cyber security weaknesses across the US economy), in my comments I anticipated what I thought might be the typical response regarding security of the power grid: “Oh my God, we have to do something about this! We need to really tighten up the cyber security standards so that the power industry (that would be you, Dear Reader) doesn’t let his happen to us!”

My response to this response, whether triggered by the White House report, Ted Koppel’s book or any number of alarmist statements, is that the situation won’t be improved by requiring say a 10-fold increase in the severity of the NERC CIP requirements. There has never been an outage caused by a cyber attack in North America; nor has there ever been even a documented penetration of a control network in a grid asset of any significance (I make this qualification because there was a penetration of a 5MW dam in New York state in 2013. I know of no other such penetration, although if anyone knows of another I’d appreciate your letting me know about that in an email). And of course, any amount of increased cyber security spending by the power industry will do nothing to mitigate the danger of a different type of event like a solar storm (and NERC is currently in the process of approving a draft standard to address solar storm risks by “hardening” grid assets).

I have long believed that the best protection against widespread outages, no matter what the cause, is microgrids. If the great majority of populated areas in North America were protected by microgrids, there could still be widespread grid events which would destroy huge fixed generation and the high-voltage transmission network - but these events wouldn’t cause widespread outages. Each microgrid would automatically activate its local generation resources – wind, solar, gas turbines – and keep on chugging. Of course, the problem at the moment is that microgrids are very expensive to implement and they’ve only been implemented in a small number of high-value locations (like the New Jersey transit system, which was knocked out by Hurricane Sandy, greatly complicating the recovery from the storm). But if we’re talking about throwing a lot more money at grid security, wouldn’t it be a lot better to throw it at a solution to all potential causes of a huge grid outage, rather than just at reducing the already-infinitesimal probability of just one of those causes – a massive cyber attack?[ii]

Before I go, I do want to point one thing out: I don’t believe for a moment that electric utilities are paying too much for OT cyber security controls now. In fact, given the ever-tightening threat environment, I think it’s inevitable they will need to spend more every year for the foreseeable future. My issue is that a large portion of what NERC entities spend on NERC CIP compliance now goes simply to pure compliance activities, not to increasing the level of cyber security. I think that virtually all NERC entities understand that this is a tough world, and they will have to increase their cyber and CIP spending every year just to stay in the same position. But they will be increasingly reluctant to do this without changes in the NERC CIP compliance regime (and in the standards themselves) that will allow a much higher percentage of every dollar they spend on CIP to go towards cyber security.

And what are those needed changes? Funny you should ask. That is the topic of a book I am working on with two co-authors. You may roll your eyes and point out that I’ve been talking about this book for close to two years. That’s true, but we now have a lot of momentum and I’m sure we’ll have something out before the end of the year. And what is the solution we’re advocating? Well, you can find most of it in my posts, although I’ll admit you’ll have to look hard (and be able to tie a lot of bits and pieces together). I hope to have everything in a one-stop-shop this year!

If you would like to comment on what you have read here, I would love to hear from you. Please email me at Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

[i] This online publication has the best articles about security of the energy sector of any publication I’ve seen, all written by either Blake or his colleague Pete Behr. They’re all very in-depth (a few even as long as my average post, heaven forbid!) and very well-researched – and this applies to all of the articles, not just those on cyber. Most online news feeds confine themselves to news feeds or reproductions of articles from other publications, so E&E News really stands out. This is a subscription-only publication, but I strongly recommend you try to get your own organization to subscribe to it. I recommended to my boss at Tom Alrich LLC that we purchase a subscription, but he hasn’t replied to my email yet.

[ii] And if you’re tempted to think that the big toughening of the NERC CIP standards would be paid for by “private industry” while the cost of implementing microgrids everywhere would have to be borne by the taxpayers, let me point out something to you: Every taxpayer is a ratepayer to their local electric utility. Where will the utilities get the money to comply with this huge increase in the cost of CIP compliance?

Friday, February 16, 2018

Lew on Patch Management Mitigation Plans

Five days ago, I wrote a post calling attention to Lew Folkerth’s December article on CIP-010 R1 (configuration management) compliance. In that post, I mentioned that you can now sign up to receive notices of new RF newsletters (where Lew’s column appears). I signed up and I’m glad I did, because yesterday I got notice of a new newsletter. This time, Lew’s article (which as usual goes under the name “The Lighthouse”) is about CIP-007 R2 patch management mitigation plans. Lew’s articles are always excellent, but this is still one of his best yet, in my opinion.

I must admit I hadn’t thought very much about patch management mitigation plans. R2.3 says “Mitigation plans shall include the Responsible Entity’s planned actions to mitigate the vulnerabilities addressed by each security patch and a timeframe to complete these mitigations.” I had always just assumed this was all you need to know about these plans.

It turns out I was wrong. Lew does an excellent job of pulling out what really needs to be in a mitigation plan and how it needs to be handled. None of this is in any way an add-on to the requirement, by the way. Lew is simply following the logical implications of what the mitigation plan needs to accomplish, both for security and compliance. Lew has always been most concerned about what makes the most sense from a security point of view. I won’t say everything he says in this article is needed strictly for compliance with the requirement, but doing it will certainly allow you to tell a good story when you get audited.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

Thursday, February 15, 2018

How Should Vendors “Comply” with CIP-013? Part 2

This is the second in a series of posts of how vendors can “comply” with CIP-013. Of course, vendors don’t have to comply at all, and they don’t have to help their power industry clients comply either. On the other hand, if they wish to remain a vendor to the power industry, a vendor is going to have to do everything they can to help their customers comply. It is inevitable that the vendor’s larger clients will have to comply with CIP-013, and they will need help in doing so.

I do want to emphasize that, if you’re part of a NERC entity who has to comply with CIP-013, you shouldn’t stop reading this post (or the subsequent ones in this series), on the idea that it doesn’t really have anything to do with you. This has everything to do with you, since – as I emphasized in the first post in the series – the only way I see a CIP-013 compliance program succeeding is if the NERC entity at least tries to partner with their vendors for compliance. You need to understand their position as much as they need to understand yours.[i]

In the first post in this series, I suggested that power industry vendors would be well advised to reach out to their customers before the customers reach out to them. I made this suggestion because I think it’s very likely that if the customer reaches out to the vendor first, it will probably be the lawyers and/or purchasing people who do this. This is because most of the discussion so far on CIP-013 compliance (with this blog hopefully being an exception!) has revolved around the question of contract language. But contract language is just one of the ways in which a utility can fulfill its obligations under CIP-013, and it’s probably the bluntest instrument available to the NERC entity to fulfill those obligations. I think that any NERC entity that focuses on contract language first, before even looking at all the options available to it to comply with R1.2, has already done both itself and its vendors a big disservice.

So the vendor should absolutely try to reach out first to their customers. How can they do that? There are of course lots of media available for this: notice on web site, email, USPS, phone call, webinar, onsite meeting, etc. While these are all good, I always favor the more personal ones above all. A webinar might be the ideal first step, since it’s delivered by a person(s) but it still has a structured content. Then the vendor could follow up with each customers by phone or in-person meeting to discuss next steps.

But before you can do a webinar, Mr./Ms. Vendor, you need to know what you’re going to say! And that means you need to have figured out your CIP-013 strategy[ii] – so that’s really the first step. How can you figure out your strategy?

This is of course rough and preliminary, but it seems to me that a vendor’s strategy should be to try to make the CIP-013 compliance process as easy as possible for the customer, period. I can assure you that your customers will be feeling every bit as uncertain and fearful about CIP-013 as you do. If you can convince them that you know the right way to cooperate for compliance, and you follow through on what you say you’ll do, what could possibly go wrong – except, perhaps, if you don’t in fact know the right way to cooperate and both of you end up at some sort of regulatory dead end?

How do you stay away from dead ends? By having a good methodology for designing your CIP-013 strategy. And what should you do first? Well, it seems to me that, if you’re going to design a strategy to help your customers do something, you need to figure out what they have to do. There’s a quick answer to that: They have to comply with CIP-013. Now that we have that out of the way, how do they comply with CIP-013?

As I discussed (at length) in this post, CIP-013 requires the NERC entity to develop and implement a supply chain cyber security risk management plan. Specifically, the plan has to address the three areas of risk listed in R1.1: (a) procuring vendor equipment and software; (b) installing vendor equipment and software; and (c) transitions from one vendor(s) to another vendor(s).

I suggest you start your CIP-013 strategy effort by brainstorming on how you can help your customers address all three of these areas of risk. To be honest, you might decide you can help customers in some ways that are either too expensive to be practical or not likely to yield a lot of benefit compared to the effort required; you can drop these ideas from your strategy.

For procuring vendor equipment and software, the NERC entity needs to address risk in six specific areas; these are listed in R1.2. The reason they are specifically listed in CIP-013 is because FERC specifically required – in Order 829 - that these areas all be addressed in the new standard. The NERC entity needs to do a lot more than simply address these six areas of risk, of course, but because they’re specifically called out you can be sure the auditors will all make sure they’ve been properly addressed in the entity’s plan, probably before they even get around to looking at anything else.

Let’s choose one of the areas in R1.2 as an example:

R1.2.4. Disclosure by vendors of known vulnerabilities related to the products or services provided to the Responsible Entity; 

How can you help your customers address this area of risk? This is of course a particularly difficult one for software vendors, since what might seem like the obvious way to help – disclosing to your customers all of the vulnerabilities in your software that you know about – would be the height of irresponsibility. If a vulnerability hasn’t been patched yet, the last thing you want to do is broadcast the existence of that vulnerability to all of your customers.

On the other hand, you might decide that you do need to provide information on all vulnerabilities (not just the ones that have already been patched, which of course can be advertised to the whole world) to at least your largest and most trusted customers. You need to decide internally in what cases you will do this, what safeguards you will require on the customer end, what alternatives to full disclosure you might suggest to customers for whom you don’t want to take the “Full Monty” approach, etc.

I hope you understand what I’m getting at here. For each of the six items in R1.2, you need to figure out a complete strategy for dealing with all of your customers (and as you can see, you will probably want to deal with particular types of customers in different ways, although the way you break down your customers is likely to vary according to the item in question). This will be an important component of your strategy, Mr./Ms. Vendor.

But this isn’t the only component of your strategy. My next post in this series will go over what else needs to be in your strategy.

If you are with a vendor to the electric power industry, and your company is trying to figure out what you will have to do to comply with CIP-013, Tom Alrich LLC would be pleased to offer you a free one-day (2-6 hours) workshop to review a) what CIP-013 requires, b) what you are likely to get asked to do by your clients, and c) what they should be asking you to do (since b and c won’t usually be the same thing). There will be no charge for my time, but I will require you to pay travel expenses at cost. Please email or call me if you would like to discuss this.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.       

[i] And of course, I’m also doing a series of posts on how NERC entities can comply with CIP-013. Here is the most recent post in that series. Vendors should read this series of posts as well!

[ii] And by the way, NERC entities also need a CIP-013 compliance strategy, which I am gradually laying out in my other series of posts. A lot of elements in both strategies should be the same – just told from different points of view. But a NERC entity’s strategy will inherently include a lot of elements that have nothing to do with vendors, since vendor risk is just one of three areas of supply chain risk that need to be addressed in the supply chain cyber security risk management plan. See this post for a short summary of those three areas, and see this post for a long, painful discussion of that topic – but which ultimately might be more enlightening.