Friday, February 16, 2018

Lew on Patch Management Mitigation Plans


Five days ago, I wrote a post calling attention to Lew Folkerth’s December article on CIP-010 R1 (configuration management) compliance. In that post, I mentioned that you can now sign up to receive notices of new RF newsletters (where Lew’s column appears). I signed up and I’m glad I did, because yesterday I got notice of a new newsletter. This time, Lew’s article (which as usual goes under the name “The Lighthouse”) is about CIP-007 R2 patch management mitigation plans. Lew’s articles are always excellent, but this is still one of his best yet, in my opinion.

I must admit I hadn’t thought very much about patch management mitigation plans. R2.3 says “Mitigation plans shall include the Responsible Entity’s planned actions to mitigate the vulnerabilities addressed by each security patch and a timeframe to complete these mitigations.” I had always just assumed this was all you need to know about these plans.

It turns out I was wrong. Lew does an excellent job of pulling out what really needs to be in a mitigation plan and how it needs to be handled. None of this is in any way an add-on to the requirement, by the way. Lew is simply following the logical implications of what the mitigation plan needs to accomplish, both for security and compliance. Lew has always been most concerned about what makes the most sense from a security point of view. I won’t say everything he says in this article is needed strictly for compliance with the requirement, but doing it will certainly allow you to tell a good story when you get audited.


If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.




Thursday, February 15, 2018

How Should Vendors “Comply” with CIP-013? Part 2



This is the second in a series of posts of how vendors can “comply” with CIP-013. Of course, vendors don’t have to comply at all, and they don’t have to help their power industry clients comply either. On the other hand, if they wish to remain a vendor to the power industry, a vendor is going to have to do everything they can to help their customers comply. It is inevitable that the vendor’s larger clients will have to comply with CIP-013, and they will need help in doing so.

I do want to emphasize that, if you’re part of a NERC entity who has to comply with CIP-013, you shouldn’t stop reading this post (or the subsequent ones in this series), on the idea that it doesn’t really have anything to do with you. This has everything to do with you, since – as I emphasized in the first post in the series – the only way I see a CIP-013 compliance program succeeding is if the NERC entity at least tries to partner with their vendors for compliance. You need to understand their position as much as they need to understand yours.[i]

In the first post in this series, I suggested that power industry vendors would be well advised to reach out to their customers before the customers reach out to them. I made this suggestion because I think it’s very likely that if the customer reaches out to the vendor first, it will probably be the lawyers and/or purchasing people who do this. This is because most of the discussion so far on CIP-013 compliance (with this blog hopefully being an exception!) has revolved around the question of contract language. But contract language is just one of the ways in which a utility can fulfill its obligations under CIP-013, and it’s probably the bluntest instrument available to the NERC entity to fulfill those obligations. I think that any NERC entity that focuses on contract language first, before even looking at all the options available to it to comply with R1.2, has already done both itself and its vendors a big disservice.

So the vendor should absolutely try to reach out first to their customers. How can they do that? There are of course lots of media available for this: notice on web site, email, USPS, phone call, webinar, onsite meeting, etc. While these are all good, I always favor the more personal ones above all. A webinar might be the ideal first step, since it’s delivered by a person(s) but it still has a structured content. Then the vendor could follow up with each customers by phone or in-person meeting to discuss next steps.

But before you can do a webinar, Mr./Ms. Vendor, you need to know what you’re going to say! And that means you need to have figured out your CIP-013 strategy[ii] – so that’s really the first step. How can you figure out your strategy?

This is of course rough and preliminary, but it seems to me that a vendor’s strategy should be to try to make the CIP-013 compliance process as easy as possible for the customer, period. I can assure you that your customers will be feeling every bit as uncertain and fearful about CIP-013 as you do. If you can convince them that you know the right way to cooperate for compliance, and you follow through on what you say you’ll do, what could possibly go wrong – except, perhaps, if you don’t in fact know the right way to cooperate and both of you end up at some sort of regulatory dead end?

How do you stay away from dead ends? By having a good methodology for designing your CIP-013 strategy. And what should you do first? Well, it seems to me that, if you’re going to design a strategy to help your customers do something, you need to figure out what they have to do. There’s a quick answer to that: They have to comply with CIP-013. Now that we have that out of the way, how do they comply with CIP-013?

As I discussed (at length) in this post, CIP-013 requires the NERC entity to develop and implement a supply chain cyber security risk management plan. Specifically, the plan has to address the three areas of risk listed in R1.1: (a) procuring vendor equipment and software; (b) installing vendor equipment and software; and (c) transitions from one vendor(s) to another vendor(s).

I suggest you start your CIP-013 strategy effort by brainstorming on how you can help your customers address all three of these areas of risk. To be honest, you might decide you can help customers in some ways that are either too expensive to be practical or not likely to yield a lot of benefit compared to the effort required; you can drop these ideas from your strategy.

For procuring vendor equipment and software, the NERC entity needs to address risk in six specific areas; these are listed in R1.2. The reason they are specifically listed in CIP-013 is because FERC specifically required – in Order 829 - that these areas all be addressed in the new standard. The NERC entity needs to do a lot more than simply address these six areas of risk, of course, but because they’re specifically called out you can be sure the auditors will all make sure they’ve been properly addressed in the entity’s plan, probably before they even get around to looking at anything else.

Let’s choose one of the areas in R1.2 as an example:

R1.2.4. Disclosure by vendors of known vulnerabilities related to the products or services provided to the Responsible Entity; 

How can you help your customers address this area of risk? This is of course a particularly difficult one for software vendors, since what might seem like the obvious way to help – disclosing to your customers all of the vulnerabilities in your software that you know about – would be the height of irresponsibility. If a vulnerability hasn’t been patched yet, the last thing you want to do is broadcast the existence of that vulnerability to all of your customers.

On the other hand, you might decide that you do need to provide information on all vulnerabilities (not just the ones that have already been patched, which of course can be advertised to the whole world) to at least your largest and most trusted customers. You need to decide internally in what cases you will do this, what safeguards you will require on the customer end, what alternatives to full disclosure you might suggest to customers for whom you don’t want to take the “Full Monty” approach, etc.

I hope you understand what I’m getting at here. For each of the six items in R1.2, you need to figure out a complete strategy for dealing with all of your customers (and as you can see, you will probably want to deal with particular types of customers in different ways, although the way you break down your customers is likely to vary according to the item in question). This will be an important component of your strategy, Mr./Ms. Vendor.

But this isn’t the only component of your strategy. My next post in this series will go over what else needs to be in your strategy.


If you are with a vendor to the electric power industry, and your company is trying to figure out what you will have to do to comply with CIP-013, Tom Alrich LLC would be pleased to offer you a free one-day (2-6 hours) workshop to review a) what CIP-013 requires, b) what you are likely to get asked to do by your clients, and c) what they should be asking you to do (since b and c won’t usually be the same thing). There will be no charge for my time, but I will require you to pay travel expenses at cost. Please email or call me if you would like to discuss this.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.       


[i] And of course, I’m also doing a series of posts on how NERC entities can comply with CIP-013. Here is the most recent post in that series. Vendors should read this series of posts as well!

[ii] And by the way, NERC entities also need a CIP-013 compliance strategy, which I am gradually laying out in my other series of posts. A lot of elements in both strategies should be the same – just told from different points of view. But a NERC entity’s strategy will inherently include a lot of elements that have nothing to do with vendors, since vendor risk is just one of three areas of supply chain risk that need to be addressed in the supply chain cyber security risk management plan. See this post for a short summary of those three areas, and see this post for a long, painful discussion of that topic – but which ultimately might be more enlightening.

Wednesday, February 14, 2018

I Forgot My Anniversary!



This might seem like an odd topic for a post on Valentine’s Day, but I’m actually referring to the anniversary of my blog. A friend asked yesterday how long I’d been writing the blog, and I at first was going to say three years - because for some reason I seem to have latched onto that number a couple years ago, and never bothered to do the advanced math since then.[i] Then I realized it’s been five years. In fact, just over two weeks ago I missed the fifth anniversary of my first post, written in a hotel room in San Diego on January 28, 2013.[ii]

So anyway, I’m proud that this blog has lasted five years, and I hope it lasts another five! And for those of you who’ve stuck with me for some or even all of the five years, thanks!

  
If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

Any opinions expressed in this blog post are quite definitely those of my employer, Tom Alrich LLC! If you disagree with what I’ve said, I suggest you take it up with them.


[i] This may also explain why for many years I’ve had the idea that I’m 42. I finally realized that had to be wrong when my son turned 30 last year – and I was sure he hadn’t been born while I was in junior high!

[ii] I had previously written posts for a Honeywell blog that no longer exists, and before that I’d written some “open letters” that Honeywell distributed. The open letters started in 2010, when I attended my first CIP drafting team meeting and found it so interesting – and consequential – that I wrote my first open letter about it.

The topic of my first post wasn't exactly timeless. I was trying to convince people that they had to take CIP version 4 seriously, since after all FERC had approved that version in April of 2012, and the compliance due date was April 1, 2014. I - and many others - felt that CIP version 5 was still a pipe dream that might never be approved by FERC. So it was much better to go with the bird in the hand (v4) rather than the two in the bush (v5). This campaign of mine culminated in what I call my "Dewey Beats Truman" post, when I confidently predicted that v4 would come into effect in 2014, with v5 following a few years later. Of course, just two months later FERC issued their NOPR saying they intended to approve v5, and v4 would go to sleep with the fishes.

Sunday, February 11, 2018

Lew Folkerth on Configuration Management



I apologize, but it seems I’ve fallen behind my minimum quarterly requirement of posts that quote from Lew Folkerth of RF. I just discovered Lew wrote a great article on configuration baselines and CIP-010 R1 for the RF Newsletter dated November/December 2017. You can find it by clicking on The Lighthouse in the table of contents on the left side of the page. I was also pleased to note that RF will now send out emails when new newsletters come out (which is bi-monthly), so neither you nor I will miss any future articles from Lew.

The article speaks for itself, but here are the points I found most interesting[i]:

  • Installed software and firmware listed in CIP-010 R1 should match software and firmware listed in CIP-007 R2 (Patch Management). Auditors check for this now, so you should definitely make sure they match on a regular basis, and even sync the two lists up if possible (page 16, last column).
  • A good tip for simplifying the job of CIP-007 R1 (Ports and Services) documentation by leveraging information from the baseline (p. 17, first column).
  • Benefits of a good baseline for incident response (p. 17, first column).
  • Lew’s recommended list of software and firmware to include in the baseline (p. 17, third column).
  • Lew recommends that firewall rules be under change management, whether or not they’re included in the baseline for the firewall.
  • The box about scripts on page 18 is worth the price of admission by itself! And that certainly doesn’t mean it’s worthless, even though admission to the article is free.
I recommend you all read the article, as well as subscribe to the newsletter.


If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

Any opinions expressed in this blog post are quite definitely those of my employer, Tom Alrich LLC! If you disagree with what I’ve said, I suggest you take the matter up with them.           

[i] A few of these aren’t new – in fact, I’ve written about them in previous posts) – but they’re worth repeating.

Thursday, February 8, 2018

How should Vendors “Comply” with CIP-013? Part I



The title of this post should give you pause. Vendors don’t have to comply with CIP-013; only NERC entities do. But this is a distinction without a difference. It has always been the intention of both FERC (in ordering NERC to develop a supply chain security standard) and the NERC standard drafting team (in developing the standard) to require vendors to be involved with compliance – although there will be no penalty to the NERC entity if they try to get a vendor to cooperate in compliance and are unsuccessful.

So it’s very legitimate to ask how a vendor can comply with CIP-013. I have had discussions with several vendors (and many more NERC entities) about this, and here is what we have come up with so far. I do want to point out that this is quite embryonic.

The most important consideration for a vendor is that it will be a big mistake if you look at CIP-013 compliance as some sort of adversary exercise, although you should be forgiven if you have thought of it that way so far. After all, a lot of the guidance so far (and that’s just one 13-page document) focuses on contract language.

My idea of a contract language “negotiation” is one in which the NERC entity’s lawyers and the vendor’s lawyers are sitting on opposite sides of an 8-foot-high brick wall. The entity’s lawyers throw some contract language to the other side and snarl “Here, put this in our contract.” The vendor’s lawyers look it over and throw something else back, saying (perhaps snarling less) “We can accept some parts without change and some with changes, but we can’t accept these other parts at all.” The entity’s lawyers look this over, and reply with their own counter-proposal. This goes back and forth, with the length of time determined by whether or not the lawyers are on salary (in which case they can do this for years) or they are paid by the hour (in which case the company will have to call an end to the fun and tell them to wrap up the “negotiation”).

Of course, this is perhaps an overly bleak view of the contract negotiation process, but I think it illustrates what I’m saying: Both the vendor and the entity should do everything they can to keep their relationship from degenerating to this point. In fact, IMHO if they first resort to contract language in their CIP-013 compliance process, both sides have already lost the game. Contract language is a nice thing to have, but it should never be the first concern.

So what’s the best way for a NERC entity and a vendor to approach the CIP-013 compliance process? I see two possible ways to get this going:

  1. The entity reaches out to the vendor and says “Let’s have a discussion on how we (note the plural pronoun!) can address CIP-013 compliance.”
  2. The vendor reaches out to all of their power industry customers (at least all who have any High or Medium impact assets where their software or hardware product is used), perhaps through an e-mailing or even a – gasp! – USPS mailing, and actually suggests how both sides could work together to achieve the goal of CIP-013 compliance.
So here’s a little quiz: Which of these two actions is more likely to result in a fruitful partnership to achieve CIP-013 compliance? I think they are both good options, but I would strongly recommend that the vendor be the first to reach out. Because if the entity reaches out first, I can almost bet that most of them will do that in the form of a demand for particular contract language. This isn’t because they’re sociopaths, but because NERC compliance up until now has primarily been seen as a legal exercise, whether for NERC CIP or the other NERC standards.

If the vendor reaches out first, and if they don’t immediately focus on contract language (if they do, they’ve obviously decided that it’s important that CIP-013 compliance be made as painful as possible, probably for reasons of job security of the people writing the document), then they can set the terms of engagement. The vendor can suggest actions that favor its strengths – i.e. developing, supporting and securing great hardware and/or software products – and don’t favor the utility’s strength (which is the fact that they are, after all, the buyer, and that money is flowing from them to the vendor, not vice versa. Plus the vendor doesn’t have a monopoly, at least in any electric power product market that I know of).

What should the vendor suggest to their power industry clients? I’ll leave that for the next post in this series. 


If you are with a vendor to the electric power industry, and your company is trying to figure out what you will have to do to comply with CIP-013, Tom Alrich LLC would be pleased to offer you a free one-day (2-6 hours) workshop to review a) what CIP-013 requires, b) what you are likely to get asked to do by your clients, and c) what they should be asking you to do (since b and c won’t usually be the same thing). There will be no charge for my time, but I will require you to pay travel expenses at cost. Please email or call me if you would like to discuss this.

If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.

Any opinions expressed in this blog post are quite definitely those of my employer, Tom Alrich LLC! If you disagree with what I’ve said, I suggest you take it up with them.


Tuesday, February 6, 2018

I’m an ICS “Influencer”!



I was honored to be asked by Indegy – a company you should really learn about, especially if you own or operate power plants – to join nine other industrial cyber security “influencers” in giving our ideas on what you, i.e. a staff member in an organization operating ICS, can expect in 2018. While I can’t tell you whether your taxes will be lower or you’ll receive a bigger bonus this year, I can talk about what I think is an important development – which anyone who follows this blog might have already heard about once or twice J. You can read the post, on Indegy’s blog, here.


If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. To discuss this, you can email me at the same address or call me at 312-515-8996.


Friday, February 2, 2018

Complying with CIP-013, Part 3: What is a good Implementation?


If you’ve read the first two posts in this series (here and here), you realize that for these posts I’ve tried to forget everything I know or have written about CIP-013, and just work through the words of the requirements (and the Purpose statement at the beginning of the standard). My goal has been to mine what is actually written in the requirements and get it all out on the table. The reason I’m doing this is because nothing is binding on the auditors (and therefore on the NERC entity) except what is in the requirements. So it’s important to put in a lot of effort to pull apart the requirements to find everything that’s possible to find in them – before we start going beyond what the wording says and start to think about the best way to actually comply with the requirements.

In our last episode, we wallowed in R1 and found – to my genuine surprise – that there is something important missing from R1.1: While this requirement mandates that the NERC entity “identify and assess” cyber security risks in the supply chain, it doesn’t require them to do anything about them! Of course, this is clearly just a mistake, and I highly recommended that everyone assume the word “mitigate” is also in there, whether or not this problem ever gets fixed in a formal manner.

Now we’re ready to go on to R2, which on the surface seems quite simple:

R2. Each Responsible Entity shall implement its supply chain cyber security risk management plan(s) specified in Requirement R1.

Is this really that simple? Yes and no. It isn’t simple because of what I just pointed out about R1.1: Your supply chain cyber security plan must include mitigation of the risks you’ve identified and assessed, even though that step isn’t included in the requirement part; so you need to mitigate risks when you implement the plan as well. But it is simple because it doesn’t mandate anything about how the plan needs to be implemented.

It seems you’re on your own in this requirement. You have to implement the plan, but you might take one year or 50 years. You might complete one part before starting on the next one, or you might address all parts simultaneously and try to complete them all at the same time. You might prototype your plan in one part of the company before moving it to others, etc. It seems the details are all up to you.

Indeed, this idea is reinforced in spades by the Implementation Guidance document put out by the CIP-013 drafting team last year. The document says literally nothing about what must be done to implement the plan, beyond repeating the substance of the note that appears below R2, emphasizing that the entity isn’t required to renegotiate or abrogate existing vendor contracts. It seems the drafting team couldn’t think of anything important to say about implementation.

However, let’s be realistic: The details of implementing a NERC CIP standard are never up to you. Last fall, I looked at the industry’s early experience with audits of CIP-014; like CIP-013, this standard requires the entity to develop a plan (in this case, a physical security plan for key substations) and then implement it. In two posts (here and here), I told several stories that point to the same conclusion: Even though R2 lacks any details about what constitutes a good vs. a bad implementation, the auditors are going to have their own ideas, and they won’t hesitate to make these known to the entity if they think they haven’t done a good job of implementation[i]. They hopefully won’t actually issue a Potential Non-Compliance (PNC) finding, as they did to one of the entities discussed in the second post. But they will very likely identify an Area of Concern and require the entity to fix the problem identified.

So what are the auditors going to look to when they decide whether or not an entity has properly implemented their supply chain cyber security risk management plan from CIP-013 R1? Darned if I know, but there is one thing of which I’m sure: I’m sure that auditors will look at the actual timeline for completing the implementation, and how that compares with what is in the plan.

Say your plan lays out a two-year implementation schedule, and your next audit is scheduled for about 1 ½ years after you start the implementation. If the auditors show up on that date and decide there’s no way your implementation can be completed by the two-year mark, they will very likely issue an Area of Concern, meaning you will need to develop a new plan with a realistic implementation date. And you’d better make that new date, or you’ll be in trouble three years later at your next audit.

So what does this mean for your plan? Should you always sandbag the finish date in the plan, making it at least a year or two after the date you think you’ll really finish? I don’t really think this is a great idea. I think you should include in your plan what you honestly believe is the date you will finish your implementation. However, I also think you should keep very close tabs on the implementation. Whenever it begins to look like you will not be able to meet your original finish date, you should immediately revise the plan to reflect the new date (of course, you should have change control to document that you made this change. You certainly don’t want to be in the position of needing to convince the auditor that the revised plan was actually the original one!).

Of course, if the auditor doesn’t agree with your excuse for why the implementation date had to be moved back (e.g. “My dog ate our supply chain cyber security risk management plan and we didn’t have any backup. This forced a six-month implementation delay as we re-drafted the plan from scratch.”), they may disagree with you. But IMHO there is no way they can issue you a PNC because of this, since R2 is completely silent on what characterizes a good vs. a bad implementation.[ii]

What’s the moral of this story? Even though R2 looks like it’s wide open to whatever interpretation you might want to give it, this doesn’t mean you should simply do whatever you want when you implement your supply chain cyber security risk management plan.  The auditors will always have their idea of how a plan should be implemented. It is almost certain that they will make their ideas clear to you in your audit, even if – hopefully – they don’t resort to a PNC to do that.

But what if it didn’t need to be this way? What if there were a way you could get your auditors (or at least your Regional Entity) to weigh in on whether and how your plan should be changed, before you started to implement it? That is the question I discussed in this recent post. There needs to be some way a NERC entity can get their CIP-013 R1 plan reviewed by their Region before they embark on implementing it. Moreover, there needs to be some way the Region can check in on the implementation of that plan as it is in progress, to make sure the entity hasn’t made some big mistake that might force it to re-do some or all of what it has already done.

The post described, with the assistance of an auditor, the Entity Development teams that are already implemented in one NERC Region and being implemented in another. The members of these teams aren’t auditors, but they help entities comply with the NERC CIP standards (and possibly other NERC standards). It seems to me that these teams would be the ideal organization to review an entity’s CIP-013 R1 plan as well as its implementation, before the entity has invested a sizable sum in implementation of what could turn out to be a flawed plan

I believe that some mechanism like this needs to be in place, in order for CIP-013 (as well as other plan-based standards like CIP-014 and the upcoming CIP-012, along with plan-based requirements like CIP-003 R2 and CIP-010 R4) to be successful. I hope you agree with me on this (and if not, that you’ll let me know). I hope NERC does, too!


If you would like to comment on what you have read here, I would love to hear from you. Please email me at tom@tomalrich.com. Please keep in mind that Tom Alrich LLC can help you with NERC CIP issues or challenges like what is discussed in this post. You can email me at the same address or call me at 312-515-8996.

[i] To be honest, the auditor issues in the second of the two posts cited (i.e. Part 3 of the series on lessons from CIP-014) had to do mostly with the plan itself, not with its implementation. But that is just because of timing: The two audits discussed in that post both occurred before the entity had even been able to start to implement their plan. Of course, this was very beneficial, since – had they implemented the plan, which was later determined to be insufficient, they might have been forced to re-do some or all of their implementation, using a new plan that was deemed sufficient.

[ii] They might conceivably issue you a PNC for violating CIP-011 R1.2, since the plan presumably included BES Cyber System Information and you obviously didn’t have a very good plan for protecting BCSI from your dog.