1 Introduction
Adequate security of information in Industrial Control Systems (ICS) and supporting Critical Infrastructures (CI) is a fundamental management responsibility. ICS employees and supervisors must be constantly aware of the status of their information security controls, in order to make informed judgments and investments that appropriately mitigate risks to an acceptable level. Cybersecurity self-assessment tools realize risk assessment and risk management procedures, provide automated solutions for CI operators and owners to determine the current status of their information security programs and, where necessary, pinpoint specific targets for improvement. Self-assessment tools utilize extensive and structured questionnaires containing specific control objectives and security measures against which any ICS or group of interconnected ICS systems can be tested and evaluated [1].
1.1 Motivation and Contribution
This paper aims to provide a review and analysis of available cybersecurity Self-Assessment tools, which can be used by ICS owners and CI operators. These tools support organizational risk management and enforce cybersecurity by identifying operating weaknesses, employee’s security awareness and by evaluating implementation of effective control practices to protect ICS against realistic threats and associated risks. In addition, we deepen our research into questionnaire content analysis, which is used by the examined self-assessment tools, with the purpose to create a classification on questions content, according to the Core Functions presented by the newly published “Cybersecurity Framework” (v.1.1) of National Institute of Standards and Technology (NIST), which promotes the protection and resilience of critical infrastructures [2].
1.2 Outline/Structure
The structure of this paper after the introductory part is as following: Sect. 2 presents Security Challenges for ICS and related work on ICS Cybersecurity Risk Assessment and Management. Section 3 presents four developed self-assessment tools and provides a comprehensive comparison. In Sect. 4, the analysis is extended to questionnaire content analysis and classification. Finally, Sect. 5 extracts main conclusions and importance evaluation of using Cybersecurity Self-Assessment Tools for risk management purposes.
2 Cybersecurity Challenges for Industrial Control Systems
ICS is a general term describing cyber-physical and automation systems responsible for data acquisition, visualization and control of processes found in industrial sectors and supporting CIs. They play a critical role, not only in maintaining the business automation, but also in ensuring functional and technical safety, preventing large industrial accidents and environmental disasters [4]. In the past, ICS had little resemblance to the traditional information technology (IT) systems, since they were isolated systems, running proprietary control protocols and using specialized hardware & software [1]. Widely available, low-cost Internet Protocol devices are now replacing proprietary solutions, which increase their functionality and interoperability, along with the possibility of cyber security vulnerabilities and incidents. Moreover, the goals of safety and efficiency sometimes conflict with security in the design and operation, as ICS have unique execution criticality and reliability requirements (24 × 365) and change management can jeopardize their integrity and performance [3].
The trend towards integrating ICS systems with IT networks provides significantly less isolation from the outside world, creating a greater need to secure these systems from remote, external risks. Threats to both ICS and IT systems can come from numerous sources, including malicious intruders, terrorist groups, disgruntled employees, accidents and others [4]. Therefore, ICS have greater security challenges to confront, since they have not achieved yet the same level of cybersecurity maturity as other cyber or IT resources.
2.1 Related Work on ICS Security Management
Over the last decade, a number of standards and directives dealing with cybersecurity of ICS systems have emerged. In 2004, NIST published the System Protection Profile for Industrial Control Systems, which covered the risks of ICS systems [5]. In 2007, the US President’s Critical Infrastructure Protection Board and the Department of Energy outlined the steps an organization must undertake to improve the security of ICS networks by introducing 21 Steps to Improve Cyber Security of SCADA Networks [6]. In 2008, the UK Centre for Protection of National Infrastructure (CPNI) produced a Good Practice Guide for Process Control and SCADA Security encapsulating best security practices [7]. In 2013, the European Union Agency for Network and Information Security (ENISA) released the recommendations for Europe on ICS security [8] and three years later published security good practices for ICS/SCADA Systems [9]. In 2014, North American Electric Reliability Corporation (NERC) introduced the development of a wide range of standards covering ICS cyber security [10]. NIST has released a comprehensive guidance on wide range of security issues, and technical, operational and management security controls, last updated in 2015 [1]. Finally, in 2018 the ISA99 committee published ISA/IEC 62443 security standard, which suggests the adoption of a flexible framework in order CI to address and mitigate current and future security vulnerabilities in industrial automation and control systems [11]. Over and above to these guidance work, scientific research has developed various CIP tools, able to model CI characteristics, their interdependencies and the impact of potential failures in their systems. In previous work [12], a review of sixty-eight available in literature tools, frameworks and methodologies for CI protection were analyzed and classified. However, these tools do not focus only on ICS systems; instead, they examine CIs entities as a whole.
Risk assessment is generally understood as the process of identifying, estimating and prioritizing risks to the organizational assets and operations [1]. This is an essential activity within security management as it provides the foundation for risk identification and treatment with the adoption of effective cybersecurity measures. Although several methods and tools are available in literature for conducting risk assessments, the particularities of SCADA often prevent the straight forward application and adjustment is required to fit the context of SCADA systems. Therefore, focused on ICS systems, a detailed overview of 24 risk assessment methods developed for SCADA systems was presented by Cherdantseva et al. [13]. This work pinpointed that there was no software prototype or automated tool for the vast majority of the methods examined for ICS, in order to support their implementation. Instead, in several methods the development of software prototype was outlined as a subject for future work. Our literature research revealed that despite exhaustive work on ICS cybersecurity protection guidance, risk assessment and management tools, no research has been presented related to self-assessment tools analysis and their complementary effect on ICS cybersecurity and efficient risk management.
Self-assessments usually provide an additional tool for organizations to determine current status of their information security programs, improve staff security awareness, prepare organization before security audits and establish new targets for improvement [3]. Most self-assessment methods utilize an extensive questionnaire survey, containing specific audit objectives, for testing and evaluating control systems or group of interconnected systems. These questionnaires do not establish new security requirements. Instead, their control objectives and techniques are abstracted directly from long-standing requirements and established standards, as found in statute, policy, and guidance on security. For a self-assessment to be effective, a complementary risk assessment should be conducted by security experts in parallel or in advance. Therefore, a self-assessment does not eliminate the need for a risk assessment within the organization Risk Management Program.
Since 2003, NIST has introduced the first Automated Security Self-Evaluation Tool to automate the process of completing a system self-assessment, contained in NIST Special Publication 800-26, which was later retired and replaced by NIST SP 800-53A [3]. Since then several security self-assessment tools have been developed, evolved and enhanced with functionalities which are presented, analyzed and compared in Sect. 3. As part of its efforts to increase awareness, understanding and reducing cyber risks to critical infrastructures, NIST has also developed a voluntary framework, based on existing standards, guidelines and practices [2]. NIST Cybersecurity Framework creates a solid basis for managing cybersecurity risks related to critical infrastructure. The framework provides a risk-based approach for cybersecurity through five core functions: (i) identify, (ii) protect, (iii) detect, (iv) respond, and (v) recover. These core functions represent the 5 primary pillars for a successful and holistic cybersecurity program. They aid organizations in easily expressing their management of cybersecurity risk at a high level and enabling risk management decisions. A short description of each function follows:
- (1)
The Identify Function assists in developing an organizational understanding to managing cybersecurity risk to systems, people, assets, data and capabilities.
- (2)
The Protect Function outlines appropriate safeguards to ensure delivery of critical infrastructure services.
- (3)
The Detect Function defines the appropriate activities to identify the occurrence of a cybersecurity event. It enables timely discovery of cybersecurity events.
- (4)
The Respond Function includes appropriate activities towards a detected security incident, by enhancing the ability to contain impact of any cybersecurity incident.
- (5)
The Recover Function identifies appropriate activities to maintain plans for resilience and to restore any capabilities or services that were impaired due to a cybersecurity incident, by supporting timely recovery to normal operations.
3 ICS Cyber Security Self-assessment Tools
In this section we briefly present four self-assessment tools that have been developed with the scope to support cybersecurity management in critical CIs and provide specific analysis, or a dedicated section for ICS & SCADA evaluation.
- (i)
Control System Cyber Security Self-Assessment Tool (CS2SAT): A desktop software tool that gathers information about the facility of ICS, guides users through a step-by-step process to collect specific control system information and makes appropriate recommendations for improving system’s cyber-security. The purpose of CS2SAT is to provide organizations that use ICS to control any physical process with a self-assessment tool for evaluating the programmatic and certain aspects of security [14]. It is designed as a self-contained tool to assist individuals in identifying cyber security vulnerabilities and then it provides a comprehensive evaluation of implemented security programs and comparison to existing industry standards and regulations. The CS2SAT also provides recommendations from a database of industry available cyber-security practices. Each recommendation is linked to a set of actions that can be applied to remediate specific security vulnerabilities [14].
- (ii)
Cyber Security Evaluation Tool (CSET): A desktop software tool, which helps through a step-by-step process, owners to assess information and operational systems cybersecurity practices, by asking a series of detailed questions about system components and architectures, as well as operational policies and procedures [15]. These questions are derived from accepted industry cybersecurity standards. CSET includes a dedicated section to support ICS and SCADA security analysis for a tailored assessment of cyber vulnerabilities. Once ICS standards have been selected and the resulting questionnaire is answered, CSET creates a compliance summary, compiles variance statistics, ranks top areas of concern, and generates security recommendations.
- (iii)
SCADA Security Assessment Tool (SSAT): A tool developed by UK Centre for the Protection of National Infrastructure (CPNI) for SCADA utilities and CIs. According to CPNI, it provides a high-level snap-shot of the information assurance of an organization’s ICS that are deemed to constitute the UK critical national infrastructure [16]. Moreover, it contains 99 questions divided into various categories for physical, personnel and electronic evaluation performance, based upon CPNI Good Practice Guidance and international good practices. SSAT output result is a performance scoring, aggregating users answering on specific targeted questions and providing high level understanding of SCADA/ICS security status [17]. Finally, SSAT is not a standalone self-assessment tool, so it is less robust tool than the previous examined ones.
- (iv)
Cyber Resilience Review Self-Assessment Package (CRR): An interview-based assessment able to evaluate a CIs organization’s cybersecurity practices and operational resilience. It has a dedicated section for control management and can be either conducted as a self-assessment or as on-site assessment facilitated by cybersecurity professionals [18]. CRR focuses on key areas that typically contribute to the overall cyber resilience and measures essential cybersecurity capabilities to provide indicators of an organization’s operational resilience during normal operations and during times of operational stress. CRR can evaluate cyber resilience capabilities of a wide range of organizations both in terms of different critical services or CI sectors and in terms of organizational size and maturity [19].
3.1 Cybersecurity Self-assessment Tools Comparison
Cybersecurity self-assessment tools comparison
Tool description | CS2SAT | CSET | SSAT | CRR |
|---|---|---|---|---|
Type | Desktop software application tool | Desktop software application tool | Questionnaire XLS assisted Tool | Questionnaire PDF assisted Tool |
Developer | Department of Energy National Laboratories | ICS-CERT/DHS | CPNI | US-CERT/DHS Carnegie Mellon University |
Origin | USA | USA | UK | USA |
Description | Self-contained tool step-by-step process | Self-contained tool step-by-step process | SSAT Questionnaire which links directly to the CPNI SCADA security good practice | Self-contained tool |
Step Process | 6 | 5 | 1 | 1 |
Survey Method | Structured Questionnaire | Structured Questionnaire | Structured Questionnaire | Structured Questionnaire |
Security Expertise Needed | YES | NO | YES | NO |
Standards Compliance | NERC CIP, NIST SSP-CIPCS, NIST SSP-ICS, NIST SP 800-53, DoD 8500.2 ISO/IEC 15408 | DHS Cat. of CS NERC CIP 002-009 NIST SP 800-82 NIST SP 800-53 NRC Reg. Guide 5.7 CNSSI 1253 INGAA Control Security Guidelines NISTIR 7628 Guide | CPNI Good Practices NIST SPP-CIPCS NIST SPP-ICS ISO/IEC 15408 NERC CIP 002-009 NIST SP 800-53 DoD IA | NIST SP 800-18 NIST SP 800-30 NERC CIP FISCAM Clinger-Cohen law GISRA law FIPS 102 OMB Circul. A-130 |
Checks ICS Compliance with Security Standard | YES | YES | NO | NO |
Database of industry available cyber-security practices | YES | YES | NO | NO |
Sector average score | ΝΟ | YES | YES | NO |
Recommendation List | YES | YES | YES | YES |
Type of Result | Full Performance Evaluation | Full Performance Evaluation & Compliance of Selected Std | Scoring Result | Full Performance Evaluation |
From graphical facilities, CSET and SSAT contain a graphical user interface that allows users to diagram network topology and identify the “criticality” of the network components. Moreover, in CSET user can import a pre-built template diagram or import an MS Visio diagram. One main difference occurs in the presentation of the results, where CSET gives a full report of evaluation performance with compliance analysis according to selected standards. Less detailed report is produced by CS2SAT, while CCR report focus more on resilience and contingency reporting analysis and recommendations. Finally, the SSAT provides a simple scoring result with limited technical analysis and recommendations. Overall the CSET is the most technical complete tool, which covers all particular issues of ICS control and adjusts to users’ needs for every standard compliance. In addition, it can be characterized as the most user-friendly, although sometimes, its detailed analysis can be time consuming for users.
Last but not least, one common functionality of all the above presented tools is that they base they evaluation on a well-structured and specific targeted questionnaire to assess the security programs and organization risk management effectives. Therefore, in the following section we will further investigate questionnaire functionalities, design and characteristics.
4 Questionnaire Content Analysis
As described in Table 1, for every Self-Assessment tool a structured questionnaire is used as a survey method for collecting valuable information for self-evaluation purposes. This is a common technique for collecting information and completing an internal assessment of the security controls designed, applied and performed. These questionnaires can serve for many purposes. First, they can be used by management team and experts who know their agency’s systems and security controls to gain a general understanding of security assurance and make informed decisions about the agency improvement needs. Second, they can be used as a guide for thoroughly evaluating the status of security for a system. Third, they can enhance and support employees’ security awareness. Finally, the results of such thorough reviews provide a much more reliable measure of security effectiveness and may be used to (1) fulfill reporting requirements; (2) prepare for audits; and (3) identify resource needs. Therefore, the completed self-assessment questionnaires are a useful resource for compiling agency reports, such as: security program management and security planned activities [15].
In this section we analyze and compare the questionnaires used is the above described tools used for assessing ICS security, which are namely: (i) NIST Security Self-Assessment Questionnaire which is used in the first three examined tools; (ii) CSET Scada Self-Assessment Questionnaire when not using NIST Cybersecurity Framework; and (iii) CRR Self-Assessment Questionnaire used by the CRR tool.
4.1 Overview of SCADA Self-assessment Questionnaires
NIST Security Self-assessment Questionnaire:
NIST topic areas of questions
NIST questionnaire screenshot
CSET SCADA Self-assessment Questionnaire:
CSET topic areas of questions
CRR Self-assessment Questionnaire:
CRR questionnaire domain composition
CRR questionnaire domain | Goals | Goal practices | Questions |
|---|---|---|---|
Asset Management (AM) | 7 | 29 | 78 |
Controls Management (CM) | 4 | 16 | 38 |
Configuration and Change Management (CCM) | 3 | 23 | 37 |
Vulnerability Management (VM) | 4 | 15 | 47 |
Incident Management (IM) | 5 | 23 | 36 |
Service Continuity Management (SCM) | 4 | 15 | 31 |
Risk Management (RM) | 5 | 13 | 26 |
External Dependencies Management (EDM) | 5 | 14 | 27 |
Training and Awareness (TA) | 2 | 11 | 24 |
Situational Awareness (SA) | 3 | 8 | 21 |
TOTAL | 42 | 167 | 365 |
Each domain is composed of a purpose statement, a set of specific goals and associated practice questions unique to the domain, and a standard set of maturity indicator level questions. The MIL scale uses six maturity levels, which are: (i) Incomplete, (ii) Performed, (iii) Planned, (iv) Managed, (v) Measured, and (vi) Defined. The CRR divides assets into four categories: People, Information, Technology, and Facilities. Some questions require a separate answer for each of the four assets, while other questions refer to all assets.
4.2 Questionnaires Comparison
Questionnaire analysis
Questionnaire analysis | NIST | CSET | CRR |
|---|---|---|---|
Number of Available Questions | 258 | 1030 | 365 |
Question Type | Close Ended, Scaled Answering (L1-L5) | Open & Close Ended, YES/NO Answering | Open & Close Ended, YES/NO Answering |
Link to supplementary information or explicatory info provided | No | Yes | Yes |
Additional Comments Allowed | Yes | Yes | Yes |
Complementary data requested based on user’s answering allowed | No | Yes | No |
Static/Dynamic security flow analysis | Static Flow | Dynamic Flow | Static Flow |
NIST has a static flow of questioning structure for analyzing the 17 topic areas presented in Fig. 1, while question type permits a scaled answering with 5 implementation levels. CSET is the most detailed and advanced questionnaire with a dynamic flow of questions, and interaction based on user’s selection. It can accept additional information either in the form of comments, data, files, graphs, diagrams and other material, while the user can override any question he considers as irrelevant. CRR uses both close and open-ended questions from ten thematic domains and can provide additional information to respondents to assist and facilitate their assessment.
Questionnaire analysis based on NIST cybersecurity framework
Questions related to the Identify Function vary from 16% or 41q in NIST Questionnaire, 19% or 194q in CSET and 32% or 116q in CRR, which indicates that organizational understanding to cybersecurity management is more trivial to assess. On the other hand, the questions dealing with Response and Recovery Function keep a low as a percentage, despite resilience and contingency necessity in ICS and CI facilities.
The greatest gravity of self-assessment questionnaires is given to protective measures and controls, related to less importance given on managerial and operational practices as included in the identify function. We can also realize that response and recovery investigation functions are significantly less examined, despite being an essential function for organization’s resilience. This area should be further enriched in the future with additional content to assess specific areas to self-assessment questionnaires and related tools.
5 Conclusions
Adequate security of information in ICS supporting CIs is a fundamental management responsibility. ICS employees must be constantly aware of the status of their information security controls, in order to make informed judgments and appropriately mitigate risks to an acceptable level. There are several methods and tools for agency officials to help determine the current status of their security programs relative to existing policy. Ideally many of these methods and tools would be implemented on an ongoing basis to systematically identify weaknesses and where necessary, establish targets for continuing improvement.
Self-assessment tools provide a tailored assessment for CI operators and owners for assessing cyber vulnerabilities. Based on a selectable array of cybersecurity standards, these tools provide structured questionnaires to build organizational knowledge and create a cybersecurity compliance report with compiled statistics and security recommendations. Since self-assessment tools do not generate a complex risk assessment, they will not provide a detailed architectural analysis of the network or detailed hardware/software configuration review. Therefore, periodic onsite reviews and inspections must still be conducted using a holistic approach including facility inspection, interviews, and examination of facility practices and penetration testing.
From Self-Assessment Tools Comparison, commonalities and differences have been exhibited along with main tools’ functionalities. In conclusion, the CSET is the most technical complete tool, which covers all particular issues of ICS control and adjusts to users’ needs for every standard compliance. In addition, it is characterized as the most user-friendly, although sometimes its detailed analysis can be time consuming for users.
It is important to note that self-assessment tool is not intended to provide an all-inclusive list of control objectives and related techniques. Accordingly, it should be used in conjunction with the more detailed guidance listed in cybersecurity standards and government/legal mandates. In addition, specific technical controls, such as those related to individual technologies or vendors, are not specifically provided due to their volume and dynamic nature.
On the other side, while comparing Cybersecurity self-assessment questionnaires, we have found a diversified number of available questions, however the majority of them focuses on protection measures and technical safeguards to ensure cybersecurity performance. Response and recovery investigation are less examined, despite being an essential function for organization’s resilience, so this area should be further enriched in the future with additional content to assess specific areas to self-assessment questionnaires and related tools.
After all, self-assessment questionnaires are only one component of the overall cyber security assessment and should be complemented with a robust cyber security evaluation program within the organization. A self-assessment cannot reveal all types of security weaknesses and should not be the sole means of determining an organization’s security posture. It should also be noted that an agency might have additional laws, regulations, or policies that establish specific requirements for confidentiality, integrity, or availability. Each agency should decide if additional security controls should be added to the questionnaire and, if so, customize the questionnaire appropriately.