Software Quality Standards – ISO 5055 – CISQ

Software Quality Standards – ISO 5055

CISQ developed Automated Source Code measurement standards for Reliability, Security, Performance Efficiency, and Maintainability, which were approved as OMG standards. They were upgraded to incorporate both IT and embedded software weaknesses and then combined into the Automated Source Code Quality Measures standard. OMG submitted this latter standard to the International Organization for Standards, which adopted it as ISO/IEC 5055:2021. Since the OMG and ISO Automated Source Code Quality Measures standards are identical, this remainder of this page will refer to them as ISO 5055.

READ THE STANDARD

What is ISO/IEC 5055:2021?

ISO/IEC 5055:2021 (which will be referred to as ISO 5055 hereafter) is an ISO standard for measuring the internal structure of a software product on four business-critical factors: Security, Reliability, Performance Efficiency, and Maintainability. These are the factors that determine how trustworthy, dependable, and resilient a software system will be.

Why is ISO 5055 important?

Before ISO 5055, there was no international standard for measuring the quality and integrity of a software system by analyzing its internal construction to detect severe structural weaknesses. This would be equivalent to evaluating a house by its outward appearance without ever checking its internal structure for wood rot. Existing software standards measured after-the-fact operational measures that assessed the costly operational effects of severe weaknesses. ISO 5055 provides before-the-fact measures of the product’s software during development to identify and eliminate structural weaknesses before they cause operational problems.

How are ISO 5055 measures computed?

A team of international software experts at CISQ sorted through a wide range of software weaknesses and selected the most dangerous ones for inclusion in ISO 5055 measures, where ‘danger’ was based on knowing a weakness had to be removed from the software to avoid damaging business operations or excessive IT costs. The search for severe weaknesses must be conducted across the entire stack of software technologies and their interconnections that compose a modern system. Counts of the weaknesses are calculated for each of the four factors that may then be transformed into comparative measures such as the density of weaknesses or the sigma level achieved by the product.

What are some examples of ‘dangerous’ weaknesses?

The ISO 5055 weaknesses include serious problems at both the architectural and component levels to provide a complete evaluation of the factors determining a system’s trustworthiness, dependability, and resilience.  ‘Ban Unintended Paths’ is an architectural weakness that violates security and data protection controls by allowing a path from the user interface directly to the database without passing through user authentication routines. ‘Improper Resource Shutdown or Release’ is a reliability weakness that has frozen customer-facing systems during critical business hours. The other weaknesses comprising ISO 5055 have similar undesirable impacts on business operations and IT costs.

How can we use these measures?

ISO 5055 measures can be used to set measurable targets for ensuring the trustworthiness, dependability, and resilience of software systems. These targets can be written into requests for proposals, statements of work, and contracts as acceptance criteria for software products delivered by system integrators, software vendors, and other third-party suppliers. They can also be used with internal software teams to establish release criteria or improvement targets.  Some of the weaknesses contained in ISO 5055, such as the most dangerous security and reliability weaknesses, should be marked as ‘unacceptable,’ and software cannot be put into operation until they have been removed.

Who should be using ISO 5055?

CIOs/CSOs can use ISO 5055 to:

1. Evaluate the risk in their various applications and allocate resources to those most requiring correction
2. Assess the level of technical debt in their applications and estimate their future cost of ownership
3. Determine whether improvements in their development practices are producing higher quality, less risky code

Vendors and procurement teams can use ISO 5055 to:

1. Set quality targets for the source code delivered from their contractors.
2. Evaluate the quality of the source code received from their vendors and require correction when necessary
3. List weaknesses that must not exist in source code delivered from contractors

Software engineers can use ISO 5055 to:

1. Evaluate the source code they are submitting to a build
2. Learn about critical weaknesses they must avoid creating in their source code
3. Become certified as a ‘Dependable Developer’ by demonstrating knowledge of the weaknesses in ISO 5055

How do I implement ISO 5055?

The ISO 5055 standard is implemented by vendors of static analysis technology that detect, report, and measure its weaknesses across the entire technology stack and its interconnections. IT organizations should select a vendor technology that is endorsed by the Consortium for Information and Software Quality (www.it-cisq.org), the originator of the ISO 5055 standard, as being conformant with ISO 5055. Static analysis at the system level should become a standard task in vendor acceptance, application modernization, and quality assurance processes.

How does 5055 relate to other standards?

The four measures in ISO 5055 evaluate 4 of the 8 quality characteristics specified in ISO/IEC 25010, the ISO quality model for software systems. Each of the ISO 5055 measures conforms to the definition of its associated quality characteristic in ISO/IEC 25010.  The set of weaknesses in each measure were checked to ensure coverage of the entire domain of issues in the associated quality characteristic. ISO 5055 measures supplement ISO/IEC 25023 (which defines software measures mainly at the operational rather than the product level) by developing structural and architectural measures from the product’s software. Each ISO 5055 weakness is included in the Common Weakness Enumeration Repository (mitre.cwe.org), a listing of the most severe software weaknesses, which is an International Telecommunication Union (ITU) standard.

Measuring Code Quality

Development teams can use code quality standards to evaluate the structural quality of software ahead of each release. By applying standards earlier in the software development lifecycle, a codebase can be carried over to other products, developed further, or open sourced with greater confidence, resulting in less technical debt and complexity. The measures are designed to be automated on source code through static analysis and provide an industry-wide foundation for benchmarking, setting quality targets, providing visibility, and tracking improvement progress.

Built on Common Weakness Enumeration (CWE)

Each code quality measure for Reliability, Performance Efficiency, Security, and Maintainability is comprised of a set of weaknesses (CWEs) in the Common Weakness Enumeration (CWE). The CWE is a reference point for developers and tools and codifies over 800 known software weaknesses. The coding rules enshrined in CISQ standards include such CWEs as SQL Injection and Buffer Overflow. CISQ identified the most critical and impactful CWEs and standardized them for automation under each quality characteristic. THE CWE Repository is managed my MITRE corporation with funding by the Department of Homeland Security (DHS) and the National Institute of Standards and Technology (NIST).

Download a list of the CWEs in each code quality measure

SUPPLEMENTING ISO 25000 BY MEASURING QUALITY AT THE SOURCE CODE Level

The ISO/IEC 25000 series of standards, also known as SQuaRE (System and Software Quality Requirements and Evaluation), contains a framework to evaluate software product quality. ISO/IEC 25010 defines a set of eight software quality characteristics, or system “-ilities,” i.e. security, reliability, and maintainability. ISO/IEC 25023 describes how to apply the quality characteristics to measure product quality. However, the measures defined in 25023 largely measure quality at the behavioral level rather than at the level of specific quality problems in the source code. To supplement the level of measurement in 25023, CISQ defined source code level measures of four quality characteristics — Reliability, Performance Efficiency, Security, and Maintainability as outlined above. Dr. Bill Curtis, Founding Executive Director of CISQ, is the American lead on ISO 25000 and we seek to supplement ISO 25000 with actionable, automated standard measures of quality.

Coverage at the Unit and System Level

The following table shows a snapshot of software engineering rules contained in the measurement of each code quality characteristic at the code unit level and system level.

Software Quality Characteristic
Coding Practices Unit Level
Architectural Practices System Level

Reliability

• Protecting state in multi-threaded environments
• Safe use of inheritance and polymorphism
• Resource bounds management, Complex code
• Managing allocated resources, Timeouts

• Multi-layer design compliance
• Software manages data integrity and consistency
• Exception handling through transactions
• Class architecture compliance

Performance Efficiency

• Compliance with Object-Oriented best practices
• Compliance with SQL best practices
• Expensive computations in loops
• Static connections versus connection pools
• Compliance with garbage collection best practices

• Appropriate interactions with expensive or remote resources
• Data access performance and data management
• Memory, network and disk space management
• Centralized handling of client requests
• Use of middle tier components vs. procedures/DB functions

Security

• Use of hard-coded credentials
• Buffer overflows
• Missing initialization
• Improper validation of array index
• Improper locking
• Uncontrolled format string

• Input validation
• SQL injection
• Cross-site scripting
• Failure to use vetted libraries or frameworks
• Secure architecture design compliance

Maintainability

• Unstructured and duplicated code
• High cyclomatic complexity
• Controlled level of dynamic coding
• Over-parameterization of methods
• Hard coding of literals
• Excessive component size

• Duplicated business logic
• Compliance with initial architecture design
• Strict hierarchy of calling between architectural layers
• Excessive horizontal layers
• Excessive multi-tier fan-in/fan-out