Software Testing Standards : An Ultimate Guide [2019]

Whenever we think of some protocols or guidelines which are globally acceptable and ensure quality, interoperability, excellence, and correctness, the term Standards comes to our mind. In this article, we are going to discuss in details about Software testing standards along with an in-depth knowledge of: –

• What are the standards?
• What is the basis for standards?
• Importance of Standards
• Software Testing Standards
• What is ISO? IEEE? CMM? SEI? ANSI?

 

What are the Software Testing Standards?

Can we imagine a day without standards? It will be really difficult, maybe not possible! Life would be insecure, and nothing would fit perfectly. There is standard for everything like dimensions for a bed, the car you drive, the size of the stairs you are walking through, the electrical equipment you use, etc. The list of standards is endless. You cannot go through a day without being in contact with standards.

In technical terms – Standards can be defined as a set of obligatory requirements established by a general agreement and maintained by a recognized body to impose a uniform disciplined practice.

To get a clear picture let us consider an example, for a global company such as Xiaomi Corporation, who developed the Redmi smartphones. It is working across multiple countries such as India, Malaysia, Singapore, Indonesia, Philippines, and South Africa. This means that they are often working across multiple languages, cultures, and ways of operating. Although this could be a potentially dangerous recipe for mistakes and delays when it comes to the development and production of such products. In such cases, the company must follow some protocol/guidelines to ensure that different business units communicate better, reduce the potential for confusion, and quality is never compromised and to make their product globally acceptable. These guidelines/protocols is referred to as “Standards“.

There are different standards for different products. For example, Xiaomi Corporation may be following below Standards which are applicable for smartphones

• ISO/IEC 24755:2007 Standard defines screen icons and symbols with their related functions for Redmi phones.
• ISO 9241 standard evaluate the software quality of apps
• ISO 9126 checks external quality such as reliability, usability, portability in different types of smartphones Android or iOS.

In the same manner – ISO / IEC / IEEE with the number 29119 is intended for software testing as a compilation of internationally approved standards in software tests that are followed for any SDLC model in software development for any organization. When you implement the standards, you adopt internationally recognized test standards that will eventually offer your organization a quality focus for the test.

What is the basis for Standards?

Here we will consider some points that constitute the pillar for any Standard: –

• Communication: There is a very important role in common terminology in defining Standards. The Incident Command System (ICS) is a standardized approach that establishes common terminology that allows various incident management and support organizations to work together in a wide variety of area.
• Certification/compliance schemes: Product certification is the process of certifying a certain product has passed the quality control tests, the performance tests, and meets the qualification criteria. Most of the certification bodies are accredited according to ISO / IEC 17065.
• Guidelines for the benchmark of good industry practice: Guidelines define “What they are and how to use them?” Benchmarking is the continuous process of comparing business performance metrics and processes with the best-known methods of the industry. The dimensions typically measured are time, quality, and cost.
• Importance of interoperability and consistency: These terms play a key role in designing a Standard. Interoperability refers to the basic ability of a system to easily connect and communicate with each other, even if they were developed by very different manufacturers in different countries. Consistency is the condition of a product of being consistent in different types of environment.

Importance of Standards:

Now million dollar question is – what is the importance of Standards? Why standards are required? Here is the answer – Standards provide mutually agreed guidelines and instructions to people and organizations with a basis for mutual comprehension and are used as tools to facilitate measurement. Here are a few more reasons-

• Standards facilitate business interaction
• Instruct companies to comply with relevant laws and regulations
• They are responsible for the introduction of innovative products to market
• Aims to provide interoperability between new and existing products, processes and services.

 

 

What is ISO? IEEE? CMM? SEI? ANSI?

Many organizations around the globe develop and implement different standards to improve the quality needs of their software. Here we will discuss some of the organizations along with their standards w.r.t Software Development.

• ISO: The International Organization for Standardization (ISO) having headquartered in Geneva, Switzerland, and works in 164 countries. Over twenty thousand standards have been developed and publish by ISO covering everything from technology, software quality to manufactured product, agriculture, and healthcare.
• IEEE: The Institute of Electrical and Electronics Engineers (IEEE) having its corporate office in New York City and its operations center in Piscataway, New Jersey.  As of 2018, it has more than 423,000 members in over 160 countries around the world. IEEE publishes tutorials and standards related to electrical and electronics engineering and computer science fields.
• CMM: The Capability Maturity Model (CMM) is a development model created by the software community in 1986. The US. Department of Defense funded this research.
• SEI and ANSI: The Software Engineering Institute (SEI) is headquartered in Pittsburgh, Pennsylvania. This project was initiated by the U.S. Defense Department to improve software development processes. The American National Standards Institute (ANSI) publishes some software-related standards in conjunction with the IEEE and ASQ. Headquarter of this organization is in Washington, DC, and operations office is located in New York City.

ISO Software Testing Standards

We have five standards within the ISO/IEC 29119 international software testing standard:

• ISO/IEC 29119-1: This was published in Sep 2013 and deals with concepts and definitions of software.
• ISO/IEC 29119-2: This Standard deals with test processes in a product and was published in Sep 2013.
• ISO/IEC 29119-3: This was published in Sep 2013 and deals with test documentation of the product.
• ISO/IEC 29119-4: This Standard was published one year later in 2014 and deals with testing techniques and strategies used in software testing.
• ISO/IEC 29119-5: This Standard was published in the year 2015 and deals with keyword-based software testing.

ISO/IEC 9126:

This ISO standard deals with the following key aspects to determine the quality of a software application −

• Overview of Quality model
• Defining External metrics
• Defining Internal metrics
• Quality standard in use metrics

This standard ensures that the software must have the following characteristics to ensure the quality:

• Functionality: Functionality is defined as a set of attributes that perform the specified task. Some of the attributes of functionality are suitability, accuracy, interoperability, security, and functionality compliance.
• Reliability: Reliability is the capacity of software to maintain a failure-free software operation for a specified period of time in a specified environment. Some of the attributes of reliability are Maturity, fault tolerance, recoverability, and reliability compliance.
• Usability: Usability refers to the effort a user needs to put while exploring products or systems, including websites, software, devices, etc. Some of the attributes of usability are understandability, learnability, operability, attractiveness, and Usability compliance.
• Efficiency: Efficiency is defined as the ratio of output to input expressed in terms of percentage. It is the calculation between the level of performance of the software and the number of resources used, under predefined conditions. Some of the attributes of efficiency are time behavior, resource utilization, and efficiency compliance.
• Maintainability: Maintainability is the amount of effort needed to make specified modifications. It is the measures of the time and speed with which a system needs to restore after a failure occurs. Some of the attributes of maintainability are analyzability, changeability, stability, testability, and maintainability compliance.
• Portability: Portability is the ability of software to be transferred from one environment to another. Some of the attributes of portability are adaptability, installability, co-existence, replaceability, and portability compliance.

ISO/IEC 9241-11:

This standard deal with the extent that a product may be utilized by a specific user to attain some goals with effectiveness, potency, and satisfaction during a fixed context of use.

This standard suggested a framework that identifies and describes the usability components and the relationship between them. User performance and satisfaction are the key points for usability in this standard. According to ISO 9241-11, usability depends on the context of use and therefore the level of usability can amendment because the context changes.

ISO/IEC 25000:2005:

ISO/IEC 25000:2005 this standard provides the guidelines for Software Quality Requirements and Evaluation (SQuaRE). SQuaRE replaces ISO-9126 and ISO-14598 old ISO Standards.

This standard helps in organizing and enhancing the method associated with code quality necessities and their evaluations. SQuaRE is divided into sub-parts such as −

• ISO 2500n − This standard is known as Quality Management Division
• ISO 2501n − This standard is known as Quality Model Division
• ISO 2502n − This standard is known as Quality Measurement Division
• ISO 2503n − This standard is known as Quality Requirements Division
• ISO 2504n − This standard is known as Quality Evaluation Division

The main contents provided by SQuaRE are −

• Terms and definitions used for the product
• Reference Models used for development
• A general guide for the users
• Individual division guides for the users
• All Standard related to Requirement Engineering (i.e. specification, planning, measurement, and evaluation process)

ISO/IEC 12119:

This standard is applicable to software packages. It defines the requirements for software packages and instructions on how to test a software package against these requirements. This standard does not deal with the production process only with software packages as offered and delivered. The main contents are related to the following items −

• Establishment of the requirements for software packages.
• Define and describe instructions for testing a delivered software package against the specified requirements.

 

IEEE- Standard for Software Test Documentation:

IEEE series defines an internationally-agreed set of standards for software testing of software testing standards. The main purpose of the IEEE series is to provide such guidelines that can be used by any organization when performing any form of software testing.

• IEEE 829: IEEE Standard defines a for the format for software test documentation. These documents are used in different stages of software testing.
• IEEE 1061: This standard defines the approach for constructing quality requirements, analyzing, identifying, implementing validating the process, and product of software quality metrics.
• IEEE 1059: This standard provides a complete guide for Software Verification and Validation Plans.
• IEEE 1008: This standard provides a complete standard for unit testing.
• IEEE 1012: This standard provides a complete standard for Software Verification and Validation.

 

CMM- Capability Maturity Model:

Capability Maturity Model was developed by the software community in 1986. It describes the principles and practices for the underlying software development lifecycle. It is meant to assist software system organizations to improve the maturity of their software system processes in terms of an evolutionary path from ad hoc, chaotic processes to mature, disciplined software processes.

The Capability Maturity Model is organized into five maturity levels:

• Initial Level: Defines the disciplined process, standard, predictable process, consistent process, continuously Improving process, etc.
• Repeatable Level: This level specifies the key practice areas such as requirements management, Software project tracking & oversight, Software project planning, Software subcontract management, Software configuration management, Software quality assurance, etc.
• Defined: This level of CMM concentrates on the key practice areas which are organization process focus, software product engineering, organization process definition, integrated software management, training program, intergroup coordination, Peer reviews, etc.
• Manageable: The key practice areas for this level can be quantitative process management and software quality management.
• Optimizing: This level specifies guidelines for defect prevention, technology change management, and process change management.

 

SEI and ANSI:

Software Engineering Institute (SEI) initiated by the U.S. Defense Department to motivate the improvement of software development processes. The SEI addresses significant and generalized software engineering problems for:

• Creation of prototypes and open source software
• Identify and add value to new innovative technologies, emergent or underutilized technologies
• Motivating research
• Improvement and adaptation of existing solutions

The SEI Capability Maturity Model (SEI CMM) helped organizations to improve the quality of the software they develop. SEI was established to optimize the process of acquiring, developing, and maintaining heavily software-reliant systems. SEI advocates industry-wide adoption of the CMM. The SEI CMM is similar to ISO 9001, one of the ISO 9000 series of standard.

The American National Standards Institute (ANSI) is a nonprofit organization with a private membership that coordinates the development of the voluntary national standards of the United States and is the member of the United States of the International Organization for Standardization (ISO).

ANSI Standards for Software:

American National Standards Institute publishes some standards related to the software in conjunction with IEEE and ASQ (American Society for Quality).

The ANSI / IEEE 829-1983 standard describes a test plan such as:

“A document that describes the focus, scope, timeline, resources and of the planned test activities. Identify the characteristics to be tested, test elements, the test tasks, who will perform each task and any risk that requires contingency planning.”

• The standards should provide a description of the characteristics from which competitive and interoperable implementations can be developed.
• ANSI strongly discourages any use of software that is inconsistent with this set of guidelines.
• ANSI recommends that copyrighted software be included for informational purposes only or in ways that do not require particular implementations of the standard.
• The object code should never be included in a standard as a regulatory requirement. ANSI recognizes that there may be circumstances in which software is included if it is accompanied by the appropriate legal permits. It can facilitate the development of multiple, competitive and interoperable implementations of the standard.

Finally, we can conclude by saying that “Standards play a vital role in bringing new ideas to market faster. They are the priceless ingredients that underpin the dissemination of all emerging technologies. They are a form of knowledge – setting out a way forward for new technologies as they are being developed.” (Heseltine, 2012).