QM

quality measure

QME

quality measure element

ID

Name

Description

Measurement function

FCp-1-G

Functional completeness

What proportion of the specified product functions are implemented?

X = 1 - A/B

A

= Number of functions missing

B

= Number of functions specified

NOTE 1 The denominator B must be non-zero. If B = 0, this measure is not applicable.

NOTE 2 Functions can be specified in a requirement specification, a design specification, a user manual, or all of these.

NOTE 3 A missing function is detected when the system or software product cannot perform a function that is specified.

NOTE 4 The functions are implemented to meet tasks and intended user objectives. The tasks and intended user objectives can be specified in a requirement specification, a design specification, user documentation, a test document, or all of these.

NOTE 5 ISO/IEC 14143 can be used to identify the functions of a product.

FCp-2-G

Functional

requirement

completeness

What proportion of the specified functional requirements have been implemented?

X = 1 - A/B

A

= Number of functional requirements no t implemented

B

= Total number of specified functional requirements

ID

Name

Description

Measurement function

FCr-1-G

Functional correctness

What proportion of functions provides the correct results?

X = 1 - A/B

A

= Number of incorrect functions

B

= Number of functions considered

NOTE 1 An incorrect function does not provide a reasonable and acceptable outcome to achieve the specific intended objective.

NOTE 2 The functions considered for evaluation can be all the functions of a product or a specific set of

functions required for a particular usage.

NOTE 3 The developer or tester possibly examines an individual function by reviewing or testing and determines whether the function successfully provides suitable outcomes to specific objectives as defined in the requirements specification or not. In such a case, the degree of correctness is determined per an individual function.

NOTE 4 The result of functional accuracy measurement can be included in the result of functional correctness measurement.

FCr-2-G

Functional

accuracy

How accurate are the results provided by a specific function?

X = 1 - A/B

A

= Number of test cases that do not meet the specified threshold

B

= Number of test cases evaluating the specific function

NOTE 1 Higher values of X (e.g., above 0.9) are generally indicative of high accuracy, depending on context.

EXAMPLE If 10 out of 100 input images are incorrectly classified by an object recognition function, the functional accuracy is 0.9.

NOTE 2 Functional correctness is to measure whether the product's functions are provided correctly, while functional accuracy is to focus on measuring the accuracy of a specific function.

FCr-3-G

Functional

precision

How precise are the results provided by a specific function?

X = 1 - A/B

A

= Number of test cases that do not meet the required degree of precision

B

= Number of test cases evaluating the specific function

NOTE 1 The precision indicates how well the test results fall within the acceptable range of error. The acceptable range is defined based on specific testing objectives and requirements (e.g., Acceptable range: ±0.5 °C). The closer the measured values are to a central value (i.e., the smaller the standard deviation), the higher the precision.

NOTE 2 If the output values are not dense but sparse for given input values, we can say that the function has low precision. For example, the function that uses floating-point values produces imprecise results because floating-point numbers are represented as approximations. When measuring precision, such cases can be taken into account where the output is sparse due to rounding, approximation, or physical representation (e.g., floor function), despite the expectation of dense or continuous values.

NOTE 3 The value of X ranges from 0 to 1, and higher values are preferable. However, this interpretation depends on how close the mean values are to the measured values, which in turn depends on the actual standard deviation.

ID

Name

Description

Measurement function

FAp-1-G

Functional appropriateness

What proportion of functions are traceable to the specified requirements?

X = A/B

A

= The number of functions which are traceable to the specified requirements

B

= Number of functions considered

NOTE 1 The functions considered for evaluation can be all the functions of a product or a specific set of functions required for a particular usage.

NOTE 2 Even if the value of this measure is good, when the requirements specification does not adequately cover the functionality, appropriateness will be reduced by the lack or inefficiency of function to accomplish the task and objectives during operation. Therefore, when defining requirements, it is helpful to eager to adequately define stakeholder needs and requirements (e.g., use prototype and defining requirements iteratively).

PTb-1-G

Mean system wait time

What is the mean time between the arrival of a request and the point when the processing begins?

Ai

= Time between the arrival of an i-th request and the point when the processing begins

n

= Number of requests measured

PTb-2-G

Mean response time

How long is the mean time for the product to respond to a user request to perform specified tasks under specified conditions?

Ai

= Time taken by the product to respond to a user i-th request to perform a specific user task

n

= Number of responses measured

NOTE 1 In the case of a pipeline (e.g. a systems chain), the elapsed time in each stage of the pipeline has to be considered and bottlenecks in one stage can affect overall turnaround time.

NOTE 2 It is a good practice to use this measure in conjunction with specified payload and/or workload.

NOTE 3 See Figure 4 below for a visual distinction between response time and turnaround time.

PTb-3-G

Response time adequacy

What proportion of the product response times meet the specified response time requirements?

X = 1 - A/B

A

= Number of response instances that do not meet the specified response time requirements

B

= Number of response time requirements

NOTE 1 Response time is the duration it takes for the initial response to be received after a request has been initiated, whereas turnaround time represents the total duration required for a task to be initiated and completed.

NOTE 2 An alternative to this measure is nth percentile response time under expected load conditions. It is also useful to apply it to individual functions or classes of functions.

PTb-4-G

Mean

turnaround time

What is the mean time taken for completion of a job or an asynchronous process?

Ai

= Time of job submission or time of starting asynchronous process i

Bi

= Time of completing the job or asynchronous process i

n

= Number of jobs or asynchronous processes

PTb-5-G

Turnaround time adequacy

What proportion of the product turnaround time meets the specified turnaround time requirements?

X = 1 - A/B

A

= Number of turnaround instances that do not meet the specified turnaround time requirements

B

= Number of turnaround time measured

Figure 4 — Comparison Between Response Time and Turnaround Time

PTb-6-G

Mean throughput

What is the mean number of jobs completed per unit of time?

Ai

= Number of jobs completed during i-t h observation time

Bi

= The length of time in observation period i

n

= Number of observations

NOTE 1 Jobs could be fine-grained operations, like microprocessor operations, coarse-grained transaction processing units, like those defined by the Transaction Processing Performance Council (TPC), or higher-level abstractions, like functions. So, the results of this measure, when used in different contexts, are interpreted appropriately taking the context into account.

NOTE 2 When such a target threshold is specified as one of the requirements, the adequacy value is expected to be greater than or equal to 1 to satisfy the requirement. For example, if the required throughput is 100 requests per second and the system achieves a mean throughput of 120, the adequacy value would be 1.2, which satisfies the requirement.

PTb-7-G

Throughput adequacy

What proportion of the product throughput meets the specified throughput requirements?

X = 1 - A/B

A

= Number of throughput that does not meet throughput requirements

B

= Number of throughput requirements

ID

Name

Description

Measurement function

PRu-1-G

Peak

processor utilization

How much of the maximum processing utilization is used to execute the specific functions of the product?

X = Max (A1, A2, A3, …, An)

Ai

= Maximum processor utilization rate during each execution of a specific product function i

n

= Number of observations

NOTE 1 Result value varies from greater than 0 to 1. Usually, the smaller is better.

NOTE 2 In order for the results to be statistically meaningful, it is generally a good practice to repeat the measurement at least five times, therefore it is desirable n is more than 5.

NOTE 3 The interpretation of peak utilization values depends on the system’s operational context. For general-purpose systems, sustained usage above 90% possibly indicate potential resource strain. However, in performance-optimized environments (e.g., HPC), high utilization is sometimes both intended and desirable. Users are advised to compare observed utilization levels against their specific system requirements and operational goals.

PRu-2-G

Mean processor utilization

What is the mean value of the processor utilization used to execute the specific functions of the product?

Ai

= Percentage of time a core or processor is occupied compared to the total time the core or processor is available for use in i-th observation

n

= Number of 1observations

NOTE 1 It is essential not only to measure the maximum processor utilization but also to measure the average processor utilization.

NOTE 2 It is worth considering better measures, such as median, because mean values are sensitive to outliers.

PRu-3-G

Peak

memory utilization

How much of the maximum memory utilization is used to execute the specific functions of the product?

X = Max (A1, A2, A3, …, An)

Ai

= Maximum size of memory used to execute the specific functions of the product in i-th observation

n

= Number of observations

PRu-4-G

Mean memory utilization

What is the mean value of the memory used to execute the specific functions of the product?

Ai

= Size of memory used to execute the specific functions of the product in i-th observation

n

= Number of observations

NOTE 1 Linux systems do not have a specific metric to measure the actual memory usage of a process. As an alternative, available memory can be measured instead of memory usage.

NOTE 2 To understand how much the product utilizes system resources to the maximum, it is essential not only to measure the average memory utilization but also to observe the maximum memory utilization.

PRu-5-G

Mean wait time

What is the mean time between the arrival of a request and the point when the processing begins?

Ai

= Time between the arrival of a request and the point when the processing begins in i-th observation

n

= Number of requests measured

PRu-6-G

Peak

I/O devices utilization

How much is the maximum I/O device utilization rate used to execute the function of the product?

X = Max (A1/T1, A2/T2, , …, An/Tn)

Ai

= Busy time of I/O device during the execution of the specific functions of the product in i-th observation

Ti

= Total observation time in i-th observation

n

= Number of observations

NOTE Utilization rate is measured as (Disk Busy Time / Total Observation Time) × 100 %. Busy time means the period during which a system or a device is working.

PRu-7-G

Mean

I/O devices utilization

What is the mean utilization rate of I/O devices during the operational period of the system or product?

Ai

= Utilization rate of I/O devices during i-th measurement interval

n

= Number of observations or intervals measured

PRu-8-G

Peak

bandwidth

utilization

How much is the maximum bandwidth utilized to execute the specific functions of the product?

X = Max (A1, A2, A3, …, An)

Ai

= Maximum bandwidth during the execution of the specific functions of the product in i-th observation

n

= Number of observations

NOTE 1 The result of X is expected to be below specified bandwidth capacity.

NOTE 2 The measurer has to consider the possible communication traffic limitations (e.g. dropping or throttling) which can affect the resulting statistical values including average.

PRu-9-G

Mean bandwidth utilization

What is the mean value of the bandwidth utilization used to execute the specific functions of the product?

Ai

= Size of bandwidth used to execute the specific functions of the product in i-th observation

n

= Number of observations

NOTE Bandwidth utilization is commonly measured in Mbps. Alternatively, it can be expressed as a percentage of maximum capacity to reflect relative usage.

PRu-10-G

Peak

energy

consumption

utilization

How much is the maximum energy consumed to execute the specific functions of the product?

X = Max (A1, A2, A3, …, An)

Ai

= Maximum energy utilization rate consumed during the execution of the specific functions of the product in i-th observation

n

= Number of observations

NOTE In physical deployments (e.g., IoT, edge devices, on-premise servers), power meters and smart PDUs can be used to directly measure energy consumption during specific function execution. In cloud environments, direct measurement is not possible. However, major cloud providers such as AWS, Azure, and Google Cloud provide tools or APIs (e.g., Cloud Carbon Footprint, Sustainability Calculator) that enable energy consumption estimation based on resource usage. These allow for approximation of peak and mean energy consumption per function or service.

PRu-11-G

Mean energy consumption utilization

What is the mean value of the energy consumption used to execute the specific functions of the product?

Ai

= Size of energy consumption used to execute the specific functions of the product in i-th observation

n

= Number of observations

PRu-12-G

Resource utilization

adequacy

What proportion of the resource utilization meets the specified utilization requirements?

X = 1 - A/B

A

= Number of resource utilizations that do not meet resource utilization requirements

B

= Number of resource utilization requirements

PCa-1-G

Maximum capacity used

What is the maximum amount of throughput under the given resources?

X = Max (A1, A2, A3, … An)

Ai

= Maximum number of transactions completed during observation time under the given resources in i-th observation

n

= Number of observations

NOTE 1 Result value varies from 0 to maximum limit. Usually, the larger is better.

NOTE 2 This measure can be useful only if there is sufficient workload to test.

NOTE 3 Task can be alternatively used, as well as transaction.

NOTE 4 The capacity of a product refers to the maximum amount of throughput that can be achieved under specific conditions within a given period. This capacity can be influenced by various factors such as hardware specifications, software configuration, network bandwidth, and other resources, which collectively determine the specified conditions.

PCa-2-G

Maximum user access capability used

How many users can access the product simultaneously at a certain time at maximum?

X = Max (A1, A2, A3, …, An)

Ai

= Maximum number of users who can simultaneously access the product at i-th observation

n

= Number of observations

NOTE 1 Result value varies from 0 to maximum limit. Usually, the result of a larger value is better.

NOTE 2 In terms of performance efficiency, the term ‘users’ typically refers to concurrent users, which means the number of users who access the system simultaneously and request specific tasks.

PCa-3-G

User access

adequacy

What proportion of user access can the product accommodate?

X = A/B

A

= Number of user accesses that the product can accommodate

B

= Number of targeted user accesses that the product should accommodate utilizing available resources

NOTE 1 This measure is designed to measure whether the product can cope with the user access increase during the specific period.

NOTE 2 Result value varies from 0 to maximum limit. Usually, the larger is better.

NOTE 3 This measure indicates the degree to which the capability of software or system has enough capacity to accept accesses from a lot of users, even during the rapid increase of users in a given moment, e.g. an extremely large number of users could simultaneously access the system or software in an instance through the internet. If the value of X is less than 1, it means that the product does not accommodate the targeted user accesses, indicating in sufficient capacity under the given conditions.

PCa-4-G

Capacity adequacy

What proportion of the product capacity meets the specified capacity requirements?

X = 1 - A/B

A

= Number of capacities that do not meet capacity requirements

B

= Number of capacity requirements

NOTE If x is smaller than 1, it means the product fails to meet capacity requirements.

ID

Name

Description

Measurement function

CCo-1-G

Co-existence

with other

products

What proportion of specified capability of products can share the environment with this product without adverse impact on their quality characteristic or functionality?

X = A/B

A

= Number of other products with which this product can interact without negatively affecting their quality characteristics or functionality in the same environment

B

= Number of other products required to co-exist with this product in the same environment

NOTE 1 A negative impact on quality characteristics or functionality refers to measurable issues such as performance degradation (e.g., slower response time or reduced throughput), increased error rates, or failure to meet functional requirements.

ID

Name

Description

Measurement function

CIn-1-G

Data formats exchangeability

What proportion of the specified data formats is exchangeable with other products?

X = A/B

A

= Number of data formats exchangeable with other products

B

= Number of data formats specified to be exchangeable

CIn-2-G

Data exchange protocol sufficiency

What proportion of the specified data exchange protocols is supported?

X = A/B

A

= Number of data exchange protocols supported

B

= Number of data exchange protocols specified to be supported

NOTE For the details of data quality, refer to Con-I-1 in ISO/IEC 25024.

CIn-3-G

External interface completeness

What proportion of the specified external interfaces (interfaces with other products) is implemented?

X = A/B

A

= Number of external interfaces that are implemented

B

= Number of external interfaces specified

ID

Name

Description

Measurement function

IAr-1-G

Description completeness

What proportion of usage scenarios is described in the product description or user documentation?

X = A/B

A

= Number of usage scenarios described in the product description or user documentation explicitly

B

= Number of usage scenarios of the product

IAr-2-G

Demonstration coverage

What proportion of tasks have demonstration features for users to recognize the appropriateness?

X = A/B

A

= Number of tasks with demonstration features

B

= Number of tasks that could benefit from demonstration features

ID

Name

Description

Measurement function

ILe-1-G

User guidance

completeness

What proportion of functions is explained in sufficient detail in user documentation and/or help facility to enable the user to apply the specified functions?

X = A/B

A

= Number of fufnctions described in user documentation and/or help facility as required

B

= Number of functions implemented that are required to be documented

NOTE 1 Learnability is strongly related to appropriateness recognizability and appropriateness recognizability measurements can be indicators of the learnability potential of the software.

NOTE 2 Help facility includes, for example, online help, operational guide video, operational instruction system, etc.

ILe-2-G

Entry fields default

What proportion of entry fields that could have default values are automatically filled with default values?

X = A/B

A

= Number of entry fields whose default values have been automatically filled in during operation

B

= Number of entry fields that could have default values

NOTE The default values for entry fields are helpful for beginners to learn how to operate the product comprehensively and quickly.

ILe-3-G

Self-explanatory user interface

What proportion of user interfaces presented to the user enables specific tasks to be completed by a first-time user without prior study or training or seeking external assistance?

Ai

= Number of user interfaces for task i that can be understood without study, training, or external assistance

Bi

= Number of user interfaces used for task i

n

= Total number of tasks

NOTE 1 This measure is particularly relevant for public systems and websites.

NOTE 2 User interfaces can include GUI(Graphical User Interface), AUI(Auditory UI) and T and H UI(Tactile and Haptic UI), command lines, and so on.

ID

Name

Description

Measurement function

IOp-1-G

Operational consistency

To what extent do interactive tasks have a behaviour and appearance that is consistent both within the task and across similar tasks?

X = 1 - A/B

A

= Number of interactive tasks with interfaces that exhibit or create inconsistencies in the product's operation to complete the task

B

= Total number of interactive tasks in the product

IOp-2-G

Message clarity

What proportion of messages from a product conveys the right outcome or instructions to the user?

X = A/B

A

= Number of messages that convey the right outcome or instructions to the user

B

= Number of messages implemented

IOp-3-S

Functional customizability

What proportion of functions and operational procedures can users customize for their convenience?

X = A/B

A

= Number of functions and operational procedures that can be customized for the user's convenience

B

= Number of functions and operational procedures for which users could benefit from customization

IOp-4-S

User interface customizability

What proportion of user interface elements can be customized by users?

X = A/B

A

= Number of user interface elements that can be customized by users

B

= Number of user interface elements

IOp-5-S

Monitoring

capability

What proportion of functions can be monitored during operation?

X = A/B

A

= Number of functions having state monitoring capability

B

= Number of functions that could benefit from monitoring capability

NOTE 1 Monitoring and management of the operational state of some functions are very important in case of, for example, distributed systems, embedded systems, and so on.

NOTE 2 For a better measurement, it is helpful to find which function has a benefit from monitoring capability from a view of usability during operational scenario review or operational testing by the user. Such a function is also able to be specified as requirements.

IOp-6-S

Undo capability

What proportion of tasks that have a significant consequence to users provides an option for re-confirmation or undo capability?

X = A/B

A

= Number of tasks that provide undo capability or prompt for re-confirmation

B

= Number of tasks for which users could benefit from having re-confirmation or undo capability

IOp-7-S

Understandable categorization

of information

To what extent does the product organize information in categories that are understandable for their tasks?

X = A/B

A

= Number of information structures that are familiar and convenient for the intended users

B

= Number of information structures used

EXAMPLE The online shop of a department store organizes the goods in a similar way to the physical layout of the goods in the store.

IOp-8-S

Appearance consistency

What proportion of user interfaces with similar items has a similar appearance?

X = 1 - A/B

A

= Number of user interfaces with similar items but with different appearances

B

= Number of user interfaces with similar items

EXAMPLE The “OK” and “Cancel” screens are always located at the same position.

IOp-9-S

Input device

support

To what extent can tasks be initiated by all appropriate input modalities (such as keyboard, mouse, or voice)?

X = A/B

A

= Number of tasks that can be initiated by all appropriate input modalities

B

= Number of tasks supported by the system

EXAMPLE Within a search form, the search button can be activated by using the mouse or by pressing the “Enter” key on the keyboard.

ID

Name

Description

Measurement function

IEp-1-G

Avoidance of use error

What proportion of user-inappropriate actions and inputs are validated to avoid operation errors that can influence product malfunctions?

X = A/B

A

= Number of actions and inputs that are validated to avoid operation error that can influence the product malfunctions

B

= Number of user actions and inputs that are specified to be validated to avoid operation errors that can influence product malfunctions

NOTE 1 This includes the system requesting confirmation before carrying out an action that cannot be undone and that would have significant consequences.

EXAMPLE When erasing files within an application, the user is required to confirm each deletion.

NOTE 2 For a better measurement, it is helpful to find user actions and inputs where a user often makes an error during operational testing. Such protections against erroneous user actions and inputs are also able to be specified as requirements.

NOTE 3 It is helpful to involve representative users from early phase of development and to observe their operation behaviour for specifying which user actions and inputs are to be validated for operational error prevention.

IEp-2-G

Error message resolvability

What proportion of the error messages state the reason why the error occurred and how to resolve it?

X = A/B

A

= Number of error messages which state the reason for occurrence and suggest the ways of resolution where this is possible

B

= Number of error messages implemented

IEp-3-G

User entry

error

correction

To what extent does the product correct detected user entry errors or suggest corrections for them?

X = A/B

A

= Number of user entry errors that the product corrects or for which it suggests corrections

B

= Number of entry errors detected

NOTE For the details of related data quality, refer to Cre-I-1 in ISO/IEC 25024.

IEp-4-G

User error recoverability

What proportion of user operation errors can be corrected by the product to resume nominal operation without human intervention?

X = A/B

A

= Number of user operation errors corrected by the product to resume nominal operation without human intervention

B

= Number of user operation errors that were made during the operation

ID

Name

Description

Measurement function

IUe-1-G

Engaging user interfaces

To what extent are the user interfaces, including functions or information, provided in an inviting and motivating manner?

X = A/B

A

= Number of interfaces that motivate user engagement

B

= Number of interfaces required to motivate user engagement

NOTE 1 An inherent or behaviour user engagement quality measure is used to assess the appearance of the user interfaces and will be influenced by factors such as screen design and color. This is particularly important for consumer products.

NOTE 2 Good colour combinations can help users quickly read the text or identify the image. Then, it can be helpful for better aesthetics measurement to address bad colour combinations, such as light blue on grey, red on orange, green on blue, and so on.

NOTE 3 This quality measure often depends on an individual of users. Then, either expert usability designers or testers on behalf of users, or representatives from target user groups are expected to be involved to measure this.

ID

Name

Description

Measurement function

IIn-1-G

Language inclusivity for the widest range of users

To what extent is the product accessible to users from specified language backgrounds?

X = A/B

A

= Number of functions successfully usable by users from specified language backgrounds

B

= Number of functions implemented

NOTE 1 For this quality measure, there is a need to define target users of the product that use language backgrounds.

NOTE 2 Inclusivity focuses on how products can be inclusive of all users, while user assistance focuses on how products can be helpful to disabled users. For example, to support inclusivity, a product is provided in multiple languages or content that is appropriate for users from different cultural backgrounds. To support user assistance, assistive technology can allow blind users to hear the content of a product or service, or deaf users to see the content of a product.

IIn-2-G

Culture inclusivity for the widest range of users

To what extent is the product accessible to users from specified cultural groups?

X = A/B

A

= Number of functions successfully usable by users from specified cultural groups

B

= Number of functions implemented

NOTE For this quality measure, there is a need to define target users of the product to culture groups.

EXAMPLE The meaning of colors can differ across cultures (e.g., white symbolizes purity in some cultures but mourning in others), and suitable character sizes can vary depending on the language (e.g., alphanumeric vs. Kanji characters).

ID

Name

Description

Measurement function

IUa-1-G

Assistance for users with disabilities and diverse users

To what extent can potential users with specific disabilities successfully use the product with assistive technology?

X = A/B

A

= Number of functions that the users with specific disabilities successfully use the product with assistive technology

B

= Number of functions implemented

NOTE 1 To define criteria for whether the product is 'successfully completing’ accessibility-related standards, for example, ISO/IEC 40500 ISO 9241-171, and ISO/IEC/IEEE 2651n series can be used.

NOTE 2 Specific disabilities include cognitive disability, physical disability, hearing/voice disability, and visual disability.

NOTE 3 The range of capabilities includes disabilities associated with age.

NOTE 4 Any person becomes possibly a user with limited cognitive, physical, hearing, or visual ability under specific situations or environments, for example, in darkness, in low atmospheric pressure at high altitude, in water, and so on.

IUa-2-G

Assistance and support for users who require any support for use of system or products

To what extent can many kinds of users in a variety of environment successfully use the product with assistive technology or some support

X = A/B

A

= Number of functions that the users successfully use the product with assistive technology

B

= Number of functions implemented

ID

Name

Description

Measurement function

ISd-1-G

Presentation of understandable information for user tasks

What proportion of user tasks are provided with understandable information for performing tasks?

X = A/B

A

= Number of user tasks with understandable information for performing the tasks

B

= Number of user tasks

NOTE The level of understandable information can be influenced by the user's background knowledge or proficiency in performing the task, and these factors can be taken into account when applying this measure more effectively.

ID

Name

Description

Measurement function

RFa-1-G

Fault resolution rate

What proportion of detected reliability-related faults have been resolved during development and operation?

X = A/B

A

= Number of faults resolved during development and operation

B

= Number of faults detected during development and operation

NOTE Reliability-related faults refer to faults where a system or product experiences unexpected interruptions or impairments, such as program abnormal termination/interruption, abnormal data loss, system recovery errors, and data integrity errors.

RFa-2-G

Mean time between failures (MTBF)

What is the MTBF during the product operation?

X = A/B

A

= Operation time

B

= Number of product failures that occurred during the operation time

NOTE 1 Result value varies from 0 to infinite. Usually, the larger is better.

NOTE 2 MTBF itself can be used to compare the reliabilities of different systems or software products.

RFa-3-G

MTBF improvement

Has the MTBF improved across operational periods?

X = A/B

A

= MTBF measured during current operation time

B

= MTBF measured during previous operation time

NOTE If X is greater than 1, it indicates that the MTBF is improving. If X equals 1, the MTBF shows no change. If X is less than 1, it indicates that the MTBF is worsening.

RFa-4-G

Failure rate

What proportion of the observation period corresponds to the time in which failures are detected?

X = A/B

A

= Amount of time in which failures are detected

B

= Duration of observation period

NOTE 1 The period used in this measure could be different for testing and operations purposes, which refers to actual usage or testing time.

NOTE 2 A reliability estimation model can use this measure as an input.

NOTE 3 The usefulness of this quality measure depends on the adequacy of test cases or the extent of system usage during testing, e.g. normal, exceptional, and abnormal cases.

RFa-5-G

Failure rate improvement

Has the failure rate improved across operational periods?

X = A/B

A

= Failure rate measured during current operation time

B

= Failure rate measured during previous operation time

NOTE If X is greater than 1, it indicates that the failure rate is worsening. If X equals 1, the failure rate shows no change. If X is less than 1, it indicates that the failure rate is improving.

ID

Name

Description

Measurement function

RAv-1-G

Product availability

For what proportion of the scheduled product operational time is the product available?

X = A/B

A

= Product operation time provided

B

= Product operation time specified in the operation schedule

NOTE 1 This measure can be extended to special days, such as holidays and weekends, in addition to regular operational days.

NOTE 2 Generally, product availability can be measured during operation. When measuring product availability during the testing phase, B can be defined as the observation time during testing.

RAv-2-G

Mean down time

How long does the product stay unavailable when a failure occurs?

X = A/B

A

= Total down time

B

= Number of breakdowns observed

NOTE 1 Result value varies from 0 to infinite. Usually, the smaller is better.

NOTE 2 Externally, availability can be assessed by the proportion of total time during which the system, product, or component is in an up state. Availability is therefore a combination of maturity (which governs the frequency of failure), fault tolerance, and recoverability (which governs the length of downtime following each failure).

ID

Name

Description

Measurement function

RFt-1-G

Fault avoidance

What proportion of fault patterns has been brought under control to avoid critical and serious failures?

X = A/B

A

= Number of fault patterns under control

B

= Number of fault patterns causing failure during testing

RFt-2-G

Fault identification

What proportion of fault patterns has been identified by performing testing?

X = A/B

A

= Number of test cases that have detected faults during testing or analysis

B

= Number of test cases

RFt-3-G

Redundancy of components

What proportion of product components is duplicated redundantly to avoid product failure?

X = A/B

A

= Number of components duplicated

B

= Number of components to be required for duplication

NOTE 1 For example, in many safety-critical systems, some parts of the control system could be duplicated to increase the reliability of the system.

NOTE 2 To enhance system reliability, redundancy can be implemented for servers, DBMS, storage, processes, and other components. As a result, the unit of components can vary depending on the context.

RFt-4-G

Mean fault notification time

How quickly does the product report the occurrence of faults?

Ai

= Time at which the fault i is reported by the product

Bi

= Time at which fault i is detected

n

= Number of faults detected

NOTE Result value varies from 0 to infinite. Usually, the closer to 0 the better.

ID

Name

Description

Measurement function

RRe-1-G

Mean recovery time

How long does it take for the product to recover from failure?

Ai

= Total time to recover the downed product and re-initiate operation for i-th failure

n

= Number of failures

NOTE 1 Result value varies from 0 to infinite. Usually, the smaller is better.

NOTE 2 When this quality measure is compared to a target threshold for mean recovery time, that is specified in agreed requirements by the acquirer and supplier, the measure can be used to examine conformance.

RRe-2-G

Mean recovery time by component recovery level

How long does it take for the component to recover from failure at a specific recovery level?

Ai

= Total time required to recover the failed component at the specified recovery level and re-initiate its operation for failure i

n

= Number of failures

NOTE 'Mean recovery time' is to measure the recovery time for the product as a whole, including its components, while 'Mean recovery time by component recovery level' is to measure recovery time specifically based on the recovery level of each component within the product.

RRe-3-G

Backup data completeness

What proportion of data items is backed up regularly?

X = A/B

A

= Number of data items backed up regularly

B

= Number of data items requiring backup for error recovery

SCo-1-G

Access controllability

What proportion of confidential data items are protected from unauthorized access?

X = 1 - A/B

A

= Number of confidential data items that can be accessed without authorization

B

= Number of data items that require access control

SCo-2-G

Access control mechanism sufficiency

To what extent are the access control mechanisms implemented to protect the product from unauthorized access?

X = A/B

A

= Number of access control mechanisms actually implemented

B

= Number of access control mechanisms required to be implemented for the product

NOTE Access control mechanisms to protect the product include user role-based access restrictions, URL direct access restrictions, automatic session termination for web pages, screen lock after a period of inactivity, and access restrictions based on specified IP or MAC addresses.

SCo-3-G

Data encryption correctness

How correctly is the encryption/decryption of data items implemented as stated in the requirement specification?

X = A/B

A

= Number of data items encrypted/decryption correctly

B

= Number of data items that require encryption/decryption

NOTE For the details of related data quality, refer to Cnf-I-1 in ISO/IEC 25024.

SCo-4-G

Strength of cryptographic algorithms

What proportion of cryptographic algorithms has been well-vetted?

X = 1 - A/B

A

= Number of cryptographic algorithms broken or unacceptably risky in use

B

= Number of cryptographic algorithms used

NOTE 1 It is important to select a well-vetted algorithm that is currently considered to be strong by experts in the field and to select well-tested implementations. As with some cryptographic mechanisms, the source code has to be available for analysis. For example, US government systems require FIPS 140-2 certification.

NOTE 2 There are other ways of measuring the strength of cryptographic algorithms, for example, using ethical hacking.

NOTE 3 The term "well-vetted" refers to components that have been thoroughly examined or tested to ensure they are not broken or unacceptably risky for use.

SCo-5-G

One-way encryption algorithm

To what extent has a one-way encryption algorithm been applied to data that could pose security risks if decrypted when leaked?

X = A/B

A

= Number of data items encrypted using a one-way encryption algorithm

B

= Number of data items required to be encrypted using a one-way encryption algorithm

NOTE 1 Encrypted data, such as user passwords and biometric information used for authentication and identification, can pose security risks if decrypted when leaked. Therefore, it must be encrypted using a one-way encryption algorithm to prevent decryption.

NOTE 2 One-way encryption algorithms generate the same output for identical input values during encryption and must be used with a seed value.

SCo-6-G

Data transmission protection

To what extent are communication paths within the product components or with other systems implemented through secure communication channels?

X = A/B

A

= Number of communication paths transmitting data through a secure cryptographic communication channel

B

= Total number of communication paths transmitting data

NOTE 1 Communication paths transmitting data can include paths between servers and clients, agents and servers, or clients and databases.

NOTE 2 Secure cryptographic communication channels are implemented using protocols such as TLS or SSH.

SCo-7-S

Minimization of personal data collection

To what extent does the product collect unnecessary personal data among the collected personal data items?

X = 1- A/B

A

= Number of unnecessary personal data items among the collected personal data

B

= Total number of collected personal data items

NOTE 1 Personal data item can include name, age, blood type, address, etc.

NOTE 2 If the product requests blood type information during personal data registration but does not utilize or provide it, this is considered unnecessary personal data collection.

ID

Name

Description

Measurement function

SIn-1-G

Data integrity

To what extent is data corruption modification or deletion by unauthorized access prevented?

X = 1 - A/B

A

= Number of data items that are corrupted by unauthorized access, modification or deletion

B

= Number of components of the product where an internal data corruption prevention method is necessary or required

SIn-2-G

Internal data corruption prevention

To what extent are the available prevention methods for internal data corruption implemented?

X = A/B

A

= Number of internal data corruption prevention methods implemented

B

= Number of components of the product where an internal data corruption prevention method is necessary or required

NOTE 1 Internal data refers to data that is generated, collected, stored, and managed within a system. Examples of internal data include employee information, customer data, operational data, and so on.

NOTE 2 Examples of methods for data corruption prevention are backing up data frequently, comparing data to reference data periodically, storing data in multiple mirror sites, use of RAID systems, data validation, and integrity checks, Antivirus and Anti-malware Software, use of network security programs, regular updates to software, access control management, transaction logging and monitoring and so on.

SIn-3-G

Important executable file integrity

To what extent are important executable files protected from modification or deletion by unauthorized access?

X = 1- A/B

A

= Number of important executable files that were modified or deleted due to unauthorized access

B

= Total number of important executable files that must be protected from modification or deletion

NOTE Important executable files can be modified or corrupted to bypass genuine license checks or cause data leaks of specific information.

SIn-4-G

Response to integrity corruption

To what extent has the product implemented responses to handle corruption of data or important executable files caused by unauthorized access?

X = A/B

A

= Number of response actions actually implemented when detecting integrity corruption

B

= Number of response actions specified to be performed when detecting integrity corruption

NOTE When integrity corruption is detected, the product can perform actions such as disabling execution, displaying an integrity corruption detection message, or recovering corrupted data.

ID

Name

Description

Measurement function

SNo-1-G

Non-repudiation assurance

What proportion of actions or events is proved that they have taken place so that they cannot be repudiated later?

X = A/B

A

= Number of actions or events that have been successfully proven to have occurred using the specified non-repudiation methods

B

= Number of actions or events requiring non-repudiation using the specified methods

NOTE 1 Certificates and security algorithms are also helpful in improving non-repudiation.

NOTE 2 Examples of the specified methods are digital signature, digital certificates, logging, timestamping, blockchain, and so on.

SNo-2-G

Non-repudiation implementation completeness

To what extent are non-repudiation methods (e.g., digital signatures, digital certificates) completely implemented in a secure and reliable manner?

X = A/B

A

= Number of non-repudiation methods completely implemented using secure and reliable methods

B

= Total number of non-repudiation methods required to be implemented

NOTE Complete and secure implementation of non-repudiation methods involves the use of strong cryptographic algorithms (e.g., RSA, ECDSA, SHA-256 or higher), adherence to validity periods, and the issuance of certificates from internationally trusted root and subordinate certification authorities.

SNo-3-G

Utilization of trusted timestamps

To what extent are trusted timestamps utilized in functions that rely on time information within the product?

X = A/B

A

= Number of functions using trusted timestamps

B

= Number of functions requiring trusted timestamps

NOTE 1 Examples of functions using trusted time information include audit logs and digital signatures.

NOTE 2 The source of a trusted timestamp can be a reliable NTP server or an NTP server located in a physically secure environment. Time information that can be freely modified, such as the client PC clock, cannot be considered a trusted timestamp.

ID

Name

Description

Measurement function

SAc-1-G

User audit trail completeness

How complete is the audit trail concerning the user access or activities to the product or data?

X = A/B

A

= Number of user accesses or activities that are recorded in logs

B

= Number of user accesses or activities that are required to be recorded in logs

NOTE For audit tracing, logs can include activities and events, timestamps, errors and warnings, and so on.

SAc-2-G

Audit log retention

For what percent of the required retention period is the product log retained in stable storage?

X = A/B

A

= Number of product log types (e.g. configuration management logs, login/logout logs, user data access logs, etc.) retained in stable storage for the specified retention period

B

= Number of product log types required to be retained in stable storage for the specified retention period

NOTE 1 A stable storage is a classification of computer data storage technology that guarantees atomicity for any given write operation and allows software to be written that is robust against some hardware and power failures. Most often, stable storage functionality is achieved by mirroring data on separate disks via RAID technology.

NOTE 2 Result value varies from 0 to infinite. Usually, larger than 1 is better.

SAc-3-G

Mechanism for audit log

To what extent are mechanisms for stable log retention implemented in the product?

X = A/B

A

= Number of implemented mechanisms for secure and stable audit log retention

B

= Number of mechanisms required for secure and stable audit log retention

NOTE Mechanisms for stable audit log retention can include notifying administrators via email, SMS, or alerts when storage space exceeds a critical threshold, deleting old data (only exceeding the specified retention period) when storage space is insufficient, or halting product operation when the storage space is completely full.

ID

Name

Description

Measurement function

SAu-1-G

Authentication mechanism sufficiency

How well does the product authenticate the identity of a subject?

X = A/B

A

= Number of authentication mechanisms implemented (e.g., User ID/password, IC card, or biometric authentication)

B

= Number of authentication mechanisms specified

NOTE 1 What is relevant for security is the strength of the authentication model and the ability to have multi-level multi-factor authentication and threat detection. Several factors and the degree of authenticity of the provided protocol can also be used as authenticity measures.

NOTE 2 Examples of biometric authentication include fingerprint scanning, facial recognition, iris scanning, and vein recognition.

SAu-2-G

Authentication rules conformity

What proportion of the required authentication rules is implemented in the product?

X = A/B

A

= Number of authentication rules implemented

B

= Number of authentication rules specified

SAu-3-G

Authentication protection mechanism

To what extent are authentication protection mechanisms implemented to prevent security threats during the authentication process?

X = A/B

A

= Number of implemented authentication protection mechanisms

B

= Number of mechanisms required for secure authentication protection

NOTE Authentication protection mechanisms for secure authentication can include locking an account after consecutive authentication failures (requiring manual unlocking by an administrator) or disabling it for a specified time, enforcing password expiration and mandatory changes upon reaching the expiration period, displaying failure messages that do not reveal the username or password, and blocking duplicate logins if necessary.

ID

Name

Description

Measurement function

SRe-1-G

Resistance to hacker attacks

To what extent did the product resist penetration attempts during security testing?

X = A/B

A

= Number of penetration attempts that were unsuccessful (i.e., did not penetrate the product)

B

= Total number of penetration attempts during security testing

SRe-2-G

Use of secure middleware and operating systems

To what extent does the product rely on middleware or operating systems with known security vulnerabilities for its operation?

X = 1 - A/B

A

= Number of middleware or operating systems with known security vulnerabilities

B

= Total number of middleware or operating systems used for product operation

NOTE Information about insecure middleware (e.g., web servers, WAS, open source, databases) or operating systems can be found on platforms such as the CVE website.

SRe-3-G

Middleware information disclosure

To what extent does the middleware required for product operation expose important information?

X = 1 - A/B

A

= Number of middleware exposing important information

B

= Total number of middleware used for product operation

NOTE If important information such as passwords or encryption keys is stored in plaintext, or if database passwords use well-known default values, these situations can pose security threats to the entire IT system.

ID

Name

Description

Measurement function

MMo-1-G

Coupling of components

How strongly are the components free from impacts from changes to other components in a product?

X = A/B

A

= Number of components which are implemented with no impact on others

B

= Number of specified components which are required to be independent

NOTE Such a threshold is helpful to determine whether the degree of impact from changes of other components is minimal or not, for example, the frequency of changes of the component caused by changes of other components or the number of externally shared databases that the component directly accesses.

MMo-2-G

Acceptable cyclomatic complexity

How many product modules have acceptable cyclomatic complexity?

X = 1 - A/B

A

= Number of software modules that have a cyclomatic complexity score that exceeds the specified threshold

B

= Number of software modules implemented

NOTE 1 Such a threshold is used to determine whether a value of cyclomatic complexity is acceptable or not for each module. This is defined by each project or organization and is possibly a different value for a programming language, a type of module, or a function.

NOTE 2 This scope of quality measure is software rather than the product which is different from other measures.

ID

Name

Description

Measurement function

MRe-1-G

Reusability of assets

How many assets in a product can be reusable?

X = A/B

A

= Number of assets which are designed and implemented to be reusable

B

= Number of assets required to be reusable in a product

NOTE In this measure, assets could be work products such as requirements documents, source code modules, testing modules, specific hardware, etc.

MRe-2-G

Coding rules conformity

How many modules conform to the required coding rules?

X = A/B

A

= Number of product modules conforming to coding rules for a specific system

B

= Number of product modules implemented

NOTE 1 Coding rules for a specific system might include rules that contribute to, for example, modularity, traceability, and conciseness.

NOTE 2 This quality measure can also be applied to different characteristics and subcharacteristics such as analysability.

ID

Name

Description

Measurement function

MAn-1-G

Product log completeness

To what extent does the product record its operations in logs so that they are to be traceable?

X = A/B

A

= Number of operations that are recorded in logs

B

= Number of operations for which audit trails are required

MAn-2-G

Diagnosis function effectiveness

What proportion of the diagnosis functions meets the requirements of causal analysis?

X = A/B

A

= Number of diagnostic functions useful for causal analysis

B

= Number of diagnostic functions implemented

MAn-3-G

Diagnosis function sufficiency

What proportion of the required diagnosis functions have been implemented?

X = A/B

A

= Number of diagnostic functions implemented

B

= Number of diagnostic functions required

NOTE Analysability measures are used to assess such attributes as the maintainer’s or user’s effort or resources used when trying to diagnose deficiencies or causes of failures or for identifying parts to be modified.

ID

Name

Description

Measurement function

MMd-1-G

Modification efficiency

How efficiently are the modifications made compared to the expected time?

Aij

= Actual time taken to perform the j-th modification of type i

Bij

= Expected time required to perform the j-th modification of type im = Number of modification types

n

= Number of modifications for each modification type

NOTE 1 A value of X less than 1 indicates that modifications were completed within the allowed duration. A lower value generally implies more efficient modification, though the level of efficiency can vary depending on the context.

NOTE 2 Expected time for making a specific type of modification can be based on historical data or industry averages.

NOTE 3 Modifications for interaction capability are to improve the user interface and the overall user experience. On the other hand, modification for maintainability is targeted to the system's codebase or architecture making it easier to update, modify, or extend in the future.

MMd-2-G

Modification capability

To what extent are the required modifications made within a specified duration?

X = A/B

A

= Number of modifications successfully completed within the specified duration without causing any incidents or failures

B

= Total number of modifications made

ID

Name

Description

Measurement function

MTe-1-G

Test function completeness

How completely are test functions implemented?

X = A/B

A

= Number of test functions implemented as specified

B

= Number of test functions specified

MTe-2-G

Code coverage

How completely is the product code tested against a specific test coverage criterion?

X = A/B

A

= Number of unique code structural elements covered by test cases

B

= Number of unique code structural elements in the product’s code

NOTE Code structural elements can include statements, branches, loops, and other relevant constructs depending on the selected test coverage criterion. For the details of test coverage, refer to ISO/IEC 29119-4.

MTe-3-G

Testable dependency

How independently can a product be tested?

X = A/B

A

= Number of tests that can be simulated by stub among the tests that depend on other products

B

= Number of tests which depend on other products

NOTE A stub is a skeletal or special-purpose implementation of a software module used to develop or test a module that calls or is otherwise dependent on it.

MTe-4-G

Test restartability

How easy is it to resume the test from the point where it was paused?

X = A/B

A

= Number of cases in which testers can pause and restart executing test runs at desired points to check step-by-step

B

= Number of cases in which executing test run can be paused

MTe-5-G

Test coverage

To what extent of the test cases cover the product functions, capabilities, and states?

X = A/B

A

= Number of test cases that actually cover the requirements

B

= Number of test cases required to cover requirements that define the product's functions, capabilities, and states

MTe-6-G

Test case execution coverage

What proportion of the designed test cases are executed?

X = A/B

A

= Number of test cases that are executed

B

= Total number of test cases designed

MTe-7-G

Test case pass rate

What proportion of test cases have passed successfully?

X = A/B

A

= Number of test cases passed

B

= Total number of test cases executed

ID

Name

Description

Measurement function

FAd-1-G

Hardware environmental adaptability

What proportion of hardware environments can the product adapt itself to?

X = 1 - A/B

A

= Number of functions that were not completed or results failed tests in different hardware environments

B

= Number of functions which were tested in different hardware environments

FAd-2-G

System software environmental adaptability

What is the proportion of system software environments to which the product can adapt itself?

X = 1 - A/B

A

= Number of functions that were not completed or results failed tests in different system software environments

B

= Number of functions that were tested in different system software environments

NOTE 1 When a user has to apply an adaptation procedure other than previously provided by software for a specific adaptation need, the user’s effort required for adapting has to be measured.

NOTE 2 System software is a type of software that serves as an intermediary between computer hardware and application programs. It plays a crucial role in supporting and managing the core functions of a computer system.

NOTE 3 System software can include operating systems, middleware, database management systems, compilers, network management systems, etc.

FAd-3-G

Operational environment adaptability

What proportion of operational environments can the product successfully adapt to?

X = 1 - A/B

A

= Number of functions that were not completed or results failed tests in different operational user’s environments

B

= Number of functions which were tested in different operational user’s environment

NOTE A testing environment refers to the environment in which the product is installed to perform testing, typically referred to as a testbed. On the other hand, an operational environment is the environment where the product is deployed and used, which can differ from the testing environment.

ID

Name

Description

Measurement function

FSc-1-G

Scalability

scale-out

To what extent does the product support scale-out compared to the configuration setting?

X = A/B

A

= Number of resources that are available automatically and immediately

B

= Number of configured resources

NOTE A product is already designed to handle larger workloads and is capable of accommodating additional resources to support scalability.

FSc-2-G

Scalability

scale-up

To what extent does the product support scale-up compared to the configuration setting?

X = A/B

A

= Amount of resources that are available automatically and immediately

B

= Amount of configured resources

NOTE “Scale-out” refers to handling increased workloads by adding multiple identical servers or nodes, each performing the same tasks. In contrast, “scale-up” refers to enhancing the performance of a single server or node by upgrading its hardware components such as CPU, memory, or storage.

ID

Name

Description

Measurement function

FIn-1-G

Installation time efficiency

How efficient is the actual installation time compared to the expected time?

Ai

= Total work time spent for making an installation i

Bi

= Expected time for making an installation i

n

= Number of installations measured

NOTE 1 X greater than 1 represents inefficient installation, and X less than 1 represents efficient installation.

NOTE 2 Expected time for making an installation can be based on historical data or industry averages.

FIn-2-G

Installation customizability

Can users customize the installation procedure for their convenience?

X = A/B

A

= Number of cases in which a user succeeds in customizing the installation procedure

B

= Number of cases in which a user attempted to customize the installation procedure for the user’s convenience

NOTE Such changes in installation procedure can be recognized as customization of installation by the user.

ID

Name

Description

Measurement function

FRe-1-G

Usage similarity

What proportion of user functions of the replaced product can be performed without any additional learning or workaround?

X = A/B

A

= Number of user functions that can be performed without any additional learning or workaround

B

= Number of user functions in the replaced software product

NOTE User functions are those that users can call and use to perform their intended tasks including user interfaces.

FRe-2-G

Product quality equivalence

What proportion of the quality measures is satisfied after replacing the previously specified product with this one?

X = A/B

A

= Number of quality measures of the new product that are better or equal to the replaced product

B

= Number of quality measures of the replaced specified product that are relevant

NOTE Some of the critical product qualities relevant to replaceability are interoperability, security, and performance efficiency.

FRe-3-G

Functional inclusiveness

Can similar functions easily be used after replacing the previously specified product with this one?

X = A/B

A

= Number of similar functions that can be easily used in the replaced product

B

= Number of similar functions which have to be used in the replaced product

FRe-4-G

Ease of data use

Can the same data be used after replacing the previously specified product with this one?

X = A/B

A

= Number of data items that can be used continuously as before

B

= Number of data items which have to be used continuously in the replaced product

ID

Name

Description

Measurement function

SOp-1-G

Domain hazard

coverage

What proportion of internal safety hazards can the product operate within its specified safety parameters during test?

X = A/B

A

= Number of internal safety hazards in which the product operated within its specified safety parameters during test

B

= Number of internal safety hazards tested

NOTE 1 To measure this quality measure, it is good practice to identify potential hazards in advance based on IEC 61508. The hazard analysis and risk assessment (HARA) is in the context of the operational design domain(ODD).

NOTE 2 Internal safety hazards are unsafe conditions created by the way the product operates or when the product tries to operate outside its safe operational parameters. It is good practice to implement controls to ensure safe operation against each identified internal safety hazard.

SOp-2-G

Coverage of successful behaviours to treat domain hazards

Under what proportion of operational safety hazards can the product operate within its specified safety parameters during test?

X = A/B

A

= Number of operational safety hazards in which the product operated within its specified safety parameters during test

B

= Number of operational safety hazards tested under

SOp-3-G

Operational domain coverage

Is the product capable of addressing potential hazards in the operational domain?

X = A/B

A

= Number of potential hazards that have been addressed in the operational domain

B

= Number of potential hazards in the operational domain

NOTE The ‘address’ means the hazard can be identified during the hazard analysis and mitigated in the safety requirement and design. The operational domain is the environment within which a functional unit is expected to operate and perform its required function. For instance, if a self-driving car system is suitable only for urban driving, then the urban environment becomes the operational domain for that system.

SOp-4-G

Coverage of successful behaviours to treat hazards in the operational domain

Is the product capable of treating potential hazards in the operational domain enough successfully during dynamic testing and operation?

X = 1 - A/B

A

= Number of operation scenarios in which the product failed to behave to treat potential hazards during dynamic testing or operation, although those hazards have been addressed in the operational domain

B

= Number of operation scenarios corresponding to identified potential hazards in the operational domain

NOTE Safety hazards refer to potential risks or dangers to safety, operational scenarios describe specific conditions or events during system operation, and the operational domain defines the broader environment or context in which the system functions.

ID

Name

Description

Measurement function

SRi-1-G

Risk identification

coverage

Has the risk associated with specific conditions (hazards, events, etc.) that could occur during product operation been identified?

X = A/B

A

= Number of risk scenarios that are identified as test cases for the product.

B

= Number of specified risk scenarios where risks could occur

NOTE It is desirable to identify risks and hazard scenarios through the process of hazard analysis and risk assessment (e.g., FEMA).

SRi-2-G

Coverage of successful behaviours to handle risks of product operation

Has the risk associated with specific conditions (hazards, events, etc.) that potentially occur during product operation been successfully handled during dynamic testing and operation?

X = 1 - A/B

A

= Number of risk scenarios of operation that failed to be handled during dynamic testing or operation

B

= Number of specified risk scenarios of operation where identified risks potentially occur

SRi-3-G

Level of safety risk

What is the level of safety risk presented by specific risk scenario?

Ai

= Impact of the i-th risk

Bi

= Likelihood of the i-th risk

n

= Total number of risks included in the specific scenario

NOTE 1 Mean level of risk can be computed at the scenario level and then aggregated to measure the total risk across scenarios in the operational domain.

NOTE 2 Risk measure scales can be created by each user of these measures or use a scale such as: 1 = no safety risk, 2 = slight safety risk, 3 = moderate safety risk, 4 = significant safety risk, 5 = dangerous safety risk, 6 = extreme safety risk, 7 = extreme harm cannot be avoided.

NOTE 3 In each scenario, if any risk item has a risk score in a specified dangerous zone, the highest score can the automatically assigned to the full risk scenario.

SRi-4-G

Maximum level of safety risk

What is the maximum level of safety risk presented by all risk scenarios?

Ain

= Risk rating for risk k, calculated as the product of probability and impact for each individual risk in risk scenario i

k

= Number of risks in i-th scenario

n

= Number of scenarios

ID

Name

Description

Measurement function

SFa-1-G

Malfunction

recoverability

Can the product automatically transition to a safe mode in case of malfunction?

X = A/B

A

= Number of risk scenarios in which the product automatically transitions to a safe mode when the scenario leads to malfunctions

B

= Number of specified risk scenarios

NOTE Malfunction means the inability of a functional unit to perform its required function under specified conditions.

ID

Name

Description

Measurement function

SHa-1-G

Hazard warning

responsiveness

Can the product provide a hazard warning when risk scenarios occur?

X = A/B

A

= Number of warnings provided within the specified safe time frame

B

= Number of warnings that are expected to be provided within the specified time frame

ID

Name

Description

Measurement function

SSi-1-G

System safety

integration

What proportion of safety-related functions operate correctly after integration with another product or system?

X = A/B

A

= Number of safety-related functions in specified product that successfully pass the safety related testing when applied to the integrated system

B

= Total number of safety-related functions in the specified product when applied to the integrated system

NOTE 1 Even if the individual systems are considered safe, the integrated system is not always safe due to interactions between them. The component can be a module, subsystem, or another system.

NOTE 2 In general, X is expected to be non-zero, as a zero value means there are system problems.

Functional Suitability

Functional completeness

FCp-1-G

Functional completeness

Both

HR

FCp-2-G

Functional requirement completeness

Both

HR

Functional correctness

FCr-1-G

Functional correctness

Both

HR

FCr-2-G

Functional accuracy

Both

HR

FCr-3-G

Functional precision

Both

HR

Functional appropriateness

FAp-1-G

Functional appropriateness

Both

HR

Performance efficiency

Time behaviour

PTb-1-G

Mean system wait time

Both

HR

PTb-2-G

Mean response time

Both

HR

PTb-3-G

Response time adequacy

Both

R

PTb-4-G

Mean turnaround time

Both

R

PTb-5-G

Turnaround time adequacy

Both

R

PTb-6-G

Mean throughput

Both

R

PTb-7-G

Throughput adequacy

Both

R

Resource utilization

PRu-1-G

Peak processor utilization

Behavioural

HR

PRu-2-G

Mean processor utilization

Behavioural

R

PRu-3-G

Peak memory utilization

Behavioural

R

PRu-4-G

Mean memory utilization

Behavioural

R

PRu-5-G

Mean wait time

Behavioural

R

PRu-6-G

Peak I/O device utilization

Behavioural

R

PRu-7-G

Mean I/O devices utilization

Behavioural

R

PRu-8-G

Peak bandwidth utilization

Behavioural

UD

PRu-9-G

Mean bandwidth utilization

Behavioural

UD

PRu-10-G

Peak energy consumption utilization

Behavioural

R

PRu-11-G

Mean energy consumption utilization

Behavioural

R

PRu-12-G

Resource utilization adequacy

Behavioural

R

Capacity

PCa-1-G

Maximum capacity used

Both

R

PCa-2-G

Maximum user access capacity used

Both

R

PCa-3-G

User access adequacy

External

UD

PCa-4-G

Capacity adequacy

Both

R

Compatibility

Co-existence

CCo-1-G

Co-existence with other products

External

HR

Interoperability

CIn-1-G

Data formats exchangeability

Both

HR

CIn-2-G

Data exchange protocol sufficiency

Both

R

CIn-3-G

External interface completeness

Both

HR

Interaction capability

Appropriateness recognizability

IAr-1-G

Description completeness

Both

HR

IAr-2-G

Demonstration coverage

Both

UD

Learnability

ILe-1-G

User guidance completeness

Both

HR

ILe-2-G

Entry fields default

Both

R

ILe-3-G

Error messages resolvability

Both

R

ILe-4-G

Self-explanatory user interface

Both

UD

Operability

IOp-1-G

Operational consistency

Both

HR

IOp-2-G

Message clarity

Both

R

IOp-3-S

Functional customizability

Both

UD

IOp-4-S

User interface customizability

Both

UD

IOp-5-S

Monitoring capability

Both

UD

IOp-6-S

Undo capability

Both

R

IOp-7-S

Understandable categorization of information

Both

R

IOp-8-S

Appearance consistency

Both

UD

IOp-9-S

Input device support

Both

UD

User error

protection

IEp-1-G

Avoidance of user operation error

Both

HR

IEp-2-G

Error message resolvability

Both

HR

IEp-3-G

User entry error correction

Both

R

IEp-4-G

User error recoverability

Both

R

User engagement

IUe-1-G

Engaging user interfaces

Both

UD

Inclusivity

IIn-1-G

Language inclusivity for the widest range of users

Both

HR

IIn-2-G

Culture inclusivity for the widest range of users

Both

HR

User assistance

IUa-1-G

Assistance for users with disabilities and diverse users

Both

R

Interaction capability

Self-

descriptiveness

ISd-1-G

Presentation of understandable information for

user tasks

Both

R

Reliability

Faultlessness

RFa-1-G

Fault resolution rate

Both

HR

RFa-2-G

Mean time between failures (MTBF)

Behavioural

HR

RFa-3-G

MTBF improvement

Behavioural

R

RFa-4-G

Failure rate

Behavioural

R

RFa-5-G

Failure rate improvement

Behavioural

R

Availability

RAv-1-G

Product availability

Behavioural

HR

RAv-2-G

Mean down time

Behavioural

R

Fault tolerance

RFt-1-G

Fault avoidance

Behavioural

HR

RFt-2-G

Fault identification

Behavioural

HR

RFt-3-G

Redundancy of components

Both

R

RFt-4-G

Mean fault notification time

Behavioural

UD

Recoverability

RRe-1-G

Mean recovery time

Behavioural

HR

RRe-2-G

Mean recovery time by component recovery level

Behavioural

HR

RRe-3-G

Backup data completeness

Both

R

Security

Confidentiality

SCo-1-G

Access controllability

Both

HR

SCo-2-G

Access control

mechanism

sufficiency

Both

HR

SCo-3-G

Data encryption correctness

Both

R

SCo-4-G

Strength of

cryptographic

algorithms

Both

R

SCo-5-G

One-way

encryption

algorithm

Both

UD

SCo-6-G

Data

transmission

protection

Both

R

SCo-7-S

Minimization of

personal data

collection

Both

R

Security

Integrity

SIn-1-G

Data integrity

Both

HR

SIn-2-G

Internal data corruption prevention

Both

R

SIn-3-G

Important

executable file

integrity

Both

R

SIn-4-G

Response to

integrity

corruption

Both

R

Non-repudiation

SNo-1-G

Non-repudiation assurance

Both

R

SNo-2-G

Non-repudiation

implementation

completeness

Both

R

SNo-3-G

Utilization of

trusted

timestamps

Both

R

Accountability

SAc-1-G

User audit trail completeness

Both

HR

SAc-2-G

Audit log retention

Both

R

SAc-3-G

Mechanism for

audit log

Both

R

Authenticity

SAu-1-G

Authentication mechanism sufficiency

Both

HR

SAu-2-G

Authentication rules conformity

Both

R

Sau-3-G

Authentication

protection

mechanism

Both

R

Resistance

SRe-1-G

Resistance to hacker attacks

Both

HR

SRe-2-G

Use of secure

middleware and

operating

systems

Both

R

SRe-3-G

Middleware

information

disclosure

Both

R

Maintainability

Modularity

MMo-1-G

Coupling of components

Both

R

MMo-2-G

Acceptable cyclomatic complexity

Inherent

UD

Reusability

MRe-1-G

Reusability of assets

Both

HR

MRe-2-G

Coding rules conformity

Inherent

R

Analysability

MAn-1-G

Product log completeness

Both

HR

MAn-2-G

Diagnosis function effectiveness

Both

R

MAn-3-G

Diagnosis function sufficiency

Both

R

Modifiability

MMd-1-G

Modification efficiency

Both

HR

MMd-2-G

Modification capability

Both

UD

Testability

MTe-1-G

Test function completeness

Both

R

MTe-2-G

Code coverage

Behavioural

R

MTe-3-G

Testable dependency

Behavioural

R

MTe-4-G

Test restartability

Both

UD

MTe-5-G

Test coverage

Behavioural

R

MTe-6-G

Test case execution coverage

Behavioural

R

MTe-7-G

Test case pass rate

Behavioural

R

Flexibility

Adaptability

FAd-1-G

Hardware environmental adaptability

Behavioural

HR

FAd-2-G

System software environmental adaptability

Behavioural

HR

FAd-3-G

Operational environment adaptability

Behavioural

UD

Scalability

FSc-1-G

Scalability scale-out

Behavioural

HR

FSc-2-G

Scalability scale-up

Behavioural

HR

Installability

FIn-1-G

Installation time efficiency

Behavioural

R

FIn-2-G

Installation customizability

Behavioural

R

Replaceability

FRe-1-G

Usage similarity

Both

HR

FRe-2-G

Product quality equivalence

Both

R

FRe-3-G

Functional inclusiveness

Behavioural

R

FRe-4-G

Ease of data use

Behavioural

R

Safety

Operational constraint

SOp-1-G

Domain hazard coverage

Behavioural

HR

SOp-2-G

Coverage of successful behaviours to treat domain hazards

Both

R

SOp-3-G

Operational domain coverage

Behavioural

HR

SOp-4-G

Coverage of successful behaviours to treat hazards in the operational domain

Both

R

Risk identification

SRi-1-G

Risk identification coverage

Behavioural

HR

SRi-2-G

Coverage of successful behaviours to handle risks of product operation

Both

R

SRi-3-G

Level of safety risk

Both

R

SRi-4-G

Maximum level of safety risk

Both

R

Fail safe

SFa-1-G

Malfunction recoverability

Behavioural

R

Hazard warning

SHa-1-G

Hazard warning responsiveness

Behavioural

HR

Safe integration

SSi-1-G

System safety integration

Behavioural

R

n1

is the number of distinct operators (i.e. the number of distinct operator words that are prepared and reserved by the programming language in a program source code);

n2

is the number of distinct operands (i.e. the number of distinct operand words that are defined by the programmer in a program source code);

N1

is the total number of operators (i.e. the number of occurrences of distinct operators in a program source code);

N2

is the total number of operands (i.e. the number of occurrences of distinct operands in a program source code).

ID

Quality measure

name

Product Quality

Quality-in-Use

Analysis/Design

Implementation

Testing

Operation

FCp-1-G

Functional completeness

O

FCp-2-G

Functional requirement completeness

O

FCr-1-G

Functional correctness

O

O

FCr-2-G

Functional accuracy

O

O

FCr-3-G

Functional precision

O

O

FAp-1-G

Functional appropriateness

O

O

PTb-1-G

Mean system wait time

O

O

PTb-2-G

Mean response time

O

O

PTb-3-G

Response time adequacy

O

O

PTb-4-G

Mean turnaround time

O

O

PTb-5-G

Turnaround time adequacy

O

O

PTb-6-G

Mean throughput

O

O

PTb-7-G

Throughput adequacy

O

O

PRu-1-G

Peak processor utilization

O

O

PRu-2-G

Mean processor utilization

O

O

PRu-3-G

Peak memory utilization

O

O

PRu-4-G

Mean memory utilization

O

O

PRu-5-G

Mean wait time

O

O

PRu-6-G

Peak I/O device utilization

O

O

PRu-7-G

Mean I/O devices utilization

O

O

PRu-8-G

Peak bandwidth utilization

O

O

PRu-9-G

Mean bandwidth utilization

O

O

PRu-10-G

Peak energy consumption utilization

O

O

PRu-11-G

Mean energy consumption utilization

O

O

PRu-12-G

Resource utilization adequacy

O

O

PCa-1-G

Maximum capacity used

O

O

PCa-2-G

Maximum user access capability used

O

O

PCa-3-G

User access adequacy

O

O

PCa-4-G

Capacity adequacy

O

O

CCo-1-G

Co-existence with other products

O

O

CIn-1-G

Data formats exchangeability

O

O

CIn-2-G

Data exchange protocol sufficiency

O

O

CIn-3-G

External

interface completeness

O

IAr-1-G

Description completeness

O

O

IAr-2-G

Demonstration coverage

O

O

ILe-1-G

User guidance completeness

O

O

ILe-2-G

Entry fields default

O

O

ILe-3-G

Error messages resolvability

O

O

ILe-4-G

Self-explanatory user interface

O

O

IOp-1-G

Operational consistency

O

O

IOp-2-G

Message clarity

O

O

IOp-3-S

Functional customizability

O

O

IOp-4-S

User interface customizability

O

O

IOp-5-S

Monitoring capability

O

O

IOp-6-S

Undo capability

O

O

IOp-7-S

Understandable categorization of information

O

O

IOp-8-S

Appearance consistency

O

O

IOp-9-S

Input device support

O

O

IEp-1-G

Avoidance of user operation error

O

O

IEp-2-G

Error message resolvability

O

O

IEp-3-G

User entry error correction

O

O

IEp-4-G

User error recoverability

O

O

IUe-1-G

Engaging user interfaces

O

O

IIn-1-G

Language inclusivity for the widest range of users

O

O

IIn-2-G

Culture inclusivity for the widest range of users

O

O

IUa-1-G

Assistance for users with disabilities and diverse users

O

O

ISd-1-G

Presentation of understandable information for user tasks

O

O

RFa-1-G

Fault resolution rate

O

O

RFa-2-G

Mean time between failures (MTBF)

O

O

RFa-3-G

MTBF improvement

O

O

RFa-4-G

Failure rate

O

O

RFa-5-G

Failure rate improvement

O

O

RAv-1-G

Product availability

O

O

RAv-2-G

Mean down time

O

O

RFt-1-G

Fault avoidance

O

O

RFt-2-G

Fault identification

O

O

RFt-3-G

Redundancy of components

O

O

RFt-4-G

Mean fault notification time

O

O

RRe-1-G

Mean recovery time

O

O

RR3-2-G

Mean recovery time by component recovery level

O

O

RRe-3-G

Backup data completeness

O

O

SCo-1-G

Access controllability

O

O

SCo-2-G

Access control

mechanism

sufficiency

O

O

SCo-3-G

Data encryption correctness

O

O

SCo-4-G

Strength of

cryptographic

algorithms

O

O

SCo-5-G

One-way

encryption

algorithm

O

O

SCo-6-G

Data

transmission

protection

O

O

SCo-7-S

Minimization of

personal data

collection

O

O

SIn-1-G

Data integrity

O

O

SIn-2-G

Internal data corruption prevention

O

O

SIn-3-G

Important

executable file

integrity

O

O

SIn-4-G

Response to

integrity

corruption

O

O

SNo-1-G

Non-repudiation assurance

O

O

SNo-2-G

Non-repudiation

implementation

completeness

O

O

SNo-3-G

Utilization of

trusted

timestamps

O

O

SAc-1-G

User audit trail completeness

O

O

SAc-2-G

Audit log retention

O

O

SAc-3-G

Mechanism for

audit log

O

O

SAu-1-G

Authentication mechanism sufficiency

O

O

SAu-2-G

Authentication rules conformity

O

O

O

SAu-3-G

Authentication

protection

mechanism

O

O

SRe-1-G

Resistance to hacker attacks

O

O

O

SRe-2-G

Use of secure

middleware and

operating

systems

O

O

SRe-3-G

Middleware

information

disclosure

O

O

MMo-1-G

Coupling of components

O

O

MMo-2-G

Acceptable

cyclomatic

complexity

O

O

MRe-1-G

Reusability of assets

O

O

MRe-2-G

Coding rules conformity

O

O

MAn-1-G

Product log completeness

O

O

MAn-2-G

Diagnosis function effectiveness

O

O

MAn-3-G

Diagnosis function sufficiency

O

O

MMd-1-G

Modification efficiency

O

O

MMd-2-G

Modification capability

O

O

MTe-1-G

Test function completeness

O

O

MTe-2-G

Code coverage

O

O

MTe-3-G

Testable dependency

O

MTe-4-G

Test restartability

O

O

MTe-5-G

Test coverage

O

O

MTe-6-G

Test case execution coverage

O

O

MTe-7-G

Test case pass rate

O

O

FAd-1-G

Hardware environmental adaptability

O

O

FAd-2-G

System software environmental adaptability

O

O

FAd-3-G

Operational environment adaptability

O

O

FSc-1-G