John Nolan Most companies who have a QMS (Quality Management System) certified according to ISO 9004 or ISO 9001:2015 are accustomed to performing against set objectives and KPIs (key performance indicators), and also demonstrating continual improvement, which we examined in the article ISO 9001 – Where does quality management fit into your organization? But, how can the QMS itself, and its effectiveness, be measured, and is there any one method of doing this that will provide an organization’s top management with one true indicator of QMS performance?
Why is measurement needed?
Section 9.1.1 of the ISO 9001:2015 standard suggests that “the organization shall evaluate the performance and the effectiveness of the quality management system.” While this is not prescriptive in terms of measuring the QMS by one true indicator, most top management teams prefer that one true measure is established to ascertain the effectiveness of the system, as well as whether month-on-month improvement – and value for money – are attained. You can find more details on this from the article Monitoring and measurement: The basis for evidence-based decisions. So, if you are a quality professional given that task, where do you begin, and what criteria should you consider?
What measurement fits best?
There have been many opinions over the years on how quality is best measured, particularly in manufacturing processes, although with the increase in service industries in recent years this has also become a point of discussion there. Manufacturing organizations traditionally may like to use measures such as “defects per million opportunities,” while a service organization may choose to measure customer satisfaction via a survey. Each organization must decide what is most important to it, but let us consider some of the options available to measure QMS performance:
- Customer satisfaction is at the heart of the ISO 9001:2015 standard, and many organizations think the answer over measurement should lie there. We discussed some of the factors that affect this element in the article Main elements of handling customer satisfaction in ISO 9001, but how can customer satisfaction be measured? Perhaps your organization can produce a questionnaire for customers and measure satisfaction against a target or objective that is known to be standard in your sector. This method is particularly useful, for example, when measuring customer satisfaction in a call center or other “one-to-one client-facing” business sector.
- Use one “defining” quality measure. This is particularly useful and prevalent in production-based environments where repeatable processes are used. Measuring test “pass rates,” for example, can give a key indicator of how a process within the QMS is performing, or devising a DPMO (defects per million opportunities) measure, where a key part of your process captures critical fails before they reach your external customer, can be an effective measure of your QMS. Ultimately, if this measure does not improve month over month, you must examine the key components of your process. Whether it is training, knowledge, methods, materials, or failure of your risk and/or corrective action process, you must analyze and modify your process using the QMS tools at your disposal to improve results.
- Derive your quality management success from your financial performance. Many top managers think that these two things are inextricably linked, and while it’s true that a dissatisfied customer will not return, this is a very reactive outlook – by the time that sales are falling, it is generally too late to recover customers who have changed brands. Therefore, assuming your QMS is effective because sales have increased over the last month is a dangerous thing to do, and while some businesses use this one indicator to manage their QMS, it is not recommended. The article How to write Good Quality Objectives can help provide some further ideas on this topic.
- RMA (Return Material Authorization) Rates. This is another common measure of QMS performance in manufacturing plants. By measuring the rate of returns from customers (in or out of warranty), many organizations deem that the effectiveness of the QMS is measured. Again, there is merit in this key performance indicator, but once again it means measuring failures after they have happened and negatively affected your customers, as opposed to preventing failure – which protects your client base.
- Cost of quality. Many organizations measure the cost of “reworking” product, whether due to design or manufacturing error or oversight. Again, while there is a reactive element to this method, it is undoubtedly a valid measure of how your QMS – including design and delivery, risk, and corrective action processes – are working. The article How to use root cause analysis to support corrective actions in your QMS can help you understand more about the corrective action process.
Choosing the measure that is right for your organization
Every organization must decide what works best for them, but personally, I like to measure the “cost of quality” as a primary QMS specifically for a manufacturing organization, supported by all the other options above measured at a secondary level. You can refer to the article Analysis of data obtained from Monitoring and Measurement to help here, too. That way, the true “cost of quality” can be presented to top management in dollar form, which usually gives the incentive to drive initiatives from a strategic level. At a secondary level, using elements such as customer feedback, internal test fails, and RMA analysis will help you to improve your “cost of quality,” while also protecting your customers and improving the processes that design and deliver your product. In this way, you can ensure continual improvement, customer satisfaction, and improved product design – all elements that validate good QMS performance – and your board of directors can be satisfied that they have one “true” measure that informs them as to how the QMS is performing, and can be easily used to ascertain if continual improvement is occurring.
Use this free online training ISO 9001:2015 Foundations course to learn more about the measurement requirements.
