Measurement system analysis by attribute, an effective tool to ensure the quality of the visual inspection process within an organization

. A successful quality assurance program in any organization needs good measurement systems. The concept of measurement system refers to the manpower, machines, materials, methods and mother nature involved in obtaining measurements. Measurement system analysis (MSA) is a set of procedures which are used to determine the amount of variation due to the measurement system and if the measurement data are valid. Most problematic measurement system issues come when the results of a measurement system are qualitative values such as “pass” or “fail” (attribute data) rather than quantitative values (variable data). Because in industry many measurement systems deal with qualitative data, the assessment of the visual inspection process is a contemporary approach for quality assurance in most manufacturing organizations. The aim of this paper is to apply the MSA attribute study in a local company from the Sibiu region supplying products for automotive industry, to ensure the quality of the visual inspection process for one of their parts, respectively the exterior lighting projectors (Day Running Light, DRLs). The output data were analyzed using Minitab software. The conclusion was that the visual inspection process must be improved by appraisers training and developing panel with samples for the most common defect types of DRLs.


Basics about measurement system analysis by attribute
In our day to day work we often collect and use data to make important decisions about our processes. Unreliable data can lead us to believe there is a problem with the process when actually everything is okay, or it may prevent us from spotting a problem. This is likely to cost the business money, through unnecessary scrap or rework, or unnecessary improvement projects, or through being unaware that there is a problem with a process or product and risking customer complaints or more serious problems such as safety incidents. Therefore, before collecting and using data to make decisions about any process or product (part), it is important to verify that the measurement system is good enough by doing a Measurement System Analysis [1].
The concept of measurement system refers to the manpower, machines, materials, methods and mother-nature involved in obtaining measurements and a measurement system analysis (MSA) is a set of procedures which are used to determine the amount of variation due to the measurement system [2]. The principle of such an analysis is to ask two or three persons (appraisers) to measure more parts several times and to calculate some indices/metrics to look at how much variation there is in the results and after, based on some criteria, to decide whether this amount of variation is acceptable or not. Although the principles are similar, there are 2 main types of measurement system analysis: • variable measurement system for continuous data; in this case the MSA method called Gauge R&R is used, that is the short for Gauge Repeatability and Reproducibility. • attribute measurement system for discrete data; in this case it is used the MSA method called Attribute Agreement Analysis. Most problematic measurement system issues come when the results of measurements are qualitative values such as "pass" or "fail" (attribute data), often the result of the human visual inspection, rather than quantitative values (variable data). This is because it is very difficult for appraisers (the people who normally make the measurements) to apply the same operational definition of what is "good" and what is "bad". Because in industry many measurement systems deal with qualitative data, the assessment of the visual inspection process is a contemporary approach for quality assurance in most organizations [3][4][5].
The measurement system analysis by attribute is a straightforward method that can be used to assess the reliability of a qualitative measurement system. Generally, this method does the following operations: classifies a part as either conforming (C) or nonconforming (NC), classifies a part into one of multiple categories and counts the number of "nonconformities" per part inspected.
The Attribute Agreement Analysis is a set of trials conducted to assess the appraiser ability to categorize parts -refers to whatever is being assessed. The method requires that multiple appraisers (two or more) to categorize, independently and in a random order, multiple parts (minimum 20 parts) multiple number of times (two or more trials) without being able to remember their previous results.
In addition to the appraisers, when running an Attribute Agreement Analysis, there must exist an expert person to evaluate each part and determine the correct result for each; the expert decision is referred to as the "standard" and this evaluation is used to assess the level of agreement between the appraisers and the standard. Based on how often the appraisers agree with themselves, with the standard and each other, several measures/statistics are then calculated (table 1).
The key in all measurement system analysis is to have a valid assessment method and clear criteria to evaluate the measurement system. The decision criteria from AIAG MSA reference manual [2] are presented in table 1.

Case study 2.1 Visual inspection of Day Running Lights (DRLs)
An investigation was conducted in a local company from Sibiu region supplying parts for the automotive industry. Its long industrial tradition and mastery of a broad technologies portfolio have positioned it as a key reference in the automotive sector. The company is the provider for 1 of every 3 cars manufactured worldwide.
The aim of the paper was to apply the measurement system analysis by attribute to study the visual inspection process for one of their products specifically the exterior lighting projectors -Daytime running lights, DRLs (figure 1). These parts were made mandatory for all new vehicles in Europe starting by 2011 and are used by car manufacturers as a way of differentiating their brands and models. More and more new designs and shapes determine the styling of DRLs.  As in other industry sectors, the goal of the automotive industry is to manufacture defects free products. The product audits, conducted by quality internal auditors on finished DRLs, have detected that the significant frequency of non-conforming DRLs is related to their aspect, respectively foreign bodies inside the projector, scratches and black dots (figure 2).
In order that the finished DRLs to be send directly to the customer and not re-verified at the "Quality Wall", unless they may have different nonconformities that can not be visually detected within the assembly line, the appraisers from the final workplace must be tested for their capability to perform the visual inspection by using the measurement system analysis by attribute.

Experimental work
In order to assess the visual inspection process for the DRLs, the results were processed with the help of the Minitab software with its module called Attribute Agreement Analysis [6]. Minitab produces both numerical and graphical analysis.
We look at the graphical output first (figure 3): the graph from the left side shows the agreement within the appraisers (analogous to Repeatability) and the graph from the right shows the agreement between the appraisers and the standard. The dots represent the actual agreement for each appraiser and the crosses represent the bounds of a 95% confidence interval prediction for the agreement average. Minitab also produces some numerical output that provides more detail on the results seen in the graphs. Analyzing results of the research, presented from table 2 to table 6, the conclusions are: • Within Appraisers (individual repeatability of appraisers) is between 80% and 90% for appraiser 2 and 3 (figure 3 -left side and table 2), so only acceptable: this means that these appraisers are not consistent with themselves; • Each Appraiser vs. Standard (individual effectiveness of appraisers) is under 80% for all appraisers, so not acceptable (figure 3 -right side and table 3); this means that appraisers are not in agreement with the true status (standard) of the DRLs; • Assessment Disagreement (table 4) results show that all appraisers classified good DRLs as bad (#NC/C) so the percent is above 5%, all appraisers classified bad DRLs as good (#C/NC) so the percent is above 2% and all appraisers were inconsistent in their judgments (#Mixed), so not acceptable; • Between Appraisers (reproducibility of measurement system) results show that all three appraisers agreed with each other on both assessments to 36.76% (on 11 out of 30 inspected DRLs), so not acceptable (table 5); • The last metric, All Appraisers versus Standard (overall effectiveness of the measurement system) tells that for 11 out of 30 DRL inspected, all three appraisers agreed with the true status of product, which represents 36.76%, so not acceptable (table 6).   So, after the initial study the following problems were identified: risk of ppm DRLs at the customer, risk of supra inspection and inconsistency of the appraisers to repeat their decisions for the same inspected part. In conclusion, the results show that there is a great problem with the ability of appraisers to assess carefully DRLs so, ways of improving this situation had to be considered by appraiser training and developing visual aids.
To address the problem, the company's engineering and quality representatives worked to improve the panel with DRL samples (figure 4 -right side) with the most common defect types of DRLs and all appraisers were trained in this specification and the actual requirements were discussed.

Fig. 4. Panel with DRL samples (initial -left side, improved -right side)
In the weeks after the training session, the study was performed again using 20 of the original parts, with the results presented in figure 5 and table 7 to 11. This study shows that the inspection process was improved substantially.

Conclusions
Human measurement systems are often used to perform visual inspection, so their assessment is important to discover if there are problems and it is necessary to implement a (re)training session. Measurement system analysis by attribute is just one of many such tools an organization may use to better understand their inspection processes and verify the validity and utility of the data collected through their measurement systems. This is a straightforward method that can be used to assess the accuracy of appraisers and the types of mistakes they are likely to make. The paper demonstrated an effective mean by which to evaluate the accuracy of the inspection people to assess DRL quality through visual inspections, using attributive MSA technique. Periodic inspector training along with developing visual aid samples are crucial to ensure any visual inspection method be efficient.