Measurement System Analysis of uncertainty is one topic in Lean Six Sigma training that is too often ignored or under-taught. I believe that it is under-taught because most instructors have never used or understood it. Therefore, this blog post intends to dive deep into what it is and why you should learn about it. First of all, this methodology only works on measurement systems that allow re-measuring the same sample and the reporting of a continuous value for the measurement output. It is not for destructive sampling or for attribute or classification gauges.
Measurement System Analysis or Gage Study
I first learned about this topic as a Measurement System Analysis or MSA. The term gage study is only one piece of the effort, but many consider gage study and MSA to be synonyms. Measurement System Analysis is probably the best term for the topic because its title is inclusive to more than just the gauge. A true MSA includes estimating the impact of the gauge, the fixturing or setup of the gauge, the operator, and the variation over time.
The best reference book
The best reference for Measurement System Analysis is the Automotive Industry Action Group (AIAG) blue book titled “Measurement System Analysis” 4th ed. This book has all the information on the definitions and methods that are in common use along with very good examples. When I taught at Bechtel, we required every Lean Six Sigma black belt to obtain a copy of this book. Right now I teach out of Forrest Breyfogle’s book Integrated Enterprise Excellence Volume 3, which has licensed portions of the AIAG material so that it is also an excellent reference book.
Ratio of items to appraisers to repeat measurements
The general answer to this is to measure multiple items, multiple times, by multiple people. The most common ratios when all of the analysis was performed by hand were 10 items, two measurements, by three people. I seem to recall that this ratio provided a good balance of the uncertainties, but it is far from a requirement. You do need a minimum of two appraisers and a minimum of two measurements of each item by each appraiser, but the number of items to be measured can vary a good deal.
My guidance to most people is to consider where you believe the greatest sources of variation exist and increase the factor that would allow the best estimation of that factor.
- If the difference in appraisers (people) is expected to be high, use three or four appraisers of widely different experience levels
- If you expect that slight differences in the measured items may impact the reported measurement, make sure you include a full range of 8-12 items that include at least one item outside of the specification.
- If you believe that there may be a time-to-time change in the measurement system because of adjustments or setups methods, then include at least three replicated measurements from each appraiser of each item.
There are two accepted measurement system analysis methods: ANOVA and X-bar-R. I have always trusted in the ANOVA method because it provides the ability to evaluate an interaction between the appraiser and the items. The X-bar-R method is preferred when you are doing it by hand or using a calculator. When using a statistical analysis program, I recommend always using the ANOVA method.
What is precision or Gauge Repeatability and Reproducibility?
There are two components of a uncertainty in a measurement system analysis: Repeatability and Reproducibility. The combination of repeatability and reproducibility is labeled as the Precision of a measurement system.
Repeatability is the variation found in a measurement system when the same item is measured over and over again without changing its position or who appraised it and all at the same time. Get it: repeated measurements. This uncertainty estimate is really the smallest error you can get on a measurement system without fundamentally changing the equipment or the measurement process. It is reported as a standard deviation.
Repeatability: variation in measuring something the same way, same person, same time, same conditions, same, same, same.
Reproducibility is the variation found when trying to reproduce the measurement under different conditions. These different conditions will include the difference in appraisers, the difference in fixturing or positioning the item in the measurement tool, different times, and different calibrations. When this value is high, it implies that the measurement process is inadequate. It is reported as a standard deviation.
Reproducibility: variation in measuring something in different ways, different times, different people, different, different, different.
The precision is considered the true measurement system uncertainty. Precision is the square root of the sum of the squared repeatability value and squared reproducibility value. The precision value is the one that is compared to the specification and the process variability to determine the goodness. A large precision value may derive from a large repeatability value, a large reproducibility value, or both values being large. The component that has the largest contribution to the precision is where you address improvements.
Precision: The total measurement system variation estimate. It includes repeatability and reproducibility.
What is good for a measurement system?
This is probably the most difficult part of the entire Measurement System Analysis concept to get across to students because the definition of good can depend on who taught you. If you consider the AIAG reference books, they list multiple methods to judge goodness, but I believe there are only two that matter: percent of the tolerance and percent of the process. Comparison of the precision value to the requirements or specifications will tell you about the ability of the measurement system to be used for Quality Assurance. Comparison of the precision to the process variation reported by the measurement system tells you about the ability of the measurement system to be used for Quality Control.
AIAG provides a standard measure of goodness as a ratio of the precision to a variation source. For Quality Assurance usage, the ratio is (6*precision / width of the specification) which is reported as a % of tolerance. For Quality Control and SPC usage, the ration is (precision / std-deviation of the process) which is reported as a % of the process.
For both percentages we apply the following general rules
- >30% as unacceptable
- 20%-30% as marginal
- 10%-20% as acceptable
- < 10% as excellent.
Some organizations consider 20% as a maximum allowable, but I have seen no single business sector or industry that has a standard, it has always seemed company dependent.
Measures of goodness that have little value
Many books recommend using the % of the test as a measure of goodness. This is reported as the (precision / standard deviation of the measurements in the test) which is reported as a % of test. This is not considered as a good measure of goodness because the value will change each time you run the MSA and use different parts. This is not a good characteristic, since it means that I can choose items that have widely varying values and make the % better than if I choose a series of items that have similar values.
A second common measure of goodness is the number of categories, which again is a ratio of the precision and the standard deviation of the tested items. I have the same problem with this measure as I do the % of the test.