Pareto Analysis: Tools for New Product Testing

Testing is, of course, a critical component to product development. To better understand how it can be proactively employed to help ensure product quality and longevity, we reached out to John McCloy, Founder & President of Engineered Assurance and author of the article on the industry’s 1% failure rate that was published in the March issue. Our conversation evolved into discussion surrounding ways that testing and Pareto Analysis are being used in early upstream stages of product development.

Mr. McCloy stressed that when it comes to analyzing the problems uncovered in your test data don’t confuse the elements of frequency and importance. A poignant example he pointed out during our conversation is that for years, the focus on metal-on-metal hip implants was the common problem of stem breaks and screw fractures on the acetabular cup. Implant wear later became the culprit for massive recalls.

We thought you would find our conversation useful. Mr. McCloy’s insight on the topic follows.

BONEZONE: First, what role does testing play in new product development?

McCloy: Like all industries, the idea is to accelerate product development while improving quality. For some time now, testing has provided the real world baseline data needed for simulation. As new implants and devices are designed, computer-aided design (CAD) models are run through simulation software. This allows multiple design alternatives to be evaluated quickly, verified and validated as they are subjected to the stresses, strains and cycle data provided by testing. As a result, devices are developed at a fraction of the time and cost associated with building and testing multiple physical prototypes.

Like I said, this practice has been around for decades and is certainly not new. But as software tools, methodologies and best practices evolve, new product development will continue to look to testing to provide real-world baseline data.

BONEZONE: Where can one learn more about this in relationship to the orthopaedic industry?

McCloy: The American Society for Quality (ASQ) is an excellent resource. They provide many benefits and tools for the design, manufacture and testing of medical implants.


Frequency and importance don’t always go hand-in-hand. Remember, the most common problems aren’t always the most important problems.


BONEZONE: You mentioned verification and validation. We know from conversations with readers that confusion remains about V&V. Can you simply define the difference?

McCloy: While it’s true that the terms are often used interchangeably, verification and validation are complementary yet independent procedures. They are used together as a means to confirm that a product meets all intended requirements and specifications. Validation provides assurance that a product satisfies external or customer requirements, while verification is generally an internal process used to determine whether a product complies with a regulation, requirement, specification or other conditions. As your readers know, verification and validation are critical components of a quality management system—like ISO 9000. As such, it’s generally recommended that they be performed by a neutral third party.

BONEZONE: So, is implant failure information shared among the industry?

McCloy: Yes, to some extent. FDA publishes Manufacturer and User Facility Device Experience reports—MAUDE—that are freely available to all. These include internal recall and recovery of explant failures while providing insight into cause and frequency. When employed with a Pareto Analysis, weighted with severity, management can act quickly and decisively.

BONEZONE: How is Pareto Analysis used today?

McCloy: Pareto Analysis is based on the Pareto Principle—often referred to as the 80/20 rule. The idea is that for many events, roughly 80% of the effects come from 20% of the cause. I’m sure you’ve heard the expression that in business, 80% of sales come from 20% percent of customers.

Pareto Analysis is a decision-making process used for the selection of a limited number of tasks that produce significant overall effect. So in the business example, we would use Pareto Analysis to find the 20% of clients upon whom we should focus our time and energy.

With regard to medical implants or devices, we can say that 80% of the cost associated with defects will be attributed to 20% of those defects. In other words, some matter more. Instead of a knee-jerk reaction to something that happens frequently or trying to address everything, Pareto Analysis allows us to assign weights and prioritize based on importance. It allows manufacturers to identify and concentrate on fixing that 20% with the biggest consequences.

BONEZONE: Can you give an example?

McCloy: Each year, something like 320,000 people in the U.S. alone undergo hip reconstruction. With those numbers, the opportunity for failure and subsequent litigation costs are staggering. And when this in fact happened, most of the attention was focused on fixing the most common problems: stem breaks and screw fractures on the acetabular cup. This went on for years before massive recalls took place. As it turned out, metal-on-metal wear was the real culprit, and should have been the focus from the start. This issue was noticed two to three years ahead of the recall, but was not elevated as a top priority.

BONEZONE:So, how does someone use Pareto Analysis, in this case?

McCloy: Pareto Analysis allows us to focus on addressing the real problem, saving time and money. The first step is to construct a bar chart and capture all of the cases and reasons for defect. Was there a design flaw? Was improper material used? Was the implant or device poorly manufactured? Was it improperly installed? Did the anchoring screws fail? Or, was it a matter of a noncompliant patient who may have been overweight or had other issues?

This provides a look at the frequency and importance of failure. So, for example, while we might be tempted to redesign bone screws, Pareto Analysis allows us to understand and address the most significant areas. With an artificial hip, it may be that metal fatigue was not the most common defect, but was the most significant. The manufacturer could then concentrate on investigating why this was happening. Were design modifications needed? Were there better material choices, or did the plastic liner separating the femoral head and acetabular component fail? Frequency and importance don’t always go hand-in-hand. Remember, the most common problems aren’t always the most important ones; a weighted Pareto Chart emphasizes the most important factors.

BONEZONE: Why is a Pareto Chart useful?

McCloy: A Pareto Chart is basically a way to capture and document all reasons for defect. Usually represented as a bar chart, it is a simple yet effective way of organizing information in this manner to help visualize the 20% most critical factors. While it’s important to employ this type of thinking early in the quality improvement initiative, a Pareto Chart is useful throughout the entire lifecycle of a product. Constructing a chart isn’t difficult. It’s just a matter of collecting and organizing data. Of course, software automates the process, today.


Exhibit 1: Example of Data Collection Using a Pareto Chart





Exhibit 2: Example of Sorting Data based
on Most Costly Issue Using a Pareto Chart




BONEZONE: Can you give us an example of a Pareto Chart in action?

McCloy: Say you have had problems with a bone screw that has a history of breaking. We’ll construct a Pareto Chart to look at all of the cases and reasons for defect. I’ll use my previous examples: Was there a design flaw? Was improper material used? Were the screws poorly manufactured? Was it over-torqued or otherwise improperly installed? Or was it a matter of a noncompliant patient who may have been overweight or had bone issues?

This allows manufacturers to take a look at the frequency and importance of failure. But frequency and importance don’t always go hand-in-hand. So, while we might see that the most common reason for screw failure can be attributed to postoperative infection or failure to employ stress analysis in the design stage, Pareto Analysis allows us to prioritize efforts on the most important and costly issues; in this case, material choice and design. (See Exhibits 1 and 2.)

BONEZONE: Circling back to your prior comments, how does this apply to validation and product development?

McCloy: Product development is a continuous decision-making process. Trade-offs are made along the way to balance cost, manufacturability and quality. The idea is that the product is not over- or under-designed. For example, is it required that we construct the implant from titanium, or will a less-costly material allow us to meet quality and cost targets? Pareto Analysis allows us to assign weights to the severity of failures. In this scenario, Pareto Analysis allows us to quantifiably measure the ramifications if the device fails as a result of not using titanium.

BONEZONE: What steps should engineers take when validating devices, using this method?

McCloy: Experience is the key—there’s no substitute for it. It’s also a good idea to have tests conducted by a disinterested third party; therefore, it’s often best to subcontract to an accredited, independent test lab.

BONEZONE: Many of our readers handle a portion of testing inhouse. In your experience, is there a mistake or a cautionary tale that you can provide to device companies when handling Pareto Analysis on their own?

McCloy: Pareto Analysis works best when not subjected to outside influences. In other words, one must enter into it objectively, allowing the data to speak for itself. Don’t allow preconceived conclusions or internal politics to corrupt the findings. I’ve said it before and I cannot emphasize it enough—the most frequent failures aren’t always the most important ones. This is why I recommend outsourcing to a disinterested third-party test lab.

BONEZONE: Why is this a smart tool to use?

McCloy: Earlier, we discussed verification and validation. Pareto Analysis provides internal and external agreement that a selected implant or device will meet internal and external requirements. Displaying data on a Pareto Chart will help build consensus while allowing manufacturers to concentrate resources, time and money to address the most significant issues.

BONEZONE: Are there instances where Pareto Analysis isn’t the best option?

McCloy: Yes; as I mentioned earlier, experience is absolutely critical. If a company has no experience in this area, it’s best to go another route. In cases where product history is available, it’s a good idea to leverage that data.

BONEZONE: So what’s the better option: failure analysis or Pareto Analysis?

McCloy: All testing is important, and it’s important to note that failure analysis and Pareto Analysis aren’t mutually exclusive. As with anything, the more information one has, the better are the decisions that can be made. Much like a detective inspecting a crime scene, failure analysis analyzes explants (failed implants) to determine exactly why the implant failed. Although the process takes place after the fact, information is fed back to the medical community, allowing the necessary design, material, manufacturing or installation changes to be made. Pareto Analysis, on the other hand, allows us to understand long-term ramifications of design and material decisions to optimize designs and avoid problems down the road. In my mind, each has its place in helping to ensure better, longer-lasting medical implants and devices.

John McCloy is Founder and President of Engineered Assurance, LLC. Mr. McCloy is the past owner of Accutek Testing Laboratory, which was a mechanical and metallurgical testing laboratory for clients from the medical, aerospace and defense industry. Over 12 years, Mr. McCloy built the lab to more than 50 employees. He has since retired to part-time consulting.

Mr. McCloy remains an active member of the American Society of Testing Materials (ASTM), the American Society of Metals (ASM) and the American Welding Society (AWS). Additionally, Mr. McCloy maintains numerous industry and scholastic licenses and certifications including PE, CWI and MBA. He can be reached by phone, 513.543.8146, or This email address is being protected from spambots. You need JavaScript enabled to view it..

Engineered Assurance, LLC