Design Controls: Design Verification and Validation

This editorial is the fifth in a series of six articles focused upon the design controls process from a standpoint of design verification and design validation in the orthopaedic medical device marketplace. First, let’s review some pertinent definitions of terms from the 21 CFR, Part 820.30 and ISO 13485:2003, Section 7.3.  Our article begins after the review of definitions.

Design Verification (expanded): confirmation that the design output meets the design input requirements. The results of the design verification, including identification of the design, method(s), the date, and the individual(s) performing the verification, shall be documented in the DHF. Risk analysis is often considered to be part of design verification. Bench-top tests and quality inspections are also common forms of verification and may serve as mitigating actions as part of the Failure Mode and Effects Analysis. A design matrix (a table in which the design inputs are matched with the design outputs) is another tool used in design verification. Design verification precedes design validation and is not a substitute. The two are distinctly different and should be treated as such. The Quality System Regulation defines design verification as confirmation by examination and provision of objective evidence that specified requirements have been fulfilled.

Verification: confirmation by examination and provision of objective evidence that specified requirements have been fulfilled.

Validation: confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use can be consistently fulfilled.

Process Validation: establishing by objective evidence that a process consistently produces a result or product meeting its predetermined specifications.

Design Validation: establishing by objective evidence that device specifications conform with user needs and the intended use(s) of the medical device. Design validation shall ensure that devices conform to defined user needs and intended uses and shall include testing of production units under actual or simulated use conditions. Design validation shall include software validation and risk analysis, where appropriate. The results of the design validation, including identification of the design, methods, the date, and the individuals performing the validation, shall be documented in the Design History File. According to the Quality System Regulation, validation testing must be performed on actual production units or their equivalents. When equivalents are used, the manufacturer must provide documentation that clearly demonstrates the equality. There can be a tendency to manufacture test units under special conditions that do not mimic actual production. An example would be the manufacturing engineer performing final assembly on a device where this would normally be done by a manufacturing operator. The manufacturing engineer’s skill level and knowledge base are most likely different from those of the operator. Companies should strive to produce their test units in an environment as close to the final production environment as possible.

A Firm Foundation of Objective Evidence: Manufacturers and Specification Developers shall establish and maintain procedures for verifying and then validating the device design.

Design verification is most always done according to specifications. Therefore, to control the specifications and increase the probability of achieving desired safety and performance characteristics, device, software, labeling, packaging and any other specifications should be complete and thoroughly reviewed before development commences. As the hardware and software designs evolve, they should be evaluated in comparison to their current specifications as well.

Design verification should be achieved with test equipment calibrated and controlled according to quality system requirements. Otherwise, there is limited confidence in the data.

Verification and validation can also be done according to a written protocol. The protocol(s) should include defined conditions for the testing. The protocol(s) should be approved before use. Test protocol(s) are not perfect for a design, particularly a new design. Therefore, the designers and other verification personnel carefully annotate any ongoing changes to a protocol. Likewise, the verification personnel should record technical comments about any deviations or other events that occurred during testing. The slightest problem should not be ignored. During design reviews, the comments, notes and deviations may be as important as test data from the formal protocol(s).

It should be noted that design changes start to be realized as inputs to the design are approved by the appropriate individuals. All design changes are accounted for and are eventually documented in the Design History File (DHF). Design changes are, of course, possible after commercialization activities begin, and should be handled in such a way as to document the change and then initialize a “trigger” to pull concerning re-verification and potentially re-validation of the design. 


Design Evaluation vs. Specifications (Performance Standards)

The original design of devices and any subsequent changes should be verified by appropriate and formal laboratory, animal and in vitro testing. Risk analysis should be conducted to identify possible hazards associated with the design. Failure Mode Effects Analysis and Fault Tree Analysis are examples of risk analysis techniques. Verification testing can begin with prototypes (sometimes called breadboards), and may be repeated as design changes are realized and accounted for. Some examples of verification tests for orthopaedic devices could be:

  • Comparative testing with a predicate device or a legacy product type
  • Simulated-use testing with prototypes
  • Animal model testing
  • Biocompatibility
  • Compatibility with other devices
  • Reliability testing
  • Performance/functionality testing
  • Material compatibility
  • Environmental emissions

Once the design is translated into physical form, its safety, performance and reliability should be verified by testing under simulated use conditions. Such verification may include in vitro and in vivo testing. Appropriate laboratory and animal testing followed by analysis of the results should be carefully performed before clinical testing or commercial distribution of the devices. The manufacturer should be assured that the design is safe and effective to the extent that can be determined by various scientific tests and analysis before clinical testing on humans or use by humans. It should be noted at this juncture that design verification should ideally involve personnel other than those responsible for the design under review. Removing as much potential bias as possible will help to remove uncertainty and other subsequent aspects of risk.
 

Risk-Based Decisionmaking

A widely used risk-based evaluation technique is Failure Mode and Effects Analysis (FMEA), in which failures are assumed to occur. FMEA is useful for evaluating reliability, safety and general quality in which, for example, the evaluator assumes that:

  • Each component fails,
  • Each subsystem or subassembly fails,
  • The operator makes errors, and
  • The power source is interrupted and immediately restarted

The probability of each failure actually occurring and then the resulting effects are analyzed toward eventual mitigation. Then, where needed and feasible, hazards and faulty performance are designed out of the device or reduced, or are compensated or prevented/reduced by interlocks, warning signs, explicit instructions, alarms, etc. Risks, of course, cannot always be totally removed from medical devices, but they should be known and controlled to the extent feasible with cutting edge technology. Mitigation is sometimes the reduction of risks through compromise rather than elimination. The idea is not to promote one method above the other, because a reasonable amount of both actual testing and FMEA should be performed before a device is presented for design validation.

Aside from FMEA, there are other human factor and validation process techniques that can be used in developing an overall risk analysis. These techniques include timelines, workload analysis, failure analysis, alternative calculations, testing including animal testing, auditing the design output (including the DHF at prescribed intervals), design reviews, demonstrations and comparing a new design to a proven design, etc.

During design reviews, all evaluation results should be evaluated by a cross-functional team that will compare the tests and FMEA results with design specifications, including safety and performance standards, to make sure that the desired level of essential quality has been designed into the device. Also, the appropriate design of manufacturing processes, including validation, where appropriate, is needed to assure that production can achieve the level of quality designed into the device.
 

The Medical Device Directive (MDD) - The Essential Requirements and Design Control Impact

This is the most important section of the MDD, because the essential requirements are the legal requirements that must be met by the end of the original transition period. (June 1, 1998 –You may recall from an earlier article that this was the end of the one-year grace period that FDA gave industry to establish design controls, and to be formally judged by regulatory bodies for same.)

These requirements are divided into six general requirements and eight design and construction requirements. These design requirements are basically verified using testing and other evaluation techniques and, in some cases, require subsequent design validation to meet user needs and intended use.


The requirements are:

  • The devices must be designed and manufactured in such a way that, when used under the conditions and purposes intended, they will not compromise the health or safety of patients, users or other personnel – verification and subsequent validation
  • Safety principles must be utilized for the design and construction, and they should include state of the art technologies - verification
  • The devices must meet all claimed performance criteria - verification
  • The devices must continue to function as intended, without compromising safety or health, when subjected to normal conditions of use - validation
  • The devices must not be adversely affected during defined transport and storage conditions - verification
  • Any undesirable side effects must constitute an acceptable risk when weighed against intended performance - validation

For example, these detailed requirements define a variety of performance criteria to be met (as verifiably applicable and, in some cases, requiring subsequent validation), including:

  • Chemical, physical and biological properties - verification
  • Infection and microbial contamination - verification
  • Construction and environmental properties - verification
  • Properties for devices with a measuring function - verification
  • Protection against radiation – verification and subsequent validation
  • Protection against electrical, mechanical, thermal risks, energy supplies or energy substances – verification and subsequent validation
  • Labeling requirements and instructions for use – verification and subsequent validation
  • If applicable, demonstration of conformance with essential requirements based on clinical data - validation
     

Software Validation

Software is verified and reviewed vs. the software specifications during ongoing development of the device design. When a "final" prototype is available, the software and hardware are validated to make certain specified user needs for the device and process capability are met.

Before testing the software in actual use, the detailed code should be visually reviewed and verified vs. essential requirements.

The validation process is planned and executed such that all relevant elements of the software and hardware are exercised and evaluated. The testing of software usually involves use of an emulator and should include testing of the software in the finished device.

Testing includes normal operation of the complete device, and this phase of the validation program may be completed first to be certain that the device meets fundamental performance, safety and labeling specifications. Concurrently or afterward, the combined system of hardware and software should be challenged with abnormal inputs and conditions, and the finished device is summarily validated, meaning that the software becomes part of the total functionality and performance of the device.

As appropriate, these inputs and conditions include such items as:

  • Operator errors
  • Induced failure of sensors and cables or other interconnects
  • Induced failure of output equipment
  • Exposure to static electricity
  • Power loss and restart
  • Simultaneous inputs or interrupts
  • Deliberate application of none, low, high, positive, negative and extremely high input values

The results of the software and combined device system validation are included in the design reviews and subsequent issues of the DHF.


Labeling Verification and Subsequent Validation Impact

During verification, the complete device is put into effect such that all labeling, displays and outputs are generated, reviewed and the results documented. During the verification, all displayed prompts and instructions are checked against the manufacturer's and FDA's labeling requirements and the operator manual. The format and content of the labeling (e.g. instructions for use) can have a substantial effect on usability and on finished device safety and effectiveness. Design validation activities often include evaluation of labels and labeling to ensure that they are understandable and useful for typical gradated errors.

Displayed text should be short and to the point so that human verification competency can be established and documented in an empirical manner.

During verification, all prompts and instructions should be followed exactly by the device test or other operators, and such action should result in correct operation of the device. Prompts and instructions should appropriately match the instructions in the operator's manual.

The overall device specifications usually have requirements that cover user/operator error prevention and control. Along with operator training, such errors are controlled by:

  • Instruction manuals
  • Clear, concise and accurate device labels
  • Display of adequate prompts and correct instructions
  • Status (history) reports
  • Exclusion of certain erroneous inputs or actions
  • Human factors design

A checklist or matrix may be used to aid in the review and verification of labeling.


The Medical Device Directive: Clinical Evaluation and the Design Validation Impact

Clauses in the MDD provide details on the requirements pertinent to devices intended for clinical investigation.


These mandates include the compilation of data, statements of confidentiality and the credible documentation of:

  • Product identification information and data
  • Clinical investigation plan, including the purpose, grounds, scope and number of devices
  • The names of the medical practitioners and/or authorized individuals associated with the investigation
  • The identification of the institution responsible for the investigation
  • The location, commencement dates and scheduled duration of the investigation
  • A statement that the investigation will be performed in accordance with the ethical requirements defined in the Helsinki Declaration
  • A statement that the device conforms to the essential requirements in Annex I and that every precautionhas been taken to protect the health and safety of patients

Whereas verification is a detailed examination of aspects of a design at various stages in the development, design validation is a cumulative summation of all efforts to assure that the design will conform with user needs and intended use(s), given expected variations in components, materials, manufacturing processes and the use environment.


Design Validation

Planning for validation should begin early in the design process. The performance characteristics that are to be assessed should be identified, and validation methods and acceptance criteria should be established. For complex designs, a schedule of validation activities and organizational or individual responsibilities will facilitate maintaining control over the process. The validation plan should be reviewed for appropriateness, completeness and to ensure that user needs and intended uses are addressed.

Validation may expose deficiencies in the original assumptions concerning user needs and intended uses. A formal review process should be used to resolve any such deficiencies. As with verification, the perception of a deficiency might be judged to be insignificant or erroneous, or a corrective action may be required.

Many medical devices do not require clinical trials. However, all devices require clinical evaluation and should be tested in the actual or simulated use environment as a part of validation. This testing should involve devices that are manufactured using the same methods and procedures expected to be used for ongoing production. While testing is always a part of validation, additional validation methods are often used in conjunction with testing, including analysis and inspection methods, compilation of relevant scientific literature, provision of historical evidence that similar designs and/or materials are clinically safe and full clinical investigations or clinical trials.

Historically, some manufacturers have used their best assembly workers or skilled lab technicians to fabricate test articles, but this practice can obscure problems in the manufacturing process. It may be beneficial to ask the best workers to evaluate and critique the manufacturing process by trying it out, but pilot productionshould simulate as closely as possible the actual manufacturing conditions.

Validation should also address product packaging and labeling. These components of the design may have significant human factor implications, and may affect product performance in unexpected ways. For example, packaging materials have been known to cause electrostatic discharge (ESD) failures in electronic devices. If the unit under test is delivered to the test site in the test engineer's briefcase, the packaging problem may not become evident until after release to market.

Validation should include simulation of the expected environmental conditions such as temperature, humidity, shock and vibration, corrosive atmospheres, etc. For some classes of device, environmental stresses encountered during shipment and installation far exceed those encountered during actual use, and should be addressed during validation.

Particular care should be taken to distinguish among customers, users and patients to ensure that validation addresses the needs of all relevant parties. For a consumer device, the customer, user and patient may all be the same person. At the other extreme, the person who buys the device may be different from the person who routinely uses it on patients in a clinical setting. Hospital administrators, biomedical engineers, health insurance underwriters, physicians, nurses, medical technicians and patients have distinct and sometimes competing needs with respect to a device design.

Validation is a compilation of the results of all validation activities. For a complex design, the detailed results may be contained in a variety of separate documents and summarized in a validation report. Supporting information should be explicitly referenced in the validation report and either included as an appendix or available in the design history file.


Conclusion

Throughout product and process design controls, FDA expects manufacturers to delineate and then manage risk. The human factors focus for risk management should be aligned with use-related errors. Risk management of use-related errors involves identifying and describing use scenarios that result in hazards, assessing risk, introducing user interface design changes that eliminate or mitigate risk and verifying that design changes do mitigate risk situations.


Ultimately, risk control measures are used to prevent new hazards from being introduced. Validating that the final device can be used safely and effectively is the culmination of risk mitigation throughout the design process, with the user needs and intended use being met.

FDA anticipates that human factors should be addressed by the designing company to be commensurate with a device's inherent risk: the greater the risk associated with error, the more human factors effort is warranted. The manufacturer then must decide what human factors work together to satisfy that what they have done is sufficient to ensure that their device can be used safely and effectively.


John Gagliardi has had success over the past 40 years in the Medical Device and Pharmaceutical industries because of his practical approach to process-orientation and business. He has been actively involved in research and development, quality assurance, training, operations, process architecture, FDA inspections and regulatory affairs. John specializes in building systems in a compliant and business-ready manner. Email John at This email address is being protected from spambots. You need JavaScript enabled to view it..

MidWest Process Innovation, LLC
513-573-0085 (phone)
www.midwestprocessinnovation.com
 

Looking Back at Design Controls

Part I: A Business-Critical Tool. (BONEZONE Fall 2009) A systematic and organized approach is the goal in taking product-feasible ideas and bringing a finished device to market. The synergistic and harmonized relationship of all “players” involved is the key to commercializing devices, and meeting customer needs and regulatory requirements.

Part II: The Requirement for Planning. (BONEZONE Winter 2009) Having a written plan is not only a regulatory requirement, but also the “point of order” that is necessary to stay on course and control the destiny of an idea.

Part III: Inputs and Outputs in Synergy. (BONEZONE Spring 2010) Development of a solid foundation of requirements is the single most important design control activity.

Part IV: Design Review. (BONEZONE Summer 2010) Design reviews may have an internal and external focus. The internal centers on design feasibility and produceability with respect to manufacturing and support capabilities. The external centers upon user requirements.

Part V: Verification and Validation. (Current issue) Design verification is almost always done vs. specifications. Therefore, to control the specifications and increase the probability of achieving desired safety and performance characteristics, device, software, labeling, packaging and any other specifications should be complete and thoroughly reviewed before development commences.

Coming in the Next Issue:

Part VI: The Wrap-up - Transfer, Changes, the History File. (BONEZONE Winter 2010)
 

Review these articles online by visiting the BONEZONE journal archives at www.orthoworld.com.