Source: National Mastitis Council
Introduction and Scope: Teat disinfectant products used to prevent intramammary infections (IMIs) are important to the health of dairy production animals, the quality of the milk they produce, and the economic welfare of milk producers. Therefore, properly evaluating the efficacy and safety of these products is a responsibility that teat disinfectant manufacturers have to animals, consumers, and producers. The purpose of this document is to outline the basic requirements for proper evaluation of teat disinfectants in order to ensure scientific validity and consistent reporting, while at the same time allowing the researcher to have practical flexibility in the design and implementation of their study. The scope of this document is not intended to include all requirements that may be relevant to a specific study due to the location of the intended product market, but rather to outline what the National Mastitis Council has determined to be the basic components of a valid study design, and proper implementation and reporting.
1.0 Study Organization and Management:
1.1 Personnel: The study shall be implemented by qualified study participants whose roles are clearly defined in the study protocol and agreed upon prior to the initiation of the study. The qualifications and trainings of these participants shall be documented. The following criteria shall be met for each participant’s role;
1.1.1 The Sponsor is the individual, company, institution, or organization that takes responsibility for the initiation, management, and financing of the study. It is the responsibility of the Sponsor to recruit qualified participants/Investigators to implement the study in accordance with this document. In addition, the study participants chosen by the Sponsor shall be free from any conflicts of interest that may diminish confidence in data produced during the study.
1.1.2 Principal Investigator(s): This individual’s or the individuals’ role in the study is to ensure proper design and implementation of the study including sample size calculation, inclusion and exclusion criteria of both field sites and animals, randomization, blinding, data collection and integrity, data analysis, and reporting. This role shall be fulfilled by an individual or individuals who are considered experts in the field of mastitis and/or epidemiology.
1.1.3 Laboratory testing shall be conducted by a laboratory or laboratories with expertise in the identification of mastitis‐causing pathogens and diagnosis of IMIs. In addition, the laboratory shall have an established and documented quality system that ensures the accuracy of the test results and completeness of associated records. Whenever possible, the qualifications of the laboratory shall be documented by a certificate of accreditation to an applicable quality standard, and/or demonstrated by participation in a proficiency testing program.
1.1.4 Quality Assurance Officer (optional but recommended): This individual’s role is to confirm that the conduct of the study is in accordance with this document and the specific requirements defined within the study protocol. The individual assigned to this role shall be experienced in conducting mastitis and/or epidemiological research, and shall have training/experience in auditing field and research activities. In addition, this individual shall not be a participant in the implementation of the study itself. The frequency of compliance audits shall be defined in the protocol, but shall take place at the following minimum time points;
1.1.4.1 Upon finalization of the study protocol, but prior to implementation, to ensure that the protocol meets the requirements of this document and other specified requirements defined by the Sponsor and that all study participants are qualified to properly implement the study.
1.1.4.2 During the field phase of the study to ensure proper study implementation including randomization, inclusion and exclusion criteria of field sites and animals, blinding, sample and data collection and integrity, and test product quality.
1.1.4.3 Upon completion of the final report to ensure that data was qualified and all eligible data was used in the analysis, proper statistical techniques were employed, and the study report accurately represents the data and analysis.
1.2 Study Conduct: The study shall be conducted according to a protocol that is approved by both the Sponsor, Principal Investigator, and appropriate regulatory authority (if applicable) prior to the start of the study.
1.2.1 All on‐farm and laboratory personnel shall be trained on the study protocol and associated standard procedures used to execute the study.
1.2.2 Protocol Amendments: The approved study protocol may need to be modified as the study is implemented. Whenever possible these amendments shall be planned and well thought‐out. In these situations:
1.2.2.1 The amendment shall be described in writing.
1.2.2.2 There shall be agreement between the Sponsor, Principal Investigator, and regulatory authority (if applicable).
1.2.2.3 The amendment(s) shall become part of the study protocol.
1.2.2.4 The amendments shall be communicated to all study participants.
1.2.3 Protocol Deviations: On occasion, deviations from study procedures are not planned. In these situations:
1.2.3.1 The deviations shall be documented in writing.
1.2.3.2 They shall be evaluated by the Principal Investigator for their impact on the validity of study outcomes.
1.2.3.3 The Principal Investigator’s evaluation shall be documented.
1.2.3.4 The deviation and evaluation shall be reflected in the final study report
2.0 Protocol Requirements
2.1 Study Objectives: All objectives of the study shall be defined in the study protocol and must include the following;
2.1.1 To evaluate the efficacy of the Product A (the experimental product) in reducing the incidence of IMIs compared with either Product B (the positive control product), or no product (negative control).
2.1.1.1 A statement as to whether the comparison will be determined using the sum of all IMIs or whether IMIs caused by specific organisms or groups of organisms will be used for the comparison. If the comparison will be made using specific organisms or groups, these shall be defined in the protocol.
2.1.1.2 A statement as to what type of comparison will be made (i.e. non‐ inferiority, superiority, equivalence).
2.1.2 To evaluate the safety of applying the Product A (the experimental product) to animals’ teats compared with either Product B (the positive control product), or no product (negative control). NOTE: The objective to evaluate safety may be conducted in a separate study, but must be conducted.
2.2 Study Personnel: The individual(s) assigned to the roles of Sponsor, Principal Investigator, Laboratory Manager, and Quality Assurance Officer shall be documented in the study protocol along with a description of their responsibilities as they relate to the study planning, field conduct, laboratory testing, and reporting of study outcomes.
2.3 Study Products
2.3.1 The positive control product used shall be commercially available and registered with the local or corresponding regulatory agency, and shall be approved by the Principal Investigator. The Principal Investigator shall consider previously‐ conducted clinical and/or lab efficacy and safety studies when choosing the positive control product.
2.3.2 Description and use: Both the control and experimental products shall be described by their commercial product name or unique identifier, active ingredient(s) and its concentration, and intended use. During the study, the experimental product shall be used in accordance with its intended use in the market, and the control product shall be used as recommended by its manufacturer. The description of product use shall include, but is not limited to; mixing or dilution requirements, recommended application methods and apparatus (if required), contact duration, and time of application (pre‐ or post‐ milking).
2.3.3 Both the experimental and control products shall be obtained from a source whose procedures ensure that the product used is of the same character of the product that is, or will be, manufactured for commercial use. Records shall be available to document the experimental product manufacturing specifications (certificate of analysis). These specifications shall be the same as the product used for both the efficacy and safety investigations.
2.4 Study Design: Both split‐udder and split‐herd designs are considered acceptable as long as specific conditions are met to ensure that bias is not introduced to the evaluation. In order to achieve an acceptable level of statistical power and have realistic study designs that can be practically implemented in the commercial dairy industry, it is most common that each animal’s quarter is considered an experimental unit. The NMC allows each animal’s quarter to be considered an experimental unit as long as the specific conditions listed below are met. For all designs, external validity can be achieved by implementation of the study on more than one farm. The NMC allows a single farm to be used for the purpose of this evaluation in order to maintain a practical study size and encourage field testing of commercial products.
2.4.1 Design Type I ‐ Split‐udder: In a split‐udder design, the quarters within a study animal are randomly assigned to different study treatments. This design is statistically most desirable because it controls for animal‐level and pen‐level bias, since each animal receives both study treatments and the animals are co‐ mingled in housing areas and during milking. While the split‐udder design is more desirable statistically, implementation is often not practical in a commercial production setting, causing delays in milking procedures and misapplication of treatments. NOTE: For more practical implementation, the NMC allows diagonal teats to be consistently assigned a treatment since animal and pen‐level bias is prevented and this ensures that bias is not introduced due to the same treatment always being assigned to the front or rear teats.
2.4.2 Design Type II ‐ Split‐herd with animal‐level treatment allocation: In a split‐herd design with animal‐level treatment allocation, all the quarters on an animal receive the same study treatment. The animals are randomly allocated to each study group and are co‐mingled in housing areas and during milking. This design ensures a uniform environment and management practices across study groups, but does not control for animal‐level bias.
2.4.3 Design Type III ‐ Split‐herd with pen‐level treatment allocation: As an alternative to Type I and Type II designs, the NMC considers a split‐herd design with pen‐ level treatment allocation acceptable. This design, when properly implemented, is both statistically valid and practical to implement in a commercial setting. In this design, all the teats on all the animals within an animal pen receive an assigned study treatment. It is recognized that in this study design each animal pen would be traditionally considered an experimental unit. Therefore, when using this design, The NMC requires strict balancing of study pens based on animal‐level characteristics, along with a uniform housing environment and management protocols across study groups. This ensures there is no pen‐level bias introduced to the evaluation and that the study treatment is the only difference between the pens. While this design requires initial coordination to balance study pens, milking procedures can generally remain uninterrupted and there is less risk of misapplication of treatments. The NMC considers these benefits important to the final evaluation of the study outcomes.
2.5 Sample Size Calculation: The target sample size shall be estimated based on the risk of type II error (desired study power), the confidence level chosen, and the incidence rate of new IMIs expected in the positive control group.
2.5.1 The minimum criteria below shall be used to calculate the target sample size:
2.5.1.1 Type II error shall be no less than 80%
2.5.1.2 Confidence level shall be no less than 95%
2.5.2 If implementing a non‐inferiority study design, the margin of non‐inferiority (difference in proportion of new IMIs for the control and experimental products) shall also be used in the estimation and the following minimum criteria shall be met (Ceballos‐Marquez et al. 2013):
2.5.2.1 The control product have a documented efficacy of 70% or greater compared to a negative control
2.5.2.2 Non‐inferiority margin shall be no less than 0.30
NOTE: The actual power of the study will be dependent on the incidence rate of new IMIs observed in the study. To ensure a successful study conduct, it is important for the incidence rate to be monitored and the study duration to be adjusted to meet minimum power requirements. Therefore, a midpoint evaluation of new IMI is justified and should be conducted according to a predetermined schedule.
2.6 Blinding Methods:
2.6.1 Since it is common that products can be visually identified as being different, blinding field personnel is often not possible. However, product containers on farm shall not be identified with a commercial label, but rather with non‐ descript labels (e.g. Product A, Product B).
2.6.2 Laboratory personnel shall be blinded to the identity of the study group a given sample was collected from. This blinding requirement applies to anyone having an influence on the testing procedure and/or final reported test result.
2.7 Study Site: The inclusion and exclusion criteria for the study site(s), including geographic location and animal housing/management, shall be defined such that;
2.7.1 Study animals and their housing are managed uniformly and humanely to ensure the study is conducted in an ethical and unbiased manner.
2.7.2 A system is established to ensure unique identification of animals in order to facilitate accurate treatment allocation and data capture.
2.7.3 Health records, including mastitis history, are adequately maintained in order to accurately enroll animals into the study.
2.7.4 Milking equipment is functioning properly as defined by ISO 5707:2007.
2.7.5 Clinical mastitis is detected effectively and uniformly across study pens, and the procedure for doing so is documented. Milking staff shall have been effectively trained in the procedure.
2.8 Animal Selection and Quarter Eligibility: The inclusion and exclusion criteria for animals and individual quarters shall be defined to ensure at least the following criteria are met;
2.8.1 The target population shall be defined, including lactation and lactation stage.
2.8.2 Animals known to be unhealthy, or show signs or symptoms of potentially having a disease, including mastitis, or abnormal teats shall not be enrolled.
2.8.3 An enrolled animal must not have received antibiotics or anti‐inflammatory products within 14 days prior to enrollment.
2.8.4 An animal or quarter must be able to complete enough data collection events to provide meaningful data for evaluation of study outcomes.
2.8.5 An enrolled quarter must not have been diagnosed with an IMI within the past 30 days before enrollment.
2.9 Randomization and Balancing of Study Groups: Proper allocation of experimental units to avoid introducing bias into the study is critical to obtaining valid study outcomes. The same is true for balancing study groups when using Design Type III. The procedures used to allocate experimental units and balance study groups shall be described in both the protocol and final report. The following criteria shall be met for each Study Design Type:
2.9.1 Design Type I: Quarters shall be allocated to study groups randomly or using a diagonal teat allocation system.
2.9.2 Design Type II: Animals shall be allocated randomly.
2.9.3 Design Type III: Pens shall be balanced with random allocation of animals thereafter.
2.9.3.1 Initial Balancing: The following pen‐level characteristics must not be different (α = 0.05) following the completion of balancing study groups. a) Lactation stage (days in milk) b) Lactation group (1, 2, 3+) c) Milk production d) Infection status based on the outcome groups defined in the study objectives section of the protocol (see section 2.1b above).
2.9.3.2 Once initial balancing meets the criteria defined above, animals shall be randomly allocated to each group.
2.10 Defining a New Intramammary Infection: Proper definition of new IMIs is important so that the study outcomes reflect the ability of a product to prevent mastitis and are not skewed by environmental contamination or chronic infections. The following criteria shall be used to define an IMI and new IMI:
2.10.1 An IMI is detected by confirming the presence of a mastitis‐causing pathogen in a milk sample. A reasonable balance of sensitivity and specificity is obtained from a single quarter milk sample when the presence of ≥ 1 CFU is detected in 0.01 ml (10µl) of milk (NMC Factsheet: Interpreting Bacteriological Culture Results to Diagnose Bovine Intramammary Infection, 2012).
2.11 Detection of Intramammary Infections: In order to efficiently and accurately measure the incidence rates of new IMIs in the study groups, it is important that effective detection procedures be uniformly applied across all milkings and study groups, and proper techniques are used for sample collection.
2.11.1 Clinical mastitis: Abnormal milk, and/or inflammation or redness of a quarter are signs of a clinical mastitis event. Prior to each milking, each quarter shall be observed for these signs by visual inspection of each quarter and the milk obtained during fore‐striping.
NOTE: A strip cup or other dark surface will aid in detecting abnormal milk and will increase uniformity of detection between milking personnel.
2.11.2 Sub‐clinical mastitis: New IMIs in quarters that do not show obvious clinical symptoms must be detected proactively by collecting quarter milk samples at defined intervals and testing them for the presence of mastitis pathogens. Approaches to detecting sub‐clinical are described as follows:
2.11.2.1 Direct pathogen identification and diagnosis using a validated diagnostic test method.
2.11.2.2 Use of somatic cell count (SCC) prescreening before diagnosis: A less expensive and valid way to detect IMIs is to pre‐screen each quarter milk sample for elevated SCC, followed by bacteriologic culture on the samples with elevated SCC (e.g. > 100,000 cells/ml for heifers and >200,000 cells/ml for cows) (Ceballos‐Marquez et al. 2013).
2.11.3 Sample collection procedure and interval:
2.11.3.1 Milk samples shall be collected aseptically from each individual quarter. Samples shall only be collected after adequately fore‐ stripping or milking has been conducted.
2.11.3.2 The interval for sample collection shall effectively detect new subclinical IMIs. This interval must be justified by the principal investigator.
2.12 Teat condition monitoring: Observations of both teat skin and teat ends shall be conducted using an established scoring system (Mein et al., 2001).
2.12.1 At a minimum, observations shall be made at the following time points in order to evaluate both immediate and long‐term effects on teat health:
2.12.1.1 Prior to the start of the study to obtain a reference
2.12.1.2 At approximately two weeks after the start of the study to ensure there are no immediate effects.
2.12.1.3 At the conclusion of the study.
3.0 Laboratory Testing
3.1 Identification of mastitis pathogens: The laboratory shall culture a pre‐defined volume of well mixed milk from each sample and identification shall be conducted using standard methodologies described in the current version of the National Mastitis Council Laboratory Handbook. Alternative test methodologies may be used if they are validated and shown to accurately identify the pathogens or pathogen groups defined in the objectives section of the protocol.
3.2 The SCC measurement shall be conducted using a validated test method (e.g. Fossomatic, DeLaval Cell Counter).
4.0 Statistical Analysis: The analysis of study data shall be conducted by an investigator with expertise in analyzing generalized linear mixed models and controlling for repeated measures and correlation of events within study animals.
4.1 Time at risk: Animals may enter and leave the study at various times; therefore not all quarters are equally at risk for becoming infected. Therefore, the time‐at‐risk of each quarter shall be accounted for within the analysis.
4.2 Accounting for correlations within study animal: Host‐level factors influence the likelihood of an animal developing a new IMI, and this varies from animal to animal. Correlation within study animal shall be accounted for within the analysis.
4.3 Covariates: Additional covariates shall be considered and their significance determined. At a minimum the following variables shall be considered:
4.3.1 Location of teat (front or hind)
4.3.2 Parity/lactation/age of animal
4.3.3 Lactation stage (days in milk)
4.3.4 Week of study
5.0 Records and Report: Although the NMC is not a regulatory authority and does not review study records of reports, it recognizes the need for completeness and accuracy in order for the study Sponsor, and industry as a whole, to be assured that the study was conducted according to the study protocol. A list of records that are generally part of a well‐documented study report includes the following:
- Approved protocol
- Approved protocol amendments
- Approved protocol deviations, including a description of the impact on study validity
- The standard operating procedures employed during the conduct of the study
- Personnel qualifications (Curriculum vitae)
- Personnel training records as they relate to the protocol, and associated on‐farm and laboratory standard procedures
- Laboratory qualifications (e.g. accreditation certificate, proficiency testing summary, or a third‐ party audit)
- Audit reports (if a Quality Assurance Officer is employed)
- Significant communications (e.g. emails, memos, faxes, meeting minutes, summaries of phone/video conferences)
- Justification and approval of the positive control product chosen
- Product certificates of analysis
- Product storage conditions
- Study site description demonstrating that the criteria for enrollment were met
- Animal records demonstrating eligibility for enrollment, including health, age/lactation, date of birth, and lactation stage at enrollment
- Milking equipment function checks
- Initial pen balancing data and analysis (if Design III was used)
- Clinical observations to detect mastitis events
- Sample collection and integrity information
- Teat condition scores
- Laboratory testing records and results demonstrating that standard testing procedures were followed
- The final data set used for the statistical analysis, including identification of animals lost to follow‐up
- The programing code used for the statistical analysis of the final data set
- A final summary report, approved by the Principal Investigator
6.0 References
1. National Mastitis Council (NMC). 1999. Laboratory handbook on bovine mastitis. 222 p.
2. Lopez-Benavides, M., I. Dohoo, D. Scholl, J. Middleton, and R. Perez. 2012. Interpreting bacteriological culture results to diagnose bovine intramammary infections. National Mastitis Council – Research Committee Report. Online. Available: http://nmconline.org/docs/InterpretBactResults.pdf
3. Mein, G. A., F. Neijenhuis, W. F. Morgan, D. J. Reinemann, J. E. Hillerton, J. R. Baines, I. Ohnstad, M. D. Rasmussen, L. Timms, J. S. Britt, N. Farnsworth, N. Cook, and T. Hemling. 2001. Evaluation of bovine teat condition in commercial dairy herds: 1. Non-infectious factors Pages 347-351 in 2nd International Symposium on Mastitis and Milk Quality, Vancouver, British Columbia.
4. GCP – VICH GL9
5. Ceballos-Marquez, A., T. Hemling, B. J. Rauch, M. Lopez-Benavides, and Y. H. Schukken. 2013. Noninferiority trial on the efficacy of premilking teat disinfectant against naturally occurring new intramammary infections using a novel 2-step diagnostic process. Journal of Dairy Science 96:1-12.
