Animals > Veterinary Biologics > Guidelines / Forms 

Veterinary Biologics Guideline 3.19E

Guidelines for Licensing of Veterinary Diagnostic Test Kits in Canada

TABLE OF CONTENTS

INTRODUCTION

Under the Canadian Health of Animals Act (64.(1)(s)) and Regulations⁽¹⁾, diagnostic test kits must be demonstrated by the manufacturer to be pure, potent, safe, and effective according to the label claim. To assist manufacturers in fulfilling the requirements described in the Regulations, the following guidelines have been prepared by the Veterinary Biologics Section (VBS) and the Biologics Evaluation Laboratory (BEL). Their purpose is to standardize the information submitted by a manufacturer in support of an application for a Veterinary Biologics Product Licence or a Permit to Import a Veterinary Biologic for a diagnostic test kit.

Correspondence and documentation supporting an application for licensure should be sent to the following address:

Veterinary Biologics Section
Canadian Food Inspection Agency
2 Constellation Crescent (Floor 8)
Ottawa, Ontario K1A 0Y9
Telephone: 613-221-7566
Facsimile: 613-228-6612

GENERAL REQUIREMENTS

In support of an application to license a diagnostic test kit, the manufacturer must submit the following documents and data for review to VBS: 1) relevant publications and references summarizing the scientific veterinary medical rationale for using a kit to diagnose disease in the target animal species, 2) an Outline of Production, 3) data from the testing of pre-licensing serial(s) demonstrating the minimum detectable levels of antigen or antibody, diagnostic thresholds, reproducibility, stability, effectiveness of the kit according to the label claim, and suitability, 4) results of the potency test of pre-licensing serial(s), 5) results of master seed and cell line certification and quality control testing, 6) Special Outline(s) describing the reference panels of samples used in evaluating the kit, and 7) a draft of the packaging and labels for each component of the kit, including a product monograph.

During the review process, a manufacturer may also be required by VBS to submit samples of master seeds and cell lines to BEL for confirmatory testing. Pre-licensing serial(s) of the kit, together with panels of reference test samples, positive, negative and other control reagents, may also be tested by BEL in order to verify the absence of technical problems when using a kit under laboratory conditions. During pre-license testing by BEL, a kit may be evaluated using reference test samples that are not included in the manufacturer's panel. Prior to the release of a kit for sale, results of a manufacturer's serial release tests must be submitted to VBS for approval. Also, samples of each new serial and the relevant panels of reference test samples must be submitted to BEL. Each year, a specified proportion of serials will be tested by BEL prior to release for sale by VBS.

Because of the large number of diagnostic test kits imported from the United States, these guidelines take into account the procedures and formats specified in Title 9, Code of Federal Regulations (9CFR) Parts (104, 113.3 (b)(7), 113.4, 113.27 (b), 113.50, 113.51, 113.52, 113.53, 113.55, 114.9 (f))⁽²⁾. For the importation of kits already licensed by the United States Department of Agriculture (USDA), the material reviewed by VBS should include photocopies of the following additional documents: 1) USDA-APHIS Veterinary Biologics Establishment Licence, 2) USDA-APHIS Veterinary Biological Product Licence, 3) Veterinary Biologics Information (Form CFIA 1503) (designates a Canadian importer), 4) Application to Import Veterinary Biologics into Canada (Form CFIA 1493), and 5) APHIS Form 2008 showing the disposition of the USDA for each serial. Generally, kits already licensed and tested by the USDA will not be re-tested by BEL. However, U.S. manufacturers may be required by VBS to submit samples to BEL, e.g. to verify that the kit can diagnose Canadian strains of the infectious agents of concern.

Obtaining a Veterinary Biologics Product Licence or a Permit to Import a Veterinary Biologic for a diagnostic test kit does not necessarily qualify a product for use in a Canadian Food Inspection Agency federal disease control program. Approval for use in such a program is made by the Science Advisory and Management Division (SAMD) and the Animal Health Division (AHD), and may necessitate the manufacturer submitting additional data, kits and reference test samples in order to verify: 1) the sensitivity, specificity and reproducibility of the test results from a kit; 2) the level of reactivity of the positive, negative and other control reagents, and the reference panel of test samples to be used in serial release; and 3) the absence of technical problems in using the kit under laboratory conditions. The specific material and information required for a licensed kit to be accepted and used in a "program" will be negotiated with each manufacturer on a case-by-case basis by SAMD and AHD.

The documents prepared in support of licensing a kit should follow the recommended format described in these guidelines. For some kits, the manufacturer may need to include additional headings. Copies of the original records, documenting the development and evaluation of the kit, should be maintained on the manufacturer's licensed premises and be available for inspection by VBS at all times. Specific details of the material to be submitted by the manufacturer are described below.

I. BACKGROUND AND RATIONALE FOR KIT USE

Provide a summary of the scientific veterinary medical rationale for using the kit to diagnose disease in the target animal species. Include the relevant publications and references with the submission.

II. OUTLINE OF PRODUCTION

The Outline of Production describes in detail the methods of manufacture to be used in the preparation, standardization, and testing of the reagents and equipment making up a kit. The recommended format and content of the Outline of Production for diagnostic test kits are described in Appendix I.

III. PRE-LICENSE DATA REQUIREMENTS

In support of an application to license a diagnostic test kit, a manufacturer should produce pre-licensing serial(s) to generate data demonstrating product potency, ability to detect antigen or antibody, diagnostic threshold, reproducibility, stability, efficacy according to the label claim, and suitability for use. These serials should be at a potency or level of reactivity equal to that proposed in the Outline of Production at the release of a serial for sale. The results of the potency test(s) of the pre-licensing serial(s), must be submitted to VBS as a condition for their use in field testing by laboratories outside the manufacturer's establishment. The method(s) of test interpretation (positive, negative, suspicious, etc.) should be the same as those described in both the Outline of Production and product monograph. Data submitted to VBS from these experiments and tests, should include a description of the serial used in the testing (date of production, date of use, trade name, scientific product name, pre-licensing serial number, and potency test results). During the review of this data, samples of master seeds and cell lines (when applicable), and pre-licensing serial(s) together with panels of reference test samples, positive, negative and other control reagents may be required by VBS to be submitted to BEL for testing.

A. Pre-license Testing by BEL

Samples should be submitted to BEL as follows.

1) Master Seeds - The manufacturer is responsible for certification and quality control testing of master seeds or cell lines. The results of such testing should be submitted to VBS. Samples of any master or working seed stocks of the microorganism or cell line used in the production of the kit should be submitted to BEL. BEL conducts independent tests to monitor the manufacturer's quality control and confirm the certification of the master seed. These tests may not be the same as those of the manufacturer. If the master seeds and cell lines have previously been approved by the USDA Center for Veterinary Biologics (CVB), in a manner accepted by VBS/BEL, there may be no need to conduct further confirmatory tests, as long as they are not manipulated in any way before being used in the production of the kit.

2) Pre-licensing serial(s) - BEL will test the pre-licensing serials to verify the absence of technical problems in using a kit under laboratory conditions. For each pre-licensing serial(s), the manufacturer should supply BEL with samples of the kit, together with sufficient volumes of the panel of reference test samples for the potency test described in the Outline of Production, positive, negative and other control reagents. During pre-license testing, a kit may be evaluated by BEL using additional reference test samples that are not included in the manufacturer's panel. Thus, the number of kits and volume of reference test samples submitted to BEL by the manufacturer will be negotiated on a case-by-case basis. For the importation of kits already licensed, tested, and found satisfactory by the USDA-CVB, samples need not be submitted to BEL for re-testing. However, under special circumstances, U.S. manufacturers may be required by VBS to submit additional samples to BEL for testing.

B. Reference Panel Preparation

Various reference panels of test samples (serum, blood, leukocytes, etc.) are used to evaluate different aspects of a kit (potency, suitability, stability, etc.). The development of reference panels of test samples, and lists of their members and related information should be fully described in Special Outline(s) and submitted to VBS. The samples making up a panel should not be part of the kit. Addition or removal of test samples from a panel must be documented and approved by VBS which would necessitate changes in the Outline of Production and/or a Special Outline. The history of panel members required by VBS and the factors to be considered in their preparation are described in Appendix II.

C. Potency

Potency tests are performed by the manufacturer on each serial of a kit to ensure that the components are functioning as expected. Results from potency tests must be submitted to VBS as a condition for the release of the pre-licensing serial(s) for field testing, and as a condition for the release of each serial for sale. The tests are based on measuring the level of reactivity (titre of antibody or nanogram of antigen, etc.) of: 1) the kit positive control(s) (minimum reactivity), 2) the kit negative control(s) (maximum reactivity), 3) other control reagents, and 4) an independent reference panel of test samples (serum, blood, leukocytes, etc.) from animals classified as diseased or non-diseased. The test results for a serial are compared with minimum and maximum pass-fail criteria determined by the manufacturer and specified in the Outline of Production prior to licensing. To pass, all samples must fall within the specified range. One re-test is allowed.

There should be a sufficient number of relevant samples on a panel to detect changes in the potency (level of reactivity), and therefore the effectiveness of the various components of the kit. To establish the limits of their range of reactivity, each sample should be tested with a pre-licensing serial of the kit, multiple times, by more than one analyst, on different days. Addition or removal of test samples from a panel used to test the potency of a kit should be documented by the manufacturer and approved by VBS, and will necessitate changes in the Outline of Production and the Special Outline describing the history of the panel members.

D. Minimum Detectable Antigen Or Antibody

Describe the design and results of the experiments performed using a pre-licensing serial, to confirm the minimum antigen or antibody (expressed as a titre of antibody or nanogram of antigen, etc.) detectable by the kit. This should be established using test samples (antigen or antibody) made up of various concentrations. The preparation and method of determining the concentration of the antibody or antigen in the test samples (negative to high) should be described.

E. Diagnostic Threshold

Describe the design and results of the experiment performed using a pre-licensing serial, to confirm the diagnostic threshold, e.g. cut-off, and methods of interpreting the test. The diagnostic threshold of a kit should be determined using animals having experimental or natural infections of the disease, and animals that are determined to be disease negative by some recognized defined criteria. Samples used to determine the diagnostic threshold should be identified and described as indicated in Appendix II. For kits in which the method of interpretation is a quantitative outcome (optical density, etc.), receiver operating characteristic curves can be used to demonstrate the relationship between the estimates of sensitivity, specificity, and the cut-off values chosen as diagnostic thresholds^(3),(4),(5).

F. Reproducibility

Describe the design and results of the experiments performed using a pre-licensing serial, to confirm the level and consistency of reactivity of positive, weak positive, and negative samples. Samples used to test reproducibility of the kit should be identified and described as indicated in Appendix II. For kits having a quantitative outcome, with antigen or antibody fixed to a solid phase, samples of the control reagents should be tested a minimum of four times on at least five different solid phase units. For each sample tested, the manufacturer should report the number of replications, the mean, the standard deviation, the coefficient of variation, and the 95% confidence limits.

G. Stability

Describe the design and results of the experiments performed using a pre-license experimental serial, to establish the stability and expiration date of a kit serial including: 1) the number of days over which stability is tested, 2) the temperatures (2-4^ºC and 22-37ºC) at which stability is tested, and 3) the test statistics used to demonstrate that there is no significant difference between the test days and temperatures. Samples used to test product stability should be identified and described as indicated in Appendix II. The expiration date of each kit serial should be computed from the date of the start of the potency test for that serial. Each serial should be tested for potency by the manufacturer and submitted to VBS at release and at the estimated expiration date, until a valid stability record has been established. The expiration date for each component of a kit should be based on that of the shortest-dated component.

H. Efficacy

Efficacy data must be established for each label claim made by the manufacturer including the following: a) clinical form of a disease, b) strain of the organism, c) type of sample (blood, serum, leukocytes, etc.) used to perform the test, d) species (bovine, porcine, canine, feline, equine, etc.), and e) age group. The process of determining the efficacy of a kit involves both laboratory and field testing. Samples (serum, blood, leukocytes, etc.) from the target animal species, are collected and classified into four categories: those having or not having the disease in question according to some recognized reference standard procedure, and those testing positive or negative using a pre-licensing serial of the kit under evaluation.

Studies measuring the sensitivity of the kit should include experimental data demonstrating the ability of the kit to detect: a) the disease in experimentally infected animals, e.g. a temporal study demonstrating the kits ability to detect antibody over so many days post experimental infection, b) multiple relevant strains of the infectious disease agent, and c) disease samples collected in the field from multiple geographical locations. Studies measuring the specificity of the kit should include experimental data demonstrating the ability of the kit to detect disease in animals in the laboratory and the field that are definitely negative by some defined criteria (culture negative, specific pathogen free, etc.). Kits for the detection of antibody should be tested against immune sera to related agents or antigens to detect cross reactivity. The kit should also be tested against sera from animals that have received commonly used immunizing products. Kits for the detection of antigens should be tested against other related antigens to detect false positive reactions.

1) Test Materials - The pre-licensing serial used in efficacy testing should be at a potency level equal to that proposed in the Outline of Production at the release of a kit serial for sale. Reference samples used to test product efficacy should be identified and described as indicated in Appendix II. The reference standard against which the kit results are compared, may be based on microbiology, pathology, clinical signs, or banks of other tests. It may be pathognomonic (absolute predictor of the disease state, i.e. culture of the microorganism) or surrogate (detects secondary changes due to disease, i.e. antibody)⁶. The data submitted should include a description of the reagents and procedures for testing a sample with the reference standard, including the pass-fail criteria.

2) Statistics Used to Measure Kit Efficacy - The results of efficacy testing of the kit should include the following: a) a list of the demographic characteristics, other relevant characteristics, kit test results, reference standard results for each animal participating in the study, and b) for each farm and geographical location from which a reasonable sample of data have been collected and for all samples together: fourfold tables, estimates of sensitivity, specificity, proportion of the samples testing in the suspicious category (if applicable), positive predictive value and negative predictive value, 95% confidence limits for estimating a proportion based on a normal approximation to the binomial distribution, kappa statistic, standard error for the kappa statistic, odds ratio, and/or McNemar's chi-square test^(6),(7).

The selection of the reference standard for comparison with the kit determines the label claim that can be made by the manufacturer and the type of statistics (sensitivity, specificity, extent of agreement, 95% confidence limits, etc.) that should be used to describe and support the efficacy claim in the product monograph:

A description of the reference standard used for comparison with the kit and a summary table of the statistics generated by the manufacturer in support of the effectiveness of the kit should be included in the product monograph for products sold in Canada. For example:

^a Kit results / Reference Test results.
^b Data from a representative study.

3) Sample Size - The sample size used to establish the sensitivity, specificity and the extent of agreement between the kit and the reference standard is left to the judgement of the manufacturer. To assist the consumer in determining the extent to which the effectiveness of the kit has been measured, the manufacturer will be asked to calculate the 95% confidence limits for the sensitivity and specificity, and/or an estimate of the standard error for the kappa statistic. These should be included in the product monograph (see above example). It should be noted that small sample sizes lead to wide confidence limits and an inability to precisely identify the sensitivity or specificity of the kit.

4) Blinding - Describe the methods used to blind the personnel carrying out the kit and reference standard procedures on the test samples. This should include a list of the various aspects of the test procedure and personnel which are blinded.

I. Suitability: Kit Performance At An Independent Laboratory

Describe the design and results of the experiments organized by the manufacturer using a pre-licensing serial, to evaluate the kit under typical laboratory settings outside the manufacturer's establishment. Each participating laboratory should be experienced in using the proposed technology, and independent from the manufacturer who has developed the kit. Samples used to test the suitability of the kit should be identified and described as indicated in Appendix II. Such testing should only be carried out following the approval of the Outline of Production and other pre-license data by VBS.

IV. SERIAL RELEASE TESTING

In the Outline of Production (Appendix I) describe the methods and number of kits from each new serial that will be collected and stored by the manufacturer, and submitted to BEL for testing prior to serial release.

A. Potency Testing by the Manufacturer

To release a kit serial for sale, the manufacturer must submit potency test results to VBS using the same methods and reference panel of test samples, and pass-fail criteria described in the Outline of Production (Appendix I). To pass, all samples must fall within the specified range. One re-test is allowed.

B. Testing by BEL

For each new serial produced, a minimum of two kits, together with sufficient volumes of the panel of reference test samples for the potency test described in the Outline of Production, positive, negative and other control reagents should be submitted to BEL for serial release testing. The manufacturer should keep an equivalent number of kits, pending a request from VBS/BEL to submit additional samples. For a proportion of the serials produced each year, BEL will test to verify the absence of technical problems in using the kit under laboratory conditions. For the importation of kits already licensed, tested, and found satisfactory by the USDA, samples need not be submitted to BEL for re-testing. However, under special circumstances, U.S. manufacturers may be required by VBS to submit additional samples to BEL for testing.

If a kit is to be used in a Canadian Food Inspection Agency federal disease control program the manufacturer may be required to submit additional kits and reference test samples for serial release. The specific material and information required for a "program" kit will be negotiated with each manufacturer on a case-by-case basis by SAMD and AHD.

V. LABEL REQUIREMENTS

A draft of the packaging and labels of each component of the kit, and product monograph should be submitted to VBS for approval. These labels should include: 1) the intended use, 2) any precautions, 3) the Canadian Food Inspection Agency or United States Department Agriculture Veterinary Biologics Establishment License Number, 4) the expiration date, 5) the serial number, and 6) the statements "For Veterinary Use Only" and "For In Vitro Use Only" or equivalent. The tradename of the kit must not be more prominent than the assigned name (see Appendix I).

The product monograph should include a brief description of the kit including: 1) the principles of the test, 2) whether it is to be used to detect antibody or antigen, 3) the tissue samples used for testing (blood, serum, leukocytes, etc.), 4) the equipment and reagents making up the kit, 5) a description of the test procedures and methods of interpretations, 6) a hypothetical example of the results and interpretation, 7) a list of any precautions and test limitations, and 8) a table listing the results of the studies (see  part III  H 2(c)) performed by the manufacturer and submitted to VBS, to measure the effectiveness of the kit in detecting diseased animals. If the product is to be sold in the province of Québec the labelling and product monograph should be in French, or in both French and English, in order to comply with provincial legislation.

VI. LABORATORY BIOSAFETY

All aspects of the development and evaluation of the kit must meet the standards and regulations for laboratory safety described in the most current edition of the Laboratory Biosafety Guidelines (Public Health Agency of Canada and Medical Research Council of Canada), relevant federal, provincial legislation and local safety authorities. The guidelines do not reference all animal pathogens. However, the manufacturer is responsible for ensuring that the appropriate containment level is used to handle any live organisms used in developing and evaluating the kit. To obtain Laboratory Biosafety Guidelines and Material Safety Data Sheets contact:

Office of Biosafety
Laboratory Centre for Disease Control
Public Health Agency of Canada
Ottawa, Ontario K1A 0L2
Canada

VII. USING ANIMALS IN KIT DEVELOPMENT AND EVALUATION

All aspects involved in the proposed use of animals in the development and evaluation of the kit must meet the standards and guidelines for the care and maintenance of experimental animals as described by the Canadian Council on Animal Care, relevant provincial legislation and local animal care authorities. To obtain the Guide to the Care and Use of Experimental Animals: Volume I and II contact:

Canadian Council on Animal Care
1105 - 151 Slater Street
Ottawa, Ontario K1P 5H3
Canada

APPENDIX I - Outline of Production for Diagnostic Test Kits

Format of the Outline of Production
To assist the manufacturer, a recommended format for the preparation of an Outline of Production is described below. For some kits, the manufacturer may need to include additional headings.

The title page of an Outline of Production should contain the following (see attached example in this Appendix): a) the name and address of the manufacturer, b) the scientific name, and if known at the time of submission, the trade name of the kit, c) the Canadian Food Inspection Agency Veterinary Biologics Establishment Licence Number or the USDA-APHIS Veterinary Biologics Establishment License Number, and d) the preparation date of the Outline of Production.

On each page of the Outline of Production the manufacturer should include (see attached example in this Appendix): a) the scientific name, and if known, the trade name of the kit, b) the Canadian Food Inspection Agency Veterinary Biologics Establishment Licence Number or the USDA-APHIS Veterinary Biologics Establishment License Number, c) the date the Outline of Production was prepared, d) the date a page was superseded if such was the case, and e) the signature of the manufacturer's representative. Within the Outline of Production, number the pages consecutively.

All changes in production and testing procedures must be documented and submitted for approval. Minor changes can be documented by submitting the amended Outline of Production pages with an attached SUMMARY OF CHANGES (see attached example in this Appendix). Amended pages should be numbered the same as those being superseded. They should bear the date prepared, and the date of the pages being superseded. If one page is superseded by more than one page, the page number should be followed by letters. Major changes may necessitate a complete revision of the production outline, or designation as a new product. Proposals for major revisions should be submitted in advance to VBS for review and comment on the changes.

The Outline of Production should include the items listed below:

RECOMMENDED FORMAT FOR OUTLINE OF PRODUCTION TITLE PAGE:

RECOMMENDED FORMAT FOR OUTLINE OF PRODUCTION FOLLOWING THE TITLE PAGE:

RECOMMENDED FORMAT FOR SUMMARY OF CHANGES TO OUTLINE OF PRODUCTION:
The following is an example of the format for documenting amendments to an Outline of Production. Amendments are summarized on pages which are added to the end of the original Outline of Production.

APPENDIX II - Reference Panel Preparation

Various reference panels of test samples (serum, blood, leukocytes, etc.) are used to evaluate different aspects of a kit (potency, suitability, stability, etc.). The development of reference panels of test samples, lists of their members and related information should be fully described in Special Outline(s) and submitted to VBS. The samples making up a panel should not be part of the kit. Addition or removal of test samples from a panel must be documented and approved by VBS which would necessitate changes in the Outline of Production or the Special Outline. The history of panel members required by VBS and the factors to be considered in their preparation are described below.

1. Volume - A reference panel of test samples of sufficient volume should be available over a sufficient period of time to evaluate the consistency of a kit from serial to serial and through the end of stability testing. Samples from the reference panel should be available for testing on request by VBS or BEL.

2. Type - A separate reference panel of test samples should be prepared to evaluate each of the different types of samples (blood, serum, leukocytes, etc.) described in the label claim.

3. Identification - In a Special Outline, list the following information concerning the test samples in the reference panel: identification, type of sample, individual animal identification, source (geographical location including country and farm address), disease status of animal, specific method used to make the diagnosis (clinical signs, post mortem lesions, culture, serological tests, etc.), information on factors that could affect the status of the diagnostic test (i.e. vaccination status), information specifically relating to the label claim (i.e. age of the animal, species, clinical form of the disease, antibody titre), date sample collected, sample storage history, and date sample prepared for the reference panel. Each panel should be identified by a unique code and date of preparation.

4. Specificity - Reference panels should include samples produced from: a) experimentally infected animals and/or culture positive and/or serologically confirmed positive or negative animals, b) animals infected with the relevant strains of the infectious disease agent from different geographical locations, c) animals harbouring related agents or antigens, and d) animals that have received commonly used immunizing products.

5. Range of Reactivity - The reference panel should be made up of different samples that give strong positive, weak positive and negative reactions. The positive samples should consistently give a reaction above a pre-established positive diagnostic threshold, and should cover the full range of positive reactions described by the manufacturer in the product monograph. The negative samples should consistently give a reaction that falls below the pre-established negative threshold. Each sample should be tested using the kit, multiple times, by more than one analyst, on different days, to determine the limits of their range of reactivity.

6. Sample Size - For each type of sample (blood, serum, leukocytes, etc.) described in the label claim, the manufacturer should include a sufficient number of relevant samples to detect changes in the potency (level of reactivity) of the various components of the kit.

7. Adding or Removing a Test Sample from the Reference Panel - Describe the test procedures used to add, remove and re-test samples from the reference panel. The test procedures used to add samples to the panel should be those recognized by the veterinary medical community as providing the most definitive diagnosis for the disease at the time of licensing. They may be based on microbiology, pathology, clinical signs, or banks of other reference tests. They may be pathognomonic (absolute predictor of the disease state, ie. culture of the microorganism) or surrogate (detects secondary changes due to disease, i.e. antibody). The data submitted should include a description of the reagents and procedures for testing a sample, including the pass-fail criteria.

8. Method of Storing the Reference Panel - Describe the method used to store the panel members (frozen, lyophilized, etc.).

APPENDIX III - References

1. Canadian Food Inspection Agency, formerly, Agriculture and Agri-Food Canada. Summary of requirements and guidelines for veterinary biologics in Canada. Ottawa, Ontario: Veterinary Biologics Section, May 18, 1996.
2. United States Department of Agriculture. Code of Federal Regulations 9, Chapter I, Animal and Animal Products. Washington D.C,: Office of the Federal Registrar, National Archives and Records service, General Services Administration, 1995; 114.9.: 652-653.
3. Metz CE. Basic principles of ROC analysis. Seminars in Nuclear Medicine. 1978; VIII, 4 (October): 283-298.
4. Knapp RG, Miller MC. Defining Normality Using the Predictive Value Method. In: Clinical Epidemiology and Biostatistics. Malvern, Pennsylvania: Harwal Publishing Company, 1992: 55-56.
5. Kraemer HC. Assessment of 2 X 2 associations: generalization of signal-detection methodology. The American Statistician 1988; 42, 1: 37-49.
6. Martin SW, Meek AH, Willeberg P. Veterinary epidemiology, principles and methods. Ames, Iowa: Iowa State University Press. 1987: 62-78.
7. Fleiss JL. Statistical methods for rates and proportions, 2nd ed. In: Wiley Series in Probability and Mathematical Statistics. Toronto: John Wiley & Sons. 1981: 14.