Skip directly to search Skip directly to A to Z list Skip directly to site content
CDC Home

Guidelines for Evaluating Surveillance Systems

Prepared by Douglas N. Klaucke, M.D.* James W. Buehler, M.D.* Stephen B. Thacker, M.D.** R. Gibson Parrish, M.D.** Frederick L. Trowbridge, M.D.*** Ruth L. Berkelman, M.D.* and the Surveillance Coordination Group

INTRODUCTION

This document describes the evaluation of epidemiologic surveillance systems. Its purpose is to promote the best use of public health resources through the development of effective and efficient surveillance systems. It can serve as a guide for persons conducting their first evaluation and as a reference for those who are already familiar with the evaluation process.

Epidemiologic surveillance is the ongoing and systematic collection, analysis, and interpretation of health data in the process of describing and monitoring a health event. This information is used for planning, implementing, and evaluating public health interventions and programs. Surveillance data are used both to determine the need for public health action and to assess the effectiveness of programs.

The evaluation of surveillance systems should promote the best use of public health resources by ensuring that only important problems are under surveillance and that surveillance systems operate efficiently. Insofar as possible, the evaluation of surveillance systems should include recommendations for improving quality and efficiency, e.g., eliminating unnecessary duplication. Most importantly, an evaluation should assess whether a system is serving a useful public health function and is meeting the system's objectives.

Because surveillance systems vary widely in methodology, scope, and objectives, characteristics that are important to one system may be less important to another. Efforts to improve certain attributes--such as the ability of a system to detect a health event (sensitivity)--may detract from other attributes, such as simplicity or timeliness. Thus, the success of an individual surveillance system depends on the proper balance of characteristics, and the strength of an evaluation depends on the ability of the evaluator to assess these characteristics with respect to the system's requirements. In an effort to accommodate to these objectives, any approach to evaluation must be flexible. With this in mind, the guidelines that follow describe many measures that can be applied to surveillance systems, with the clear understanding that all measures will not be appropriate for all systems. ORGANIZATION OF THIS DOCUMENT

This document begins with an outline of the tasks involved in doing an evaluation, which is followed by sections describing each element of an evaluation. The first such section addresses the public health importance of the disease or health condition under surveillance. The second provides a framework for describing the components of a surveillance system. Subsequent sections focus on the attributes of a surveillance system (simplicity, flexibility, acceptability, sensitivity, predictive value positive, representativeness, and timeliness) and demonstrate how these combine to affect the usefulness and cost of a system. The document concludes with a discussion of resources needed to operate the surveillance system and sections detailing conclusions and recommendations.

As feedback is received on evaluations based on the guidelines given in this document, relevant examples will be incorporated into future updates. OUTLINE OF TASKS FOR EVALUATING A SURVEILLANCE SYSTEM

  1. Describe the public health importance of the health event. The following are the three most important categories to consider:

    1. Total number of cases, incidence, and prevalence

    2. Indices of severity such as the mortality rate and the case-fatality ratio

    3. Preventability

  2. Describe the system to be evaluated.

    1. List the objectives of the system.

    2. Describe the health event(s) under surveillance. State the case definition for each health event.

    3. Draw a flow chart of the system.

    4. Describe the components and operation of the system.

      1. What is the population under surveillance?

      2. What is the period of time of the data collection?

      3. What information is collected?

      4. Who provides the surveillance information?

      5. How is the information transferred?

      6. How is the information stored?

      7. Who analyzes the data?

      8. How are the data analyzed and how often?

        1. How often are reports disseminated? j. To whom are reports distributed? k. How are the reports distributed?

  3. Indicate the level of usefulness by describing actions taken as a result of the data from the surveillance system. Characterize the entities that have used the data to make decisions and take actions. List other anticipated uses of the data.

  4. Evaluate the system for each of the following attributes:

    1. Simplicity

    2. Flexibility

    3. Acceptability

    4. Sensitivity

    5. Predictive value positive

    6. Representativeness

    7. Timeliness

  5. Describe the resources used to operate the system (direct costs).

  6. List your conclusions and recommendations. State whether the system is meeting its objectives, and address the need to continue and/or modify the surveillance system. PUBLIC HEALTH IMPORTANCE Task

Describe the public health importance of the health event. Definition

The public health importance of a health event and the need to have that health event under surveillance can be described in several ways. Health events that affect many people or require large expenditures of resources clearly have public health importance. However, health events that affect relatively few persons may also be important, especially if the events cluster in time and place--e.g., a limited outbreak of a severe disease. At other times, public concerns may focus attention on a particular health event, creating or heightening the sense of importance. Diseases that are now rare because of successful control measures may be perceived as "unimportant," but their level of importance should be assessed in light of their potential to re-emerge. Finally, the public health importance of a health event is influenced by its preventability. Measures

Parameters for measuring the importance of a health event--and, therefore, the surveillance system with which it is monitored--include:

  1. Total number of cases, incidence, and prevalence

  2. Indices of severity, e.g., the case-fatality ratio

  3. Mortality rate

  4. An index of lost productivity, e.g., bed-disability days

  5. An index of premature mortality, e.g., years of potential life lost (YPLL)

  6. Medical costs

  7. Preventability These measures of importance do not take into account the

effect of existing control measures. For example, the number of cases of vaccine-preventable illness has declined following the implementation of school immunization laws, and the public health importance of these diseases would be underestimated by case counts alone. In such instances, it may be possible to estimate the number of cases that would be expected in the absence of control programs (1).

Preventability can be defined at several levels--from preventing the occurrence of disease (primary prevention); through early detection and intervention with the aim of reversing, halting, or at least retarding the progress of a condition (secondary prevention); to minimizing the effects of disease and disability among those already ill (tertiary prevention). From the perspective of surveillance, preventability reflects the potential for effective public health intervention at any of these levels.

Attempts have been made to quantify the public health importance of various diseases and health conditions. Dean et al. describe such an approach using a score that takes into account age-specific mortality and morbidity rates as well as health- care costs (2). SYSTEM DESCRIPTION Tasks

  1. List the objectives of the system.

  2. Describe the health event(s) under surveillance. State the case definition for each health event.

  3. Describe the components and operation of the system.

  4. Draw a flow chart of the system.

Methods

Objectives may include detecting or monitoring outbreaks, monitoring trends, identifying contacts and administering prophylaxis, enrolling case-patients in a study, and generating hypotheses about etiology. The objectives of the system define a framework for evaluating the specific components.

The next task is to describe the components of a surveillance system. This can be done by answering the following questions:

  1. What is the population under surveillance?

  2. What is the period of time of the data collection?

  3. What information is collected?

  4. Who provides the surveillance information? What is the data source?

  5. How is the information transferred?

  6. How is the information stored?

  7. Who analyzes the data?

  8. How are the data analyzed, and how often?

    1. Are there preliminary and final tabulations, analyses, and reports? j. How often are reports disseminated? k. To whom are reports distributed? l. How are the reports distributed?

      It is often useful to list the discrete steps in the processing

of the health-event reports by the system, and then to depict these steps in a flow chart (Figure 1). USEFULNESS Tasks

  1. Describe the actions that have been taken as a result of the data from the surveillance system.

  2. Describe who has used the data to make decisions and take actions.

  3. List other anticipated uses of the data.

Definition

A surveillance system is useful if it contributes to the prevention and control of adverse health events, including an improved understanding of the public health implications of such events. A surveillance system can also be useful if it helps to determine that an adverse health event previously thought to be unimportant is actually important. Methods

An assessment of the usefulness of a surveillance system should begin with a review of the objectives of the system and should consider the dependence of policy decisions and control measures on surveillance. Depending on the objectives of a particular surveillance system, the system may be considered useful if it satisfactorily addresses at least one of the following questions. Does the system:

  1. Detect trends signaling changes in the occurrence of disease?

  2. Detect epidemics?

  3. Provide estimates of the magnitude of morbidity and mortality related to the health problem under surveillance?

  4. Stimulate epidemiologic research likely to lead to control or prevention?

  5. Identify risk factors associated with disease occurrence?

  6. Permit assessment of the effects of control measures?

  7. Lead to improved clinical practice by the health-care providers who are the constituents of the surveillance system?

Discussion

Usefulness may be affected by all the attributes of surveillance. Increased sensitivity may afford a greater opportunity for identifying epidemics and understanding the natural course of an adverse health event in a community. Improved timeliness allows control and prevention activities to be initiated earlier. Increased predictive value positive enables public health officials to focus on productive activities. A representative surveillance system will better characterize the epidemiologic characteristics of a health event in a defined population. Systems that are simple, flexible, and acceptable also tend to be more useful. SYSTEM ATTRIBUTES Task

Evaluate the system for each of the following attributes:

  1. Simplicity

  2. Flexibility

  3. Acceptability

  4. Sensitivity

  5. Predictive value positive

  6. Representativeness

  7. Timeliness

Simplicity Definition

The simplicity of a surveillance system refers to both its structure and ease of operation. Surveillance systems should be as simple as possible while still meeting their objectives. Methods

A chart describing the flow of information and the lines of response in a surveillance system can help assess the simplicity or complexity of a surveillance system. A flow chart for a generic surveillance system is illustrated in Figure 1.

The following measures might be considered in evaluating the simplicity of a system:

  1. Amount and type of information necessary to establish the diagnosis

  2. Number and type of reporting sources

  3. Method(s) of transmitting case information/data

  4. Number of organizations involved in receiving case reports

  5. Staff training requirements

  6. Type and extent of data analysis

  7. Number and type of users of compiled case information

  8. Method of distributing reports or case information to these users

    1. Time spent with the following tasks:

    1. Maintaining the system

    2. Collecting case information

    3. Transmitting case information

    4. Analyzing case information

    5. Preparing and disseminating surveillance reports

Discussion

It may be useful to think of the simplicity of a surveillance system from two perspectives: the design of the system and the size of the system. An example of a system that is simple in design is one whose case definition is easy to apply and in which the person identifying the case will also be the one analyzing and using the information. A more complex system might involve some of the following:

  1. Special laboratory tests to confirm the case

  2. Telephone contact or a home visit by a public health nurse to collect detailed information

  3. Multiple levels of reporting (e.g., with the Notifiable Diseases Reporting System, case reports may start with the doctor who makes the diagnosis and pass through county and state health departments before going to the Centers for Disease Control) Simplicity is closely related to timeliness and will affect the

amount of resources that are required to operate the system. B. Flexibility Definition

A flexible surveillance system can adapt to changing information needs or operating conditions with little additional cost in time, personnel, or allocated funds. Flexible systems can accommodate, for example, new diseases and health conditions, changes in case definitions, and variations in reporting sources. Methods

Flexibility is probably best judged retrospectively, by observing how a system responded to a new demand. For example, when acquired immunodeficiency syndrome (AIDS) emerged in 1981, the existing notifiable disease reporting system of state health departments was used to report cases, and AIDS surveillance has adapted to rapidly advancing knowledge about the disease, its diagnosis, and its risk factors. Another example is the capacity of the gonorrhea surveillance system to accommodate special surveillance for penicillinase-producing Neisseria gonor- rhoeae. Discussion

Unless efforts have been made to adapt a system to another disease, it may be difficult to assess the flexibility of that system. In the absence of practical experience, one can look at the design and workings of a system. Generally, simpler systems will be more flexible--fewer components will need to be modified when adapting the system for use with another disease. C. Acceptability Definition

Acceptability reflects the willingness of individuals and organizations to participate in the surveillance system. Methods

In terms of evaluating a surveillance system, acceptability refers to the willingness to use the system by: a) persons outside the sponsoring agency, e.g., those who are asked to do something for the system and b) persons in the sponsoring agency that operates the system. To assess acceptability, one must consider the points of interaction between the system and its participants (Figure 1), including persons with the condition and those reporting cases.

Quantitative indicators of acceptability include:

  1. Subject or agency participation rates

  2. If participation is high, how quickly it was achieved

  3. Interview completion rates and question refusal rates (if the system involves interviews with subjects)

  4. Completeness of report forms

  5. Physician, laboratory, or hospital/facility reporting rates

  6. Timeliness of reporting Some of these measures may be obtained from a review of

surveillance report forms, while others would require special studies or surveys.

Discussion

Acceptability is a largely subjective attribute that encompasses the willingness of persons on whom the system depends to provide accurate, consistent, complete, and timely data. Some factors influencing the acceptability of a particular system are:

  1. The public health importance of the health event

  2. Recognition by the system of the individual's contribution

  3. Responsiveness of the system to suggestions or comments

  4. Time burden relative to available time

  5. Federal and state legislative restrictions on data collection and assurance of confidentiality

  6. Federal and state legislative requirements for reporting

Sensitivity Definition

The sensitivity of a surveillance system can be considered on two levels. First, at the level of case reporting, the proportion of cases of a disease or health condition detected by the surveillance system can be evaluated. In Table 1 this is represented by A/(A+C). Second, the system can be evaluated for its ability to detect epidemics (3). Methods

The sensitivity of a surveillance system is affected by the likelihood that:

  1. Persons with certain diseases or health conditions seek medical care;

  2. The diseases or conditions will be diagnosed, reflecting the skill of care providers and the sensitivity of diagnostic tests; and

  3. The case will be reported to the system, given the diagnosis. These three conditions can be extended by analogy to

surveillance systems that do not fit the traditional disease care-provider model. For example, the sensitivity of a telephone-based surveillance system of morbidity or risk factors is affected by:

  1. The number of people who have telephones, who are at home when the call is placed, and who agree to participate;

  2. The ability of persons to understand the questions and correctly identify their status; and

  3. The willingness of respondents to report their status. The extent to which these questions are explored depends on the

system and on the resources available for the evaluation. The measurement of sensitivity in a surveillance system requires a) the validation of information collected by the system and b) the collection of information external to the system to determine the frequency of the condition in a community (4). From a practical standpoint, the primary emphasis in assessing sensitivity--assuming that most reported cases are correctly classified--is to estimate the proportion of the total number of cases in the community being detected by the system. Discussion

A surveillance system that does not have high sensitivity can still be useful in monitoring trends, as long as the sensitivity remains reasonably constant. Questions concerning sensitivity in surveillance systems most commonly arise when changes in disease occurrence are noted. Changes in sensitivity can be precipitated by such events as heightened awareness of a disease, introduction of new diagnostic tests, and changes in the method of conducting surveillance. A search for such surveillance "artifacts" is often an initial step in outbreak investigations. E. Predictive Value Positive Definition

Predictive value positive (PVP) is the proportion of persons identified as having cases who actually do have the condition under surveillance (5). In Table 1 this is represented by A/(A+B). Methods

In assessing PVP, primary emphasis is placed on the confirmation of cases reported through the surveillance system. Its effect on the use of public health resources can be considered on two levels. At the level of an individual case, PVP affects the amount of resources used for case investigations. For example, in some states every reported case of type A hepatitis is promptly investigated by a public health nurse, and family members at risk are referred for prophylactic treatment with immune globulin. A surveillance system with low PVP--and therefore frequent "false-positive" case reports--would lead to wasted resources.

The other level is that of detection of epidemics. A high rate of erroneous case reports may trigger an inappropriate outbreak investigation. Therefore, the proportion of epidemics identified by the surveillance system that are true epidemics is needed to assess this attribute.

Calculating the PVP may require that records be kept of all interventions initiated because of information obtained from the surveillance system. A record of the number of case investigations done and the proportion of persons who actually had the condition under surveillance would allow the calculation of the PVP at the level of case detection. Personnel activity reports, travel records, and telephone logbooks may all be useful in estimating the PVP at the epidemic detection level. Discussion

PVP is important because a low value means that a) non-cases are being investigated and b) epidemics may be mistakenly identified. PFalse-positiveP' reports may lead to unnecessary intervention, and falsely detected PepidemicsP' may lead to costly investigations and undue concern in the community. A surveillance system with high PVP will lead to fewer wild-goose chases and wasted resources.

An example of a surveillance evaluation that examined PVP was reported by Barker et al. They reviewed hospital charts to determine the proportion of persons admitted with a diagnosis of stroke who had the diagnosis confirmed (6). Of 1,604 patients admitted to seven acute-care hospitals with a stroke-related diagnosis, 903 (PVP = 56%) were subsequently confirmed to have had strokes.

The PVP for a health event is closely related to the clarity and specificity of the case definition. Good communication between the persons who report cases and the receiving agency also can improve PVP.

The PVP reflects the sensitivity and specificity of the case definition and the prevalence of the condition in the population (Table 1). The PVP increases with increasing specificity and prevalence. F. Representativeness Definition

A surveillance system that is representative accurately describes a) the occurrence of a health event over time and b) its distribution in the population by place and person. Methods

Representativeness is assessed by comparing the characteristics of reported events to all such actual events. Although the latter information is generally not known, some judgment of the representativeness of surveillance data is possible, based on knowledge of:

  1. Characteristics of the population--e.g., age, socioeconomic status, geographic location (5);

  2. Natural history of the condition--e.g., latency period, mode of transmission, fatal outcome;

  3. Prevailing medical practices--e.g., sites performing diagnostic tests, physician-referral patterns (7,8);

  4. Multiple sources of data--e.g., mortality rates for comparison with incidence data, laboratory reports for comparison with physician reports. Representativeness can be examined through special studies that

seek to identify a probability sample of all cases.

Quality of data is an important part of representativeness. Much of the discussion in this document focuses on the identification and classification of cases. However, most surveillance systems rely on more than simple case counts. Information commonly collected includes the demographic characteristics of affected persons, details about the health event, and notification of the presence or absence of potential risk factors. The quality and usefulness and representativeness of this information depends on its completeness and validity.

Quality of data is influenced by the clarity of surveillance forms, the quality of training and supervision of persons who complete surveillance forms, and the care exercised in data management. A review of these facets of a surveillance system provides an indirect measure of quality of data. Examining the percentage of unknown or blank responses to items on surveillance forms or questionnaires is straightforward. Assessing the reliability and validity of responses would require such special studies as chart reviews or re-interviews of respondents. Discussion

In order to generalize findings from surveillance data to the population at large, the data from a surveillance system should reflect the population characteristics that are important to the goals and objectives of that system. These characteristics generally relate to time, place, and person. An important result of evaluating the representativeness of a surveillance system is the identification of population subgroups that may be systematically excluded from the reporting system. This process allows appropriate modification of data collection and more accurate projection of incidence of the health event in the target population.

For example, an evaluation of reporting of hepatitis in a county in Washington State suggested that cases of type B hepatitis were under-reported among homosexual males and that cases of type non A-non B hepatitis were under-reported among persons given blood transfusions. The importance of these risk factors as contributors to the occurrence of these diseases was apparently underestimated by the selective under-reporting of certain types of hepatitis cases (9).

Errors and bias can make their way into a surveillance system at any stage. Because surveillance data are used to identify high-risk groups, to target interventions, and to evaluate interventions, it is important to be aware of the strengths and limitations of the information in the system.

So far the discussion of attributes has been aimed at the information collected for cases, but in many surveillance systems morbidity and mortality rates are calculated. The denominators for these rate calculations are often obtained from a completely separate data system maintained by another agency, e.g., the Bureau of the Census. Thought should be given to the comparability of categories (e.g., race, age, residence) on which the numerators and denominators of rate calculations are based. G. Timeliness Definition

Timeliness reflects the speed or delay between steps in a surveillance system. Methods

The major steps in a surveillance system are shown in Figure 2. The time interval linking any two of the steps in this figure can be examined. The interval usually considered first is the amount of time between the onset of an adverse health event and the report of the event to the public health agency responsible for instituting control and prevention measures. Another aspect of timeliness is the time required for the identification of trends, outbreaks, or the effect of control measures. With acute diseases, the onset of symptoms is usually used. Sometimes the date of exposure is used. With chronic diseases, it may be more useful to look at elapsed time from diagnosis rather than to estimate an onset date. Discussion

The timeliness of a surveillance system should be evaluated in terms of availability of information for disease control--either for immediate control efforts or for long-term program planning.

For example, a study of a surveillance system for Shigella infections indicated that the typical case of shigellosis was brought to the attention of health officials 11 days after onset of symptoms--a period sufficient for the occurrence of secondary and tertiary transmission. This suggests that the level of timeliness was not satisfactory for effective disease control (10). In contrast, when there is a long period of latency between exposure and appearance of disease, the rapid identification of cases of illness may not be as important as the rapid availability of exposure data to provide a basis for interrupting and preventing exposures that lead to disease. In another time frame, surveillance data are being used by public health agencies to track progress toward the 1990 Objectives for the Nation and to plan for the Year 2000 Objectives.

The need for rapidity of response in a surveillance system depends on the nature of the public health problem under surveillance and the objectives of that system. Recently, computer technology has been integrated into surveillance systems and may promote timeliness (11,12). RESOURCES FOR SYSTEM OPERATION Task

Describe the resources that are used to operate the system (direct costs). Definitions

This document covers only the resources directly required to operate a surveillance system. These are sometimes referred to as Pdirect costsP' and include the personnel and financial resources expended in collecting, processing, analyzing, and disseminating the surveillance data. Methods

In estimating these resources consider the following:

  1. Personnel requirements A first step is to estimate the time it takes to operate the

system (e.g., person-time expended per year of operation). If desired, these measures can be converted to dollar estimates by multiplying the person-time by appropriate salary and benefit figures.

b. Other resources These may include the cost of travel, training, supplies, equipment, and services (e.g., mail, telephone, and computer time).

The application of these resources at all levels of the public health system--from the local health-care provider to municipal, county, state, and Federal health agencies--should be considered.

The costs of surveillance systems from two studies are illustrated in Tables 2 and 3 below (7,13). Discussion

This approach to assessment of resources includes only those personnel and material resources required for the operation of surveillance and excludes a broader definition of costs that might be considered in a more comprehensive evaluation. Estimating the overall costs of a surveillance system can be a complex process. The estimates may include the estimation of a) indirect costs, such as follow-up laboratory tests or treatment incurred as a result of surveillance; b) costs of secondary data sources (e.g., vital statistics or survey data); and c) costs averted (benefits) by surveillance.

Costs are often judged relative to benefits, but few evaluations of surveillance systems are likely to include a formal cost-benefit analysis, and such analyses are beyond the scope of this document. Estimating benefits (e.g., savings resulting from preventing morbidity through surveillance data) may be possible in some instances, although this approach does not take into account the full spectrum of benefits that may result from surveillance systems. More realistically, costs should be judged with respect to the objectives and usefulness of a surveillance system. Examples

Examples of resource estimation for surveillance systems operated in Vermont and Kentucky follow. Vermont example (7)

Two methods of collecting surveillance data in Vermont have been compared. The PpassiveP' system was already in place and consisted of unsolicited reports of notifiable diseases to the district offices or state health department. The PactiveP' system was implemented in a probability sample of physician practices. Each week, a health department employee called these practitioners to solicit reports of selected notifiable diseases.

In comparing the two systems an attempt was made to estimate their costs. The estimates of resources directly applied to the surveillance systems are shown in Table 2. Kentucky example (13)

Another example is provided by an assessment of the costs of a surveillance system involving the active solicitation of case reports of type A hepatitis in Kentucky. Table 3 summarizes the costs of this system. The resources that went into the direct operation of the system were for personnel and telephone and were estimated as $3,764 and $535, respectively. This system found nine more cases than would have been found through the passive surveillance system, and an estimated seven hepatitis cases were prevented through prophylaxis of the contacts of the nine case-patients. CONCLUSIONS AND RECOMMENDATIONS Tasks

List your conclusions and recommendations. These should state whether the system is addressing an important public health problem and is meeting its objectives. Recommendations should address the continuation and/or modification of the surveillance system. Discussion

The attributes and costs of a surveillance system are interdependent. Before recommending changes in a system, interactions among the attributes and costs should be considered to ensure that benefits resulting from strengthening one attribute do not adversely affect another attribute.

Efforts to increase sensitivity, PVP, timeliness, and representativeness tend to increase the cost of a surveillance system, although savings in efficiency with automation may offset some of these costs (12).

As sensitivity and PVP approach 100%, a surveillance system is more likely to be representative of the population being monitored. However, as sensitivity increases, PVP may decrease. Efforts to increase sensitivity and PVP tend to make a surveillance system more complex--potentially decreasing its acceptability, timeliness, and flexibility. For example, a study comparing health-department-initiated (active) surveillance and provider-initiated (passive) surveillance did not improve timeliness, despite increased sensitivity (8). SUMMARY STATEMENT

Evaluating surveillance systems is not easy. There is no perfect system; trade-offs must always be made. Each system is unique and therefore requires a balancing of the effort and resources put into each of its components if the system is to achieve its intended goal.

This document has presented guidelines--not absolutes--for the evaluation of surveillance systems. Attributes have been described that can be examined and evaluated to assess a system's ability to achieve the objectives for which it was designed.

Our goal has been to make the evaluation process more explicit and objective. Suggestions on how we may improve these guidelines would be welcomed.

References

  1. Hinman AR, Koplan JP. Pertussis and pertussis vaccine: reanalysis of benefits, risks, and costs. JAMA 1984;251:3109-13.

  2. Dean AG, West DJ, Weir WM. Measuring loss of life, health, and income due to disease and injury. Public Health Rep 1982;97:38-47.

  3. Weinstein MC, Fineberg HV. Clinical decision analysis. Philadelphia: W.B. Saunders Co, 1980:84-94.

  4. Chandra Sekar C, Deming WE. On a method of estimating birth and death rates and the extent of registration. J Am Stat Assoc 1949;44:101-15.

  5. Kimball AM, Thacker SB, Levy ME. Shigella surveillance in a large metropolitan area: assessment of a passive reporting system. Am J Public Health 1980;70:164-6.

  6. Barker WH, Feldt KS, Feibel J et al. Assessment of hospital admission surveillance of stroke in a metropolitan community. Am J Chron Dis 1984; 37:609-15.

  7. Vogt RL, Larue D, Klaucke DN, Jillson DA. Comparison of active and passive surveillance systems of primary care providers for hepatitis, measles, rubella and salmonellosis in Vermont. Am J Public Health 1983;73:795-7.

  8. Thacker SB, Redmond S, Rothenberg R et al. A controlled trial of disease surveillance strategies. Am J Prev Med 1986;2:345-50.

  9. Alter MJ, Mares A. Hadler Sc et al. The effect of under reporting on the apparent incidence and epidemiology of acute viral hepatitis. Am J Epidemiol 1987;125:133-9.

  10. Rosenberg ML. Shigella surveillance in the United States, 1975. J Infect Dis 1977;136:458-9.

  11. Marks JS, Hogelin GC, Gentry EM et al. The behavioral risk factor surveys: I. State-specific prevalence estimates of behavioral risk factors. Am J Prev Med 1985;1:1-8.

  12. Graitcer PL, Burton AH. The epidemiologic surveillance project: a computer-based system for disease surveillance. Am J Prev Med 1987;3:123-7.

  13. Hinds MW, Skaggs JW, Bergeisen GH. Benefit-cost analysis of active surveillance of primary care physicians for hepatitis A. Am J Public Health 1985;75:176-7. *Epidemiology Program Office, CDC **Center for Environmental Health and Injury Control, CDC ***Center for Health Promotion and Education, CDC



Use of trade names and commercial sources is for identification only and does not imply endorsement by the U.S. Department of Health and Human Services.

References to non-CDC sites on the Internet are provided as a service to MMWR readers and do not constitute or imply endorsement of these organizations or their programs by CDC or the U.S. Department of Health and Human Services. CDC is not responsible for the content of pages found at these sites. URL addresses listed in MMWR were current as of the date of publication.


All MMWR HTML versions of articles are electronic conversions from typeset documents. This conversion might result in character translation or format errors in the HTML version. Users are referred to the electronic PDF version (http://www.cdc.gov/mmwr) and/or the original MMWR paper copy for printable versions of official text, figures, and tables. An original paper copy of this issue can be obtained from the Superintendent of Documents, U.S. Government Printing Office (GPO), Washington, DC 20402-9371; telephone: (202) 512-1800. Contact GPO for current prices.

**Questions or messages regarding errors in formatting should be addressed to mmwrq@cdc.gov.

 
USA.gov: The U.S. Government's Official Web PortalDepartment of Health and Human Services
Centers for Disease Control and Prevention   1600 Clifton Rd. Atlanta, GA 30333, USA
800-CDC-INFO (800-232-4636) TTY: (888) 232-6348 - Contact CDC–INFO
A-Z Index
  1. A
  2. B
  3. C
  4. D
  5. E
  6. F
  7. G
  8. H
  9. I
  10. J
  11. K
  12. L
  13. M
  14. N
  15. O
  16. P
  17. Q
  18. R
  19. S
  20. T
  21. U
  22. V
  23. W
  24. X
  25. Y
  26. Z
  27. #