Skip directly to search Skip directly to A to Z list Skip directly to navigation Skip directly to site content Skip directly to page options
CDC Home

Using Surveillance Data in Your Evaluation

This is an archived document. The links and content are no longer being updated.

Return to Guide Main Menu

Text for PowerPoint Slides

Slide 1: CDC’s Framework for Program Evaluation in Public Health

Step 4.  Gather Credible Evidence Using surveillance data in your evaluation

Linda Leary
Field Services Evaluation Branch

Lori Armstrong
Surveillance, Epidemiology, and Outbreak Investigations Branch
Surveillance Team

Speaker Notes: Now that you are well underway in writing your Evaluation Plan, we are now at Step 4 of the CDC Framework, “Gathering Credible Evidence.”  Today, we wanted to center your attention on how to “Use surveillance data in your evaluation.”

 

Slide 2: Evaluation is an Essential Organizational Practice

Program evaluation is not practiced consistently across program
areas, nor is it well-integrated into the day-to-day management of most programs.

Program evaluation is necessary to fulfill CDC’s operating principles for public health, which include:

  • Using science as a basis for decision-making and action;
  • Expanding the quest for social equity;
  • Performing effectively as a service agency;
  • Making efforts outcome-oriented; and
  • Being accountable

Speaker Notes: These operating principles imply several ways to improve how public health activities are planned and managed. They underscore the need for programs to develop clear plans, inclusive partnerships, and feedback systems that allow learning and ongoing improvement to occur. One way to ensure that new and existing programs honor these principles is for each program to conduct routinely practical evaluations that inform their management and improve their effectiveness.

Back to Top of Page

 

Slide 3: Framework for program evaluation in public health

This slide illustrates the framework for program evaluation in public health. The framework comprises steps in program evaluation practice and standards for effective program evaluation. Adhering to the steps and standards of this framework will allow an understanding of each program's context and will improve how program evaluations are conceived and conducted.

The steps are as follows:

The second element of the framework is a set of 30 standards for assessing the quality of evaluation activities; these standards are organized into the following four groups:

http://www.cdc.gov/eval/framework.htm

The framework was developed to:

  • Summarize and organize the essential elements of program evaluation
  • Provide a common frame of reference for conducting evaluations
  • Clarify the steps in program evaluation
  • Review standards for effective program evaluation
  • Address misconceptions about the purposes and methods of program evaluation.

Persons involved in evaluation should strive to collect information that will convey a well-rounded picture of the program and be seen as credible by the evaluation’s primary users.  Information should be perceived by stakeholders as believable and relevant for answering their questions.  Having credible evidence strengthens evaluation judgments and the recommendations that follow them. 

Back to Top of Page

 

Slide 4: Six Steps in Program Evaluation

  1. Engage stakeholders
  2. Describe the program
  3. Focus the evaluation design
  4. Gather credible evidence
  5. Justify conclusions
  6. Ensure use and share lessons learned

webinar site:
http://www.cdc.gov/tb/Program_Evaluation/default.htm

Speakers Notes: The first three steps have been described in detail by Anne Powers, Battelle and Tom Chapel.  These sessions were recorded as webinars, and if you missed the previous sessions, please feel free to log onto:  and you can view these sessions at any time.

Today we will focus on Gathering credible evidence

Persons involved in evaluation should strive to collect information that will convey a well-rounded picture of the program and be seen as credible by the evaluation’s primary users.  Information should be perceived by stakeholders as believable and relevant for answering their questions.  Having credible evidence strengthens evaluation judgments and the recommendations that follow them. 

 

Slide 5: Overview

  • Define surveillance data and their use

  • Using surveillance data to determine evaluation focus and develop evaluation questions

  • Using surveillance data as a source of credible evidence

  • Issues and concerns on data quality and completeness, and what you can do about it

 

Slide 6: How can a local TB program use surveillance data to conduct evaluations?

  1. The national TB program requires each grantee to write an evaluation plan by Dec. 16, 2005.
  • Surveillance data can help ensure the right evaluation questions are identified, and
  • Surveillance data can be used to set meaningful benchmarks for progress.

Back to Top of Page

 

Slide 7: How can a local TB program use surveillance data to conduct evaluations?

  1. TB surveillance data can also be used as one source of credible evidence to compare:
  • Local objectives against national objectives;
  • Local area objectives within a state; and
  • Look at the trend of data over a period of time.

 

Slide 8: Surveillance Definition

  • Ongoing systematic collection, analysis, and interpretation of outcome-specific data.
  • For use in planning, implementation, and evaluation of public health practice.

Speaker Notes: What is public health surveillance?

Public health surveillance is the systematic, ongoing collection, analysis, interpretation, and dissemination of health data.  The purpose of public health surveillance is to gain knowledge of the patterns of disease, injury, and other health problems in a community so that we can work toward controlling and preventing them.

 

Slide 9: Uses of Surveillance Data

Public health surveillance is an important part of the information feedback loop that links the public, health care providers, and health care agencies.  These data can be useful in posing questions for your evaluation design, such as:

  • How to identify people and groups of people at risk for disease?

  • How to prioritize health needs in persons at higher risk for disease exposure or infection accompanied by a plan for follow-up?

  • How to incorporate surveillance data as a source of information for program planning and evaluation?

  • How to use indicators to determine if a condition exists or certain results have been achieved?

 

Slide 10: Indicators

Indicators are at the heart of a performance monitoring system.

  • They provide a basis for collecting credible evidence that is valid and reliable for evaluation.
  • Describes progress in achieving its objectives.
  • Tells us what to measure to determine whether the objective has been achieved.

 

Slide 11: National TB Program Objectives and Indicators

National TB Program Objectives National TB Program Indicators
Increase % of TB pts who complete TB treatment within 12 mos. Completion of therapy rates
Increase % of TB pts with initial positive cultures who also are tested and receive drug susceptibility results Percent of cultured confirmed drug cases with susceptibility results
Decrease the TB case rate in reporting areas TB case rate
At least 90% of core RVCT data are complete. Completeness of RVCT reporting

Speaker Notes: Here is a sample of a few TB program indicators and how they relate to our national TB objectives.

Let’s look at the TB case rate indicator.  This indicator could point to many questions in your evaluation design, such as:

  • How do we gain knowledge about case rates in racial groups in your state and local areas?
  • How do we compare case rates in specific local areas?
  • How does your state case rate look over the past x of years compared to the national case rate? 

These are the type of questions stakeholders need answers to, in order to, make future programmatic decisions.

 

Slide 12: Tuberculosis Cases, Rate per 100,000, United States, 2000-2004

Speaker Notes: Case Rate – is the measure of probability of the occurrence of disease in a population of 100,000. Rates are expressed as number of cases reported each calendar year per 100,000.  Also, rates improve one’s ability to make comparisons.

Here we are looking at the US case rate over a four year period.  There are many ways to analyze cases rates and develop evaluation questions appropriate to your evaluation design.  For instance:

State rates can be compared among states and to the rate of the US.

  • Evaluation Question - How does my state compare to the US case rate?
  • Evaluation Question - And, how does my state compare to other states and states within my region?

Rates can be judged whether higher or lower in a specific geographic area, such as county, state, or zip code area.

  • Evaluation Question - Does the TB rates vary within the state, county, or zip code area?

Cases rates can be compared among racial groups.

  • Evaluation Question - Which race group within my state has the highest case rate?

Back to Top of Page

 

Slide 13: Race/Ethnicity – Year 2002

  State A State B  
  No. Cases Rate No. Cases Rate US Rate**
Hispanic 2 2 16 23 10
Black, Non-Hispanic (NH) 1 4 16 5 13
Asian. Pacific Islander, Native Hawaiian, NH 137 22 14 39 28
White, NH 6 2 100 3 2
Other 2   0    
Total 148 11.9 146 3.6 5.2

* rate per 100,000

Speaker Notes: Test question: Which target population should stakeholders focus their evaluation efforts?

In looking at both states, the distribution of TB among racial groups is very different between the two states.

State A

  • The largest number of TB cases were found in Asian, Pacific Islander population – 137 cases
  • Also, the highest case rate for State A is found in the APINH population – 22 per 100,000
  • APINH cases rate for State A [22 per 100,000] is little less then the US case rate for APINHs at 28 per 100,000.
  • The APINH case rate for State A [22 per 100,000] is 4.2 times higher than the national case rate of 5.2 per 1000,000.

State B

  • The largest number of TB cases was found in whites, non-Hispanic which is 100 cases.
  • But, the highest cases rates were found in APINH [39 per 100,000] and Hispanics [23 per 100,000].
  • Even though the cases rate is lowest in whites, NH [3.6 per 100,000], the largest burden of disease is found among this population.

Answer: The Evaluation Team will determine where to direct their time and resources based on the goals of their program and the design of their evaluation.

  • This table was easily constructed by retrieving the US Census Bureau population data and the TB case rate data can be retrieved from the national annual TB surveillance data.

 

Slide 14: How do you calculate the TB Case Rate?

RVCT variable item #6 – month year counted

  • Data is obtained from the RVCT.
    [The health department is responsible for: counting TB case by month year, verifying the case as TB, and including it in the official case count.]
  • The national annual surveillance report publishes the total number of TB cases by state.

Population estimates from the US Census Bureau
http://www.census.gov/popest

Speaker Notes:

Information to calculate case rates are found in two sources:

  • RVCT data
  • Census data

 

Slide 15: Calculation of TB Case Rate Year 2000 – State X

  • Population of State X = 554,817
  • # of TB Cases for Year 2000 State X = 42

42 / 554,817 x 100,000 = 7.6 per 100,000

 

Slide 16: Guidance documents in measuring outcomes

  • Using the RVCT as an Evaluation Tool
  • Exercises from the Program Manager’s Course.
  • Request copies of draft documents: lsl1@cdc.gov

Speakers Notes: There are two additional documents we wanted to share with you.

Using the RVCT as an Evaluation Tool”  — This document was drafted by RTI and gives five examples of objectives and strategies related to timely identification and treatment of people with active TB.  The document identifies objectives and strategies, types of indicators, identifies variables from the RVCT to calculate indicators, provides formulas for the calculation, and analyzes the result.

Also provided are exercises from the Program Manager’s course which walk you through the steps of measuring outcomes and it give principles of data gathering, analysis, and data interpretation.

If you would like a copy of these documents, please email me and I can forward the documents to you.

Back to Top of Page

 

Slide 17: Who is OTIS?

 

Slide 18: The Online TB Information System (OTIS)

  • A query-based public use data set of national TB surveillance data from 1993 through 2003
  • It includes the 50 states, District of Columbia, and Puerto Rico
  • Maintains confidentiality by de-identifying, aggregating, or suppressing data
  • Performs queries and ad hoc cross-tabulations on 22 variables from the RVCT
  • Supplements annual surveillance report
  • “Housed” on WONDER server
  • http://wonder.cdc.gov/tb.html

Speaker Notes: OTIS is another source you can use to access and summarize your surveillance data to meet your evaluation needs.

 

Slide 19: OTIS: current status

  • Each state signed a data release agreement form to accept or refuse participation in OTIS.
  • States reviewed OTIS for several months
  • Data were checked and changes made based on your suggestions
  • Release date: November 2005
  • Anticipate 2004 data will be available March 2006

 

Slide 20: CDC Wonder — TB OTIS Request — Screen Capture

Speaker Notes: This is the initial request screen that you see when you log onto OTIS.

The Online Tuberculosis Information System (OTIS) contains information on verified tuberculosis (TB) cases reported to the Centers for Disease Control and Prevention (CDC) by state health departments, the District of Columbia and Puerto Rico from 1993 through 2003.  These data were extracted from the CDC national TB surveillance system.  Data for 22 variables are included in the data set and users are able to produce cross-tabulations with multi-level stratification.  The data are updated on a regular basis.

 

Slide 21: TB OTIS Request — Screen Capture

Speaker Notes: Let’s switch to completion of therapy percentage using OTIS.

Completion of Therapy (COT) in less than one year

Completion of Therapy (COT) indicates if the patient completed TB therapy in less than or equal to one year for patients with "uncomplicated" TB. Displayed as a percentage, COT is not a categorical variable.

  1. Group Results – click STATE
  2. Check the Completion of therapy box.
  3. To end this query – press send.

 

Slide 22: TB OTIS Results - Screen Capture

Speakers Notes: This is the Table page of your results.

On the Request screen, you can select "By-Variables," which serve as keys (indexes) for organizing your data. For example you can select to show the results grouped by Year and by Sex, so that the data results display in a table, summarized (stratified) by the values in the Year and Sex variables. The table's first column is the first By-Variable selection (Year values in the given example), the second column is the second By-Variable selection (Sex values in this example), with a row for each grouping (each combination) of the selected By-Variable values (this example yields a row for each combination of Year and Sex values). These groupings shape the charts and maps you can create for your data. If you choose to export your data results to a file, then you see a line-listing structured by your "By-Variable" choices.

 

Slide 23: TB OTIS Charts - Screen Capture

Speakers Notes: This is a screen capture of the OTIS Web Application. If you click on chart, you can select chart options, chart colors, and chart type.

This section allows you to select charts to create, depending on the information available for the current request.

By-variables allows you change the order that by-variables are charted. The primary by-variable values are represented by tick marks on the domain axis (x axis). The secondary by-variable values, if they exist, are represented by extra sets of bars or lines, one for each secondary by-variable value.

Measures allows you to pick which measure(s) to chart. You can pick one or more items from the measures list. By default one chart will be created for each measure. You can also choose to chart all selected measures on the same chart. See "Combine Measures in one Chart" below.

 

Slide 24: TB OTIS Maps - Screen Capture

This is a screen capture of the OTIS Web Application. If you click on the map you’ll get this.   There are multiple mapping options too.

First, you must send a request for data, and you must select at least one geographical location as a "By-Variable" for grouping your data, such as Region or State.

Next Click on the Map tab near the top of the screen, to go to the Map Options page, where you can set which data items to map, the number of quantiles (or set your own break points) for your data, and control your map's size, color, legends and more.

 

Slide 25: Issues of Data Completeness and Quality

Speakers Notes: Now we will look at data completeness and data quality.

A very important component of using your data for credible evidence is making sure you have high quality data.  Data quality refers to the appropriateness and integrity of information used in an evaluation.  High-quality data are reliable, valid, and informative for their intended use. Since data accuracy is so important, it is one element of the “Standards for Effective Program Evaluation.”   It is very important to enter your data completely and accurately, in order to, produce accurate results.  If not, “garbage in, garbage out.”

Back to Top of Page

 

Slide 26: Data Completeness

  • TIMS data are evaluated for overall completeness each month
  • The goal for percent completeness of each RVCT (TIMS) variable is based on GPRA goals or on tradition
  • Each RVCT variable has a completeness goal, usually 95% to 99%

Missing and Unknown Reports (MUNK)

  • MUNK Reports provide information on RVCT (TIMS) variables that are below the completeness goal
  • MUNK reports are sent to each TB program or recording area 1 to 3 times/year and to the FSEB Consultant for that area.

 

Slide 27: Missing and Unknown Variables 2004 RVCT Data

RVCT Variable Completeness Goal (%) Average % completeness among all reporting areas (%)
TB skin test 95 93.1
Year of previous TB 99 98
Month/year arrived in US (FB) 95 86.3
Excess alcohol use 99 96.6
Injecting drug use 99 96.5
Non-injecting drug use 99 96.1

Speakers notes: This slide shows the completeness goals and average percent completeness of all reporting areas for the 2004 surveillance data for a few selected variables.  

A good strategy is to:

  • update TIMS as correct information is received throughout the current year.
  • and to update even if the correct information is received after the reporting year.

If your MUNK report indicates that you have problems in data completeness, you can focus your evaluation on data reporting as one of your evaluation objectives.  

 

Slide 28: Data Quality

• High quality RVCT data are essential for proper interpretation of data for TB control programs

• Inconsistent or inaccurate data can lead to inappropriate associations and unrecognized biases in research

• Examples of miscoded information

  • People from India were coded as American Indian - (reclassified to Native American/Alaskan Native in 2003)
  • Young children were coded as alcoholics or IDUs - (Children born to IDUs and alcoholics are not drug abusers or alcoholics)
  • People born in the Philippines or Japan were coded as “Native Hawaiian and Other Pacific Islander”

Speaker notes: For the last example I’d like to go into more detail.

 

Slide 29: Native Hawaiian and Other Pacific Islander Miscoded information

In preparation for the 2005 World TB Day MMWR article, we found 37 foreign-born TB cases who were incorrectly coded as “Native Hawaiian and Other Pacific Islander”

Incorrect NH/OPI cases were coded in TIMS as foreign born and from other countries such as Philippines, Japan, Laos, Vietnam, India, Somalia, Iran, Ecuador, Brazil

 

Slide 30: TB cases by Asian, NH/OPI race, United States, 2004 (provisional data)

  Uncorrected Corrected*
Race N Rate/100,000 N Rate/100,000
Asian 3,222 26.7 3,235 26.9
Native Hawaiian and Other Pacific Islander 103 25.9 66 16.6

Speaker notes: With the first analysis of the provisional data we found 3,222 cases recorded as Asian for a rate of 26.7/100,000 and 103 Native Hawaiian and other Pacific Islanders for a rate of 25.9/100,000.  Upon closer inspection we found that 37 NHOPI cases were recorded as foreign-born and that their country of origin was not Hawaii or one of the other Pacific Islands.  We contacted the reporting area and one-by-one each case was resolved.  29 cases were really Asian, and the other 8 were white, black, or Hispanic. 

The corrected case count and rate for the Native Hawaiian and Other Pacific Islander race category resulted in a 36% change in the case rate from 26.9/100,000 to 16.6/100,000.  Because we were not able to confirm the status of all NH/OPI cases in time for publication, this rate was never reported for the provisional data in the MMWR. 

Final case count for NH/OPI in the Annual Report:  65 cases, 16.3/100,000 rate.

Back to Top of Page

 

Slide 31: Native Hawaiian and Other Pacific Islander: Correct Definition

  • First introduced as a race code in 2003
  • TIMS Race and Ethnicity Classification (9/4/2002) lists all the islands classified as Other Pacific Islands and countries that are considered Asian.
  • Examples
  • – NH/OPI: Carolinian, Chamorro, Fijian, Guamanian, Kiribati, Papua New Guinean, Polynesian, Saipanese, Samoan, Solomon Islander, Hawaiian Islands
  • – Asian: Filipino, Bhutanese, Asian Indian, Singaporean, Taiwanese, Okinawan, Nepalese, Hmong, Iwo Jiman

Speaker notes: There are too many islands and Asian countries for all of them to be listed here.  For a complete list see the SURVS-TB/TIMS revision.

If unsure see the documentation.

 

Slide 32: Summary

  1. Follow the six steps in the CDC Framework in conducting your evaluation.
  2. Use surveillance data as one source of credible evidence to use in evaluation. Also, consider using OTIS as another source of gathering information.
  3. Strive for data quality. Quality refers to the appropriateness and integrity of information used. Good decisions are based on accurate data.

 

Slide 33: Contacts

Linda Leary 404.639.8342 lsl1@cdc.gov
Lori Armstrong 404.639.8860 lra0@cdc.gov

 

Slide 34: Questions ?????

 

 
Contact Us:
  • Centers for Disease Control and Prevention
    Division of Tuberculosis Elimination (DTBE)
    1600 Clifton Rd., NE
    MS E10
    Atlanta, GA 30333
  • 800-CDC-INFO
    (800-232-4636)
    TTY: (888) 232-6348
  • Contact CDC–INFO
USA.gov: The U.S. Government's Official Web PortalDepartment of Health and Human Services
Centers for Disease Control and Prevention   1600 Clifton Road Atlanta, GA 30329-4027, USA
800-CDC-INFO (800-232-4636) TTY: (888) 232-6348 - Contact CDC–INFO
A-Z Index
  1. A
  2. B
  3. C
  4. D
  5. E
  6. F
  7. G
  8. H
  9. I
  10. J
  11. K
  12. L
  13. M
  14. N
  15. O
  16. P
  17. Q
  18. R
  19. S
  20. T
  21. U
  22. V
  23. W
  24. X
  25. Y
  26. Z
  27. #