Developing Objectives and Evaluation Questions for TB Program Evaluation
Updated September 10, 2020
The purpose of this tool is to assist State and Local public health programs as they develop objectives and questions to guide evaluation of their tuberculosis (TB) prevention and treatment programs.
When evaluating TB programs, appropriate objectives and corresponding evaluation questions help to ensure that the evaluation produces useful information for program improvement. One method to ensure that objectives can be used to track progress and meet the expectations of the evaluation is to make the objectives Specific, Measurable, Achievable, Relevant, and Time bound (SMART).
SMART objectives and evaluation questions can be developed during brainstorming sessions and discussions with internal and external stakeholders.
Taking the time to develop an evaluation plan that includes priority objectives and evaluation questions with shorter term timelines that reflect annual report submission deadlines might lead to more feasible evaluation projects.
Below are examples of SMART objectives and associated evaluation questions. Relevant national TB program objectives and National Tuberculosis Indicators Project (NTIP) indicators are listed under each focus area. Although each example includes several evaluation questions, in practice, a smaller number of questions would be chosen for an evaluation plan. These examples are not meant to be copied directly but are intended to give ideas that can be applied to develop objectives and questions specific to each TB program.
TB programs can use the SMART criteria and the CDC Evaluation Question Checklistpdf icon to assess whether objectives and evaluation questions are feasible, relevant, specific, well-planned, and useful.
Focus Area | Increase Surveillance Data Reporting of Human Immunodeficiency Virus (HIV) Test Results
Relevant National TB Objective | Increase the proportion of TB patients with a positive or negative HIV test result that are reported to the TB program to 99% by 2025.
Relevant NTIP Indicator | Known HIV Status
Example TB Program SMART Objectives |
Determine by December 2021 (time bound) why or why not (measurable and achievable) patients treated for TB in 2021 have or do not have an HIV test result recorded (specific and relevant).
Identify by December 2021 (time bound) patient and provider barriers to HIV testing (specific, measurable, achievable, and relevant).
Example Evaluation Questions |
- To what extent are patients offered an HIV test, by provider type (e.g. clinics, hospitals, private providers)?
- What challenges do providers experience in receiving and reporting HIV test results?
- What are patient-related barriers to obtaining HIV tests? (Consider where HIV tests are performed and the costs per HIV test per patient)
- What are practice-related barriers to administering HIV tests?
- What are the barriers to obtaining and recording HIV test results for surveillance?
Focus Area | Increase Timely Documentation of Sputum Culture Conversion Results
Relevant National TB Objective | For TB patients with positive sputum culture results, increase the proportion of documented conversion to negative results within 60 days of treatment initiation to 83% by 2025.
Relevant NTIP Indicator | Sputum Culture Conversion
Example TB Program SMART Objectives |
By March 2021 (time bound), identify barriers to timely documentation of sputum culture conversion results by providers (measurable, achievable, specific, and relevant).
By June 2021 (time bound), identify two strategies to improve sputum culture conversion results reporting to the state health department (specific, measurable, achievable, and relevant).
Example Evaluation Questions |
- When and how are sputum specimen(s) sent to the laboratory for processing?
- How frequently are sputum culture results reported to the TB case manager/state TB staff? How many months after starting TB treatment are results reported?
- What challenges do TB program staff identify in sputum culture collection, processing, and reporting?
- What factors lead to a longer sputum culture conversion time (e.g., clinical characteristics of TB patients, treatment regimen, program/lab operational protocols)?
- To what extent did implementing the Sputum Collection and Reporting Training for State Nurses Program improve the documentation of sputum culture conversion (e.g., improved quality of sputum culture collection, clearer protocols, improved efficiency of data reporting)?
Focus Area | Implementation of new Electronic Directly Observed Therapy (eDOT) program
Relevant NTIP Indicator | Sputum Culture Conversion
Example TB Program SMART Objectives |
By January 2021 (time bound), identify the benefits and limitations (measurable and achievable) of offering electronic Directly Observed Therapy (specific and relevant).
By March 2021 (time bound), identify challenges and facilitators (measurable and achievable) of offering electronic Directly Observed Therapy (eDOT) (specific and relevant).
By June 2021 (time bound), initiate a strategy to begin implementing eDOT for TB patients beginning treatment in 2021 (specific, measurable, achievable, and relevant).
By August 2021 (time bound), initiate a strategy to identify and enroll TB patients who are most likely to adhere to telehealth to participate in the eDOT program (specific, measurable, achievable, and relevant).
Example Evaluation Questions |
- On average, how far do staff travel and how much time is spent to conduct field DOT each week (e.g., traveling to observe DOT at a patient’s location in a week)?
- What are the health department costs associated with conducting field DOT?
- On average, how far do patients travel to come to the TB clinic? How long does it take patients to travel to the TB clinic?
- What are the health department rules and regulations for using eDOT?
- What equipment would be needed to implement an eDOT program? (e.g., webcams, software)
- How would the health department identify eligible patients for an eDOT program?
Focus Area | Evaluate current eDOT strategy
Relevant NTIP Indicator | Treatment Initiation, Treatment Completion
Example TB Program SMART Objectives |
By July 2021 (time bound), describe characteristics of current electronic directly observed therapy (eDOT) strategy (specific, measurable, achievable, and relevant).
By August 2021 (time bound), determine the effectiveness of the current eDOT strategy (specific and relevant) to improve on patient outcomes and benefit program capacity (measurable and achievable).
Example Evaluation Questions |
- What percentage of TB treatment doses are observed by eDOT compared to doses observed by in-person DOT over the last year?
- How much time was spent traveling for each in-person field dose (so that time per dose can be calculated)?
- What is the cost per observed dose per TB patient using eDOT compared with the cost per observed dose per TB patient using in-person DOT?
- How many doses were administered via synchronous (real-time interaction) vs. asynchronous (not real-time interaction) eDOT?
- What percentage of patients with TB completed treatment within 12 months while participating in eDOT compared to in-person DOT?
- How much, if at all, has eDOT use decreased staff time spent on DOT compared to in-person DOT?
- What are the benefits and challenges of using eDOT compared to in-person DOT according to patients and health care providers?
Focus Area | Submit Complete Aggregated Report for Program Evaluation (ARPE) Data to CDC via the National Tuberculosis Indicators Project (NTIP) system
Relevant National TB Objective | Ensure the 100% completeness of core Aggregate Reports for Tuberculosis Program Evaluation (ARPE) data items reported to CDC by 2025.
Relevant NTIP Indicator | ARPE
Example TB Program SMART Objectives |
By December 2021, identify barriers (measurable and achievable) that cause missing data items in ARPE data submissions to CDC from cohort period 2018-2020 (specific and relevant).
By March 2022, identify a data quality assurance strategy to address missing data item issues for cohort period 2019-2021 (specific, measurable, achievable, and relevant).
By March 2023, assess the effectiveness of newly implemented data quality assurance strategy for 2021-2022 cohort period data (specific, measurable, achievable, and relevant).
Example Evaluation Questions |
- How well are each of the core ARPE data items that need to be reported to CDC captured in the monitoring system(s)?
- What are the challenges and barriers to gathering and reporting each of the core data items?
- What strategies can help us improve ARPE data?
- Since applying a new strategy to improve data reporting, how much change is there in the timeliness, completeness, and accuracy of data exchange between health care facilities and the state health department?
Focus Area | Improve Contact Investigation Examinations
Relevant National TB Objective | For contacts to sputum Acid-Fast Bacillus (AFB) smear-positive TB cases, increase the proportion to 94% for those who are examined for infection and disease by 2025.
Relevant NTIP Indicator | Examination
Example TB Program SMART Objectives |
By March 2022 (time bound), identify barriers to completing an examination for TB disease and LTBI among contacts in different population groups (specific, measurable, achievable, and relevant).
By September 2022 (time bound), describe success of implementing targeted testing strategies to increase the proportion of contacts to sputum smear-positive TB cases examined (specific, measurable, achievable, and relevant) to inform future program planning.
Example Evaluation Questions |
- What barriers are expressed by contacts related to completing their test for TB infection and chest X-ray?
- To what extent are the proportion of contacts examined different in regard to demographic characteristics, risk type (medical and social), type of test (e.g., TST, IGRA), and examination setting (e.g., congregate settings, health care facilities, public health clinics, private providers, schools, or household)?
- How successful were targeted testing strategies in reaching population groups who are at high risk for TB disease and LTBI?