Example Program Evaluation Plan for State & Local TB Programs
Updated August 10, 2020
Disclaimer: The activities in this document are intended for example purposes only. The actual activities implemented as part of TB program evaluation should be identified by state or local TB program officials in collaboration with other stakeholders. The example provided here is not intended to be applied directly to any specific TB program.
Program Evaluation Topic/Focus Area: Completion of Treatment for Tuberculosis Disease by Using Incentives and Enablers
The purpose of this section is to provide background, purpose or rationale, and intended use of your evaluation results. What are the expected outcomes and impact, if successful?
Background and Rationale
Scenario: In August 2020, a Tuberculosis (TB) prevention and control program of a county public health department implemented activities that address barriers and improve TB completion of treatment (COT) within 12 months among patients receiving clinic-based, directly observed therapy (DOT). TB patients who receive DOT at home (or other locations) have high completion rates; therefore, the focus of this evaluation is on patients who come to the clinic for DOT.
Activities included providing bus passes for TB patients so that they can get to the clinic for DOT, expanding clinic hours so that patients have more flexibility regarding when they must arrive for DOT, and rapidly following up to reschedule patients who miss a DOT visit. This process evaluation assesses the success of those activities.
Goal: Meet the 2025 National Performance Target (95%) of patients with newly diagnosed TB disease who complete treatment within 12 months among those who are eligible to complete treatment within 12 months.
Should be specific, measurable, attainable, realistic, and time-bound (SMART).
- By December 2021, decrease the percentage of TB patients with missed doses of in-clinic DOT from 10%/week to 5%/week.
- By December 2021, increase the overall percentage of TB patients who complete treatment from 86% to 90%.
Program evaluation should ideally reflect the National Tuberculosis Indicator Project (NTIP).
Please specify the NTIP indicator addressed by this evaluation.
If the evaluation is not linked to an NTIP indicator, please provide an explanation regarding how the evaluation relates to your scope of work.
Corresponding NTIP indicator: Completion of Therapy – the proportion of patients who completed treatment within 12 months among those for whom ≤12 months of treatment was indicated.
| Evaluation Question |
Measure |
Data Source |
Method of Analysis |
Target |
Timeline of Activities and Person |
| Are bus passes, extended clinic hours, and follow-up telephone calls for missed directly observed therapy (DOT) effective ways to increase the number of patients who come to the clinic for DOT? | Proportion of patients who received bus passes
Number of patients who came to clinic during extended hours Proportion of patients who rescheduled a same-day DOT when the scheduled DOT time is missed |
Bus pass purchase receipts
Patient self-reported use of bus passes Clinic DOT logs and schedule |
Number of patients who used a bus pass out of the number of patients who received a bus pass
Number of patients who came during extended hours Number of patients who rescheduled same-day DOT out of the total number of patients who missed their scheduled DOT time |
90% of patients who received bus passes arrive at clinic by scheduled DOT time
At least 5 patients/week come to the clinic during extended hours 90% of patients who miss their scheduled DOT time are rescheduled |
August 2020–December 2021 • DOT outreach workers will supply bus passes to eligible patients • TB program manager will partner with sexually transmitted disease clinic to use their extra clinic hours for DOT patients • DOT outreach workers attempt to reschedule missed DOT visit • TB Program Evaluation Network (PEN) focal point and DOT outreach workers will develop and implement standard operating procedures for using incentives and enablers • TB PEN focal point and DOT outreach workers will develop and implement standard operating procedures for follow-up on patients who miss their scheduled DOT time. January–August 2021 |
| Was the program able to increase completion of treatment among patients TB patients? | The proportion of TB patients eligible to complete treatment within 12 months who truly completed it within 12 months | National Tuberculosis Indicator Project (NTIP)
Local surveillance system data |
Number of patients who completed treatment out of the number of patients eligible to complete treatment within 12 months | By December 2021, 90% of eligible patients complete treatment within 12 months | January–December 2021 • Status of indicator is assessed during monthly meeting with clinic staff, TB program manager, epidemiologists, and PE focal points • Epidemiologist and PE focal points review, monitor, and present NTIP data related to each indicator • Clinic staff present successes and challenges in completing treatment among TB patients |
| Evaluation Question and Related Findings | Benchmark
Met or Not Met |
| Are bus passes, extended clinic hours, and follow-up telephone calls for missed appointments effective ways to increase the number of patients who come to the clinic for DOT? | |
| • 9 of 12 (75%) patients who were given bus passes arrived at the clinic for DOT at their scheduled time. | Goal 90% Not Met |
| • 5 of 5 patients (100%) who expressed the need for extended clinic hours arrived at the clinic for DOT during extended
hours; 6 additional patients who had not previously expressed this concern also used extended clinic hours for DOT. |
Goal 95% Met |
| • 7 of 8 (88%) patients who missed DOT visits at the clinic were rescheduled by DOT workers the same day, and 1 (12%)
patient who did not answer the telephone was not rescheduled and treatment will be extended accordingly. |
Goal 90% Not Met |
| Was the program able to increase completion of treatment among TB patients? | |
| • Although we did not reach the 2025 target of 95% completion of treatment within 12 months for eligible patients, the
proportion who completed treatment increased from 86% to 90% in 2021 (among patients who could have completed treatment during the year of the evaluation). |
Goal 90% Met |
What helped facilitate completion of this evaluation?
- Optimal relationship with sexually transmitted disease (STD) clinic.
- Flexibility in rescheduling appointments and multiple contact numbers for patients (in case one number did not work).
- Weekly and monthly data review and follow-up from epidemiologist, program evaluation staff, and the TB program manager.
- Financial support from the state health department for incentives and enablers.
What major challenges did you face during this evaluation? If you did not meet your benchmark, please explain why.
- Extracting information from clinic logs was time consuming and it was difficult to find information.
- Although we knew how many bus passes were provided and to whom they were provided, there was no way to track how many bus passes were actually used (sometimes the DOT worker did not ask the patient if they used a bus pass that was provided by the clinic).
- Difficulty remembering to follow up with patients (DOT worker), no patient contact information, and patient not answering the telephone or being unreachable.
- DOT workers are already busy, and it took a lot of time to do follow up for missed appointments.
- Long-term use of free bus passes will be difficult due to the expense and lack of resources.
How did you use the findings from this program evaluation?
Findings from this evaluation were used to
- Negotiate with STD clinic manager to increase extended STD clinic hours for DOT patients. Cross-trained volunteer STD outreach workers to perform and record DOT.
- Provide training to DOT workers regarding using an appointment log or patient checklist to keep track of appointments.
- Brainstorm ways to help remind DOT workers when a patient is >30 minutes late or does not appear (e.g., put reminders on telephone calendar or cell phone alarm or check appointment log every hour).
- Justify cost of bus passes to help patients arrive at the clinic for DOT and alternative schedules for DOT workers who participate in extended STD clinic hours.
Please provide a list of final deliverables and describe how you plan to disseminate to stakeholders.
- Share information during staff meetings, budget meetings, and cohort review meetings.
- Sustain line item for bus passes in each annual budget.
- Check-in quarterly with STD clinic manager to discuss any problems related to shared clinic hours.
- Adjust DOT worker schedules to accommodate extra clinic hours.
- Add contact sheet for missed or late appointments and reschedules to electronic scheduling system.
- Revise the standard operating procedures for use of incentives and enablers and follow-up for missed appointments based on lessons learned during the evaluation period.