DTBE Evaluation Plan and Annual Report Example
Updated August 17, 2020
Disclaimer: The activities in this document are intended for example purposes only. The actual activities implemented as part of TB program evaluation should be identified by state or local TB program officials in collaboration with other stakeholders. The example provided here is not intended to be applied directly to any specific TB program.
Program Evaluation Topic/Focus Area: Identify Factors Related to Patient Completion of Treatment for Latent Tuberculosis Infection
Planned Start Date: October 1, 2020 – Planned Completion Date: August 1, 2021
Part A: Evaluation Plan
(Please update as necessary when submitting annual report.)
Please provide a background, purpose or rationale (what activity or program component are you evaluating, and why have you chosen that particular activity or program component to evaluate) and expected outcome of your project.
Background and Rationale:
Scenario: Completing treatment among patients with latent tuberculosis infection (LTBI) is needed to prevent development of active tuberculosis (TB) disease; and promote progress towards TB elimination. Despite these important goals, over the past 3 years our program has not been making progress towards meeting the 2025 national target of at least 93% of contacts to sputum acid-fast bacilli smear-positive TB patients newly diagnosed with LTBI completing treatment. While almost all patients with LTBI begin treatment, a third do not complete treatment. Understanding why this occurs will assist in designing interventions that promote completion of treatment among persons with LTBI.
Long Term Goal: Meet the 2025 National Performance Target (93%) of patients with newly diagnosed TB disease who complete treatment within 12 months among those who are eligible to complete treatment within 12 months.
Expected Outcomes\Short Term Goals:
Identify factors affecting success or failure to complete treatment among patients with LTBI. Using this information, the TB program will design and implement a plan that addresses these factors.
Please provide 1 or more objectives for each goal of the evaluation topic. Objectives should be Specific, Measurable, Achievable, Realistic, and Time-bound (SMART). Each objective must have at least 1 Evaluation Question.
- Identify facilitators, barriers, and potential solutions to barriers expressed by patients in complying with LTBI treatment during October 1, 2020–May 1, 2021.
- Identify facilitators, barriers, and potential solutions to barriers expressed by public health staff in promoting completion of LTBI treatment during October 1, 2020–May 1, 2021.
- Identify characteristics of patients succeeding or failing to comply with LTBI treatment during October 1, 2020–May 1, 2021.
- Identify public health system factors that might be related to patients succeeding or failing to comply with LTBI treatment and possible solutions to barriers during October 1, 2020–May 1, 2021.
Program evaluation should ideally reflect the National Tuberculosis Indicator Project (NTIP). Please specify the NTIP indicator addressed by this evaluation. If the evaluation is not linked to an NTIP indicator, please provide an explanation regarding how the evaluation relates to the scope of work in your cooperative agreement with the Centers for Disease Control and Prevention (CDC).
Contact Investigations: LTBI Completion of Treatment
Examination of Immigrants and Refugees: Completion of LTBI treatment among contacts of persons with LTBI
| Evaluation Question | Measure | Data Source | Method of Analysis | Target | Timeline/Persons Responsible |
|---|---|---|---|---|---|
| 1. Each evaluation question or activity must correspond to an objective | 1.1 Each evaluation question should have its own deliverable
1.2 Having multiple deliverables per question is permissible |
1.1. Each deliverable needs a data source | 1.1 Each deliverable should have a way to be calculated or measured objectively | 1.1. How will you know your deliverable was met? | 1.1 What is your anticipated timeline and who is responsible for different aspects of the evaluation? |
| 1. What are facilitators, barriers, and potential solutions to barriers expressed by patients in complying with LTBI treatment? | 75% of LTBI patients will identify facilitators, barriers, and potential solutions to barriers | Medical records of LTBI patients receiving treatment during November 1, 2020–May 1, 2021 | Each patient will be asked by nursing staff about facilitators, barriers, and potential solutions to barriers during ≥2 treatment sessions | 75% of all LTBI patients receiving treatment during November 1, 2020–May 1, 2021, will be asked to identify facilitators, barriers, and potential solutions to barriers to treatment | Develop questions, a database for responses, and instruct nurses: October 1–November 1, 2020
Data collection: November 1, 2020–May 1, 2021 Analysis: May 1–July 1, 2021 |
| 2. What are facilitators, barriers, and potential solutions to barriers expressed by public health staff in promoting completion of LTBI treatment? | 90% of health care staff identify and record facilitators, barriers, and potential solutions to barriers to patient adherence with LTBI treatment | Health care staff conducting examinations, administering medication, or overseeing directly observed treatment (DOT) or electronic DOT (eDOT) to LTBI patients November 1, 2020–May 1, 2021 | Each health care staff member will complete a questionnaire regarding facilitators, barriers, and possible solutions to barriers during November 1, 2020–May 1, 2021 | 90% of health care staff will identify and record facilitators and barriers to providing treatment | Develop questions, database for responses: October 1-November 1, 2020
Data collection: November 1, 2020–May 31, 2021 Analysis: June 1–August 1, 2021 |
| 3. What are patient characteristics among those succeeding or failing to comply with LTBI treatment? | Using the electronic surveillance system, review 50% of patients receiving treatment for LTBI November 1, 2020–May 1, 2021, and record treatment adherence and selected variables | Electronic surveillance system | The TB coordinator will conduct a review by using the electronic surveillance system for a random sample of 50% of eligible patients (patients receiving LTBI treatment during November 1, 2020–May 1, 2021) | 75% of eligible patients will have variables extracted and recorded | Select primary and secondary variables: October 1–November 1, 2020
Develop database to input abstracted data: November 1–December 1, 2020 Records review and abstraction: May 1–June 1, 2021 Analysis: June 1–July 1, 2021 |
| 4. What public health system factors do clinic administrative and management staff express as affecting whether patients succeed or fail to complete LTBI treatment, and what solutions to possible barriers to LTBI treatment completion are expressed? | Staff involved in clinic administration or management will participate in a structured meeting to discuss analyses and brainstorm about related health system factors affecting LTBI treatment completion and strategies for addressing those factors | Structured meeting | Clinic administrative and management staff will discuss perceived facilitators, barriers, and potential solutions to barriers to LTBI treatment completion | At least four staff involved in clinic administration or management will participate in the structured meeting | Develop structured meeting questions:
May 1–July 1, 2021 Conduct meeting: July 1–15, 2021 Write up meeting summary: July 15–August 1, 2021 |
Part B: Evaluation Findings and Conclusions
(This section should be updated with submission of the annual report, if new findings are available, when the evaluation is complete.)
Is this a final report of a completed evaluation or an interim report of an ongoing evaluation? Final
| Evaluation Question or Indicator Outcomes (Listed results are by evaluation question and per NTIP indicator, if appropriate.) |
Benchmark Met or Not Met |
|---|---|
| 1. What facilitators, barriers, and possible solutions to barriers were expressed by patients in complying with treatment for LTBI?
20 eligible patients; 19 interviewed Facilitators Being given a bus pass (expressed by 10 patients); later clinic hours (5 patients); Sunday clinic hours (2 patients) Barriers Cannot easily walk to the bus; clinic not open evenings Possible Solutions to Barriers eDOT; evening clinic hours |
Met — 90% of patients asked about barriers and facilitators |
| 2. What facilitators, barriers, and possible solutions were expressed by public health staff in promoting completion of LTBI treatment?
10 eligible staff; 7 interviewed. Facilitators Calling patients who are late to remind them (4 nurses); having a regular schedule so that staff can develop a relationship with the patients (2 respondents) Barriers Constantly being assigned to different areas of the public health clinic so that staff do not develop a relationship with patients (3 respondents); too little time allotted for visits so that staff are unable to have a meaningful conversation with the patient receiving DOT (2 respondents) Possible Solutions to Barriers Nursing managers improve consistent assignment of staff, increase time allotted to visits by 5 minutes each, and development of prompting questions that can guide health care staff in identifying patients’ barriers to treatment completion |
Not met — 70% of eligible staff interviewed instead of 90% because of staff turnover |
| 3. What are patient characteristics among those succeeding or failing to adhere with the LTBI treatment regimen?
20 eligible patients; all electronic records reviewed Characteristics of Adherent Patients (12 Patients) Ages 30–60 years; ability to speak English fluently; or being US-born Characteristics of Nonadherent Patients (8 Patients) Ages 20–30 years or ≥60 years; being male; not speaking English well; or being homeless for part of the treatment period |
Met — all records reviewed |
| 4. What public health system factors do clinic administrative and management staff express as affecting whether patients succeed or fail to complete LTBI treatment, and what possible solutions to barriers are proposed?
Successful Adherence Extended clinic hours Unsuccessful Adherence Lack of communication with local homeless shelters Possible Solutions to Barriers Extend clinic hours; develop association with homeless shelter staff |
Met — 5 staff participated and provided feedback in structured meeting |
What helped facilitate the completion of this evaluation? (This is helpful for other programs who might want to try a similar evaluation.)
Having a timeline and an example to follow
Attending monthly calls with the CDC program evaluation consultant
Organized and up-to-date electronic surveillance records
How did you use the findings?
We held a meeting with health care staff and clinic management on August 12, 2021, during which we reviewed all of the findings and discussed facilitators, barriers, and possible solutions to barriers. We identified multiple strategies for addressing barriers and expanding facilitators, as follows:
- Provide a bus pass for all eligible and interested patients (i.e., those expressing difficulty getting to the clinic)
- Extend clinic hours from 5:00 to 6:00 pm each day
- Develop and maintain a contact list of patients and a protocol or follow-up if they are ≥30 minutes late for an appointment
- Decrease shifting of TB staff to other clinic areas during hours when the majority of DOT patients come to the clinic
- Adopt use of culturally and linguistically appropriate materials and use of an alternative language telephone line or other interpreter service
- Establish an association with nearby homeless shelter staff
Please provide a list of final deliverables and describe how you plan to disseminate to appropriate stakeholders.
- Include funding for bus passes in the next annual budget (September 2021)
- Revise staffing structure so that the clinic can remain open an additional hour each day (Begins October 2021)
- At next staff meeting, instruct nursing staff to maintain contact sheet for DOT patients and to call them if they are ≥30 minutes late.
- Find and use TB materials in patients’ first language or develop materials in patients’ first language if none are found
- Schedule meeting with director of nearby homeless shelters to discuss areas for collaboration