5 Data Management and Evaluation
Proper data management and evaluation are critical components of effective STD prevention and control. Data management includes documentation, storage, and extraction. There is not currently a standardized way to document and evaluate IPS efforts. Programs use different data management systems, and the information they collect varies widely. Developing documentation, data collection, and evaluation plans simultaneously helps to ensure that the right information is collected in order to monitor and evaluate an IPS program’s progress, quality, and effectiveness, and identify possible improvements to the program. Program activities that are not documented or evaluated present programmatic blind spots that are often undetected and unaddressed.
Internet partner services can create unique documentation issues. For example, marginal contacts may not be able to be entered into certain data management systems (e.g., PRISM) without a field record being initiated. Some programs have worked with their IT departments to create an additional field to capture marginal information.
Another common issue is maintaining an internet-initiated case throughout the investigation. Some data systems will overwrite internet information once traditional information is found. This prevents a program from determining how many cases started as internet-only cases and how many cases were able to be notified because of information gathered through online or app searches and investigations. When you lose internet-only status, it impacts a program’s ability to evaluate IPS activities.
It is important that your program’s data system allows for the collection and extraction of IPS data variables such as profile names and email addresses. With certain apps, there may be profiles that do not have a profile name if users are not required to provide one. Users may use emojis in lieu of words in their profile or screen names, and programs have to determine how best to capture these characters so that they can be searched within the database in the future. Southern Nevada Health District is developing a list of emojis with descriptive words for each so that they can be searched (e.g., :) = happy face). Marginal information should be documented in a program’s data management system.
Below is an example of an IPS documentation form.
5.2.1 Logic models
Logic models are an effective way to describe the inputs, outputs, and desired outcomes of IPS and may help in developing evaluation plans. Below are two examples of logic models for PS.
- Increase the provision of targeted and effective health department DIS PS for:
- Primary and secondary syphilis cases
- HIV co-infected GC and syphilis cases
- GC cases with possible GC treatment failure or suspected/probable cephalosporin-resistant GC
- Link partners who have not previously been diagnosed with HIV but who test positive for HIV to care
- Within state law, increase the provision of EPT for CT and GC according to CDC guidelines
- Increase the provision of effective PS through social media websites and other digital or communication technologies (e.g., IPS)
- Link newly identified HIV-infected individuals in STD clinics to HIV care
- Link uninsured patients or uninsured partners to safety net services
- Number, timeliness, & quality of PS provided (to OP and partners), including counseling/education
- Formalization and use of a PS triage/prioritization plan
- IPS utilization
- Number of EPT distributed, by provider/mode
- Percentage of eligible who were offered and who received PS
- Number/percent of linkages made
- Linkage protocols in place
- Provider connections maintained/ developed
- Identification of new cases/infections of HIV and others STDs
- Reduction in sexual risk behaviors of OP and partners
- Decreases in reinfection of OP and partners
- Decreased morbidity/mortality related to STDs (e.g., infant mortality due to CS, PID)
- Decreased incidence, prevalence, cost
- Increases in people receiving care for HIV
- Increases in people receiving prevention care and care for other health issues (e.g., PrEP & diabetes screening)
- Increases in people receiving needed social services
- Reduced morbidity, mortality, and costs
- Reduced HIV transmission
- Increases in quality of life
- Improved social determinants of health
An example of an IPS logic model and a blank logic model for your use can also be found in Appendix L.
Once the logic model is complete, programs can consider whether to conduct a process evaluation, an outcome evaluation, or both. Both process and outcome evaluations can provide valuable insight into the effectiveness of a program.
Process evaluations can assess whether IPS activities have been implemented as intended. For example, process measures can capture whether patients are being asked about technology-based sex seeking, or if attempts have been made by the DIS to contact online or mobile-based partners. Outcome evaluations determine the effects of the intervention on the targeted population and may include numbers of partners notified, tested, infected, and treated.
5.2.2 Examples of process and outcome evaluation questions and associated indicators
The following is a sample of PS and IPS-specific process and outcome evaluation questions and possible indicators a program may choose to consider. For this example, we have used the term iPartner for clarity. It is defined as any partner met via the internet or other digital technology (e.g. mobile app). Sometimes the only means of contacting an iPartner is through these digital venues.
|Process Evaluation Questions||Indicator/Measure|
|How productive are interviews in terms of eliciting iPartners for field investigation?||Total number of interviews|
|Total number of partners, regardless of location info|
|Number of patients asked about iPartners|
|Number of iPartners for which some contact information was obtained (e.g. profile name, email address, phone number)|
|Number of iPartners initiated for field investigation|
|Outcome Evaluation Questions||Indicator/Measure|
|How productive are field investigations of iPartners in terms of notification of exposure?||Number of iPartners sent notificationA of their possible exposure|
|Number of iPartners contactedB about their possible exposure|
|Number of iPartners for whom traditional contact information is obtained (telephone number or physical address)|
|Number of iPartners notifiedC of their possible exposure|
|How productive are field investigations of iPartners in terms of testing/examination?||Number of iPartners previously diagnosed with HIV|
|Number of iPartners tested/examined (STD or HIV)|
|How productive are field investigations of iPartners in terms of identifying new infections?||Number of iPartners preventively treated for STD|
|Number of iPartners infected and number brought to treatment for STD|
|Number of iPartners newly diagnosed with HIV|
|How effective are efforts to link newly diagnosed HIV+ and previously HIV+ partners to HIV care?||Number of HIV+ partners linked to HIV care (includes newly diagnosed partners and previously positive partners who have fallen out of care)|
A—Notification refers to a message being sent to an iPartner by email, text message via a social/sexual networking site, or similar format. It does not imply that any response was received back to the DIS from the iPartner.
B—Contact refers to a message that has been sent to an iPartner and for which there is some indicator that the iPartner received and read the message, e.g., the delivery of a “read receipt” back to the DIS or direct contact by the iPartner.
C—Notify refers to a face-to-face or telephone conversation between the DIS and an iPartner notifying the iPartner of their possible exposure.
Some of the above questions can be further analyzed for more nuanced detail that may reveal useful insight. For example, partner notification methods can be compared to determine the most effective means of reaching partners, e.g., a field visit versus a telephone call versus an email versus a text message.
Measures can be monitored over time to track the status or performance of IPS activities.
As with any evaluation effort, the findings are most useful when used to inform, modify, or further develop program planning and activities and are an integral part of the quality improvement process.
For example, evaluation results can be used to
- Understand the effectiveness of IPS components (e.g., notification via email versus social/sexual networking site).
- Identify technical assistance and training needs (e.g., cultural competency training).
- Allocate program resources (e.g., more DIS allowed to conduct IPS).
- Identify gaps that could be addressed by increased resources (staff, equipment, funding).
- Identify virtual communities in need of additional services.
There are online resources related to program evaluation that programs may find useful. These resources are comprehensive and serve as a good reference for evaluating any STD/HIV intervention or activity, including PS.