5 Data Management and Evaluation

Proper data management and evaluation are critical components of effective STD prevention and control. Data management includes documentation, storage, and extraction. There is not currently a standardized way to document and evaluate IPS efforts. Programs use different data management systems, and the information they collect varies widely. Developing documentation, data collection, and evaluation plans simultaneously helps to ensure that the right information is collected in order to monitor and evaluate an IPS program’s progress, quality, and effectiveness, and identify possible improvements to the program. Program activities that are not documented or evaluated present programmatic blind spots that are often undetected and unaddressed.

5.1 Documentation

Internet partner services can create unique documentation issues. For example, marginal contacts may not be able to be entered into certain data management systems (e.g., PRISM) without a field record being initiated. Some programs have worked with their IT departments to create an additional field to capture marginal information.

Another common issue is maintaining an internet-initiated case throughout the investigation. Some data systems will overwrite internet information once traditional information is found. This prevents a program from determining how many cases started as internet-only cases and how many cases were able to be notified because of information gathered through online or app searches and investigations. When you lose internet-only status, it impacts a program’s ability to evaluate IPS activities.

It is important that your program’s data system allows for the collection and extraction of IPS data variables such as profile names and email addresses. With certain apps, there may be profiles that do not have a profile name if users are not required to provide one. Users may use emojis in lieu of words in their profile or screen names, and programs have to determine how best to capture these characters so that they can be searched within the database in the future. Southern Nevada Health District is developing a list of emojis with descriptive words for each so that they can be searched (e.g., :) = happy face). Marginal information should be documented in a program’s data management system.

Below is an example of an IPS documentation form.

Sample Log Sheet

See Appendix G and Appendix H for additional samples of documentation logs and Appendix I for examples of documentation procedures.

5.2 EVALUATION

5.2.1 Logic models

Logic models are an effective way to describe the inputs, outputs, and desired outcomes of IPS and may help in developing evaluation plans. Below are two examples of logic models for PS.

Example Logic Model: 2014 STD-AAPPS Logic Model
Assurance: Partner services and linkage to care
Activities
  1. Increase the provision of targeted and effective health department DIS PS for:
    1. Primary and secondary syphilis cases
    2. HIV co-infected GC and syphilis cases
    3. GC cases with possible GC treatment failure or suspected/probable cephalosporin-resistant GC
  2. Link partners who have not previously been diagnosed with HIV but who test positive for HIV to care
  3. Within state law, increase the provision of EPT for CT and GC according to CDC guidelines
  4. Increase the provision of effective PS through social media websites and other digital or communication technologies (e.g., IPS)
  5. Link newly identified HIV-infected individuals in STD clinics to HIV care
  6. Link uninsured patients or uninsured partners to safety net services
Outputs

Partner services

  • Number, timeliness, & quality of PS provided (to OP and partners), including counseling/education
  • Formalization and use of a PS triage/prioritization plan
  • IPS utilization
  • Number of EPT distributed, by provider/mode
  • Percentage of eligible who were offered and who received PS

 Linkage

  • Number/percent of linkages made
  • Linkage protocols in place
  • Provider connections maintained/ developed
Key Outcomes

Partner services

Short-term

  • Identification of new cases/infections of HIV and others STDs
  • Reduction in sexual risk behaviors of OP and partners
  • Decreases in reinfection of OP and partners

Long-term

  • Decreased morbidity/mortality related to STDs (e.g., infant mortality due to CS, PID)
  • Decreased incidence, prevalence, cost

Linkage activities

Short-term

  • Increases in people receiving care for HIV
  • Increases in people receiving prevention care and care for other health issues (e.g., PrEP & diabetes screening)
  • Increases in people receiving needed social services

Long-term

  • Reduced morbidity, mortality, and costs
  • Reduced HIV transmission
  • Increases in quality of life
  • Improved social determinants of health

An example of an IPS logic model and a blank logic model for your use can also be found in Appendix L.

Once the logic model is complete, programs can consider whether to conduct a process evaluation, an outcome evaluation, or both. Both process and outcome evaluations can provide valuable insight into the effectiveness of a program.

Process evaluations can assess whether IPS activities have been implemented as intended. For example, process measures can capture whether patients are being asked about technology-based sex seeking, or if attempts have been made by the DIS to contact online or mobile-based partners. Outcome evaluations determine the effects of the intervention on the targeted population and may include numbers of partners notified, tested, infected, and treated.

5.2.2 Examples of process and outcome evaluation questions and associated indicators

The following is a sample of PS and IPS-specific process and outcome evaluation questions and possible indicators a program may choose to consider. For this example, we have used the term iPartner for clarity. It is defined as any partner met via the internet or other digital technology (e.g. mobile app). Sometimes the only means of contacting an iPartner is through these digital venues.

Caption for compliance
Process Evaluation Questions Indicator/Measure
How productive are interviews in terms of eliciting iPartners for field investigation? Total number of interviews
Total number of partners, regardless of location info
Number of patients asked about iPartners
Number of iPartners for which some contact information was obtained (e.g. profile name, email address, phone number)
Number of iPartners initiated for field investigation
Outcome Evaluation Questions Indicator/Measure
How productive are field investigations of iPartners in terms of notification of exposure? Number of iPartners sent notificationA of their possible exposure
Number of iPartners contactedB about their possible exposure
Number of iPartners for whom traditional contact information is obtained (telephone number or physical address)
Number of iPartners notifiedC of their possible exposure
How productive are field investigations of iPartners in terms of testing/examination? Number of iPartners previously diagnosed with HIV
Number of iPartners tested/examined (STD or HIV)
How productive are field investigations of iPartners in terms of identifying new infections? Number of iPartners preventively treated for STD
Number of iPartners infected and number brought to treatment for STD
Number of iPartners newly diagnosed with HIV
How effective are efforts to link newly diagnosed HIV+ and previously HIV+ partners to HIV care? Number of HIV+ partners linked to HIV care (includes newly diagnosed partners and previously positive partners who have fallen out of care)

A—Notification refers to a message being sent to an iPartner by email, text message via a social/sexual networking site, or similar format. It does not imply that any response was received back to the DIS from the iPartner.

B—Contact refers to a message that has been sent to an iPartner and for which there is some indicator that the iPartner received and read the message, e.g., the delivery of a “read receipt” back to the DIS or direct contact by the iPartner.

C—Notify refers to a face-to-face or telephone conversation between the DIS and an iPartner notifying the iPartner of their possible exposure.

Some of the above questions can be further analyzed for more nuanced detail that may reveal useful insight. For example, partner notification methods can be compared to determine the most effective means of reaching partners, e.g., a field visit versus a telephone call versus an email versus a text message.

Measures can be monitored over time to track the status or performance of IPS activities.

As with any evaluation effort, the findings are most useful when used to inform, modify, or further develop program planning and activities and are an integral part of the quality improvement process.

For example, evaluation results can be used to

  • Understand the effectiveness of IPS components (e.g., notification via email versus social/sexual networking site).
  • Identify technical assistance and training needs (e.g., cultural competency training).
  • Allocate program resources (e.g., more DIS allowed to conduct IPS).
  • Identify gaps that could be addressed by increased resources (staff, equipment, funding).
  • Identify virtual communities in need of additional services.

5.3 SUPPORTING RESOURCES FOR PROGRAM EVALUATION

There are online resources related to program evaluation that programs may find useful. These resources are comprehensive and serve as a good reference for evaluating any STD/HIV intervention or activity, including PS.