5 Documentation, Data Collection and Evaluation
How information is documented influences what data is collected, which will determine exactly what pieces of a program can be monitored, evaluated, and improved upon. Conversely, the activities a program evaluates will determine what data needs to be collected and how it will be documented. Developing documentation, data collection, and evaluation plans in concert helps to ensure that the right information is collected in order to monitor and evaluate an IPS program’s progress, quality, and effectiveness, and identify possible improvements to the program. Program activities that are not documented or evaluated present programmatic blind spots that are often undetected and unaddressed.
Currently, there is not a standardized way to document and evaluate IPS efforts. Programs use different data management systems and the information they collect varies widely. Below is an example of an IPS documentation form. See Appendix F for additional samples of documentation logs and Appendix G for examples of documentation procedures.
|Sample Log Sheet
INTERNET PARTNER NOTIFICATION LOG SHEET
Initiation Date: OP Id # and Disease
Standards for the collection, sharing, and security of HIV, viral hepatitis, STD, and TB data can be found in the National Center for HIV, viral Hepatitis, STD and TB (NCHHSTP) Data Security and Confidentiality Guidelines. https://www.cdc.gov/nchhstp/programintegration/docs/pcsidatasecurityguidelines.pdfCdc-pdf
These guidelines require that all newly hired staff sign confidentiality agreements before being given access to identifiable information and require annual renewal of that agreement.39 Examples of confidentiality agreements can be found in Appendix H.
Because screen names, email addresses, and telephone numbers are considered to be personal identifying information (2 C.F.R. § 200.79), and are to be held to the same levels of confidentiality as a patient’s first name and surname, programs offering IPS typically require DIS to sign additional agreements regarding the specific use of technology. For example, programs create policies describing the acceptable use of technology for PS, including what websites, apps, and social networking sites may be used for PS, what information can be shared or discussed over these mediums, expected professional and behavioral standards, and consequences for violations of the policy. These policies may also indicate whether staff may conduct IPS from personal computers and devices and other structural limitations. For an example of an acceptable use agreement, see below and Appendix I.
|Example of Acceptable Use for Accessing Restricted Websites
Example- Maryland Department of Health and Mental Hygiene
Agreement for Accessing Restricted Websites
Staff members that perform Internet Partner Services (IPS) are often required to access websites that are restricted and may contain adult oriented material during the course of their normal job duties. This agreement has been developed to establish clear expectations when restricted sites are to be accessed while performing IPS, and the consequences for acting outside of these expectations. All staff performing IPS must sign this agreement prior to being granted access to restricted websites
gree to comply with this agreement.
Some programs may require a non-networked computer, one with Internet access that is separate from the internal network, in order to protect servers from viruses or unauthorized intrusions, and may alleviate IT networking concerns about security breaches.
Logic models are an effective way to describe the inputs, outputs, and desired outcomes of IPS and may help in developing evaluation plans.
Example Logic Model: 2014 STD-AAPPS Logic Model
Assurance: Partner services and linkage to care
An example of an IPS logic model and a blank logic model for your use can also be found in Appendix J.
Once the logic model is complete, programs can consider whether to conduct a process or an outcome evaluation, or both.
Examples of possible evaluation questions and associated indictors
Both process and outcomes evaluations can provide valuable insight into the effectiveness of a program. Process evaluations can assess whether IPS activities have been implemented as intended. For example, process measures can capture whether patients are being asked about technology-based, sex-seeking, or if attempts have been made by DIS to contact online or mobile-based partners. Outcome evaluations determine the effects of the intervention on the targeted population and may include numbers of partners notified, tested, infected, and treated.
The following is a sample of PS and IPS-specific process and outcome evaluation questions and possible indicators a program may choose to consider.
iPartner is defined as any partner met via the Internet or other digital technology (e.g. mobile app). Sometimes the only means of contacting an iPartner is through these digital venues.
|Process Evaluation Questions||Indicator/Measure|
|How productive are interviews in terms of eliciting iPartners for field investigation?||Total number of interviews|
|Total number of partners, regardless of locating info|
|Number of patients asked about iPartners|
|Number of iPartners for which some contact information was obtained (e.g. profile name, email address, phone number)|
|Number of iPartners initiated for field investigation|
|Outcome Evaluation Question examples||Indicator/Measure|
|How productive are field investigations of iPartners in terms of notification of exposure?||Number of iPartners sent notification1 of their possible exposure|
|Number of iPartners contacted2 about their possible exposure|
|Number of iPartners for whom traditional contacting information is obtained (telephone number or physical address)|
|Number of iPartners notified3 of their possible exposure|
|How productive are field investigations of iPartners in terms of testing/examination?||Number of iPartners previously diagnosed with HIV|
|Number of iPartners tested/examined (STD or HIV)|
|How productive are field investigations of iPartners in terms of identifying new infections?||Number of iPartners preventively treated for STD|
|Number of iPartners infected, brought to treatment, for STD|
|Number of iPartners newly-diagnosed with HIV|
|How effective are efforts to link newly diagnosed HIV + and previously HIV + partners to HIV care?||Number of HIV+ partners linked to HIV care (includes newly diagnosed partners and previously positive partners who have fallen out of care)|
1 – The term notification refers to a message being sent to an iPartner by email, text message via a social/sexual networking site, or similar format. It does not imply that any response was received back to the DIS from the iPartner.
2 – Contact refers to a message that has been sent to an iPartner and for which there is some indicator that the iPartner received and read the message, for example, the delivery of a “read receipt” back to the DIS or direct contact by the iPartner.
Some of the above questions can be further analyzed for more nuanced detail that may reveal useful insight. For example, partner notification methods can be compared to determine the most effective means of reaching partners, i.e., a field visit versus a telephone call versus an email versus a text message.
Measures can be monitored over time to track the status or performance of IPS activities.
As with any evaluation effort, the findings are most useful when used to inform, modify or further develop program planning and activities and are an integral part of the quality improvement process.
For example, evaluation results can be used to
- Understand the effectiveness of IPS components (e.g., notification via email versus social/sexual networking site);
- Identify technical assistance and training needs (e.g., cultural competency training);
- Allocate program resources (e.g., more DIS allowed to conduct IPS); or
- Identify gaps that could be addressed by increased resources (staff, equipment, funding)
- Identify virtual communities in need of additional services.
There are online resources that programs may find useful related to program evaluation. These resources are comprehensive and serve as a good reference for evaluating any STD/HIV intervention or activity, including Partner Services (PS).
- Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide, https://www.cdc.gov/eval/guide/index.htm
- Practical Use of Program Evaluation for STD Programs, https://www.cdc.gov/std/program/pupestd.htm
- Partner Services Evaluation Field Guide, https://effectiveinterventions.cdc.gov/en/2018-design/hiv-testing/group-1/partner-services