III. Appendixes
Appendix A. QA standards for HIV CTR data
A. Protocol for data QA
- A1. Health departments should develop and maintain a protocol for CTR data QA that is guided by the 2009 QA Standards and provides specific guidance for conducting QA at each point in the CTR data life cycle.
- A2. Health departments should provide CTR service providers with standard guidance for developing a site-specific manual of procedures.
- A3. Health departments and service providers should establish mechanisms for monitoring their policies and procedures.
B. Staffing, training, and communication
- B1. Health departments should assign appropriate staff, with clearly defined roles and responsibilities, to perform CTR data QA activities
- B2. Training in QA policies and procedures should be provided to health department staff soon after they are hired and as part of staff development activities
- B3. Health department staff should communicate with service providers regularly and provide them with continuing technical assistance to build their data QA capacity
- B4. Health department CTR staff should communicate regularly with other departmental staff.
C. Data security and client confidentiality
- C1. Training and oversight of security and confidentiality policies and procedures should take place regularly
- C2. Health departments should have a designated point person whom CTR staff can contact about issues related to the security and confidentiality of HIV/AIDS data
- C3. Health departments should periodically review evolving technology and consider how it may influence current security and confidentiality procedures
- C4. Health departments should implement client confidentiality and data security practices to maximize clients’ privacy during data collection
- C5. Health department staff should implement security and confidentiality practices to maximize clients’ privacy and the integrity of the data during data entry, management, and analysis.
D. Data collection
- D1. Health departments and CTR service providers should limit the number of required CTR variables collected to those that will be used on the basis of current plans.
- D2. Health departments should develop written guidance for service providers to explicate all required CTR variables.
- D3. Health departments should ensure that CTR data collection forms are user-friendly and designed to reduce errors in data documentation.
- D4. Health departments should require service providers to implement QA practices for data elicitation and documentation to maximize the completeness and accuracy of CTR data.
- D5. Health departments should assign priority to data collected from, or concerning, HIV-positive clients, including documentation of a confirmed test result and the provision of test results and linkages to care.
E. Data entry
- E1. Health departments should establish and communicate to service providers the requirements for data transfer and should develop a system for monitoring the transfer and the receipt of CTR data.
- E2. Health departments should establish clear procedures for integrating data and timelines that are based on the health department’s method of data entry (e.g., scanning, key entry, electronic transfer).
- E3. Health departments should develop guidance for communicating with service providers to verify and update data and to correct data errors.
- E4. Computer-based systems used for CTR should have programmed data entry features to improve data accuracy and efficiency.
F. Data management and cleaning
- F1. Health departments should develop and maintain a data dictionary that includes variable names, value labels, and definitions of all CTR data variables.
- F2. Health departments should implement data screening procedures that will identify aggregate-level and record-level data errors in the central CTR database.
- F3. Health departments should implement data-cleaning procedures that will resolve errors and result in a clean CTR database.
- F4. Health departments should routinely document all decisions and procedures related to data management and cleaning.
G. Data submission
- G1. Health departments should use secure mechanisms for submitting CTR data to CDC.
- G2. Health departments should submit data to CDC within the timeframe specified by CDC.
- G3. Health departments should maintain a log of records submitted to CDC and records that require follow-up.
H. Data analysis and utilization
- H1. Health departments should develop a CTR data analysis plan for use in monitoring and evaluating program services and programmatic outcomes.
- H2. Health department staff should develop and adhere to policies for sharing and reporting data that prevent the disclosure of individual client data.
- H3. Health departments should disseminate annual CTR data reports that are clear, unbiased, and accessible to stakeholders.
- H4. Health departments should serve as a resource to service providers for the analysis and utilization of their data.
Appendix B. Resources, by standard
A. Data QA protocol
| Resource | Description |
|---|---|
| Standard A1 | |
| Guidance to policymakers and service providers for developing HIV CTR services and policies in traditional and nontraditional settings. | |
| Recommendations to policymakers and health care providers for implementing HIV testing in health care settings. | |
| Guidance for sites using or planning to use rapid test kits to detect antibodies to HIV. | |
| Standard A2 | |
|
A template, developed by state health department staff, which service providers can tailor in developing their own data policies and procedures. |
| Detailed guidelines for service providers to help them complete data collection forms correctly; includes pictures, screenshots, and common mistakes. | |
|
An excellent example of a protocol template that can be easily tailored and that contains comprehensive information on CTR programs as well as data collection and recordkeeping requirements. |
| Standard A3 | |
| Comprehensive information developed by NASTAD to help health departments assess their responsiveness to 2001 CDC’s Revised Guidelines for HIV Counseling, Testing, and Referral; includes information about collecting and entering data. | |
B. Staffing, training, and communication
| Resource | Description |
|---|---|
| Standard B1 | |
| Includes examples of position descriptions for an epidemiology supervisor, an epidemiologist, and a data entry operator. | |
| Standard B3 | |
|
Detailed example of a health department’s training manual designed for data-entry personnel; provides step-by-step instructions and screenshots for entering data into a local software system. |
| Excellent example of clear communication between the health department and testing sites about the provision of follow-up training on the data collection guidelines for new HIV test forms. | |
| Document to help health department CTR staff track annual training and technical assistance for service providers to enhance quality assurance of CTR data. | |
|
This Web site features complete, updated list of CTR templates, forms, policies and procedures, and other materials that service providers need to fulfill their data requirements: a model of the kind of easy-to-access and easy-to-use information that health departments can make available to service providers via a Web site. |
C. Data security and client confidentiality
| Resource | Description |
|---|---|
| This Web site features a compendium of key state HIV testing laws and policies. State HIV testing laws vary, and many have been revised since the release of CDC’s 2006 Revised recommendations for HIV testing. | |
| This Web site outlines the policy requirements and staff responsibilities of the department-wide information security program; includes management, operational, and technical information. | |
|
Comprehensive guidance to help state and local health department HIV/AIDS surveillance programs ensure the security and confidentiality of their data. Surveillance programs, compared with most HIV testing programs, have strict guidelines for managing variables because they are more likely to have name-based data or other personally identifying information, but these guidelines offer useful information for CTR programs that are developing, reviewing, or updating security protocols. |
| Standard C1 | |
|
One-page agreement, including a summary of statutes pertaining to confidential public health records and penalties for unauthorized disclosure and an employee confidentiality pledge. |
|
This Web site features tools (an incident reporting form, a sensitive information inventory, a facilities security plan, guidelines for workstation and wireless security) that can be used by health departments to develop security and confidentiality training for new employees. |
| Standard C2 | |
| Agreement between CDC and PEMS users, which outlines specific agreements regarding data collection, entry, storage, use, sharing, retention and disposal, system security, and access privileges; a good example of a written agreement between a central organization and multiple grantees with respect to security and confidentiality. | |
| Information about the security components of PEMS, oversight of those components, activities related to certification and accreditation, user responsibilities, and requirements for the execution of security agreements between CDC and PEMS users. | |
| This document specifies the formal rules of behavior expected of system administrators and communicates policies and procedures to be followed. These include legal and regulatory requirements, authentication management, and information management. | |
| Standard C3 | |
|
Comprehensive guidance to help state and local health department HIV/AIDS surveillance programs ensure the security and confidentiality of their data. Surveillance programs, compared with most HIV testing programs, have strict guidelines for managing variables because they are more likely to have name-based data or other personally identifying information, but these guidelines offer useful information for CTR programs that are developing, reviewing, or updating security protocols. |
| Policy for the use of all IT resources owned by, or operated on behalf of, CDC; describes appropriate personal use of resources, prohibited uses, privacy expectations, and responsibilities. | |
| Standard C4 | |
| Guidelines for medical providers and AIDS service organizations that need to share information to facilitate medical follow-up; includes an example of a consent form, approved by legal counsel, which may be adapted for use in any community. | |
| Standard C5 | |
| Surveillance policy, including relevant security and confidentiality requirements, such as maintenance of information, security investigations, release of information, oath of confidentiality, data transfers, and access to information. | |
|
This document provides counselors and administrators with guidelines for maintaining HIV/AIDS medical records and describes records retention schedules. |
|
Comprehensive guidance to help state and local health department HIV/AIDS surveillance programs ensure the security and confidentiality of their data. Surveillance programs, compared with most HIV testing programs, have strict guidelines for managing variables because they are more likely to have name-based data or other personally identifying information, but these guidelines offer useful information for CTR programs that are developing, reviewing, or updating security protocols. |
|
This Web site outlines the policy requirements and staff responsibilities of the department-wide information security program; includes management, operational, and technical information. |
| Policy and procedures for safeguarding (storing, disseminating, and protecting) data and documents that are sensitive enough to require protection but that may not be designated as classified information. | |
| Principles of collection, use, and warehousing of electronic medical records and claims data; includes summary of related AMA policy. | |
D. Data collection
| Resource | Description |
|---|---|
| Standard D2 | |
| Complete list of National HIV Prevention Program Monitoring and Evaluation (NHME) variables (variable number, variable name, value choices [if applicable], definition, instructions, and CDC’s reporting requirements for each variable). | |
| Detailed explanation of the CTR variables that providers must collect to meet CDC requirements. This manual is focused on the client- and agency-level variables related to CTR services and based on CDC’s revised HIV Test Form (released November 2007). Although not all agencies will use the new HIV Test Form to collect CTR data, the rationales and definitions in this manual are relevant to all health departments funded by CDC to provide CTR services and collect testing data. | |
|
Detailed guidelines for service providers to help them complete data collection materials correctly; includes pictures, screenshots, and common mistakes. |
|
Guidance developed by the California Department of Public Health to help HIV counselors better understand the HIV Counseling Information Form and thus to record client information consistently; provides a detailed explanation of each item on the form and describes the role of the form in HIV counseling. |
| Table of agency ID numbers and agency names: an example of clear guidance concerning variables. | |
| Standard D3 | |
| Health departments funded by CDC to provide HIV CTR services are required to collect data about these activities. This form is designed to follow the flow of a typical CTR session which helps CTR counselors systematically collect the required CTR variables. | |
|
User-friendly form designed to reduce errors in documenting or entering data. |
| Standard D5 | |
| Detailed guidelines for service providers to help them complete data collection materials correctly; includes pictures, screenshots, and common mistakes. | |
| Excellent example of a tracking spreadsheet to facilitate follow-up with clients who test HIV positive. It includes sections to track preliminary positive rapid test results and confirmed/conventional test results. | |
| Form designed to collect behavioral details while checking for internal consistency of responses provided. | |
E. Data entry
| Resource | Description |
|---|---|
| Standard E1 | |
|
Complete, updated list of CTR templates, forms, policies and procedures, and other materials that service providers need to fulfill their data requirements: a model of the kind of easy-to-access and easy-to-use information that health departments can make available to service providers. |
| Standard E2 | |
|
Detailed instructions and graphics for CTR data entry staff of how to process forms, what information should be on forms, what errors can be found on forms, and what information should be entered in the database. |
|
Manual developed to help PEMS users understand and use the application (inclusive of all major releases of PEMS as of April 2008, as well as associated patches up to 3.3.1); includes hyperlinks that allow the document to be used as an online tool. |
| Standard E3 | |
| Example of an error report from the health department to the service provider (including errors identified on each form to assist in correction and resubmission) and a cover letter to service providers. | |
F. Data management and cleaning
| Resource | Description |
|---|---|
| Standard F1 | |
|
Complete list of National HIV Prevention Program Monitoring and Evaluation (NHME) variables (variable number, variable name, value choices [if applicable], definition, instructions, and CDC’s reporting requirements for each variable). |
| Standard F2 | |
|
Cody, R. (1999). Cody’s data cleaning techniques using SAS software. Cary, NC: SAS Institute, Inc. Step-by-step instructions for developing data cleaning programs and macros; includes validation checks (e.g., character data, numeric data, missing values, and date values), searching for duplicate records, double data entry and verification by using the COMPARE procedure, SQL-based solutions, and creating data sets by using general-purpose cleaning and validation programs. |
|
Example of an error report from the health department to the service provider (including errors identified on each form to assist in correction and resubmission) and a cover letter to service providers. |
G. Data submission
H. Data analysis and utilization
| Resource | Description |
|---|---|
| Standard H2 | |
| This Web site features guidelines for addressing both confidentiality and statistical issues in working with small numbers; explains the risk in using small numbers in tables, what constitutes a breach of confidentiality, how to reduce the risk of a breach, the reliability of statistics based on small numbers, and how to address statistical issues. | |
|
One-page checklist to help staff decide whether data can be released in response to an internal or an external request; includes questions about the release of individual information, compliance with guidelines and statutes, and the need for data-use agreements. |
|
Guidelines for medical providers and AIDS service organizations that need to share information to facilitate medical follow-up; includes an example of a consent form, approved by legal counsel, which may be adapted for use in any community. |
| Standard H3 | |
| Web site that features epidemiologic profiles, fact sheets, monthly surveillance reports, slides, HIV counseling and testing annual reports, quarterly HIV data, and many other resources. | |
| Web site that features epidemiologic profiles, HIV and STD surveillance reports, reports to stakeholders, and many other resources. | |
| Web site that features epidemiologic profiles, HIV and STD surveillance reports, and many other resources. | |
| Data (test-level or aggregate-level) on the number of tests, the results of those tests, and information about the demographics and behavioral risk factors of the persons tested; includes (see Technical Notes) the limitations and comparability of the data. | |
Appendix C. Common CTR data errors
Type of Data Error |
Examples |
Possible Resolutions |
|---|---|---|
Missing data are the result when a respondent declines to answer a question, a data collector fails to ask a question or record a respondent’s answer, or a data entry staff member skips the entry of a response. |
A CTR client answered 20 questions, but responses for only 15 variables were recorded on the form. |
Assign value for incomplete or missing data on the basis of known information (e.g., enter or correct site ZIP code information on the basis of the service provider’s location). |
Transcription or transposition errors occur when the data entry person misreads the answer recorded on the form and enters a number or letter that looks like the correct one (but is, in fact, incorrect). |
The number: |
Conduct random spot checks of records |
Out-of-Range errors occur when a number or date lies outside the span of probable or possible values. Many data entry software packages have built-in systems that check whether entered values are out of range. |
The variable is the date of most recent HIV test, but the answer is a future date (e.g., 3/3/2025). This response would be considered outside the acceptable range. |
Correct value for out-of-range data field. |
Duplication errors occur when the same information is entered in the database more than once. |
A CTR client is tested once, and data about that session are entered. The hard-copy data are not clearly marked as “Entered,” and a second person enters those same data. |
Determine whether the record is a duplicate or the ID number has been entered incorrectly. |
Incomplete errors result when a field is not filled out completely. |
The five-digit ZIP code field only contains three digits. |
Communicate with CTR service provider |
Interpretation errors occur when a counselor or client misinterprets the meaning of a variable and records incorrect data. |
For “self-reported result,” the client misinterprets “positive” to mean a good test result and “negative” to mean a bad result, and reports invalid answer. |
Communicate with CTR service provider |
Inconsistent errors require a comparison of more than one data variable to determine whether there are inconsistencies in the data collected. |
If the date of provision of results is before the date of the test, we know that one of those dates must be incorrectly recorded. However, you would not be able to determine that by looking at only one of the data values. Therefore, certain values must be reviewed together in order to reveal errors. |
Communicate with CTR service provider |
Appendix D. Methods used to develop the 2009 QA Standards
Developing the 2009 QA Standards for CTR data required several steps. (See Figure 13.) The 2009 QA Standards represent the best available evidence from the public health data literature, expert opinions, document reviews, and interviews with key staff members of health departments.
Figure 13. Steps in the development of the 2009 QA Standards
Step 1: Planning and coordination
To develop standards that CTR staff and service providers would adopt, it was important to establish the topics to be addressed and how the CTR staff and service providers would use the document. A comprehensive review of the literature was important in identifying resources that could be used to improve the quality of CTR data. After reviewing and summarizing the literature, we shared an outline of the standards with internal and external stakeholders so that they could review and provide feedback. To better understand the issues confronted by health departments, we also reviewed health department materials (e.g., progress reports, CTR QA plans, and CTR data submitted to CDC).
Step 2: Assessing current health department CTR data QA practices
Site visits, including group interviews with CTR staff, helped us document health department policies and practices during each step in the CTR data life cycle. The selection of sites entailed consideration of various factors (e.g., history of timely submission of data, completeness of data, and innovative data use practices). Several health departments were identified as sites with a history of submitting high-quality CTR data. Of these, three sites were invited (and agreed) to participate in a 4-day site visit.
Next, we developed a detailed data-collection protocol, which specified each step in preparation for the site visit, the plan and procedures for the site visit, and the procedures to be followed after the visit. All site visitors were trained in the use of the interview guides and onsite procedures. We conducted group interviews with CTR staff involved in each of the stages of the data life cycle. The interviews were audio-taped, and verbatim transcripts were produced for qualitative coding and analysis. We used the findings of this analysis in developing a draft of the 2009 QA Standard
Step 3: Translating information into user-friendly standards
In the next step, we obtained feedback from internal subject matter experts and external stakeholders during a 1½-day consultation. This consultation involved facilitated discussions designed to ensure that the draft addressed the needs of health departments and CTR programs and that it provided practical guidance for improving the quality of CTR data. Stakeholder feedback from the consultation was incorporated into the initial draft of the 2009 QA Standards, along with resources, tools, and templates. One of the primary objectives of this round of revisions was to further develop the document as a user-friendly and practical resource for health department staff.
Step 4: Piloting the transfer of the 2009 QA Standards into practice
To learn from health departments about the usefulness and applicability of the 2009 QA Standards and to facilitate the transfer of the standards to practice at the site level, CDC collaborated with three health departments to pilot the 2009 QA Standards. Health departments for this pilot were selected on the basis of their interest in participating, their geographic location, and the characteristics of their CTR data system (e.g., method of data entry). Each participating health department worked with an assigned project team member to plan, develop, and document policies and procedures. Project team members documented this process in order to understand how effectively the 2009 QA Standards met health departments’ needs as they developed and applied QA practices to CTR data processing.
The pilot process began with a 2-day introductory site visit to each of the three selected health departments. Site visits were followed by a series of seven weekly conference calls, each of which was focused on a specific aspect of the 2009 QA Standards (e.g., data collection, entry, management, submission, analysis, utilization, and security and confidentiality). Team members facilitated the conference calls with health department CTR and service provider staff. In addition to the weekly conference calls with individual sites, a cross-site conference call was conducted so that staff at each of the health department sites could share information and feedback about the 2009 QA Standards and the process of developing their protocol.
Step 5: Finalizing and disseminating the 2009 QA Standards
After the pilot, the document was revised on the basis of feedback from the health departments and lessons learned from the pilot. The document was also reviewed for clearance by the Division of HIV/AIDS Prevention and revised accordingly. The final step in the PADQUA project was the dissemination of the 2009 QA Standards, in both hard-copy and electronic formats, to health departments and CTR service providers.
