This document1 is intended to help health departments funded by the Centers for Disease Control and Prevention (CDC) improve the quality of their HIV counseling, testing, and referral (CTR) data. In addition to providing definitions of the principles, models, and concepts relevant to improving the quality of CTR data, the document describes recommended quality assurance (QA) practices and provides sample tools and templates to facilitate the implementation of QA systems and procedures for processing CTR data. The document also includes recommendations to facilitate staff training, technical assistance, communication, and support to health department staff and service providers in processing CTR data.
The goals of effective CTR monitoring and evaluation are to improve the planning and provision of CTR services and ultimately to decrease HIV morbidity and mortality. High-quality data are needed at the local, state, and national levels to effectively monitor and evaluate CTR service delivery. By providing the recommendations in this document, we intend to foster, among CTR program staff, a culture of high-quality data, including practices and policies that build organizational commitment, stakeholder involvement, and staff capacity.
The development of the 2009 QA Standards was informed by (1) a comprehensive review of the literature on QA of data and the current CTR guidance; (2) a thorough review of current CTR data practices and interviews with CTR staff from three health departments that have a record of achieving high data quality; (3) a stakeholder consultation regarding the draft, with participation from CDC staff, health department CTR staff, and local service providers; and (4) an implementation pilot with three additional health departments (for a more comprehensive description of methods, see Appendix D). If you have suggestions, questions, or comments about the content of this document or future efforts regarding the 2009 QA Standards, please send an e-mail to PEBQAS@cdc.gov or call the Program Evaluation Branch at CDC (404-639-0952).
The primary audience is HIV prevention staff at CDC-supported health departments, including staff in CTR programs. However, CTR data influence, and are influenced by, many people before and after the data reach the health department, including CTR service provider staff, HIV community planning group members, and consumers of CTR data (such as public health researchers, policymakers, and the general public). For example, although this document is specifically for health department staff, it is expected that much of the information will be disseminated to CTR service providers, who are responsible for collecting and transferring client data and who may also be analyzing their own data for local monitoring, evaluation, and planning.
The primary purpose of this document is to encourage health departments to develop a protocol for data QA—a systematic and seamless approach to collecting and processing CTR data (for more information about this protocol, see Section IIA). The 2009 QA Standards provides general guidance from which health departments can develop more specific data QA policies, operating procedures, and tools or templates in order to improve and then maintain high-quality CTR data in the context of their data collection and processing systems.
Throughout this document, we include guidance for policies and procedures to assist with the development of a protocol for high-quality data, in addition to listing resources—example forms, checklists, and reading material—after each standard. These resource documents are included on a compact disc that accompanies the print version and may be helpful in informing or expediting the development or expansion of your health department’s current data QA practices.
After reviewing the 2009 QA Standards, health departments may consider initiating a thorough self-assessment of their current CTR data collection and processing systems, including their readiness and capability to develop a protocol for data QA. After this self-assessment, health departments should be able to decide how and where to invest available resources to most effectively and efficiently improve the quality of CTR data. Health departments should also develop realistic timelines for implementing future efforts for improving data quality.
High-quality CTR data accurately represent CTR service delivery, including the number of HIV tests provided, HIV test results, and the characteristics of clients2 receiving HIV tests. High-quality data are imperative for conducting program monitoring and evaluation for decision making and planning activities. Six primary data quality principles which can be applied to CTR data are precision, reliability, validity, integrity, completeness, and timeliness. The following are descriptions and examples of each principle:
Precision: Data have a high level of precision if they are collected, analyzed, and interpreted at a level of detail that allows effective monitoring and evaluation of a CTR program and informs program management planning and decisions. For example, if a program wishes to determine the percentage of females who test positive for HIV in a jurisdiction and who are African American, the program would need to collect data on race and gender, as well as test results for all persons receiving CTR services. If this information is not collected, the program’s data would lack the precision necessary to answer this important monitoring question.
Reliability: A measure is reliable to the extent that the same, or very similar, results are obtained when the measurement is repeated. Implementing data collection protocols and procedures that yield reliable information is crucial to the overall quality of CTR data. To assist in on-going assurance of reliability, it is important that data collection and processing procedures be documented in a standard and systematic manner.
Validity: A measure is valid if it measures what it is intended to measure. For example, if you want to determine a client’s current HIV serostatus, you ask the client whether a previous HIV test result was positive or negative. If the client misinterprets the question to mean that a “negative” test result is bad and a “positive” test result is good, the response would not be a valid, or accurate, measure of HIV serostatus. To increase the validity of the response, you can clarify the meaning of “negative” and “positive.”
Integrity: Data integrity concerns the preservation of accurate data for their intended use from the time they are collected to the time they are analyzed and reported. Most CTR data stem from the interaction between a counselor and a client. Thus, the finalized data set should accurately reflect the data collected from clients during this interaction.
Completeness: Data should be collected, entered, and submitted in a manner that maximizes the completeness of the data set and minimizes missing values. The completeness of specific variables, including those required by CDC, should be prioritized for collection and reporting. Incomplete data may result from hurried or careless data collection or entry. For example, a counselor who rushes through the data collection process with a client may unintentionally skip variable fields, thus negatively affecting the completeness and quality of the CTR program’s data.
Timeliness: Data entry, management, analysis, submission, and utilization should be conducted on a timely schedule. Timely data help ensure the most up-to-date and relevant representation of CTR service delivery and, ultimately, of the HIV epidemic. For example, if a sexually transmitted disease (STD) clinic that identifies a large number of HIV-positive patients during a reporting period fails to submit data to the health department in a timely manner, the overall number of HIV diagnoses in STD clinics for that period will be underestimated, which may result in misinformed decisions about resource allocation
The primary goals, or outcomes, of a CTR program are to detect new HIV infections, provide clients with their HIV serostatus, and refer HIV-positive clients to appropriate treatment, care, and prevention services. Using CTR data to monitor the provision of CTR activities during a specific period can help program managers and staff members determine whether they are meeting their program objectives. For example, monitoring the number and characteristics of clients tested, the type of test that clients are electing to receive (anonymous or confidential), the test technology (conventional or rapid), the number of persons who receive their test results, and the number of HIV-positive clients who are referred to treatment and care provides crucial information in building and maintaining an up-to-date understanding of service gaps and capacity and in developing or revising program objectives.
Examples of monitoring questions
- How many clients received CTR services during a given period?
- What were the demographic characteristics and risk factors of CTR clients?
- How many clients tested positive for HIV?
- What were the demographic characteristics and risk factors of CTR clients who tested HIV-positive?
- In what settings were CTR services most frequently provided?
- In what settings were the highest proportions of HIV infection identified?
- What type of HIV test (rapid versus conventional) did clients receive in these settings?
- How many HIV-positive clients received their test results?
- How many HIV-positive clients were referred to medical care or prevention services?
The answers to fundamental monitoring questions can be compared with the goals of HIV program planning to evaluate the overall coverage and yield of a CTR program. This comparison can effectively identify potential gaps in CTR service or intervention delivery and assist managers in planning and budgeting CTR services for various communities and populations. CTR data can also be used to improve program operations, manage staff performance, identify training needs, document the performance of service providers, and enhance program accountability by providing evidence to stakeholders and potential funders. The collection of CTR data is also important in allowing state health departments and CDC to report program activities and data output to the general public and other stakeholders. In addition, working with grantees and other national partners, CDC has adopted HIV program performance indicators that describe CDC’s intended performance in critical areas of HIV prevention planning, service delivery, and evaluation. CDC’s HIV prevention grantees, including health departments and community-based organizations (CBOs), are required to report annually on these indicators and are accountable for achieving performance goals. The program performance indicators for CTR examine the percentage of newly identified, confirmed HIV-positive results among all tests reported by HIV CTR sites, the percentage of newly identified, confirmed HIV-positive results returned to clients and the percentage of facilities with a prevalence of HIV-positive tests equal to or greater than the jurisdiction’s target.
If program activities are not being implemented as expected, it is important to quickly identify barriers that may be affecting implementation and take corrective actions. Successfully using data to inform decision making about the funding and management of CTR programs relies on the quality of the CTR data collected, entered, submitted, analyzed, and reported. Sufficient staffing, financial resources, and an appropriate technical infrastructure are critical elements in ensuring high-quality CTR data.
Although the benefits of high-quality CTR data may be readily apparent, significant programmatic risks are involved in reporting or using poor-quality CTR data for decision making. For example, HIV program managers who use poor-quality data may base program plans on incorrect information about which communities most need HIV testing services. Misguided program targets may divert intervention or treatment services from those most in need and fail to maximize limited program resources.
The CTR data life cycle (Figure 1) illustrates how CTR data are processed from the point of data collection through data analysis and utilization, including the consideration of data security and confidentiality at every stage. The quality of CTR data can be compromised or enhanced at any point in the data life cycle, from collection to submission to CDC.
The CTR data life cycle spans three levels that represent three different organizational roles: service providers, health departments and CDC. Service providers initiate the CTR data life cycle at the data collection stage through a client-counselor interaction. As data move upstream, service providers enter data and participate in the initial data verification stage with the health department. At the health department level of the CTR data life cycle, CTR data are managed and cleaned, analyzed and periodically submitted to CDC. At the CDC level, CTR data are managed and cleaned, and a second data-verification stage takes place between CDC and the health department before CTR data are prepared for final data analysis.
Data security and client confidentiality
Data security and client confidentiality practices are critical to ensuring that CTR data are used for maximum public health benefit, with minimum risk of disclosure of client-level information. The fundamental privacy and security protections recommended in this document are presented first because of their importance during all data processes.
Data security involves the application of methods (e.g., data encryption) to protect data from unauthorized access, modification, destruction, or disclosure. New computer technologies have required health departments to develop and routinely monitor more technically sophisticated security policies and practices to prevent breaches in data security and client confidentiality. These activities include controlling who has access to data and how data are accessed or modified. Data security is an ongoing QA issue that should be considered at everystage in the data life cycle.
Client confidentiality is the protection of personal information based on a relationship of trust and with the expectation that client data will not be divulged in ways that are inconsistent with the original disclosure. This protection is not only an important consideration involving technical measures to ensure data security but also falls within the context of other potential breaches of confidentiality, such as the inappropriate disclosure of personal information by health department staff or ineffective security procedures or staff practices that allow unauthorized persons access to client data forms.
The need for CTR data QA begins with the service provider’s collection of data as part of the CTR session between a counselor and a client. Data collection involves the elicitation of client information, including client demographics, risk behaviors, prior HIV test results, and the type and result of the current HIV test. In addition, data are collected on clients’ receipt of results and referrals to care. Data may be collected electronically or on hard copy.
Data entry: transfer and integration
Once CTR data have been collected from the client, service providers are responsible for transferring completed hard-copy forms or electronic data to the health department for integration into a central database. CTR data may be entered in a computer-based application, such as Program Evaluation and Monitoring System (PEMS). Data transfer refers to the point at which the health department receives access to service provider data, whether the data have been transferred electronically or on hard copy. Note that in PEMS, data, rather than being transferred, can be accessed in real time once data-sharing permissions are established. Once CTR data have been transferred from service providers via hard-copy forms or electronically, they will then be entered and integrated into the central health department CTR database for data management and cleaning.
Data management, cleaning, and verification
Effective data management and cleaning require that health departments implement several quality checks at multiple points in the CTR data life cycle to prevent errors, identify errors, and verify and correct data. Data cleaning involves conducting standard methods for identifying and correcting missing, inaccurate, or out-of-range data values. Once data have been cleaned, additional procedures—including recoding or reformatting variables and computing new variables—may be necessary to prepare the database for analysis, reporting, and submission to CDC.
The data verification step, represented in the CTR data life cycle as a circle between data entry and management, involves the identification and resolution of data discrepancies between the health department and service providers. Data verification is critical to the overall quality of CTR data and can be implemented during both the data entry and data management phases by tracking errors to their source (e.g., data collection or data entry) and replacing invalid or missing values with accurate data. A final data verification step is also conducted by CDC after CTR data have been submitted by health departments.
Data analysis and utilization
CTR data that have been verified and cleaned are ready for analysis. Data analysis involves systematically applying statistical and other analytic techniques to describe, compare, and summarize data. At the health department, data analyses are generally designed in accordance with reporting requirements and program planning objectives. Program managers can use CTR data in setting goals or targets, evaluating progress toward goals or targets, and providing information to stakeholders for decision making (e.g., allocating resources or funding, developing or changing policy).
Health departments are required to submit a set of CTR data to CDC for national monitoring and evaluation purposes, which include reports to Congress and the public about CDC-supported HIV testing activities throughout the United States. Data submission from health departments to CDC involves additional data QA tasks, such as the preparation of standard file formats for electronic transfer over the secure data network (SDN). Although the process for submitting CTR data to CDC will depend on the data collection and entry system3 that a health department is currently using (e.g., PEMS, scanning, or XPEMS), it is very important that data be prepared and submitted according to a protocol to ensure the security and integrity of all data.
3Currently, these are the three systems that health departments use to process CTR data. (1) PEMS is a secure Web-based data collection and management application designed for health departments and community-based organizations and maintained on CDC servers. (2) Scanning refers to optical character recognition technology used to process and submit CTR data in file formats specified by CDC. (3) Agencies that use XPEMS submit data (in file formats specified by CDC) from their own data systems rather than through PEMS. Agencies that use XPEMS are responsible for modifying their systems to meet reporting requirements for national-level program monitoring.