Skip directly to local search Skip directly to A to Z list Skip directly to navigation Skip directly to site content Skip directly to page options
CDC Home

III. Methods

In August and September 2008, the Centers for Disease Control and Prevention (CDC) published new data indicating that approximately 56,300 new HIV infections occurred in the United States in 2006. This figure is roughly 40 percent higher than CDC's former estimate of 40,000 infections per year, indicating that the HIV epidemic is—and has been—worse than previously known.

Furthermore, new infections have been steadily increasing among MSM since the early 1990s, and the epidemic continues to have a disproportionately severe impact on African Americans and Latinos. Studies have found that many persons at risk for HIV infection are not being reached through current prevention efforts.

These findings have led the Division of HIV/AIDS Prevention to initiate a comprehensive review of its surveillance, research, program, and evaluation portfolios to ensure that they are appropriately configured to address the current epidemic. The review also meets the CDC-wide requirement for a Board of Scientific Counselors (BSC) administered external peer review of scientific programs, including research and non-research (i.e., public health practice), conducted or funded by CDC, to assess scientific and technical quality at least once every five years.

A. Review Process

An external review of DHAP's HIV prevention portfolio was discussed at an open session where Kevin Fenton, M.D., Director of NCHHSTP, and Richard Wolitski, Ph.D., Acting Director of DHAP, presented at the United States Conference on AIDS in September 2008. Such a review was seen as an initial step in developing DHAP's response to the revised numbers of new infections reported at the International AIDS Conference in Mexico City. The External Peer Review (EPR) described in this report was initiated to address this need. The EPR process consisted of selecting a group of external reviewers with extensive experience in the area to be reviewed, the development of a series of guiding questions to steer the review into areas that can provide meaningful input to CDC, the selection of the Peer Reviewers, development of briefing materials on the activities to be reviewed, presentation of these materials, and providing a forum for review and discussion.

The peer review was to obtain external feedback on DHAP's HIV prevention programs, including research and non-research. The review addressed DHAP's entire range of activities, including strategic planning and prioritization; surveillance programs; biomedical intervention and laboratory research; health services and operational research; behavioral, social, and structural intervention research; health department and community-based organization programs for HIV prevention; social marketing activities; capacity-building assistance; and program monitoring and evaluation.

The following aspects of the program were to be addressed:

  • Relevance to DHAP's mission
  • Scope and prioritization
  • Scientific and technical quality, approach, and direction
  • Adequacy of translation and dissemination of research findings for use in programs
  • Strengths, gaps, challenges, and opportunities
  • Extent to which the program addresses the National Center for HIV, Viral Hepatitis, STD, and TB Prevention (NCHHSTP) programmatic imperatives of program collaboration and service integration, reducing health disparities, and maximizing global synergies

Samuel Dooley, M.D., Associate DHAP Director for Science and Program Integration, was charged with the responsibility of leading the review planning and implementation. In this role, Dr. Dooley chaired an internal Peer Review Planning Committee that was entrusted with supporting the external review. The process was divided into five areas, and each area was assigned a staff liaison from the leadership at DHAP:

Steering Committee Samuel Dooley, M.D., Associate Director for Science and Program Integration
Planning Chris Cagle, Ph.D., Associate Director for Policy and Planning
Surveillance Amy Lansky, Ph.D., M.P.H., Acting Deputy Director for Surveillance, Epidemiology, and Laboratory Science
Biomedical Bernard "Bernie" Branson, M.D., Associate Director for Laboratory Diagnostics
Behavioral Cindy Lyles, Ph.D., Acting Deputy Director for Behavioral and Social Science
Program Janet Cleveland, M.S., Deputy Director for Prevention Programs

This DHAP Peer Review Planning Committee came together for weekly meetings to accomplish the following tasks:

  1. Reviewers: submit names of experts in the five areas for inclusion in the External Peer Review
  2. Questions: provide input on the questions to be developed to guide the process
  3. Briefing: coordinate with the Division and the various Branches to obtain the briefing materials to be submitted to the EPR panel members
  4. Presentation: coordinate with the Division and the various Branches to develop presentations to be made in person to their respective panels during the EPR meeting
  5. Resource: serve as a resource to their panels and answer questions as appropriate
  6. Response: coordinate the DHAP response to the findings and recommendations of the respective panels

B. Reviewer Selection Process

The DHAP Peer Review Planning Committee was interested in inviting a group of individuals with experience in HIV Prevention and CDC activities who would be able to provide input to CDC. In order to accomplish this, the committee considered that it was important for the reviewers to have a prior understanding of CDC's activities and would be able to articulate feedback useful to DHAP.

The DHAP Peer Review Planning Committee proposed criteria for selecting EPR Panel members, including the following:

  • Knowledge of CDC activities and programs
  • Experience in HIV prevention
  • Experience in similar processes in the past
  • Racial, ethnic, gender, and geographic diversity, and membership in populations historically served by CDC's programs
  • Governmental and non-governmental experience: individuals from state and local governments; academia; and community-based and national organizations, including longstanding partners of DHAP
  • Former CDC employees were considered eligible only if they had left CDC at least two years prior to being invited to participate in the review

The Planning Committee decided that approximately 70 reviewers would be ideal so that each of the review panels could consist of approximately 8–12 participants and two Co-Chairs. The Co- Chairs for each panel would also come together as a steering committee, which would be chaired by the overall EPR Chairs.

Selecting reviewers was challenging because so many people have a strong interest in HIV prevention and have expertise in the areas being covered by the review. DHAP staff, including the Director and Deputies, Associate Directors, and Branch Chiefs, were queried for suggestions, as was the NCHHSTP Office of the Director. Branch Chiefs encouraged their staff to make recommendations. Names of health department HIV program directors and surveillance coordinators were solicited from the National Alliance of State and Territorial AIDS Directors (NASTAD) and the Council of State and Territorial Epidemiologists (CSTE). An effort was also made to include members of the BSC and the CDC/HRSA Advisory Committee on HIV and STD Prevention and Treatment (CHAC).

A workgroup for each of the five review areas evaluated and prioritized the names that had been suggested. Once nominees were identified and prioritized, the list was reviewed to ensure sufficient diversity. Final prioritization of nominees was done by the DHAP Division Director, Deputies, and Associate Directors, using the same criteria listed above.

C. Chair and Co-Chair Selection Process

The Chairs of the process were entrusted with leading the reviewers to provide input to CDC that would be meaningful in the future. The Chairs were to be able to describe to the panel members and Co-Chairs the purpose of the process and maintain the necessary focus in developing findings and recommendations and serve as Chairs of the Steering Committee, whose role would be to synthesize the input from the various panels into overarching recommendations. In order to achieve this, DHAP sought two individuals with extensive and in-depth knowledge of DHAP's activities who had preferably previously served in an advisory capacity.

After an extensive vetting process, DHAP appointed Edward "Ned" Hook, M.D., and Jesse Milan, JD, to serve as overall Chairs of the process. Dr. Hook is a professor of public health at the University of Alabama at Birmingham; has extensive experience in the area of HIV prevention, surveillance and epidemiology; and currently serves as Co-Chair of the CDC/Health Resources and Services Administration (HRSA) Advisory Committee (CHAC). Mr. Milan, who is a person living with HIV for 27 years, served as AIDS Director for the City of Philadelphia, and is currently Vice President for Community Health Systems of the Altarum Institute. He served for five years as Co- Chair of the CHAC, has served as Board Chair of four nationally recognized AIDS organizations, and was project director of the CDC's National Prevention Information Network (NPIN) for seven years.

The Co-Chairs for the various panels were also selected based on their extensive experience with CDC programs, having participated in similar processes to provide input to CDC in the past, and reflected diversity in terms of organizational affiliation, geography, and demographics. The depth and breadth of experience is reflected in the selections made for the various panels as follows:


David Holtgrave, Ph.D.
Johns Hopkins Bloomberg School of Public Health

Julie Scofield
National Alliance of State and Territorial AIDS Directors

David Ernesto Munar1
AIDS Foundation of Chicago


Douglas Frye, M.D.
Los Angeles County Dept of Public Health

Amy Zapata, M.P.H.
Louisiana Office of Public Health


Kenneth Mayer, M.D.
Brown University/The Miriam Hospital

Mark Thrun, M.D.
Denver Public Health


Cynthia Gomez, M.Ed., M.P.H.
San Francisco State University

Seth Kalichman, Ph.D.
University of Connecticut


Marjorie Hill, Ph.D.
Gay Men's Health Crisis

Beth Meyerson, M.Div., Ph.D.
Policy Resource Group, LL

1 Mr. Munar agreed to serve as Co-Chair of the Planning Panel when David Holtgrave was unable to attend in person. Dr. Holtgrave participated via telephone.

D. Guiding Questions

In order to address the purpose of the EPR, DHAP staff proposed the development of a list of guiding questions, which would be presented to the EPR Reviewers prior to the meeting, and could be used to develop briefing materials and presentations. The questions (see Appendix C) represent the general areas where DHAP staff seeks input regarding programs and activities. The DHAP Liaison Committee guided the process by which the various Panel Liaisons worked with other staff to develop questions specific to each panel and provide input on questions for the Steering Committee. The questions sought to address the following issues regarding DHAP programs and activities among others:

  • Scientific validity and quality of programs and activities
  • Appropriateness of activities to address the need identified
  • Appropriate application of resources
  • Relative mix of activities across issue areas
  • Linkages to DHAP's mission, vision, and goals
  • Extent to which evaluation data is used to inform planning
  • Current gaps and recommendations to address the gaps
  • Recommendations for future programs and activities

The questions received extensive vetting from DHAP staff and from NCHHSTP leadership. Once finalized, the questions set the framework for the development of briefing materials that would assist the EPR Panel members to answer their respective questions.

E. Briefing Materials

In order to prepare for the review, DHAP staff prepared an extensive compilation of materials that would provide information to inform the process. The materials, which included information on current projects and activities and a description of the operating context of DHAP, consisted of two major components:

  • Briefing book: an exhaustive compendium of approximately 700 pages, which contained information on the EPR process, the state of the HIV/AIDS epidemic in the United States, the DHAP organization and funding, the role of DHAP in the federal effort to address the HIV/AIDS epidemic. The book also contained various sections to address each panel's topic areas in detail, including a description of the organizational units (branches), projects carried out, and narrative and funding for each project. The briefing book was made available in hard copy and CD-ROM ahead of the meeting.
  • Presentations: DHAP staff prepared various slide presentations containing some of the more salient information related to the various panels, including the vision, mission, goals, objectives, and current status of some of the more significant projects and activities. The presentations were to be made by DHAP and Branch leadership to provide an opportunity for questions and answers regarding the information presented and to clarify any issues raised during the panel discussions.

F. Conference Calls

DHAP staff coordinated various conference calls with the EPR Chairs to review the process, provide information on the steps to be carried out, and describe the expected outcomes. Rich Wolitski, DHAP Acting Director, led a conference call with Sam Dooley, EPR Planning Lead; Jesse Milan; and Ned Hook to thank them for agreeing to chair the process and to describe the charge to the reviewers, the structure of the EPR meeting in April, and the report development process.

At this and other conference calls, the Chairs and Co-Chairs were able to obtain clarification regarding the process and discuss the process to be used for arriving at findings and recommendations. The Chairs expressed reservations as to whether the Steering Committee would be able to synthesize the discussion of the various panels and answer their own questions on the last day and emphasized the need for the Steering Committee to come together repeatedly to review progress in the various panels.

The Chairs also raised the issue that each panel should decide whether CDC staff would remain in the room during the proceedings. While CDC staff would be available to answer questions, there was a concern that their presence during the discussion may distract the reviewers from the topics to be covered.

The Chairs sought clarification on the level of specificity needed from the Steering Committee and learned that it was CDC's request that the discussion be at a high level rather than project by project and that there was no expectation that the Steering Committee provide input on reallocation of resources. The feedback was expected to address priority activities, groups and types of activities to be continued and expanded, and those that should be phased out or discontinued.

G. EPR Meeting

The full group of reviewers convened on April 13–15, 2009, in Atlanta, Georgia. The reviewers attended an opening plenary session that addressed the current status of the epidemic; the role of DHAP relative to other entities involved in HIV prevention; and DHAP's mission, goals, strategic priorities, organization, and budget.

Following the opening plenary session, the group divided into the five panels, where DHAP senior staff members gave presentations to provide additional information relevant to that panel's specific subject areas and answer clarifying questions from the reviewers. Following the presentations, reviewers had ample time to discuss the questions that were posed to their panel and to begin work on their panel's report and recommendations. DHAP senior staff was available to the panels to address questions that arose throughout the meeting. The meeting concluded with a plenary session in which all reviewers had an opportunity to hear summaries of the major themes and tentative recommendations emerging from each of the panels. The Steering Committee met at lunch and at the end of the second and third days to address its own questions.

H. Adoption of Guiding Questions

As a starting point in the discussion of the various panels and the Steering Committee, the reviewers had to agree on the process to follow, which began with the review and adoption of the questions to be addressed by each of the panels. The panel was able to determine which questions, if any, would be adopted and addressed as part of the discussion. The panels and the Steering Committee were aware that they could adopt, amend, supplement, or discard the questions as appropriate. For the most part, the panels and the Steering Committee addressed the questions as originally posed. Changes to the questions are noted in the individual panel reports.

I. Discussion

Each of the panels adopted the discussion mode the members preferred, generally addressing the issues specifically raised in their questions, including issues related to those under discussion, and asking questions of CDC staff to clarify specific points. The Co-Chairs facilitated the discussion and focused the input into specific findings and recommendations as included here. Each panel was provided with a notetaker who took down questions, issues raised, and the more salient points of the discussion. The various panels were also tape recorded in order for a qualitative data analysis to be carried out, to ensure all relevant points were captured, and to analyze for themes and commonalities among the panels.

J. Report Development

Subsequent to the meeting, the Co-chairs of each panel continued work on their panel's report and recommendations with input from panel members. The Steering Committee, under the direction of the two overall Chairs, oversaw the process of integrating the five panel reports into a single, comprehensive report that also addressed cross-cutting issues. This report was developed through a three-step process:

1) Review and Synthesis of Notes: the notes from the various panels, Steering Committee and plenary sessions were used as the basis of the report, which included the compilation, harmonization, and condensation of the discussion. The notes were converted to bullets and organized under the questions to which they referred in order to address the issues within the purview of the panel or Steering Committee. 2) Qualitative Analysis of Panel and Plenary Discussion: the transcripts of the panels and Steering Committee were analyzed to determine if any important issues were not included and to analyze commonalities and themes among the panels. 3) Review and Input from Chairs and Co-Chairs: various drafts of this report were presented to the Chairs and Co-Chairs for their review and comment. 4) Supporting Materials: background materials were added, which also received input from the Chairs and Co-Chairs, and appear as part of this section and in the appendix.

K. Presentation to DHAP and Board of Scientific Counselors

Once completed and adopted by the EPR Reviewers, this report was presented to DHAP. DHAP staff prepared a programmatic response to address the issues raised and what steps would be taken to address the findings and recommendations included herein.

The final report was presented to the Board of Scientific Counselors of the Coordinating Center for Infectious Diseases at its meeting in November 2009 for its comment and final approval. External

Go to Section IV


Get Tested

Find an HIV testing site near you.

Search location, such as: "Atlanta, GA", or "30033".

For more information, please visit

Please contact with any comments, suggestions, or concerns.

Syndicate This Content Podcast button Tell us what you think button CDC-INFO The U.S. Government's Official Web PortalDepartment of Health and Human Services
Centers for Disease Control and Prevention   1600 Clifton Road Atlanta, GA 30329-4027, USA
800-CDC-INFO (800-232-4636) TTY: (888) 232-6348 - Contact CDC–INFO