Step 4 Checklist

Step 4: Gather Credible Evidence

In Step 4, evaluators build on the focus of the evaluation (Step 3) by establishing what evidence is needed to answer the evaluation questions, what, how, and when data will be collected, and from whom. The data collection strategy should be developed during this step to define methods, indicators, measures, and data sources before you begin collecting your data.

Applying the Cross-Cutting Actions and Evaluation Standards

As with all the evaluation framework steps, it is important to integrate the cross-cutting actions and evaluation standards when gathering credible evidence in Step 4. See Table 8 in the CDC Program Evaluation Framework, 2024 to determine if you have effectively applied the cross-cutting actions and evaluation standards.

Establish Expectations

☐ Engage with interest holders to determine:

• What evidence will be used to answer the evaluation question(s)

• What expectations they have about the type, quality, and quantity of data needed

• What changes, trends, or patterns would suggest the program is on track or doing well (i.e., what does success look like?)

• What types of evidence different interest holders will value

• If you will use new or existing data

• If you will use primary or secondary data

Choose Appropriate Methods

☐ Determine which quantitative and qualitative data collection methods to use by considering:

• Resource availability

• Ethical considerations

• Size and scope of the evaluation

• Validity (e.g., how well the evaluation measures what is intended)

• Reliability (e.g., ability to replicate findings)

• A review of evaluation literature on programs or interventions similar in topic, context, or stage of development to identify which methods have been effective and appropriate.

Identify Relevant Indicators

☐ Identify if the common indicator categories are included (e.g., inputs, activities and outcomes):

• Inputs: Develop measures of resources needed to conduct program activities

• Activities: Develop measures of program implementation efforts and the products of those program activities (Remember, outputs are also included in activities.)

• Outcomes: Develop measures of expected results or changes from the program, often broken out as short, intermediate, and long-term outcome. Ensure the outcome indicators show directionality.

☐ Refer to Worksheets 4B and 4C for selecting appropriate indicators

☐ Review indicators and measures for appropriateness given evaluation context (refer to Step 1 Checklist):

• Do they meet SMART (Specific, Measurable, Achievable, Relevant, Timebound) criteria?

• Are they sensitive and relevant to the community?

☐ Refer to Table 4.1 Data Collection Methods and Worksheet 4A Identifying Data Collection Sources/Methods

Determine Appropriate Data Sources, Data Quantity, and Data Quality

☐ Determine if there are existing instruments that can be adapted to fit the evaluation context

☐ Develop a protocol for instrument use to ensure consistency and completeness of data

☐Develop questions or prompts that align with the previously selected indicators

☐ Develop the format of response options for the items (such as open-ended, multiple choice, or ranking)

☐Pilot the data collection instruments with interest holders to make sure the questions are clear

☐Revise the data collection instrument based on feedback from the pilot test

☐Finalize the data collection instrument

☐ Train the data collection team to use the instruments and any associated protocols

☐Develop a data management plan, including considering data security

☐ Communicate the above information to interest holders throughout the development process