Topic Area 1: Collection and documentation of intermediate outcomes

NIOSH sees value in continuing to use the CA framework for its external program reviews. However, to successfully and efficiently implement CA moving forward, NIOSH must make a substantial culture change by proactively identifying, collecting, and documenting (IOs). The Institute defines IOs as actions taken by stakeholders in response to NIOSH products or efforts. For example, a manufacturer might commercialize a NIOSH technology prototype or an employer might adopt a NIOSH recommendation into the workplace.

IOs are critical to asserting that a NIOSH program’s contributions have had an impact on health and safety outcomes. They demonstrate that evidence-based products and findings/recommendations have been adopted by intermediaries and end users and therefore are likely to positively impact the health and safety of the users. Like others in the federal government, NIOSH has not historically collected information on such outcomes. Although the Institute has documented reach metrics such as webpage views and downloads and scholarly metrics such as citations in peer-reviewed journals, it received feedback from all five expert review panels that it needs to move beyond these types of metrics and document use and adoption.

Therefore, this learning question focuses on integrating the collection and documentation of IOs into NIOSH processes. Adding another step to any process requires an understanding of the motivators and barriers from the perspectives of key implementation groups. Therefore, NIOSH will conduct focus groups with its researchers, middle managers and program leaders and interviews with division/laboratory/office (DLO) managers, to gain their perspective about what will work and what will not work. NIOSH will use this information to develop targeted educational and awareness building material to use with those groups as appropriate during future trainings, presentations, and as reference material. However, before disseminating that material,, it will be crucial to identify NIOSH staff members to serve as evaluation resources within each DLO to help facilitate this push to increase the knowledge and value of IOs , including the role they play in the larger evaluation picture. This will ensure that every DLO has staff with some basic understanding of evaluation concepts and possible resources so that others in the DLO have an initial point of contact to seek this information.

Status

Status

Focus groups and interviews conducted, analyzed, and report complete.

Focus groups and interviews conducted, analyzed, and report complete.

Inputs

Inputs

  • Contribution analysis (CA) program reviews
  • Contribution analysis (CA) program reviews

Key Learning Question

Key Learning Question

How can NIOSH incorporate the collection and documentation of intermediate outcomes (IOs) into its processes?

How can NIOSH incorporate the collection and documentation of intermediate outcomes (IOs) into its processes?

Key Learning Activities (KLAs)

Key Learning Activities (KLAs)

  1. Assess motivations and barriers to collecting IOs. (Completion by end of FY21)
  2. Prepare for education and awareness raising activities. (Completion by end of FY23)
  3. Raise awareness and educate NIOSH staff about IOs. (Completion by end of FY24)
  1. Assess motivations and barriers to collecting IOs. (Completion by end of FY21)
  2. Prepare for education and awareness raising activities. (Completion by end of FY23)
  3. Raise awareness and educate NIOSH staff about IOs. (Completion by end of FY24)

Start of Process

Start of Process

To date, the CA framework has served as a meaningful way to assess the relevance and impact NIOSH’s research programs. However, because this framework is dependent on the Institute’s ability to demonstrate IOs (such as the adoption of NIOSH products and efforts by stakeholders), and in order to sustain implementation of this framework, the Institute must begin to build in more systematic efforts to collect and document IOs.

To date, the CA framework has served as a meaningful way to assess the relevance and impact NIOSH’s research programs. However, because this framework is dependent on the Institute’s ability to demonstrate IOs (such as the adoption of NIOSH products and efforts by stakeholders), and in order to sustain implementation of this framework, the Institute must begin to build in more systematic efforts to collect and document IOs.

Steps in Process

Steps in Process

KLA-1

  • Conduct focus groups with researchers, division/laboratory/office (DLO) branch managers, and program leaders.
  • Conduct interviews with DLO directors.

KLA-2

  • Develop further guidance around defining and documenting IOs.

KLA-3

  • Identify NIOSH staff to serve as evaluation resources within each DLO.
  • Develop and implement an awareness-raising and educational campaign around IOs.
  • Support effort to develop and/or expand upon current methods to collect IOs.

KLA-1

  • Conduct focus groups with researchers, division/laboratory/office (DLO) branch managers, and program leaders.
  • Conduct interviews with DLO directors.

KLA-2

  • Develop further guidance around defining and documenting IOs.

KLA-3

  • Identify NIOSH staff to serve as evaluation resources within each DLO.
  • Develop and implement an awareness-raising and educational campaign around IOs.
  • Support effort to develop and/or expand upon current methods to collect IOs.

Next Steps

Next Steps

Develop educational and awareness raising materials around IOs.

Develop educational and awareness raising materials around IOs.

Continue reading the plan using the buttons below

Page last reviewed: December 17, 2020