Topic Area 2: Implementation of program review recommendations

As a result of the external program reviews from 2017 through 2019, NIOSH received 60 overarching recommendations concerning Healthcare and Social Assistance (20), Exposure Assessment (11), Construction (11), Emergency Preparedness and Response (5), and Mining Programs (13), with some recommendations having multiple sub-recommendations. Although some panel recommendations were clearly stated and easily interpreted, others were somewhat obtuse. At times the recommendations suggested NIOSH take actions outside its statutory role as a federal research agency. The volume and scope of recommendations in the first round of reviews was so large that NIOSH began to suggest that panels limit the number of recommendations to no more than 10, but this yielded mixed results.

At the conclusion of each review, NIOSH and its partners tend to place great importance on the resulting scores and comparatively little importance on the recommendations. Both the external review panel report and the NIOSH response are posted to the NIOSH program evaluation webpage. The program response is sometimes treated as more of an aspirational plan than a call to action. Improving the use of the recommendations will be an important step toward moving evaluation from a validation of past work to a mechanism for improving NIOSH research programs.

NIOSH leadership recognizes the value of utilizing program review findings and recommendations. As it moves forward, NIOSH will place an emphasis on (1) understanding how to provide information to external review panels in a way that encourages them to focus their recommendations and make their implementation more feasible and (2) increasing consideration of review findings and recommendations in funding opportunities and decision-making at all levels of the institute.

As part of improving the review process, NIOSH plans to build on experience gained through incremental changes to the process in the second and third rounds of reviews. Further improving the evidence packages developed by programs under review and the guidance given to external review panels is needed. The challenge is to help external review panels focus their recommendations without compromising the independence of their reviews.

NIOSH will develop a recommendation-improvement workgroup to review materials from recent external program reviews and to identify process improvements. Materials to be reviewed include review panel criteria, guidance provided to review panels, interactions between NIOSH and review panels, panel recommendations, and NIOSH responses to reports from recent expert review panels. Additionally, the workgroup will use a combination of interviews with DLO management and focus groups of program portfolio leaders who have either been through an external program review or anticipate going through one in the future. The workgroup will produce a report outlining one or more enhancements to the current process for interactions between NIOSH and external review panels. These enhancements will increase utilization of panel findings and recommendations while maintaining the integrity of the external program review process.

Furthermore, to better implement recommendations, the Institute can benefit by taking internal steps to enhance the likelihood that NIOSH programs take advantage of the feedback received from external program review panels. NIOSH will form an implementation workgroup to look at what happens after NIOSH writes its response to expert review panel findings. The workgroup will interview program portfolio and Institute leaders who are responsible for implementation responses and decisions. These interviews will be used to identify barriers and motivators at the researcher, program, and NIOSH levels to implementing feasible review panel recommendations. Interviews will also solicit ideas on (1) how to evaluate and prioritize panel review recommendations and (2) how to leverage Institute funding opportunities and decision-making to incorporate high-priority recommendations into program activities and initiatives. Finally, the workgroup will develop a list of strategies for (1) including review recommendations into funding opportunities and decision-making at multiple levels and (2) communicating review panel recommendations to intramural and extramural researchers, DLO leaders, and other interested parties.

Status

Status

Completed review of external program review materials.

Focus groups and interviews completed

Completed review of external program review materials.

Focus groups and interviews completed

Inputs

Inputs

  • Contribution analysis program reviews
  • Contribution analysis program reviews

Key Learning Question

Key Learning Question

Question 1

  • Understand why review panels might provide recommendations that are beyond the capacity of NIOSH programs to implement. (Completion by end of FY21)
  • Refine the current process for interactions between NIOSH and review panels to increase utilization of panel findings and recommendations. (Completion by end of FY25)

Question 2

  • Identify barriers and motivators at the researcher, program, and NIOSH levels to implementing feasible review panel recommendations. (Completion by end of FY21)
  • Increase the inclusion of panel review recommendations in funding opportunities and decision-making at all levels of the institute. (Completion by end of FY25)

Question 1

  • Understand why review panels might provide recommendations that are beyond the capacity of NIOSH programs to implement. (Completion by end of FY21)
  • Refine the current process for interactions between NIOSH and review panels to increase utilization of panel findings and recommendations. (Completion by end of FY25)

Question 2

  • Identify barriers and motivators at the researcher, program, and NIOSH levels to implementing feasible review panel recommendations. (Completion by end of FY21)
  • Increase the inclusion of panel review recommendations in funding opportunities and decision-making at all levels of the institute. (Completion by end of FY25)

Key Learning Activities (KLAs)

Key Learning Activities (KLAs)

Question 1

  1. Understand why review panels might provide recommendations that are beyond the capacity of NIOSH programs to implement. (Year 1)
  2. Refine the current process for interactions between NIOSH and review panels to increase utilization of panel findings and recommendations. (Year 5)

Question 2

  1. Identify barriers and motivators at the researcher, program, and NIOSH levels to implementing feasible review panel recommendations. (Year 1)
  2. Increase the inclusion of panel review recommendations in funding opportunities and decision-making at all levels of the institute. (Year 5)

Question 1

  1. Understand why review panels might provide recommendations that are beyond the capacity of NIOSH programs to implement. (Year 1)
  2. Refine the current process for interactions between NIOSH and review panels to increase utilization of panel findings and recommendations. (Year 5)

Question 2

  1. Identify barriers and motivators at the researcher, program, and NIOSH levels to implementing feasible review panel recommendations. (Year 1)
  2. Increase the inclusion of panel review recommendations in funding opportunities and decision-making at all levels of the institute. (Year 5)

Start of Process

Start of Process

As the Institute continues to refine the framework for externally reviewing its programs, it has also recognized a need to place more emphasis on utilizing program review findings and recommendations in decision-making as opposed to using reviews simply as a mechanism to ensure its programs are programmatically relevant and impactful.

  • First, the Institute must take steps to improve its review process to encourage reviewers to offer more focused, feasible recommendations, while being careful not to infringe on the independence of reviewers.
  • Second, NIOSH must improve the incorporation of review findings and recommendations in funding opportunities and decision-making. Ideally, this would increase the cost/benefit of such reviews, improve overall programmatic relevance and impact, and allow the Institute to evaluate implementation since the last program review.

As the Institute continues to refine the framework for externally reviewing its programs, it has also recognized a need to place more emphasis on utilizing program review findings and recommendations in decision-making as opposed to using reviews simply as a mechanism to ensure its programs are programmatically relevant and impactful.

  • First, the Institute must take steps to improve its review process to encourage reviewers to offer more focused, feasible recommendations, while being careful not to infringe on the independence of reviewers.
  • Second, NIOSH must improve the incorporation of review findings and recommendations in funding opportunities and decision-making. Ideally, this would increase the cost/benefit of such reviews, improve overall programmatic relevance and impact, and allow the Institute to evaluate implementation since the last program review.

Steps in Process

Steps in Process

KLA-1

  • Review current external program review materials.
  • Interview program portfolio managers and research staff that have been or could be involved in program reviews in the future.

KLA-2

  • Make recommendations, as appropriate, to improve the utility of the external impact review process.
  • Internally review recommendations.
  • Finalize recommendations.

KLA-3

  • Interview leaders of programs and the Institute who are responsible for implementation responses and decisions.

KLA-4

  • Identify a list of strategies for including review recommendations into funding opportunities and decision-making at multiple levels within NIOSH.
  • Internally review strategies.
  • Finalize strategies.

KLA-1

  • Review current external program review materials.
  • Interview program portfolio managers and research staff that have been or could be involved in program reviews in the future.

KLA-2

  • Make recommendations, as appropriate, to improve the utility of the external impact review process.
  • Internally review recommendations.
  • Finalize recommendations.

KLA-3

  • Interview leaders of programs and the Institute who are responsible for implementation responses and decisions.

KLA-4

  • Identify a list of strategies for including review recommendations into funding opportunities and decision-making at multiple levels within NIOSH.
  • Internally review strategies.
  • Finalize strategies.

Next Steps

Next Steps

Develop recommendations and strategies.

Develop recommendations and strategies.

Continue reading the plan using the buttons below

Page last reviewed: December 17, 2020