|
|
Volume 5:
No. 4, October 2008
TOOLS AND TECHNIQUES
The National Diabetes Education Program Evaluation Framework: How to Design an Evaluation of a Multifaceted Public Health Education Program
Joanne Gallivan, MS, RD, Rachel Greenberg, MA, Clarice Brown, MS
Suggested citation for this article: Gallivan J, Greenberg R, Brown C. The National Diabetes Education Program evaluation framework: how to design an evaluation of a multifaceted public health education program. Prev Chronic Dis 2008;5(4).
http://www.cdc.gov/pcd/issues/2008/
oct/07_0191.htm.
Abstract
The National Diabetes Education Program, cosponsored by the National Institutes of Health and the Centers for Disease Control and Prevention, employs mass media communications, public-private partnerships, and dissemination of information and education tools to address the diabetes epidemic in the United States. The program’s goal is to help reduce the morbidity and mortality from diabetes and its complications by improving the treatment and outcomes for people with diabetes,
promoting early diagnosis, and preventing onset of diabetes. Evaluation is an integral component of the National Diabetes Education Program’s planning and implementation process. The program’s evaluation is based on the Centers for Disease Control and Prevention’s Framework for Program Evaluation in Public Health, which has guided program planners and evaluators in developing measurable short-term, midterm, and long-term outcomes. We describe how the National Diabetes
Education Program has applied the evaluation framework, demonstrating how multifaceted health communications programs can design program evaluations to answer key questions about program processes and outcomes.
Back to top
Introduction
Diabetes was the seventh leading cause of death in the United States in 2006; approximately 233,619 deaths
were attributed to the disease (1).
In 2007, an estimated 23.6 million people had diabetes, representing 7.8% of the US population (1). Of these, 17.9 million cases
were diagnosed, leaving 5.7 million people unaware they had diabetes. Another estimated 57 million adults aged 20 years
or older had prediabetes in 2007, putting them at increased risk for diabetes (1).
The US Department of Health and Human Services launched the National Diabetes Education Program (NDEP) in 1997 to improve diabetes management and
to help reduce the morbidity and mortality from diabetes and its complications. The leading federal government program that promotes diabetes prevention and control, NDEP is cosponsored by the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health (NIH) and the Division of Diabetes Translation of the
Centers for Disease Control and Prevention (CDC). The aim of NDEP is to improve the treatment
of diabetes and its complications, to promote early diagnosis, and to prevent the onset of diabetes (2). To reach these goals, NDEP has formulated the following program objectives:
-
Increase awareness of the seriousness of diabetes, its risk factors,
and strategies for preventing diabetes and its complications among groups at
risk.
-
Improve understanding about diabetes and its control and promote better self-management behaviors among people with diabetes and their social supporters.
-
Improve health care providers’ understanding of diabetes and its control and promote an integrated approach to care.
-
Promote health care policies that improve the quality of and access to diabetes care.
-
Reduce health disparities in racial and ethnic populations disproportionately affected by diabetes.
Guiding principles of NDEP implementation and evaluation
Four central principles guide NDEP’s planning, implementation, and evaluation activities. These principles are based on effective approaches used by the National High Blood Pressure Education Program, the National Cholesterol Education Program, and other education programs during the past 30 years (3).
The first principle is that the program must be based on scientific evidence, including epidemiologic, clinical, and demonstration studies. This evidence shows that much of the
illness and death associated with diabetes and its complications can be prevented or delayed by aggressive treatment with diet, physical activity, and pharmacologic approaches that help to normalize blood glucose levels, blood pressure, and lipids (4). Research shows that type 2 diabetes can be
prevented or delayed through modest weight loss and regular physical activity
(5). NDEP translates the science of diabetes prevention and treatment into public, patient, and professional education messages and materials, community intervention tool kits, and awareness campaigns. These are disseminated through NDEP’s publications clearinghouse, its Web sites, and its partners.
The second principle is that an effective education program must involve
various organizations that operate in partnership to achieve program goals and objectives (3).
A component of NDEP is its partnership network of approximately 200 public- and private-sector organizations. Program partners disseminate and promote NDEP’s mass media campaigns and educational messages through national, state, and local communication channels. They also provide guidance on developing
appropriate messages and strategies by participating in audience-specific work groups. These work groups meet monthly via telephone conference calls and
every 2 years at face-to-face meetings, where they develop and review the progress of their respective strategic plans, media messages, educational products, and community channel activities.
The third principle is that public, patient, and professional education must use effective communication strategies to reach selected target audiences (3). NDEP offers a wide range of resources to support 2 major public education campaigns:
“Control Your Diabetes. For Life.” and “Small Steps. Big Rewards. Prevent Type 2
Diabetes.” Each campaign offers partners a wealth of tools — brochures, tip sheets, public service advertising, health care provider tool kits, community intervention guides, and more — for conducting outreach activities in communities across the country.
Consumer materials are carefully tailored for groups at highest risk for diabetes, including older adults, African Americans, American Indians, Alaska Natives, Hispanics and Latinos, Asian Americans and Pacific Islanders, and women with a history of gestational diabetes. Many NDEP educational and promotional materials are available in as many as 15 languages, including Spanish and Tagalog.
The fourth principle is that evaluation must be an integral component of program planning and implementation and
must be used as part of an iterative process of re-planning and refining program activities (2). This principle has inspired a comprehensive approach to NDEP evaluation, encompassing both process and outcome evaluation. The process evaluation monitors program implementation and short-term effects. NDEP uses the resulting findings to identify areas in need of midcourse correction or
continuation. The outcome evaluation focuses on the midterm and longer-term effects of NDEP’s efforts, particularly NDEP’s and partner organizations’ promotion and outreach activities to target audiences. Progress on these midterm and longer-term outcomes is measured by tracking changes in consumer and health care provider awareness, knowledge, attitudes, beliefs, and behaviors
about diabetes prevention and control.
Framework for Program Evaluation in Public Health
NDEP bases its program evaluation on the Framework for Program Evaluation in Public Health (6). The framework, developed to help ensure that public health programs “remain accountable and committed to achieving measurable outcomes” (6, p.
1), encompasses 6 steps:
We describe how NDEP has applied the evaluation framework, providing a case study of how a multifaceted public health communications program can design program evaluations to help answer key questions on program processes and effects.
Back to top
Step 1: Engage Important Stakeholders
Engaging stakeholders is central to establishing a common frame of reference about the program
and the key evaluation questions to be asked, and to ensuring that the evaluation findings
are used for program improvement. In designing its evaluation, NDEP involved members of its
partnership network, which includes the leading national organizations of health professionals and volunteers concerned about diabetes, state diabetes prevention and control programs, community health professionals and organizations concerned about
diabetes, and representatives from private industry. Members of the Partnership
Network participate in NDEP’s Steering and Operations committees and in the program’s 10 work groups. NDEP engaged key stakeholders in several ways.
Stakeholders involved in program operations
These stakeholders are engaged in program evaluation through the program’s Operations Committee. The committee meets twice
per year to review the findings of process evaluation activities, such as NDEP’s semiannual Partner Activities Survey, and to review progress
made toward achieving the program’s strategic planning goals.
Stakeholders served or affected by the program
These stakeholders are the approximately 200 members of the NDEP Partnership
Network that includes all the partner organizations represented on the program’s Steering Committee, members of the work groups, and the 59 state and territorial diabetes prevention and control programs. These partners help promote and disseminate NDEP’s awareness campaigns, educational materials, and resources on diabetes prevention and control.
NDEP engages these stakeholders in program evaluation through the program’s semiannual Partner Activities Survey, which asks partners to report on their promotion and dissemination efforts related to NDEP campaigns.
Partners are sometimes engaged in NDEP’s formative research activities, such as pretesting or pilot-testing NDEP products before they are produced in final form.
Primary users of the evaluation
These stakeholders include NDEP’s Executive Committee, composed of the directors of the diabetes divisions at NIH and CDC, the chairs of the steering and operations committees, and the program directors of NDEP at NIH and CDC. NDEP formed an Evaluation Work Group that provides guidance on program evaluation design and implementation. The roles and responsibilities of the Evaluation Work Group members include
- Reviewing and updating the evaluation plan.
- Providing impact and outcome data from the organizations they represent for NDEP evaluation efforts.
- Identifying strategies to fill gaps in data for the evaluation plan.
- Reviewing and providing counsel for NDEP surveys and other research tools.
Back to top
Step 2: Describe the Program
In the CDC framework, a consensus program description helps develop the evaluation focus, associated indicators, and data sources. A strong, comprehensive program description also integrates the planning and evaluation process, as was the case for NDEP.
NDEP’s program description emerged from an extensive review of the literature on health behavior research on diabetes and from strategic planning meetings with experts in diabetes education and representatives of stakeholder groups. The review of the literature and the strategic planning meetings provided direction for developing the program’s first strategic plan (NDEP, unpublished data, 1997). The plan formulated a statement of need for the program and
presented key
objectives, target audiences, messages, strategies, and tactics it would employ.
The NDEP Evaluation Work Group provided guidance on developing outcome measures for evaluating the program by prioritizing a 3-tiered list of outcome measures for each target audience. From the list of prioritized outcome measures, the work group developed a conceptual program evaluation framework that included an overview of program resources, program activities, process goals, and intermediate and long-term goals (Figure 1).
Figure 1. National Diabetes Education Program
Conceptual Framework [A text description of this
figure is also available.]
Abbreviations: NIH, National Institutes of Health; CDC, Centers for Disease
Control and Prevention.
The evaluation work group and staff used the conceptual framework to develop the more explicit logic model. For example, in looking at the program’s glucose control component, the conceptual framework enabled NDEP to develop a logic model that included key strategies to increase knowledge, influence attitudes and beliefs, and change behaviors through its mass media messages and educational materials and activities.
As shown in Figure 2, the framework allowed NDEP to
specify the intended intermediate and long-term outcomes the program was seeking related to blood glucose control among people with diabetes.
Figure 2. Strategies and Outcomes in Glucose
Control [A text description of this figure is
also available.]
Back to top
Step 3: Focus the Evaluation
As a national education program, NDEP focuses its evaluation efforts on
monitoring and assessing what the program’s mass media campaigns, educational
materials, and other promotional activities can plausibly influence; for example, increasing knowledge of the importance of controlling risk factors for cardiovascular disease in people with diabetes, improving a person’s feeling of self-efficacy to take charge of diabetes self-management, or increasing the practice of self-monitoring
of blood glucose levels. At the same time, NDEP monitors long-term outcomes, such as prevention of diabetes-related complications, by tracking incidence and prevalence of blindness, kidney failure, amputations, and cardiovascular disease as reported
by the National Health Care Surveys conducted by CDC’s National Center for
Health Statistics (NCHS). The program monitors intermediate outcomes, such as
the health status of people with diabetes who have major risk factors that contribute to diabetes complications,
using data from the National Health and Nutrition Examination Survey (NHANES) and the
National Health Interview Survey (NHIS) conducted by NCHS and CDC’s Behavioral Risk
Factor Surveillance System (BRFSS). NDEP monitors these trends and uses the
results to determine what messages and educational activities the program needs to emphasize.
Back to top
Step 4: Gather Credible Evidence
The outcomes specified in the NDEP evaluation lend themselves to survey data
collection. Because of budget and regulatory constraints on survey research by federal government programs, NDEP devised a
multifaceted approach for measuring program outcomes
using survey data. The program uses a combination of existing data available from ongoing federal government and partner organization surveys related to diabetes and supplements these data by conducting its own survey research.
Monitor and track available data
CDC’s NCHS coordinates several surveys that incorporate diabetes-related questions, including NHANES, NHIS, and BRFSS. NDEP has used data from these surveys to track many of its outcome measures, including epidemiologic trends in diabetes and cardiovascular risk factor control as well as public knowledge,
attitudes, beliefs, and health behaviors regarding diabetes.
Add NDEP questions to ongoing health surveys
To fill gaps in national data collection on key NDEP outcome measures, the program successfully negotiated adoption of several questions into the NHANES and NHIS diabetes question sets. Responses to these questions will provide NDEP with data
about people with diabetes and people at high risk for diabetes.
Use relevant data from partners
Since the program’s inception, NDEP has worked closely with the American Diabetes Association (ADA), the leading health professional and voluntary organization concerned about diabetes, to coordinate evaluation activities. ADA has a strong market research program that includes annual surveys of people with diabetes. The association also commissions periodic physician surveys and national omnibus surveys to track knowledge, attitudes, and practices related to
diabetes among the general public and people with diabetes. ADA routinely consults with NDEP
about questions to include in these surveys and shares nonproprietary data with the program. This collaboration has provided NDEP with baseline and benchmark data for tracking changes in knowledge, attitudes, and practices for several short-term outcome measures.
Other partners, such as the American Academy of Nurse Practitioners (AANP), conduct routine surveys of their members
about knowledge, attitudes, and practices related to various health conditions. The AANP shared their findings related to diabetes management with NDEP and agreed to assist NDEP with conducting survey research among its members.
Conduct original survey research
NDEP applied for supplementary funding from the NIH Evaluation Branch, explaining the need for the research and how it would be used in program evaluation. Concurrently, NDEP sought permission from the federal Office of Management and Budget (OMB) to conduct original survey research. Federal government regulations restrict agencies from fielding survey research among 10 or more people without OMB approval. These restrictions are to minimize the “paperwork burden”
on the public.
After NDEP received funding from NIH and approval from OMB, a national telephone survey was
conducted in 2006. NDEP targeted adults aged ≥45 years, the population at highest risk for diabetes and prediabetes. Because diabetes is more prevalent among minority populations, African American and Hispanic/Latino households were oversampled. The survey results will
fill most of the remaining gaps on several key evaluation questions about the public’s knowledge of diabetes and prediabetes and will provide direction for shaping educational messages and materials.
Conduct process evaluation
NDEP supplements the extensive outcome data monitoring detailed above with an ongoing system of process evaluation, including the following:
- Tracking television, radio, and print public service advertising placements (ie, the reach and frequency of messages, in terms of the number of placements [frequency] and the size of the audience [reach]).
- Tracking results of press release dissemination though a clipping service (ie, the number of news stories placed and the geographic location and circulation of the publications).
- Tracking the number of publications ordered from the National Diabetes Information Clearinghouse, NDEP’s fulfillment organization, and the number of public inquiries to CDC’s Division of Diabetes Translation.
- Gathering monthly Web statistics on the number of unique visits to the NDEP Web site and the number of downloads of the top
100 most popular publications.
Another key element of NDEP’s process evaluation involves monitoring and tracking NDEP-related activities conducted by members of the
partnership network through a semiannual Partner Activities Survey. This online survey asks partners to report on the types of promotion and dissemination activities they have conducted in the past 6 months for each NDEP
priority campaign message. This survey also gathers feedback on partners’ satisfaction and challenges with NDEP’s campaigns, educational materials, and operations.
Back to top
Step 5: Analyze and Interpret Data
NDEP reports the results of its process and outcome evaluations to program stakeholders in many ways. One is through NDEP’s annual report on evaluation activities and the latest data on outcome measures, which is presented to the program’s
steering committee. A summary of this report is incorporated into the steering committee meeting minutes, which are disseminated to the entire
partnership network. Similarly, a report on NDEP evaluation is included on the agenda of the
partnership network’s meetings held every 2 years.
The results of the Partner Activities Survey are compiled and disseminated to all members of the
partnership network. For example, recent surveys indicate that most partners who respond to these semiannual surveys conduct activities to promote NDEP’s awareness campaigns. Many of them exhibit at conferences and meetings, participate in health fairs, make presentations, or conduct training sessions and workshops to promote NDEP’s messages and materials. Nearly all the partners
promote NDEP’s Web site resources to their colleagues and constituents.
The survey reports also provide data on partner activities to the various work groups, who use the information
to assess progress in implementing their strategic plans. NDEP has begun preparing process measure reports on media placements and publications dissemination related to each work group’s strategic plan. For example, the report to the Hispanic/Latino
work group provides process measures on all of the
public service announcements and article placements, the number of Spanish-language publications distributed by the program’s clearinghouse, and the number of Web downloads of Spanish-language materials.
In 2001 and 2004, NDEP completed and disseminated progress reports on the program’s accomplishments, covering 1997 through 2003. To mark NDEP’s 10-year anniversary, the program released a 1997-2007 report in September 2007 (7). This report presented the latest available outcome data on NDEP’s priority campaigns and summarized process measures regarding media activities, publications dissemination, and Web site usage. These reports provide members of the
partnership network
and other NDEP constituents with information about where the program has been and where it is headed.
These 3 examples illustrate that people with diabetes have a better understanding of diabetes and are taking steps to control it:
- People with diabetes have shown a dramatic increase in awareness of one of the key measures of diabetes control, the
hemoglobin A1c test. According to surveys by the ADA from 1998 to 2004, awareness of the A1c test doubled from 31% to 60% (7).
- People with diabetes report a significant increase in self-monitoring of blood glucose at least once
per day from 39% in 1997 to 63% in 2006. Patients who regularly test for glucose are more engaged and active in the management of their diabetes — a critically important step in taking control (7).
- There are also signs that people with diabetes are taking steps to improve outcomes. Comprehensive control of diabetes — control of A1c, blood pressure, and cholesterol (the ABCs of diabetes) — is a key measure of progress in controlling diabetes. Findings in the 2 most recent NHANES surveys indicate a significant increase in people with diabetes taking medications to control cholesterol and hypertension (7).
To inform the diabetes, health professional, and public health communities about NDEP’s activities, accomplishments, and lessons learned, NDEP writes and submits journal articles. NDEP staff members and partners make presentations about the program at health professional and public health meetings. The program has been on the agenda of periodic
prevention summits of the US Department of Health and Human Services, where representatives have had the
opportunity to address public health professionals about NDEP’s activities and progress.
Back to top
Step 6: Use the Findings
NDEP uses the evaluation activities and reports to assess progress toward reaching program goals and to make midcourse corrections in program activities. NDEP also updates strategic plans, including those for each work group, every 3 years. Program staff members work closely with each work group to review previous plans, assess the latest outcome and process measures, and then develop updated plans for the next 3-year period.
NDEP updates its data as new data from NHANES, NHIS, and BRFSS become available each year. The program has created a database for tracking results from these surveys and charting trends on key intermediate outcomes. These trend data are reviewed and shared with the
evaluation work group, the executive committee, and the operations committee to keep them abreast of changes in outcome measures.
NDEP also used the data to help set priorities and allocate program resources for messages and
campaigns that need to be reinforced and disseminated to selected target audiences. Similarly, results of the NDEP survey of the public’s knowledge, attitudes, and practices related to diabetes are assisting NDEP in understanding its target audiences, specifically people with diabetes, people with prediabetes, and people at risk for diabetes, and to inform the program of key issues to consider in program planning.
NDEP analyzes the results of the semiannual Partner Activities Survey to determine what, if any, changes are needed in program operations to better serve partners’ needs. The Partner Activities Survey also has provided insights
about the role partners play in promoting and disseminating NDEP messages and what gaps the national program staff needs to fill.
Back to top
Conclusion
The CDC Framework for Program Evaluation in Public Health is an effective tool for structuring an evaluation process that has a high probability of producing findings that programs can use for improvement. It has given the NDEP a roadmap for asking key evaluation questions and for identifying practical solutions to obtaining answers. The process also has been helpful in educating NDEP’s funders, partners, and other stakeholders
regarding useful, feasible, appropriate, and accurate
measures for evaluating a multifaceted public and professional education program such as NDEP, which is significantly different from a randomized clinical trial or preintervention/postintervention
study, the research model that is more familiar to stakeholders.
Engaging stakeholders in the design and implementation of NDEP’s evaluation has been a critical strategy. Without the support, guidance, and contributions of the many individuals and organizations that have been involved in the evaluation process, NDEP would have many more gaps in its evaluation framework.
Developing a logic model of the program and identifying the essential evaluation measures needed to assess program outcomes has helped to streamline the evaluation design and eliminate many irrelevant measures. As the program evolves, NDEP will add new components to the logic model, building on previous elements.
Most importantly, NDEP has taken to heart the iterative nature of evaluation as represented by the CDC
framework. The true value of this approach for NDEP will be its attention to continuous refinement of the program’s efforts based on evaluation findings —
that is, closing the loop between data and action.
Back to top
Acknowledgments
The authors thank Thomas J. Chapel, MA, MBA, senior evaluation scientist at CDC, Office of Workforce and Career Development, Atlanta, Georgia, for his thoughtful guidance and encouragement with preparation of this article.
Ms Greenberg is a consultant to the National Institute of Diabetes and Digestive and Kidney Diseases and performed the work on this article under contract no. HHSN267200700073P. Ms Brown, an employee of Social and Scientific Systems, performed the work on this article under contract no. 263-FK-417122-1. This article was funded in part by the NIH Evaluation Set-Aside Program (Agency Project # 04-207-NIDDK) administered by the Evaluation Branch, Division of Evaluation and Systematic
Assessment, Office of Portfolio Analysis and Strategic Initiatives, Office of the Director, NIH.
Back to top
Author Information
Corresponding Author: Joanne Gallivan, MS, RD, Director, National Diabetes Education Program, National Institutes of Health, Bldg 31, Room 9A06, 31 Center Dr, Bethesda, MD 20892. Telephone: 301-496-6110. Fax: 301-496-7422. E-mail: Joanne_Gallivan@nih.gov.
Author Affiliations: Rachel Greenberg, Washington, District of Columbia. Clarice Brown, Surveys and Epidemiology Services Division, Social and Scientific Systems, Silver Spring, Maryland.
Back to top
References
- National diabetes statistics fact sheet: general information and national estimates on diabetes in the United States, 2007. Bethesda (MD): National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases; 2008.
- National Diabetes Education Program. Changing the way diabetes is treated:
a progress report from the National Diabetes Education Program. Bethesda (MD): National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases; 2001.
- Office of Cancer Communications. Making health communication programs work. Bethesda (MD): National Institutes of Health, National Cancer Institute; 2002.
- Knowler WC, Barrett-Connor E, Fowler SE, Hamman RF, Lachin JM, Walker EA, et al.
Reduction in the incidence
of type 2 diabetes with lifestyle intervention or metformin. N Engl J Med
2002;346(6):393-403.
-
Intensive blood-glucose control with sulphonylureas or insulin compared with
conventional treatment and risk of complications in patients with type 2
diabetes (UKPDS 33). UK Prospective Diabetes Study (UKPDS) Group.
[Published erratum in: Lancet 1999;354(9178):602] Lancet
1998;352(9131):837-53.
- Centers for Disease Control and Prevention.
Framework for program evaluation in public health.
MMWR Recomm Rep 1999;48(RR-11):1-40.
- National Diabetes Education Program. The National Diabetes Education Program: 10
years of progress, 1997-2007. Bethesda (MD): National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases; 2007.
http://www.ndep.nih.gov/diabetes/pubs/NDEP_ProgressRpt07.pdf.
Back to top
|
|
