A Guide to Developing a TB Program Evaluation Plan -
Webinar Transcript
Return to Guide Main Menu
November 14, 2005
2:00 p.m. CST
(Extremely poor audio made transcription very difficult)
Coordinator ...Maureen Wilce, ma’am you may begin.
(Audio cut-outs)
Maureen
Hello thanks, everybody for waiting with us this afternoon. I apologize
for the little bit of delay here in getting started. I’d like
to thank everyone’s interest in joining us with our first
Webinar on the evaluation and how to conduct evaluations. As you
know, the Evaluation Working Group has been working on strategies
for evaluation capacity building for a couple of years now. One
of the things that we as a group decided, that we need to offer
some trainings to help people understand how to better conduct evaluations.
The Introduction to Program Evaluation Manual has been released
by ... CDCs within the last month and a half. This seemed like an
excellent time to begin our presentations on conducting evaluations.
We’ve got – our presentation today will be by Ann Powers
from Battell. Ann is a Senior Research Scientist and she works with
Battell in Arlington, Virginia. She’s been conducting evaluations
for more than ten years for a variety of federal clients, including
CDC, USAID and The National Institute of Health, as well as for
state, governments and foundations. Her PHD is in social psychology
from the University of Missouri. She has expertise in mixed method
evaluations and working with multiple sites. So with that, I’d
like to let her begin to talk to you about the first part of Program
Evaluation. Ann?
(Poor audio, background noise)
Ann
Okay thank you, Maureen. Thanks, everybody for coming and I’d
like to apologize – technical things going on here. So thanks
very much for joining us. What I’m kicking off today is the
first of a couple of sessions that we’re going to be doing.
Each one of the sessions will probably last about generally two
hours. What we’re trying to do is to give you an overview
of Program Evaluation, especially coming on the heels of this new
document that CDC has released in the last month and a half.
...So ... for people who are new to the field for Program Evaluation.
For those of you who are very ..., who’ve heard some of these
talks before; I hope you’ll find something new in it. What
we’ll do – I’m going to do a presentation on talking
about a new adoption of the Program Evaluation and also the first
part of the CDC Framework. Then we’ll have time at the end
for questions and if there is time still, there’s a small
example that I want to share with everybody. We’ll see how
the timing goes, especially with coming in a little bit late. Actually
now your microphones and the folks on the phones are closed. So
I would hope that if you have questions while I’m talking,
you’ll write them down and we can address them when the mikes
open.
(Poor audio, background noise)
Just to go over a little bit about the learning objectives for
the next couple of sessions. What we want to do with the first session
and that’s today, is to help you become familiar with the
six sets of the Evaluation Framework. There are a lot of other evaluation
frameworks and they’re very good ones. We’ve chosen
to use the CDC one because it’s flexible, it’s been
well used out in the field and there’s also – it can
be used with any type of service program. There’s also –
it’s very straight-forward and easy to learn and a lot of
resources to help you in terms of trying to do your own evaluations.
Including – and ... to CDC in general, I just wanted to
mention that the TB Group has an Evaluation Working Group that’s
been doing a number of products. Certainly these sessions are part
of that and a number of things that have come out and will be coming
out, that I’m going to assist you. So all of them are built
around this framework and that’s why I think useful for this
... particular application.
So today what we’re going to do is talk about the framework,
the steps of the framework and also talk more in-depth about ...
which you’ve been getting in stakeholders. Then there’s
some sessions coming up. The next session is Thursday at 11:00 I
believe and Tom....
Maureen 1:00.
(Overlapping voices, poor audio, background noise)
Ann
1:00 I’m sorry. Tom Tackle will be doing that and that’s
talking about step two and three of this training work and in more
detail and how we develop a lot of this model and how to use a ...
model and sort of very important. They’re definitely important
to public health programs. The last two sessions that we had on
this ... this was a more a tentative plan originally that we would
do two more session, talking about steps four and five in the third
session and then step six in the fourth session.
At the moment we’re thinking those are tentative –
Maureen is hearing that maybe there’s some interest in some
different – in some other topics or in additional topics.
So we’re going to call those tentative right now and if you
have ideas or suggestions about some other things that you’d
like to hear or like to learn about, just let Maureen know. You
can e-mail her or call her and let her know about some ideas that
you have about for addition sessions.
So why don’t we start with what we’re calling session
one here. Again these are your learning objectives. To become familiar
with the six steps of the CDC Evaluation Framework and then also
to learn how to identify and engage stakeholders, which is step
one. What I want to talk about today and help you learn is the basics
of what Program Evaluation is and what is isn’t. We’ll
talk about that first. So we’ll talk about the differenced
between evaluation and research and the differences between evaluation
surveillance and evaluation and monitoring.
(Audio cut outs, background noise)
I want to talk about why evaluation is important and what can
be evaluated. Also what it means to be – the utilization focus
evaluations, which are really one of the core pieces of the CDC
Framework. I want to talk about the steps ... and conducting evaluations
as outlined in the framework and also the standards for conducting
program evaluations. How you can use the standards that are part
of the framework to help you design and conduct your evaluation
and then how to identify and engage stakeholders as I said.
(Very poor audio)
So let’s start first of all with what is evaluation? In
the three perspectives here, two well-known people and third very
well-known person. The unknown person who writes a lot I gather.
Michael ..., to him evaluation is the systematic investigation of
the merit, worth or significance of an object. Like Carol said,
it’s the systematic assessment of the operation in our outcomes
of a program, a policy compared to a set of explicit or inclusive
standards as a means of contributing to the improvement of the program
or policy. Then the third, unknown, so ... evaluate ... systematic
way to determine the value of a program, program components or activities.
In all three of these definitions you do see the word systematic
and what that really means that evaluation is planned, it’s
disciplined in its objectives. One of the activities that many of
you have been doing this fall is writing an evaluation plan and
that’s part of systematizing evaluations. Another word that
you see is the word value, which is not a word that you see with
research commonly but because evaluation involves judgments and
there are values involved. Something becomes – you know you
make a judgment of value.
Evaluation is a selectable discipline and it does mean different
things to different people. The definition that I think is a very
good one and that is part of the new CDC introduction to Program
Evaluation for Public Health is the systematic collection of information
about the activity, characteristics and outcomes of programs who
make judgments about the program in truth, program effectiveness
and/or informed decisions about program development. So you can
see right there, it’s a very broad concept and useful in many
ways.
(Audio cut-outs, background noise)
So let’s talk a little bit about what evaluation isn’t
before we talk about what it is. ...This ... contrasts evaluation
and research. They are familiar in some ways in that they both employ
methods that are systematic in their investigations but there are
a lot of fundamental differences between the two. Research is generally
conducted to produce knowledge that is generalized across different
groups or different programs or different settings. Valuations are
conducted to generate findings that are intended for the use by
specific programs at a particular time.... ....is being conducted.
With research, the questions and the hypotheses are typically researcher-driven
and evaluations on the other hand, the questions for evaluations
are usually derived from the program or more importantly from the
stakeholders.
(Poor audio, background noise)
Evaluation, as I mentioned, is value-based, so it involves judgments
about quality. Research is something that we try not to do. It’s
usually conducted without judgment or bias to the best of your abilities
in any rate. So therefore for that reason, research is normally
done in a really controlled setting as much as possible to prevent
the ... from coming into the research. Evaluations, because they’re
intended to be used for program improvements, evaluation is conducted
in a real world setting and it has to take into account the environment
and the context in what you’ve based the....
Roles are much clearer with research. The researcher is –
sets out to be objective and not part of the situation but ... sometimes
experiences role conflicts that could make him be part of the program
that they’re evaluating. Sometimes we also – there’s
a difference in publication because research has a – seeks
to generalize knowledge, you often will publish those findings.
With evaluations, we’re finding they’re less often published.
There not never published but they are less often published.
(Loud background noise)
Okay this is a really quote I think what really sums up evaluation.
Research speaks to prove, evaluation speaks to improve. I know they
use similar methods; similar data announced at sometimes is the
real underlying point of doing these two activities is very different.
Evaluation is going to involve looking closely at the operation
of a program or a program initiative, which is often the case at
TB. And gaining some understanding through this examination and
then making recommendations to improve the program.
(Audio cut-outs)
I think it’s useful to really keep in mind the word improves
when we’re talking about evaluation because that is our –
what we seek to do in the ... and it’s not to criticize or
to blame or even to eliminate some part of the program, it’s
solely for improvement. So if the goal of evaluation is to improve
a program then no evaluation is good unless the findings are used
to make a difference. This is a very important point and I’ll
talk a little bit more about it as we move through the framework
and you’ll hear it through all of the sessions.
Sometimes after an evaluation is done, the reports in the papers
that are produced from it are – a report is written and then
it goes and sits on a shelf and they aren’t used. So you cant’
use the findings to improve the delivery of a program or to improve
outcomes and that’s really something that we’re thriving
to avoid in this particular case. So you want to begin evaluation
with the intention that you’re going to use the results and
as part of your planning to plan and strategize ways that you can
use those results. You can involve your stakeholders and you can
really use them to improve your programs and you’ll see that
in the framework, the last two steps or the last steps about dissemination
and utilization.
So let’s talk about Program Evaluation in comparison to
surveillance in.... These are frequently used terms and they’re
quite often used activities and surveillance typically refers to
the tracking of a disease or a risk behavior. The TIMS System is
a surveillance system but evaluation is different in surveillance
in that – evaluation is very specific. Program Evaluation
is specific to one program during a certain time period, where surveillance
tends to be a more global activity.
Monitoring is also used for tracking changes in programs and it
is simply a maze of tracking and recording and there is no evaluation
done in it. There’s no assessment about the quality of anything
in monitoring. It’s an information gathering system. What’s
important though, is that I think to mention about both of them
is that they can both be used as part of Program Evaluation. I mean
monitoring and surveillance data are important pieces. They can
be important pieces of your evaluation so we don’t want to
not use them but we want to use them to help us understand other
things.
(Audio cut-outs)
So this ... directives what can be evaluated? That’s most
everything I think. It’s all types of programs and all types
of program initiatives, which are some of the things that we’re
doing in TB. The range of activities includes services and interventions
to pieces of organizational structure, training and educational
systems, administrative systems, laboratory systems. It’s
very broad and evaluation can be adapted to all different kinds
of settings.
(Poor audio, background noise)
It is I think important – an important message for the TB
program because many of these things are things that the programs
are undertaking, particularly initiatives that may be new or may
be in the planning stage right now. That’s likely the kind
of thing that’s going to be investigated and going to be evaluated
as part of a TB program, rather than a whole – the entire
programming.... For example be interested in an outbreak investigation
rather than the entire program. This is just an example of some
of the range of things that you could evaluate if you chose to.
(Audio cut-outs, background noise)
Now this is sort of moving onto when to conduct evaluations and
how evaluation changes with this phase of the program that you happen
to be in – involved in. Now traditionally some of the evaluations
have been conducted toward the end of programs after the programs
are over with very mature programs. It’s not necessarily at
that point in order – you’re not able to make any changes
to make improvements to a program, but to be most effective in the
way that we want to see evaluation and.... To do it at all phases
for the life of the program to result in improvement.
In the conception phase, which is when you’re planning a
new program, evaluation can be used to assess motivation for starting
the program, to define the need for the program and to inform decision
makers about what a new program should look like and also to define
its objectives and expectations for the program. Also to help you
clarify some important factors that are really essential in terms
of building the program and lying down the foundation for it.
When you’re developing the program, the next phase on, you
can use evaluation to monitor your progress and also to do any course
corrections you might need or redefine program activities or expectations.
Again, find any gaps that you might have. If you move into a stable
or a mature program, again you can use evaluation findings to monitor
the program and you can also begin at this point to use it to provide
feedback to management.
You can also use – begin to think about showing program
successes at this point as well as the areas for improvement. As
the program becomes more stable, the initiative becomes more stable
and mature, you can begin to assess outcomes at this point. Then
finally at the end of the program, evaluation will help you assess
the value of the program, as well as document any lessons learned,
that you might want to use in the future.
(Background noise, audio cut-outs)
You can make an assessment of how the – if you’re
doing an intervention, how that was adapted into the particular
situation and it’s a change that’s sustainable at that
point. You can look for unexpected outcomes after a program has....
It’s also a time to assess and evaluate whether the intervention
could be replicated beyond this point and what factors would –
are necessary to be able to do that. With some of the – when
we’re talking the TB program, many of them I know are stable
or mature but you also may have an initiative or a new program or
a new effort that’s in the planning stage or in the developing
stage.
So this is kind of not – any one program could be in various
stages and it’s not an all or nothing kind of thing. So you
may have a very mature program that’s starting a new initiative
and you can kind of mix and match those things and put those together.
But the point of the slide is that the stage of the program development
is going to influence the reason for conducting Program Evaluation.
So why evaluate programs? The first reason is to gain insight
and this is to evaluate a program to judge its merit or gain insight
about the program and its operations. This can include providing
information concerning the practicality of the new approach or developing
program and to find out where the program is and where it’s
heading. Evaluation will also help you assess where we are in the
program development and how to identify information that you need
to, to move onto the next step of the program development. It also
will help you with decision making and also in terms of managing
resources and services more affectively.
So for example, an evaluation of testing and treatment for LTDI
might show how many people in high-risk populations and how many
people in low-risk populations are being tested. So you can use
that to and form decisions about whether it is useful for example
to review the testing in a low-risk population.
(Very poor audio)
The second reason is to improve practice. We can evaluate the
program for the purpose of making improvement or changes in practices.
It can improve or enhance what we choose to do and this is an important
thing to do when you’re in the middle of the implementation
of a program, to tell you how the – if there are changes that
need to be made or focus that needs to be changed. It can also help
you understand why you achieve these ... or perhaps not the success
you did if you didn’t meet your objectives and it’ll
information on stuff that you can strategize and plan and implement
changes so that you can improve the effectiveness of your program.
To assess the effects and this is – when you’re conducting
an evaluation, to see how well objectives and goals were met or
are being met. It can show how effective an intervention is at inducing
the intended changes over time that it was set up to do. It helps
to demonstrate the value of efforts. It’s very systematic
in documenting the contribution towards your goals. It’s useful
in terms of effecting – the fact is that it can count decision
makers at all levels, understand the benefits and the consequences
of the program and at very important times, at critical times, evaluation
can help you advocate for your program.
(Bad audio, background noise)
The last reason to evaluate is to build capacity, to increase
funding in ... scales and strengthen accountability. ...To help
you in terms of ... capacity about making self-directed changes,
such as developing skills or building infrastructure. So this is
the CDC Evaluation Framework. Many of you have probably have seen
this diagram or some version of it.
What I want to start moving into now is to really talk about the
pieces of – to giving you an overview of the pieces of the
evaluation and the framework. ...Is set up to be a practical tool
and it is generally non-prescriptive and it’s shown as –
it’s in a circle and this really represents the circular nature
of the evaluation process. So it shows how interdependent the steps
are. Also, although it cannot entirely illustrate it, that there
really can be conducted in a non-linear order and then you may be
conducting the steps more than one step at a time. So it has this
kind of circle to it as a way of demonstrating that it is not necessarily,
although we call them steps – step one, step, two, step three,
step four.
Although the steps can be conducted in a different order or they
can be done simultaneously, there is a reason why the steps are
sequenced in the way that they are. That’s primarily because
the earlier steps provide a foundation for the subsequent steps.
So engaging stakeholders is an important piece to help you describe
your programs. Once you develop a program description, then you
can start to think about focusing and evaluation design.
(Poor audio, background noise)
All of the steps are important in – or however they’re
being done or how they ... are doing at one time. Each one has an
important piece to add to the whole circle. The circle also represents
that the framework is iterative and that program evaluation is iterative.
The information or the findings from one evaluation or from an evaluation,
is used to be fed back into the program to make improvements. So
this ... ensures lessons learned to step six begins the cycle again
building the new cycle again.
At the core of the framework are the evaluation standards. This
is usability – I’m sorry utility, usability, propriety
and accuracy and these are four standards that are intended to maximize
the quality and effectiveness of the evaluation efforts. Each one
of these standards is applicable to each one of the steps in the
framework and I’ll talk about those in a second.
So here are the six steps. I’ll go over each one briefly
but hopefully in future presentations we’ll cover them in
more depth. Step one is the NDH stakeholders and stakeholders are
those that have a vested interest in your program and are potentially
affected by the programs and its evaluations. It’s important
to note that stakeholders can be both supporters and skeptics.
(Bad audio, background noise)
Step two is describing the programs. This is the step in the evaluation
that allows us to gain a better understanding of the program and
its activities. The meanings, the context and the ... copulation
for the program or the.... Then major program components like resources,
activities, outputs and outcomes in the relationship with each other
are defined in step two.
Step three is a very crucial step. Does it help us determine the
focus of the whole evaluation? When you’re planning an evaluation
– you know something that you have many worthwhile questions
that come in or identify all your stakeholders that you consider
for evaluation. It’s easy to want to think about evaluation
or evaluating everything and collecting a lot of information but
it all bounds on limited resources and times and staff and money.
So step three is the point at which you think carefully about what
it is you want to evaluate and set some priorities.
(Very loud background noise)
Step four is gathering kind of credible evidence. This is where
we’re working into data collection ... and this is where you’ll
identify the indicators that you’re going to use. Then once
you have indicators and you develop message for the indicators and
then analyze the data.
Step five is justifying conclusions and this is analysis and interpretations.
The interpretation part of this phase is really critical and this
is where you judge a finding against your findings against established
program benchmarks. The important step here for example is step
five, to involve the stakeholders that you’ve identified and
engaged in step one, to help you interpret the findings from your
evaluation.
(Audio cut-outs, poor audio)
Then step six is to ... you – ensure use and to share lessons
learned. This is as we’ve said all ready, a very important
part of your evaluation of planning for and including a way to use
the data to approve the program. The purpose of this slide is really
to think about and reinforce the ... the circular nature of the
framework. Really it illustrates how the steps are interdependent.
So one of the basic components of this framework is using the results
and that’s part of step six. It’s important to create
a market for your evaluation findings and that’s part of step
one, engaging the stakeholders.
Then finally part of creating a market and being responsive to
that market, is focusing an evaluation on the important questions
and answering the questions that your market or your stakeholders
would like to know. This continues on to show this iterative nature
of the framework and this is really illustrating how we –
although we refer to step three as the focus, step one and step
two are both important factors in focusing. Step one talks about
identifying your stakeholders, finding out who cares about the program
and finding out what they really – what they want to know.
How they define the program and how they define success for your
program.
Step two helps to focus the evaluation by establishing the monitor
of the markers that we view public health outcomes that your program
is interested in, in changing or creating. Step two – this
will be something that Tom will talk about on Thursday so I won’t
spend a lot of time on that.
I want to spend a little bit of time in talking about the standards
for effective evaluation. These are the things in the middle of
the circle. Good evaluations are conducted under the guidance of
standards for program evaluation that have been set forth by the
joint committee of standards for educational evaluations.
The four standards, as I mentioned, are utility, feasibility,
propriety and accuracy. What the standards do is that they serve
to guide your decision making process of each step in the framework,
so that you can ensure that your evaluations stay focused and balanced
all the way through.
(Background noise, poor audio)
On the next few slides I want to talk about the standards and
really the over ... purpose – or one of the over ... purposes
for using the standard, is to help you design the best evaluation
for your program and for your situation. Like research, there is
really no wrong and right evaluation. Instead, what you’re
trying to do when you’re designing an evaluation is to make
choices that are going to ... to your particular situation or your
particular program. When you apply the four standards, utility,
feasibility, propriety and accuracy, if you apply each one of them
during the steps of the evaluations framework, what’s going
to result is the most worthwhile evaluation for your situation,
which is really your objective here.
What’s on this slide is that under each standard there’s
sort of a general overarching question that you might ask yourself
when you’re designing the evaluation. To just show you how
that when you apply them, you end up with an evaluation that’s
useful, feasible, proper and accurate.
So to talk about each one of the standards individually: the first
one is utility and utility insures that the evaluation serves the
information needs of the intended users of the evaluation and also
the stakeholders. As we said, the evaluation does no good unless
it’s used to improve a program. So knowing who needs the information
and how the information will be used helps you to tailor the evaluation
and focus the evaluation to address a need.
So you need to take the time to find out what people need on –
and also in terms of what it is that they do and how they contribute
to the program, how they’re interpreting their productivity
to be improved. The whole point of this is really to ensure that
the evaluation results are used. So when we’re thinking about
utility, this requires asking questions like, who needs the evaluation
results, what do they need, will the evaluation provide relevance
and otherwise useful information in a timely manner for the users?
(Poor audio, difficult to understand)
Also the best way to do this is really to talk to your stakeholders
about this. This is one of those – it’s a standard that,
while you can plan, it’s very important to really have your
stakeholders address the feasibility. Feasibility is another standard
to really keep in mind when you’re planning an evaluation.
This is standard is to ensure that the evaluation conducted is realistic,
prudent, diplomatic and frugal because resources and ... in public
health programs are also limited.
Although as I mentioned, a lot of this – you know you’d
like to everything and there aren’t enough resources to do
that. So feasibility helps you to prioritize the need of your evaluation,
the need for evaluation. So the kinds of – the questions that
you want to ask is, are the activity’s plans realistic, given
the time, resources and expertise that we have available to us?
The next standard is propriety and this ensures that those of us
involved in the evaluation and either evaluators or members in the
evaluation team, behave legally, ethically and with regard to the
welfare of those involved or affected by the program and by the
evaluation. An evaluation can involve handling personal and sensitive
information and this may be very true especially with a TB program
or TB initiative because the evaluation may involve the review of
medical records and charts and the handling of information that
patients and providers may regard as highly sensitive. So these
are the – so it requires that we address this standard.
(Poor audio, background noise)
Although with Program Evaluation, generally is exempt from institutional
review board review. It is still important to really consider that
the data that we’re using for the evaluation is handled well
and that we protect the rights of privacy of the people involved.
So a couple of the questions that will help ensure that the evaluation
is conducted with propriety is, does the valuation or the methods
employed in conducting the evaluation protect the rights of individuals
and the welfare that are involve? Does the evaluation engage those
directly affected by the program as well as those affected by the
potential changes in the program? That’s program participants
and patients ... the community and again this is speaking back to
step one of engaging the stakeholders.
(Audio cut-outs, loud background noise)
Have we included people in the evaluation process and given them
a voice to the process into making decisions about their own health
and all the....
(Loud background noise)
Finally the last one is accuracy and accuracy ensures that the
evaluation conducted reveals and conveys reliable and valued information.
It’s important to have accurate information because it’s
the foundation of any ... decision ... and if you’re going
to use the results to improve a program it’s very important
that the results be accurate. So it’s important when you’re
planning you evaluation to identify ways or methods – methods
that will help us yield the most valid and reliable results. This
is a subject I think for part of a future presentation, that there
are some ways that we can indeed ensure that.
So I’m going to move into the next part of the presentation,
which is engaging stakeholders, which is step one. I’ve mentioned
it a couple of times but now we have an overview of the whole framework
and how the things fit together. What I want to talk about is how
to first of all identify the stakeholders that are essential to
the program and to the program evaluation. Then figure out who your
key stakeholders are and how to engage them in the evaluation.
I just want to repeat a definition of the stakeholder, so we’re
all on the same page. Stakeholders are people and are organizations
that are affected in the programs, are interested in the results
of the evaluation and/or have a stake in what will be done with
the results of the evaluations.
(Loud background noise)
This is a ... that the ... are able to absolutely critical evaluation
because other than ... are people, people, people and people. So
why do we engage stakeholders? This is just for the staff ... in
the framework, so that’s certainly an indication of the importance.
There are a lot of reasons why the stakeholders are important to
your evaluation and I’m going to give you a list here. First,
stakeholders are the people who will implement whatever changes
or improvements the evaluation recommends. Engaging them early on
in the process will increase their buy-in and thus risk ... increase
the chance that the recommendations from the evaluation are used.
Stakeholders are also important in providing a reality check when
you’re doing an evaluation and they’ll keep the evaluation
grounded throughout the process. Sometimes stakeholders can also
generate the need of political pressure or support or even sometimes
the resources that you need for evaluation. So they’re very
important in that respect. In addition,
(Very loud background noise)
stakeholders know the program, in many cases better than anyone
and they can provide important insights as ... at your evaluation.
So they’ll increase the credibility of the evaluation findings
once you collect them. They can assist you with the dissemination
of the findings later on. They can advocate for the program and
also they may be your funders. So your stakeholder may be able to
provide future funding.
(Loud background noise)
So who are the stakeholders? They are persons generally interested
in the program or affected by the program and there are three broad
categories that we typically will get stakeholders ... from. Those
that are involved in program operations, those that are served or
affected by the program and they will tend to do this ... the evaluation
findings. Then once we have identified who the stakeholders are,
then we want to talk about what they’re – think about
what they’re interests or perspective in the evaluation happens
to be.
(Very poor audio, loud background noise)
Do they support the program or are they skeptical about the program
or antagonistic for the program? One way you can do this, is kind
of systematically in – as I said if we have a little time
I might be able to talk more about this – is to create a list,
for example set it up into a table with the three categories of
stakeholders. Those involved, those served or affected by and ...
the evaluation and then list-out the persons or the type of persons
who are part of each group. You’re going to know at that point
while you’re doing this that stakeholders tend to fall in
more than one group. These aren’t usually....
Then once you’ve generated the list, think about what their
interest in the program is or what their interest in the valuation
is. Then also think about how you might go about involving them
in the evaluation, at which stage in the evaluation they could part
of. This is of course one of those iterated tasks where you can
begin with a list but it’s important to share the list with
your evaluation team and with some of your stakeholders to make
sure that you at least initially capture the universe of stakeholders.
So here are some examples: persons that are involved in the program
operations can be staffed, managers and other partners who are directly
involved in the day-to-day activities of getting tasks done that
are part of program operations. This can include public health nurses
and TB clinic clerks, as well as members of your own program staff.
Those that are affected by the program include clients or patients,
their families and friends as well as providers and community groups.
So this can include TB and LTDI patients, managers and coordinators
at school, prisons, shelters and other congregate segments. Also
it may include representatives of populations that are just proportionately
affected by TB.
(Poor audio, difficult to understand)
The third category, the intended users of the evaluation is going
to include policy makers, managers, administrators, advocates and
funders. So of course that involves CDC, public health managers
and administrators and also it may involve advocacy groups too are
also to the ... evaluations. Then the final census on this ... is
urging you to include both supporters and skeptics. Both have important
points of view and having skeptics involved in your evaluation is
going to increase the credibility of the findings.
The one caveat that I would say about having skeptics involved
is really to be able to consider their point of view and their motivation
and why they are skeptical. For example, it may be that your skeptics
are interested in redirecting program thoughts. You know, their
resources are limited and they’re anxious in redirecting the
funds. So it’s important to keep that sort of view point in
mind. However, it is very useful to have your skeptics involved
in the evaluation because having them involved will ultimately strengthen
the evaluation, increase its credibility and result in more improvement
to your programs.
So this slide talks about which stakeholders matter most and as
you can see by earlier slides, there are many, many potential stakeholders.
The universe of stakeholders is very large and what we want to illustrate
by this slide is, how do you go about identifying the key stakeholders
in your evaluations? That from this universe, how do you get down
to a group that gives you a wide enough viewpoint but is a workable
number? So the longer you ... is the universe of the evaluation
stakeholders. This is everybody that you and your evaluation team
have identified under the three categories of stakeholders. Then
you use this list to then narrow down the stakeholders.
The smaller boxes below is the illustration of the key stakeholders
and this is the stuff that of the universe of the stakeholders.
They can include people or persons that will enhance the credibility
of the evaluation, such as I was just mentioning; your skeptics;
your program staff; the day-to-day people that implement changes
that will result from the evaluation; advocates for the recommended
changes for the program and they often maybe go to fund, authorize
or expand the program. This is just really to think about that these
folks are your key stakeholders and the ones that are most important
to find a way to engage in the process.
So your stakeholders are important and the next sort of step is
to think about how and when to engage stakeholders and this slide
offers some suggestions. Although it’s not exhaustive list,
usually whoever is leading or coordinating the evaluation has some
responsibility for identifying and engaging the stakeholders. Stakeholders
as you’ll see from this list, it’s important that in
step one you identify them and engage them but that they are engaged
throughout the process. It doesn’t at the same time mean that
all stakeholders are involved in all stages.
(Background noise)
A very small subset may be involved in the evaluation from planning
to use but it may also be that certain stakeholders, key stakeholders
step in at certain times of the evaluation and then step out. So
it’s a ... selectable group of key stakeholders but it’s
important that you remember that key stakeholders should be involved
at every level. So some of these suggestions in terms of how to
involve your stakeholders, the first two are part of step two of
the framework. That’s going to use your stakeholders to help
you describe your program activities, the context and priorities
of the program. Then also working with your stakeholders to define
the problem that you may be confronting in your program or the initiatives
that you want to investigate.
The next three involve steps three and four of the evaluation framework.
That includes selecting evaluation questions and.... Stakeholders
can also serve as data sources. They may be someone who’s
part of an interview or in the case of ... it may be some part of
the start review. They will define what constitutes the proof of
success for your program. Interpreting findings can be parallel
to step five of the evaluation framework.
Then step six, which is dissemination use are the last two items.
That stakeholders can help you disseminate the information to their
sphere of influence and in many cases they can help you implement
the results, whether it’s a change in program operation or
a change in funding or need for political support.
So I wanted to stop and talk and take some questions if we have
some.
(Loud background noise)
Maureen MCI operator? Are you – operator?
Operator Thank you, one moment. I have no questions at this time.
(Loud background noise)
Maureen
I’m going jump in and ask a question, Ann....
Ann
Okay.
(Almost inaudible because of poor audio)
Maureen I’d like you to just quickly address how you would
think about the standards in regards to ... step one ... stakeholders.
The standards to how....?
(Loud background noise)
Ann ...
Around here and we’ll take all the standards in order. The
standards of usability utilities – feasibility for ... accuracy
– I would use them in terms of working with my stakeholders,
wanting to know what they want to know. I guess I would address
them initially, so with the utility standards, I would find out
what they think the problems are, how they would like to use the
evaluation and where we want to focus our efforts. The second one
feasibility, I would talk to stakeholders, especially those who
for example might have to – or be implementing changes or
how do the day-to-day operations and then find out, could they have
implement....? Can they implement the evaluation and really talk
to them about if they are able to implement the evaluation and on
what kind of changes they would be able to implement?
Propriety certainly is part of talking to stakeholders because
you want to be sure that you’re doing something that is responsible
and ethical and engaging your stakeholders at this point will help
you identify any further problems. Sometimes it’s difficult
from certain view points to understand what ethical and responsible
problems are and I’ve learned that. I sit on an institutional
review board and I often find there are program people and the community
reps have an entirely different point of view than I do as a PI.
So that’s a very important part of engaging your stakeholders.
(Loud background noise)
Of course accuracy, again you want to talk to them and you want
to engage them as part of both talking to them initially and then
helping you to think about interpreting your findings. Being able
to understand the context of your findings, so that any sort of
recommendations you make are really grounded in the most accurate
information that you have. Does that answer the question? ...Oh
you read the questions.... How can I get a copy of the ... for the
presentation? That’s going to phase-up correct, Maureen?
(Loud background noise, audio cut-outs)
Maureen
Right after this presentation, give us about 48 hours or so and
then you can go into login.... The logins at the bottom of the e-mail
that says how to watch the replay, you should be able to get the
slides from there. If there are any problems ... also ... e-mail
me or ... and we can definitely get....
(Very poor audio)
Ann
...Assessment and evaluation, I might ask you to.... In my mind,
I think evaluation still has ... one step beyond assessment....
It talks about making – well assessment may involve making
a judgment and evaluation makes a judgment but in many ways it’s
– the purpose is different. I mean evaluation is meant to
improve or meant to change or to make recommendations. In my mind,
assessment stops one step short of evaluation....
(Very bad audio)
Maureen
I agree with you on that and when we you were going over the reasons,
because I think evaluation, you talked about was a ... to gain insight
and a lot of times it’s.... ...Of different types and many
people use the words that are changeably ... at all, which we ...
about our language here at all because – so I think you can
think about ... being – looking at the big picture but ...
questions, evaluations can answer....
Ann
Yes and I think evaluation – it’s intended to be a
tool for improvements.
(Audio cuts-out)
Maureen
And assessment can be as well but they can be just – it’s
that they tend to be more focused.
Operator I do have a question from the phone lines, would you like
to take that?
W
Yes.
Operator
One moment, Samantha Chatogie you may ask your question.
Samantha
Sure I was wondering how this would relate to the triple A –
you know the three As, the cycle to assess, analyze and action.
Is there any relationship to the framework that you just described?
(Bad audio, also cuts-out, overlapping voices)
Ann
We’re looking at each other and going – okay that’s
I’m sure....
M ....
(Very poor audio, almost inaudible)
Ann
I’m sure that’s another framework that someone put
forward and as you can see the ... it fits well with ... framework
expansion and as ... couple ... to it but the same general process
is there ... would be to describe ... focus and then analyze would
be the – of course the step ... and then accurate steps to....
I think you know, the ... framework would probably say make sure
that ... your assessment ... that’s even the hard question.
If you’re defining assessment in a very, very broad way and
you include the stakeholder involvement in ... steps, that’s
all the ... framework ... takes out to make sure that we’re
– we emphasize. I think many evaluation frameworks ... why
do people do that....
Samantha
Okay great thank you.
Operator
Miss Karen Marsanek you may ask your question.
Sue Ann
Oh hello, this is Sue Ann Jakerson with Karen Marsanek in Alaska.
We’re wondering what CDC’s time table for training on
all the steps. I’ve seen you’ve gone through the first
two today and would CDC consider extending the deadline of December
15th until at least a month after all that training is complete?
(Poor audio)
W ...Going to take question?
(Inaudible)
W ....
(Very bad audio and cut-outs)
Maureen
We will have this training today and then on ... this week there’ll
be Tom ... training on the description and.... These two trainings
together should get you well into your evaluation plans. If you
feel that you need more help with the later steps in the plan, before,
you even ... those, you can feel free to ... into the plan and say
... their going to seek training and technical assistance on developing
our specific methods.... This should get us through to step one,
step two and step three. After that it becomes much more technical
and we would rather ... go use – how you’re going analyze
your data questions like that.
When Tom talks to you too on Thursday, I’m sure that Anne’s
all ready been saying – you’re all ready learning part
of step six. So I would like to see some of that in the plan that
we receive on the.... As far as filling in the message section,
which is what we probably won’t be able to get to ... probably
... after people will at least turn their plans. If you feel uncomfortable
with that, feel free to call ... or myself and know that we will
have some additional training on that coming up. You can say in
your plans, we’re developing these particular items and we
expect to have that done ... or whatever’s comfortable for
you, just put that in your....
(Inaudible)
Dan ....
(Poor audio, audio-cut-outs)
Maureen
Dan notes that it’s very important to speak with your program
people ... as well because they can have lots of suggestions for
ideas and one of the things that we do want to stress with everyone
is to keep the dialogue open and that we want things to be very
productive ... for everybody. ...For any ... that says please don’t
everybody call up on the 15th of December. We’re here right
now and would be happy to talk to you and....
(Really bad audio, cut-outs)
Ann
Just to build on my point, between today and Thursday that should
in terms of building your plan according to the Evaluation Framework
get you up through identifying your evaluation questions.... These
are – you know thinking about your goals, who your stakeholders
are, how you’re going to involve them and then focusing your
evaluation on really the core conceptual pieces.... ...These two
sessions should be able to get you up there and then....
M
You also may reiterate to folks that you also all ready provided
a guide and had a development evaluation plan, plus an excellent
template that’s available to use in helping in developing
the plan as well.
(Bad audio, loud background noise)
W
Did everybody hear that? That was ... comment. ...It’s ...
that we do have the guide on ... development and if you know if
you walk through the exact process. If anybody does not know whether
... that.... ...Internet Web site of ... and if you have any trouble
finding it again, call ... or myself or ... presentation....
(Inaudible)
W ....
(Very poor audio, cut-outs)
W
...Yes back in here you can just replay the entire Webinar and
... if you’d like....
M
Yes and the template’s a great source, a great resource....
(Inaudible)
W ...Captive audience. So thank you....
M You’re welcome. There’s another question ... too
right?
W No this is.... ...Other questions?
Operator There’s no other questions from the phone lines.
(Inaudible)
W ....
(Audio cut-outs)
W
Does anybody have any examples that they’d like to share
with us or any other comments about ... stakeholders ... questions....?
(Almost inaudible)
W
I mean I have a couple of examples but.... ...Examples about ...
engage their stakeholders....
Operator Question directions.
(Bad audio)
Maureen
...Simply walk us through the examples?
Ann
Yes.
(Very bad audio and cut-outs, background noise)
Maureen
Then we’ll close on that and then we’ll close on that.
I do want to say, if anybody has any suggestions or ideas for additional
presentations, please e-mail me or.... ...Call ... 446-395320. The
second presentation will be Thursday at 1:00 and we will send out
messages about additional presentations after that right? Yes we
now have to let Ann talk about ... examples here.
(Background noise, poor audio)
Ann
I just want to give a little ... example of something that I’ve
seen in another – a plan that I thought was very useful. It’s
a stakeholder assessment and engagement plan. It’s a table,
it’s a three by four table, it’s very simple and it
takes the – each separate group of stakeholders. So those
involved in program operations, those that are ... or affected by
the program and the intended users of the ... and puts them in a
real ... table. Then there are columns in this table that talks
about their interests or perspectives, their role in the evaluation
and try to engage them.
(Background noise, poor audio)
So for example, under the persons who are involved in the program
implementation, they’ve listed public health nurses, TB clinic
clerks, clinic physicians, agent managers and state managers. Their
interest or perspectives is attributing to – this is really
very thoughtful, that the staff may or may not be reductive to change.
The staff may perceive the evaluation as a personal judgment and
the staff may be questioning the validity of the information collected.
So is their entry for perspective and you ... there might be a critic
amongst that group.
(Very bad audio)
Then offering their role in the evaluation to be something about
asking the staff how they do their jobs and what sort of process
they do. Interviewing them to understand the context of the program.
Discussing their concerns about the evaluations, having them help
you collect date it with pulling records and other sorts of the
data. Then also suggesting possible modifications ... the evaluations
process.
For each one of the different kinds of stakeholders, this table
goes through this. Under two, you’ll note that I didn’t
talk about how ... engage the stakeholders because what I think
is very interesting is to think about at what point you want to
engage these certain stakeholders and what point perhaps they could
step out of the process. So the second group is those affected by
the program. This is patients, contact of cases, persons at facilities
where the investigations occur and the general public.
They may have concerns about their confidentiality a bit and burdens
of the process because they’re part of the evaluations. Their
... has – so they can fear about TB or about testing. They
want also – there’s a certain sense of security that
they want to feel secure about ... in the public in general. These
are very often the folks where the records and the data that are
going to be coming from. Also with providers, these are important
people to share your findings with.
(Poor audio, background noise)
Then the third group are the primary users of the evaluations.
These are program staff, state contacts, TB program administrators
and you’ll note that – especially in the third group,
you’ll find that there’s overlap with the first two
groups. So you’ll have stakeholders in various places. They
may want to streamline the processes. They want to identify opportunities
for transferable lessons. Their role in the evaluation may be to
define specific evaluation questions to reflect the decisions that
need to be made. Interpret findings and planning on for the use
of the evaluation climate.
So what can also – when we’re thinking about these
stakeholders, we’ve identified the stakeholder category, we’ve
talked about their interest or perspective and again to sort of
reiterate what’s on the slide, we thought about their interest
and perspective in terms of things like credibility, implementation,
advocacy and.... So you want to think about your stakeholders in
terms of how can their perspective increase the credibility of the
evaluation. How they might be part of the evaluation and how they
might advocate for you for the program for the evaluation and whether
they’re funders and they’ll advocate for funding or
make funding recommendations for example.
(Poor audio, cut-outs)
The last one is list ways to keep them engaged ... and I apologize
that you can’t see this – the thing that I’m looking
at, because I think it’s a very concise way of presenting
your stakeholders, thinking about who they are and how to involve
them. So what’s interesting about the way this is ... thinking
about this, is that ... those that are involved in program operations,
they’re probably going to be part of your evaluation nearly
all the way through. This is not always true but there are certainly
people you want to talk to in the beginning when you’re planning
and focusing. They’re probably going to be collecting data
with you. They certainly will be involved in using the recommendations
from you that....
So group one tends to be involved all the way through. Group two,
while very important and very key stakeholders, their role you might
be finding more limited, because these are the folks that you might
be collecting data from. These are the people who might be, look
to be and audience for your evaluation results. They may be only
involved in steps, for example for in six or four, five and six.
So it’s a slightly different perspective.
In the third group, the ... evaluation findings, although they
might be involved all the way through, users are – you know
you would want to talk to them very early on in the planning process.
Have a meeting with them and interview them and talk with them and
help them ... the planning process.
Then they may be less important during the whole implementation
phase – still keep and perhaps not someone you’re checking
in with on a weekly basis or monthly basis. Yet they become very
important again in the end, when you’re interpreting your
data and analyzing – you’re interpreting your data,
making recommendations, disseminating the plans and trying to get
their recommendations implemented. Again ... using the evaluation
findings may come back in. So I described that to you, I hope it’s
not completely obtuse but it’s really an illustration of how
key stakeholders – they’re important to every step but
again they make come in and out of frequent contact as you’re
planning implementing any ... results.
(Very poor audio, background noise)
Maureen
I’m just going to jump in – this is Maureen again.
Just that, Ann is looking at an example that was written from –
an example that was sent in from.... So the ... is talking about
– and you can see very clearly in the template itself and
if you follow the guidelines that she’s here giving these
today, well within talking with us, you should be able to fill ...
and follow through that table. If you have any questions or comments,
feel free to call us and we can help you think through how to –
I think everybody knows how to identify the stakeholders quite well
but it ... on how to prioritize them, how to engage them and what
sort of processes. I’m going to ask you a quick question on,
how do you find....?
(Poor audio)
Ann
Well there’s no substitute for talking to them. I think that
identifying and engaging your stakeholder is really required and
... your stakeholders. So I’ve scheduled meetings with them,
maybe to talk to them by phone. If there are public meetings that
they’re holding that are important to your program and useful
to your program, you can go to the public meetings and listen in
on what it is that they’re talking about. You can even schedule
time sometimes at public meetings and kind of have an open forum
discussion about what you might want to use in your evaluation.
Further on in that, you can – they can be part of your data
collections team. They also I think are very important members of
the – like I say ... add a data interpretations team. When
I was conducting an evaluation, an international evaluation, we
found that because there was so much variations in the context of
what was going on and the challenges that the different programs
had in 16 different countries, we relied on stakeholders in every
site to help us interpret the data and help us come up with some
of our conclusions and justify those conclusions, so that we weren’t
putting forth something that was really inaccurate because we didn’t
understand the context. ..Answer?
Maureen
Yes.
Ann
Okay, any more questions?
Operator
Thank you and ma’am we have no questions on the phone lines.
Ann
Okay well then I think if we have no more questions, I think we’ll
sign off for the day and thank you, everyone very much for your
attention, your attendance and your questions. At this time I’m
going to remind you that part two is going to be on Thursday at
1:00 p.m. and Tom Tackle will be presenting that and he’ll
talk about step two and three of the Evaluation Framework. So good
bye.
(Loud background noise)
Coordinator
Thank you, that will complete today’s conference call, you
may disconnect at this time.
Last Reviewed: 05/18/2008
Content Source: Division of Tuberculosis Elimination
National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention
|
