Framing an Economic Evaluation: PrintAll

Framing Is the First Step

The first step in an economic evaluation is to frame the study.

Decisions made at this stage will determine directly which costs and outcomes are considered relevant and which should be included in the analysis. Therefore, choices made in framing will have an impact on the final results of an analysis.

This tutorial will

identify the key points that must be addressed before the analytic process can begin, and
provide questions that can be used to assess the use of each of the identified points in the evaluation process.

Framing Questions

Study framing should answer two fundamental questions: 1) "What do we need to know?" and 2) "How are we going to find out?" Each of these questions has a set of sub questions, which are listed below.

1. What Do We Need To Know?

The challenge is to identify and define the problem.

Conducting an economic evaluation can be both time- and resource-intensive.

If the purpose of the study is to provide information to policymakers facing a deadline (e.g., the end of the state legislative session), the analyst must be able to provide results in a timely manner.

In addition, the amount of effort dedicated to the study should be commensurate with the importance of the policy decision.

Therefore, to produce the most useful and timely study results, it is important that the analyst be

methodical,
thorough, and
prudent in expending resources for the study.

Answers to these questions identify and define the problem:

What Is the Problem That Will Be Analyzed?

The term problem is used here in a broad sense to refer to any health-related situation in which policy solutions and management decisions are or could be applied.
Alternatively, the problem could be identified by asking: "What is the health outcome of interest?"

If employers' health coverage costs are too high because of hospital bills for the treatment of hypertension, they might be interested in a worksite hypertension treatment program to reduce employees' diastolic blood pressure. The health outcome of interest to evaluate the effectiveness of the program would be reduced diastolic blood pressure.

Not only is identifying the problem to be analyzed a crucial first step in conducting an economic evaluation, but it is also crucial in researching the effectiveness of available interventions for the problem.
If an intervention is efficacious in a clinical setting under ideal circumstances but not effective in a real-world setting, defining the study question appropriately will be irrelevant.
Therefore, a researcher should only spend time defining the study problem for those interventions that are effective as well as efficacious.
Why Is This Problem Important?
A problem might be considered important for multiple reasons, all of which contribute to the burden of an illness or an adverse health outcome on society. A problem might
- be highly prevalent and affect a substantial number of persons (e.g., sexually transmitted diseases [STDs]),
- affect certain persons more than others (e.g., influenza, which disproportionately affects the elderly and other immunocompromised persons),
- be costly to society as a whole (e.g., smoking-related illnesses) or place a heavy burden on public health budgets (e.g., tuberculosis),
- affect the ability of a program to continue operating (e.g., loss of funding), and
- have major policy implications (e.g., a state law mandating use of child-safety restraints appropriate for a child's age and weight).
The burden of a disease or illness includes excess morbidity and premature mortality. As a disease becomes prevalent in society, the burden of illness associated with that disease increases.

For example, in the United States, respiratory illnesses caused by smoking are highly prevalent and impose a greater burden on society than does malaria, which is low in prevalence as a result of mosquito control, improved sanitation practices, and effective drug therapy.

To calculate the burden of disease (specifically the economic burden of disease on society) analysts can justify the decision to support and carry out an economic evaluation by appropriately weighting an illness or condition in terms of its total impact.
Finally, certain health policy decisions are made concerning interventions that might not reduce the burden of disease or economic burden on society but that have major policy implications.

Funding a program that screens the nation's donor blood supply for human immunodeficiency virus (HIV) might result in a limited number of averted cases of the disease. However, such a program does have major policy implications (increasing the safety of those in need of disease-free blood).
What Aspects of the Problem Need To Be Explained?
The problem identified is probably complex, and an economic evaluation more than likely will not address all aspects of the problem. As a result, the focus of the analysis needs to include only those aspects that are relevant to the decision.
For example, if a mass media campaign were implemented to reduce the incidence of smoking among teens and adolescents, the following issues might be useful:

explaining current levels of tobacco use among respective groups,

analyzing a minor's ability to purchase tobacco products, and

determining the minor's familiarity with existing anti- and pro-tobacco media messages.

The following issues might be pieces of the complex puzzle but might be unnecessary for the analysis.

Using exposure data regarding environmental tobacco smoke.

Analyzing the effects of parental participation in tobacco prevention.

These two issues can be discussed, but probably would not be included in the analysis.
What Questions Need To Be Answered?
The study question should be stated in clear, specific terms. Examples include:
- How do the costs of screening all newborns for hearing defects compare with the costs of screening only those in neonatal intensive-care units?
- Do the benefits of vaccinating healthy adults (aged <50 years) against influenza outweigh the costs?
- How does the cost effectiveness of a mass media campaign compare with the cost effectiveness of a targeted campaign designed to decrease the incidence of adolescent smokers?
Study question(s) must be clear, concise, and comprehensive. The researcher should ask and answer the four questions below before adopting a research strategy:
1. Is the nature of the intervention or health program inherently understood from the study question?
  
  For example, is the mass media campaign a new project that will require start-up funding, or is it an expansion project that will have already absorbed the majority of the preliminary expenses?
2. What are the repercussions or consequences that might result from not adequately addressing key concerns in the study problem?
  
  For example, if the mass media program only targets high school students, will the incidence of tobacco use among middle school students increase?
3. What are the implications for extraneous factors (e.g., populations and public health budgets)?
  
  For example, which programs will be cut and who will be affected by those foregone benefits if the budget is allocated towards smoking-prevention campaigns?
4. Will the results obtained from this study be specific to this study or broadly applicable to general policy?
  
  For example, is the media campaign designed specifically for a certain urban population or age group, or could national tobacco policy benefit?

An Example: Defining the Problem

This example is abstracted from an economic evaluation conducted in 2003 to assess the risks and benefits of smallpox vaccination in the United States.

Risks and Benefits
of Preexposure and Postexposure Smallpox Vaccination

What was the problem to be analyzed?

There was a clear and increasing threat of the use of biologic weapons by terrorists on U.S. soil.
In view of this threat, the U.S. government had secured a stockpile of smallpox vaccine. However, how the vaccine should be administered sparked debate in all circles.

Why was this problem important?

Although the vaccine was effective, it contained a live virus that might potentially result in serious vaccine-related complications.

What aspects of the problem needed to be explained?

Potential risks for a person who is considering smallpox vaccination.
Potential benefits for the person receiving the vaccine.

What questions needed to be answered?

What are the specific risks for smallpox vaccination?
What are the benefits for smallpox vaccination?
Do the benefits outweigh the risks?

2. How Are We Going To Find Out?

After the study problem has been identified and defined, a research strategy must be adopted that can direct an analyst through the remainder of the framing process.

This requires that decisions be made regarding the structure of the study.

The questions below should ensure that all aspects of the research strategy are considered.

Which Intervention(s) Will Be Analyzed?
The first task is to define with precision the program(s) or intervention(s) to be evaluated. For each intervention or program included in the study, identify the following elements:
- the nature of the intervention(s) (e.g., a vaccination program for smallpox),
- the target population(s) (e.g., general population, health-care workers, and investigation team),
- the delivery site(s) (e.g., public health department clinics),
- the personnel delivering the service (e.g., public health nurses [PHNs]),
- the technology to be used (e.g., circulating smallpox vaccine), and
- the timing of the intervention (e.g., August 2003).
Precision is critical because programs that appear to be similar might have substantially different costs and outcomes.

For example, in the preceding study, program costs would be different if the vaccinations were conducted by hospital nurses rather than by public health nurses because of salary differences.

Who is the Audience?

The audience includes all persons or institutions that will be using the results of the study to make decisions. No limit exists in the size and no explicit criteria exist for the composition of the audience.

State and federal legislators, public health officials, program managers, physicians, hospital administrators, employers, the media, and the general public might all be part of a study's audience.

The audience might be key decisionmakers interested in choosing between alternative interventions or programs.

For example, policy decisionmakers might mandate the addition of fluoride to drinking water, whereas program decisionmakers might decide to implement a vaccination campaign on a universal or selective basis. The audience can make financial, clinical, or personal decisions.

Defining the study's audience also involves

defining the information needs of the audience, and
considering how the audience will use the study results.

The information needs of the audience will determine which question(s) the study seeks to answer. The ways in which the audience will use the results might affect the study perspective as well as the presentation of the results.

An Example: An Influenza Vaccination Program

Influenza affects 25% of the U.S. population each year and is the cause of approximately 20,000 deaths. Influenza is also costly to society. Influenza costs approximately $12 billion annually, with only $4.6 billion being attributed to direct medical costs, whereas the majority of the remainder is associated with productivity losses.

Because a considerable amount of the economic burden of disease associated with influenza is attributed to productivity losses, interest exists in the cost effectiveness of an influenza vaccination program targeting healthy working adults aged 18–59 years.

Two audiences are among those interested in such an influenza vaccination program and the potential impact it can have on society:

employers, who ultimately fund a large portion of the program, and
the Advisory Committee on Immunization Practices (ACIP), which is ultimately responsible for creating national immunization recommendations.

For these two audiences, the table below shows the differing information needs and uses for the results of an economic evaluation of the influenza vaccination program.

Information Needs and Uses for the Results of an Economic Evaluation

Audience	Information needs	How will the audience use the results?
Employer	Compare with other benefits already covered by employer	To determine if vaccine should be a covered benefit for employees
ACIP	Compare with other economic evaluations used to make recommendations for immunization policy	To make recommendations for national immunization policy

What is the Study Perspective?
The perspective of an economic evaluation is the viewpoint from which it is conducted.

The selection of the perspective is crucial, because it determines which costs and outcomes are considered relevant and are included in the analysis.
Different Perspectives

An evaluation can be conducted from five different perspectives (in order from the narrowest to the broadest)
Patients

The term patient refers to the person targeted by the health intervention under consideration (certain programs might refer to patients as clients).
The perspective of the patient is appropriate, for example, in a comparison of treating a disease on an outpatient versus an inpatient basis.

The difference in costs between the outpatient and inpatient options (e.g., work absenteeism, disturbance in family life, and length of recovery) might influence a patient's preference for one option or the other.

Providers

The term provider refers to all categories of persons and institutions providing health-related services.
Examples of providers include physicians, hospitals, nursing homes, public health clinics, and local and state health departments. The provider perspective can be that of:

a conglomerate of providers (i.e., a managed-care organization [MCO]), or

a provider within that conglomerate (i.e., one hospital or physician).

The provider perspective is appropriate, for example, when assessing the costs and benefits of establishing and operating a childhood immunization reminder system. From the clinic's perspective, direct medical costs would be included, but productivity losses associated with a patient's going to the clinic to receive the immunization would not be included.

Payers

The term payer refers to the person, entity, or institution ultimately responsible for the financial cost of a program, intervention, or medical procedure.
In the United States, payers include

households that pay for care out-of-pocket or through insurance premiums,

businesses that pay for part of the cost of health insurance,

private insurers (e.g., MCOs),

public insurance programs (e.g., Medicare and Medicaid), and

federal or state governments.

An analysis of the impact of a law mandating health plans to cover the cost of fertility drug treatment is an example of an economic evaluation that could be implemented from the payer perspective — in this case, the private health insurer.

Health-Care System

The health-care system perspective is broader than the perspective of the health-care provider. The health-care system perspective considers the costs and outcomes associated with providing care without differentiating between categories of providers or payers.
The health-care system perspective is useful and informative if the economic burden of a disease is frequently shared by all categories of payers and providers.

For example, the costs and outcomes of STDs are the responsibilities of

public health clinics,

public health laboratories,

primary-care practitioners,

obstetricians,

gynecologists,

family planning clinics, and

hospitals.

The use of a health-care system perspective can provide a more complete estimate of the true medical costs and outcomes of treating STDs than can an estimate from a single category of payers or providers.

Society

The societal perspective is the broadest possible perspective, because it includes all costs (no matter who incurs them) and all consequences (both good and bad), regardless of who experiences them.

For example, when estimating the cost of acquired immunodeficiency (AIDS) from a societal perspective, the analyst should consider not only diagnosis and treatment costs but also

productivity losses associated with the premature death of AIDS patients,

the financial and emotional costs incurred by families and friends caring for AIDS patients, and

the cost to families and society of caring for orphaned children of AIDS patients.

The government perspective is used in studies that are being conducted for the public sector, but the government perspective should not be confused with the societal perspective, which is broader because it encompasses all costs and consequences.

For example, costs incurred by the patient seeking care would not be included in a study conducted from a governmental or public budget perspective, but would be considered relevant from a societal perspective.
Choosing a Perspective

Which Perspective Is Appropriate?

The audience and the relevance of the perspective to the study are the most important criteria for determining which perspective is appropriate.
The analysis must reflect the perspective of persons or institutions who are affected by the outcome of interest and who bear certain costs associated with the program or intervention being evaluated.

If the entire cost of establishing and managing a patient reminder system for childhood immunizations is borne by a public health clinic, conducting an economic evaluation of this system from a patient perspective might be irrelevant.

The choice of a study perspective might also be constrained by the context of the study. The persons or institutions sponsoring the study (the audience) might want the analysis to reflect their own perspective and demand it. Therefore, the choice of a study perspective must be consistent with the audience choice.

Using Multiple Perspectives

A single study may be conducted from multiple perspectives.

The following perspectives can be used in a single study.

If the informational needs of the audience are to determine whether an influenza vaccination program should be a covered benefit for employer-sponsored health care, the perspective taken would be that of the employer. Costs would include those incurred by

the health-care system,

the provider (if reimbursed by the employer), and

the patient (employee) if they resulted in work absenteeism.

If the informational needs of the audience were used to set national immunization guidelines, then a societal perspective would be taken. This perspective includes all costs and benefits, regardless of who pays or receives them.

The extent to which a range of perspectives can be included in the analysis will partially depend on the resources and time available to conduct the study.

For the majority of public health economic evaluations, the societal perspective is appropriate, because the goal of the research is to analyze the allocation of societal resources among competing activities. If society would benefit from those resources, the government will probably play a role in implementing and funding the intervention.
Studies conducted from the societal perspective can be difficult to conduct (e.g., collecting and valuing all the relevant costs and benefits). Nonetheless, optimal practice would dictate whether they are included in the study.

Relating the Perspectives

The following diagram documents how the perspectives are related:

Perspective Diagram
The clinic perspective is contained within the health-care system perspective, which is itself contained within the societal perspective. The standalone employer and patient perspectives are also contained within the societal perspective.
What is the Study Time Frame?
The time frame corresponds to the period during which the intervention or treatment is delivered. The study time frame should take the following factors into account:
Seasonal Variation in Program Activity Level

Program activity can increase or decrease during certain periods of the year. For example, demand for influenza immunization increases during fall and winter. When program costs and outcomes are measured only at peak times or only during periods of low activity, over- or under-estimates might result.
Life Cycle of the Intervention

Costs and outcomes might differ depending on how long a program has been in operation.
During the start-up period when the activity level is low and the most efficient organizational structures and policies have not yet been identified, either of two situations can occur

costs might be lower and outcomes less optimal than in later years, or

higher costs might result because of start-up capital expenditures that are necessary to launch the program.

In contrast, a well-established program might have reached a level of activity that requires a larger staff and a more complex organizational structure with regular maintenance costs and steady-state health outcomes. The larger structure might result in

higher costs or lower costs, depending on how efficiently the program is maintained, and

better outcomes than a newer program.

Intervention costs and outcomes must be tracked during a period long enough for such variations in costs and outcomes to stabilize so that valid estimates can be obtained.
Future Advances in Technology

The time frame of a study needs to be of an appropriate length but should not extend beyond a reasonable point (i.e., the point at which exhausted resources can be justified).
Technology changes can lead to the obsolescence of the program or intervention within a limited period. The time frame chosen should take into account the assumptions made concerning the technology that will be used in the time period considered.

For example, in evaluating an influenza vaccination program, if a new, more effective vaccine conjugate is expected to be available next year, a limited time frame (6–12 months) might be chosen for the analysis.
This limited time frame should reflect a reasonable assumption regarding how long the current program will still be used.
What Is the Analytic Horizon of the Study?
The analytic horizon corresponds to the entire period during which costs and benefits related to the impact of the program or intervention of interest are measured. Therefore, the analytic horizon is relevant in studies in which the effectiveness of a program is considered:
- cost-benefit analysis (CBA),
- cost-utility analysis (CUA), or
- cost-effectiveness analysis (CEA).
The analytic horizon is usually longer than the time frame, particularly in the case of prevention programs. Benefits associated with these interventions can be realized for a substantial period after the intervention has concluded.

For example, a person who is influenced by a mass media campaign during adolescence and who never begins smoking will realize the benefits of averted lung cancer and premature death throughout his or her lifetime.

In certain cases, a disease or illness might extend longer than the lifetime of the targeted group.

For example, an intervention dealing with a congenital anomaly (e.g., sickle-cell disease) might affect the health of future generations because a specific gene is passed to future offspring by abnormalities in the blood's hemoglobin.

However, because no reliable way exists to account for the adverse effects on future generations, the study should include an analytic horizon that captures only the costs and benefits associated with substantial morbidity and premature mortality of the affected person.
The diagram below illustrates the difference between the time frame and the analytic horizon for an economic evaluation that estimated the cost effectiveness of an influenza vaccination program for healthy working adults.

Time Frame and Analytic Horizon
The time frame for the study is 3 weeks, which accounts for the actual time that the vaccine is administered to employees in an on-site health clinic. However, the analytic horizon spans the entire flu season, which accounts for the time that the benefits from receiving the vaccine are realized.
Which Study Format Should Be Chosen?
Possible study formats are a
- prospective study,
- retrospective study, or
- model.
Choosing a study format depends on data, time, and resources that are available to the study researcher. Decisionmakers and those who will use the study results might impose constraints on the study that might favor one study format over another.
Whatever format is chosen, assessing and being aware of the advantages and disadvantages of that particular format can improve the credibility of the study.
Prospective Study

A prospective study collects costs and outcomes data after the study begins. Besides baseline data that are typically collected before program onset, data collection occurs simultaneously with the implementation of the program and must last as long as all relevant costs and benefits accrue.
Prospective studies are often based on randomized clinical trials (RCTs). These trials are advantageous to the researchers because they grant them greater control over the type and quality of the data being obtained from the program being evaluated. The researcher can decide exactly which costs and benefits are necessary for the analysis and ensure that they are recorded.
The prospective study is considered a standard among study formats used in economic evaluations.
Prospective studies can have disadvantages. Because prospective studies collect cost and benefit data simultaneously, they can be resource- and time-intensive. They are also susceptible to observer bias.

Retrospective Study

A retrospective study treats costs that already have been incurred, and outcomes that have been realized when the study begins. The cost and outcome data can be extracted from different sources (e.g., previous program budgets, patient medical records, or existing databases). The data-collection phase lasts only as long as the researcher needs to extract the data from the chosen data sources.
Retrospective studies typically have the advantage of saving time and resources through the data-collection process, because when the study begins, costs and outcomes have already been incurred.
The main disadvantage of retrospective studies is that the researcher is not involved with the data-collection process. Therefore, the available data might not be as comprehensive or detailed as desired.
An Example: Prospective Versus Retrospective Study Formats

The figure below, Retrospective and Prospective Study Formats, illustrates a timeline for a comprehensive smoking-prevention program targeting 7th grade adolescents. The program has been operating since 2002.

Retrospective and Prospective Study Formats
Since January 2004, program officials have wanted to conduct an economic evaluation of the program to comply with the accountability standards required by those funding the program.
If the researchers choose to conduct a prospective evaluation

They will consider all appropriate costs and outcomes and devise ways to accurately track them.

They will choose a study time frame (e.g., 2 years).

When data collection begins (e.g., January 2004), the researchers will ensure that costs and outcomes are tracked as needed.

The data collected will be based on program activities conducted during January 2004–December 2005, which corresponds with the agreed-upon 2-year time frame.

The data collection phase will cover the same period (January 2004–December 2005).

Researchers might also choose to conduct a retrospective study in which

They should consider which costs will be included in the study.

They will agree on a 2-year study time frame.

The cost data collected would be based on program activities during January 2002–December 2003.

The data-collection phase would only last as long as the researchers require in their analysis of the 2002–2003 program records. The quality and availability of data would depend on the type of information that had been preserved in program records.
Model

A model provides a simplified yet accurate representation of reality. Models can be used in economic evaluations to represent the work of an intervention, program, or policy. A model is based on a set of assumptions derived from what is already known regarding the health issue of interest or the impact of programs and interventions.
The information used to construct a model can be obtained from different sources (e.g., clinical trials, guidelines, or previously published studies) therefore making the sources flexible.
Models are useful when

actual data is scarce, unreliable, or unavailable, or

the researcher wants to focus on elements essential to the program without being burdened by details that might be relevant to only the

geographic area in which the program operates,

population it serves, or

idiosyncrasies of its managers.

The predominant disadvantage of using a model is its dependence on estimated rather than measured values.
Common models used in economic evaluations are further explained in the Decision Analysis section of the Cost-Effectiveness Analysis Tutorial.
Advantages and Disadvantages

The study format advantages and disadvantages table below summarizes the advantages and disadvantages of each study format.

Study Format Advantages and Disadvantages
Which Costs Will Be Included?
Costs are the values of the resources used to produce a good or a service. Creating a comprehensive and explicit inventory of costs is necessary for the success of any economic evaluation (see the Cost Analysis tutorial for more information).
Costs may be categorized as either tangible costs or intangible costs.
Tangible Costs

Tangible costs are concrete and quantifiable (e.g., dollars). They comprise direct medical costs, direct nonmedical costs, and productivity losses.

Direct medical costs — those resources used specifically for medical treatment.

Examples are

drugs and pharmaceuticals,

medical supplies,

physician office visits, and

hospitalization charges.

Direct nonmedical costs — any costs incurred as a result of an intervention or illness that are not directly related to the medical care itself.

Examples are

patient travel cost to the hospital or clinic,

program administration, and

utilities associated with the facility administering the procedure

Productivity losses — resources foregone by the patient or caregiver as results of participation in the intervention or of the health condition itself.

Examples are

reduced levels of output,

time used to obtain health care, and

future wages lost because of

time spent away from work as the result of an unexpected illness,

a premature death, or

caring for a sick family member.
Intangible Costs

Intangible costs are those emotional costs associated with pain, suffering, and anxiety.

For example, an intangible cost associated with a vaccine program might be the emotional stress and pain associated with the vaccine injection.

These costs are often difficult to quantify and therefore are typically excluded in an economic evaluation.
Which Costs to Include

The type of economic evaluation that is being conducted determines which costs are included in the analysis

Cost-benefit analysis (CBA): Theoretically should include all costs, both tangible and intangible.

Cost-utility analysis (CUA): Typically includes only direct medical and nonmedical costs in the numerator of the cost utility (CU) ratio. Productivity losses and intangible costs are excluded, because they are assumed to be part of the quality of life estimate in the denominator of the CU ratio.

Cost-effectiveness analysis (CEA): Typically includes all tangible costs but excludes those that are intangible.

The table below summarizes the costs that are typically included in each type of economic evaluation.

Costs
Which Outcome Measure(s) Will Be Used?
An outcome measure is a unit used to assess the output of the program or intervention. Outcomes can be measured in monetary units (CBA), quality-adjusted health outcomes (CUA), or natural units (CEA).
When the study developer decides which outcome measure is the most appropriate for a study, the decision depends primarily upon available data and how the outcome measure will be used.
After the study developer decides that sufficient evidence exists to support the outcome measure chosen, the other components of the study need to be checked to ensure that the outcome can be placed inside the prescribed framework.

For example, if the outcome chosen is number of life years gained, the analytic horizon needs to span the person's lifetime.

Not only is the selection of outcome measure guided by the kind of data that can be reasonably collected, it is also guided by the policy question to be answered and by the analytic method used.
A lack of data to conduct a particular type of analysis might necessitate selecting another analytic method and another, perhaps less preferable, outcome measure.

For example, in a study of the cost effectiveness of programs to promote the use of bicycle helmets among children, the use of quality adjusted life years (QALYs) saved was determined to best answer the policy question, because this determination would allow both fatal and nonfatal head injuries to be included in one measure.
However, data did not exist on the severity of head injuries; only data regarding the number of head injuries were available. Thus, using a quality-adjusted measure was impossible. Therefore, two outcome measures were selected: head injuries prevented and fatalities prevented.
What are the Alternative Interventions?

If the economic evaluation involves comparing more than one intervention (e.g., CBA, CUA, and CEA), then the researcher must develop a list of feasible, mutually exclusive, and exhaustive alternatives.
For all alternative interventions, the researcher should consider which intermediate and final outcomes are important in the analysis.
Feasibility

Alternative options must be feasible.

For example, if a five-dose vaccine were approved for use among all adults, would including it in the study be feasible? Only a limited number of adults would probably take time to receive five vaccinations, especially if the delivery site was inconvenient.

Mutually Exclusive

Each strategy included must be exclusive of all other strategies on the list: strategies should not have overlapping components.

Exhaustive

The list of available interventions must be as exhaustive and complete as possible, allowing comparison of all feasible and mutually exclusive alternatives. If any key strategies that might be considered in the decisionmaking process are excluded, the evaluation will be useless.

Outcomes

Two kinds of outcomes are typically associated with a program or an intervention:

Intermediate outcomes are near-term effects of an intervention (e.g., number of vaccines administered or school children educated).

Final outcomes are the ultimate health outcomes of interest (e.g., cases of disease or death prevented, or QALYs saved).

In certain economic evaluations, the researcher must rely on intermediate outcomes because the final outcome data are insufficient. Although using final outcomes in an economic evaluation is the preferred method, intermediate outcomes may be used if a strong causal link exists between the intermediate and final outcomes.
The diagram below illustrates the difference between intermediate and final outcomes.

Example with Intermediate and Final Outcomes
Which Summary Measure(s) Should Be Used?
When the researcher determines the summary measure(s) that will be reported in the study, the decision depends on the outcome(s) of interest and will be reflected by the type of economic evaluation used in the study.
- In cost-benefit analysis (CBA), the following two summary measures are used
  
  Net benefits (NB) are calculated by subtracting costs (C) from benefits (B).
  
  Programs with positive NBs are typically considered optimal investments.
  Net present value (NPV) is a unique case of NB in that benefits and costs have been appropriately discounted.
  
  Benefit-cost (B/C) ratio is calculated by dividing benefits (B) by costs (C).
  
  Programs with B/C ratios >1 are typically considered optimal investments.
  
  NB or NPV is the preferred summary measure for CBA because the B/C ratio can be misleading regarding the total value of one program compared with another. For example:
  
  In the Net Benefits and Benefit-Cost Ratio table below, the NBs and B/C ratios indicate that both programs are considered optimal investments. When a researcher examines the NBs of Program A and Program B, Program B provides the higher NB and hence the greatest total return.
  However, when researchers compare only the B/C ratios of the two programs, Program A would be judged the better option, because information regarding the total return is not considered.
  
  Net Benefits and Benefit-Cost Ratio
- In a cost-utility analysis (CUA), the cost-utility ratio is the preferred summary measure, as in cost per QALY gained.
- Likewise, in a cost-effectiveness analysis (CEA), a cost-effectiveness ratio is the preferred summary measure (e.g., cost per case of pelvic inflammatory disease prevented).
The table below provides a synopsis of the summary measures used for each type of economic evaluation.

Summary Measures
Significance of Sensitivity Analysis

Sensitivity analysis isolates study variables or parameters, changes their values, and recalculates the study results. This process identifies the parameters with the most influence over the summary measure (or result) while assessing the impact the intervention can have by using varied parameters.
Sensitivity analyses are conducted to test the overall robustness of the results of the model. Testing the robustness of the results enhances the credibility of the study.
Sensitivity analysis can

Identify Variables With the Most Influence

The analyst should focus on the data and calculations involving the most influential variables to ensure that they are accurate.

For example, the age of the youth involved in the smoking-prevention program might have more of a bearing on the results than the person's ability to purchase tobacco products and should therefore be more closely examined.

Assess the Impact Throughout Populations

A study needs to be able to assess the generalizability or the accuracy by which its conclusions can be applicable to different populations.
If an intervention is cost effective only among a population 1) with a specific prevalence and incidence of disease or 2) living in a particular environment and expressing certain demographics, the results of the study will be of limited usefulness to decisionmakers regarding broader policy decisions.

Test Robustness of the Results

The results of a study are robust if they do not change when the study parameters are varied. Robust results lead to increased credibility of conclusions drawn from the evaluation.

For example, a mass media smoking-prevention campaign produces positive results when dealing with a cohort of persons who are at high risk (e.g., high school juniors and seniors who have previous smoking histories).
Now assume that identical conclusions can be drawn from a study of a younger population with no previous smoking history.
The conclusion can be made that results of the study are robust and therefore generalizable.

A Framing-the-Study Example: Adopting a Research Strategy

The research design of the Antigua HECD program.

Summary

The framing stage of an economic evaluation involves defining the problem to be analyzed and adopting a research strategy to analyze it.
Defining in detail the policy, program, or intervention to be analyzed is important.
Defining the audience of the study involves defining their information needs and considering ways in which they will use the results of the analysis.
The perspective from which the evaluation is conducted determines which costs and outcomes to include in the analysis.
Possible perspectives for an economic evaluation include those for the
- patient,
- provider,
- health-care system, and
- society.
The societal perspective includes all costs (no matter who incurs them) and all benefits (no matter who receives them). The societal perspective should be included whenever possible.
The study time frame should be long enough to account for seasonal variations and the life cycle of the policy, program, or intervention.
A prospective format allows greater control over the nature and quality of the cost data collected. The use of retrospective data has the advantage of saving time and resources.
Models are useful when data regarding the policy, program, or intervention of interest are scarce or unreliable.
Creating a comprehensive and explicit cost inventory is necessary for the success of any economic evaluation.
Outcomes can be measured in monetary units, health outcomes, or natural units.
Outcomes influence the summary measures chosen and will be reflected by the form of economic evaluation used in the study.
Sensitivity analysis can increase the credibility and generalizability of the study by testing the robustness of the results.

Test Your Understanding

Please enter your answer to each question before you look at "Our Answer" for each one.

To whom does the term audience for a study refer?

Our Answer

All consumers or users of the results of a study.
What is meant by the perspective of a study?

Our Answer

The viewpoint from which it is conducted. The perspective defines which costs and benefits will be included in the analysis.
Which costs should you include in a study conducted from a societal perspective?

Our Answer

All costs, no matter who incurs them.
Which perspectives on this list would include patient travel costs when determining the total cost?
Patient

Provider

Medicaid

Commercial health plan

Health-care system

Societal

Our Answer

Yes Patient

No   Provider

No   Medicaid

No   Commercial health plan

No   Health-care system

Yes Societal
What is the difference between a study's time frame and its analytic horizon?

Our Answer

The time frame is the time during which program/intervention costs and effects are tracked.
The analytic horizon is the time during which the costs and outcomes associated with the impact of the program/intervention are tracked.
Is conducting a prospective study to assess the costs and benefits of a program that was recently terminated possible? Explain.

Our Answer

No. In a prospective study, information regarding costs and benefits is collected as they are incurred. No information can be collected prospectively if no program activity occurs.
List three intermediate outcomes that could be used to assess the effectiveness of a needle-exchange program.
Our Answer
Number of new needles distributed

Number of used needles collected

Number of persons participating in the program
Give one example of a final outcome for a needle-exchange program.

Our Answer

Number of deaths from AIDS prevented

A Checklist for Framing the Study

Glossary — Framing

Analytic horizon

Period during which costs and outcomes associated with the intervention accrue.

Audience

Consumers or users of study results.

Costs

Value of the resources (e.g., personnel, buildings, vehicles, equipment, and supplies) used to produce a good or service.

Decision sciences

Interdisciplinary approach to problem-solving and decisionmaking, by using modeling techniques and analytic methods.

Economic evaluation

Use of applied analytic techniques to identify, measure, value, and compare the costs and outcomes of alternative interventions.

Economics

Social science that describes and analyzes the production, distribution, and consumption of goods and services.

Effective

The improved health outcome that a prevention strategy can achieve under typical community-based settings.

Efficacious

The improved health outcome that a prevention strategy can achieve under ideal settings.

Epidemiology

Scientific discipline that deals with the prevalence, incidence, and control of a disease within a given population.

Final outcome

Ultimate outcome of interest (e.g., years of life gained or deaths prevented).

Incidence

Number of new cases of a disease or health condition that develops within a specified population in a period (usually 1 year).

Intermediate outcome

Near-term effects of an intervention (e.g., persons screened, rate of condom use, or number of needles exchanged).

Model

Simplified yet accurate representation of a program or intervention, based on a set of assumptions.

Outcome measure

Measurement unit used to assess the effectiveness of a program or intervention.

Payer

Person or organization that provides money for health-care services.

Perspective

Viewpoint of the bearers of the costs and benefits of an intervention (e.g., society, government, health-care providers, businesses, or patients).

Prevalence

Number of existing cases of a disease or health condition in a given population.

Prospective study

Study in which the events of interest (costs and outcomes) take place during the course of the study.

Provider

Person or institution that provides a health-related service.

Randomized clinical trial (RCT)

Type of research design used for comparing treatments. Patients are assigned to a treatment group or control group by a random mechanism.

Retrospective study

Study in which the events of interest (costs and outcomes) occur before the study begins.

Societal perspective

The broadest possible perspective for an economic evaluation. It includes all program costs (no matter who incurs them) and all program consequences (no matter who experiences them).

Time frame

Period during which the intervention takes place.

Utility

A quantitative measure of the strength of a preferred health outcome in terms of personal preference.