EVALUATION BRIEFS	No. 23 | July 2011	Evaluation Technical Assistance


Using Ordered Response Options To Collect Evaluation Data

This brief focuses on using ordered response options to collect evaluation data. It includes the definition of ordered response options; how to create good sets of options; how to analyze and present data collected from them; and advantages and disadvantages of using them on evaluation surveys.


Ordered Response Options

Ordered response options are sets of possible answers for close-ended survey items that are offered to respondents in a specific, meaningful order. They allow participants to select an option on the continuum that best reflects their answer as shown in the following example:  
1.	Very satisfied.
2.	Somewhat satisfied.
3.	Somewhat dissatisfied.
4.	Very dissatisfied.

The options represent a specific order along a continuum, but not the magnitude of difference between the options (also called ordinal data). For example, the difference between “very satisfied” and “somewhat satisfied” may not be equal to the difference between “somewhat satisfied” and “somewhat dissatisfied.” 

Ordered response options are best used for assessing attitudes, beliefs, and opinions.  



Creating Good Ordered Response Options

When creating ordered response options, follow these guidelines to ensure that the data you collect are useful for your evaluation. 

Determine if you want an even or an odd number of response options. Offering an even number of options forces the respondent to make a choice that is not neutral, as shown in the following example:
1.	Never.
2.	Rarely.
3.	Sometimes.
4.	Often.

Offering an odd number of options gives the benefit of providing for a neutral option such as “neither agree nor disagree,” or an undecided option such as “undecided” or “no opinion.”

To determine if you should have an odd number of options, you need to decide if the data collected on the neutral option provides meaningful information for your evaluation.

If you decide to include a neutral option such as “neither agree nor disagree,” place this choice in the middle of the other options. The order of the responses should reflect a logical flow on the continuum as shown in the following example: 
1.	Strongly agree.
2.	Somewhat agree.
3.	Neither agree nor disagree.
4.	Somewhat disagree.
5.	Strongly disagree.

If you choose to add an “undecided” or “no opinion” option, put it at the end of the list of choices. People are less likely to pick this option when it is placed at the end of the list as shown in the following example:
1.	Excellent.
2.	Good.
3.	Fair.
4.	Poor.
5.	No opinion.

Ordered response options should be balanced and include the complete spectrum of options. This will help to reduce bias. If there are two options in the positive direction there should be two options in the negative direction as shown in the following example:

1.	All of the time.
2.	Most of the time.
3.	Seldom.
4.	Never.

Response options must be clearly distinguishable. It is difficult to differentiate between options such as “somewhat unqualified” and “not well qualified.” Options such as this will make the data difficult to analyze and use. 

Although it does not matter whether the positive or negative options are listed first in the set, the order of options should be consistent throughout the entire questionnaire.


Analyzing the Responses and Presenting the Results

When reporting the results from ordered response options, only report the number or percentage of people who chose an option and compare it to other option results.

At times, data can be simplified by combining categories. For example, “strongly agree” and “agree” can be collapsed into one category (agree) while “disagree” and “strongly disagree” can be collapsed into another category (disagree).

Advantages of Using Ordered Response Options
•	Questions can be answered and analyzed quickly. 
•	Response options can be easily modified for people from different backgrounds by using words appropriate for their vocabulary. 
•	Ordered response options produce uniform answers.

Disadvantages of Using Ordered Response Questions
•	People interpret ordered response options differently. For example, some may consider eating fresh fruit every day as “often” while others may consider eating fresh fruit once a week as “often.” 
•	There are limited statistical tests designed for analyzing ordinal data.


RESOURCES 
Dillman DA, Smyth JD, Christian LM. Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. 3rd Edition. Hoboken, NJ: John Wiley & Sons, Inc; 2009. 

Taylor-Powell, E. Program Development and Evaluation: Questionnaire Design: Asking Questions with a Purpose. University of Wisconsin-Extension; 1998. 

Singleton RA., & Straits BC.  Approaches to Social Research. 4th Edition. New York, NY: Oxford University Press, Inc; 2005.


INFORMATION

For more information on creating high-quality survey questions, see Evaluation Brief #15.



For further information or assistance, contact the Evaluation Research Team at ert@cdc.gov. You can also contact us via our Web site.


National Center for Chronic Disease Prevention and Health Promotion
Division of Adolescent and School Health
Centers for Disease Control and Prevention