9.1: Creating Comparison Files for Reliability Tests

Step 1: The first step in using the Reliability Agent is to create two comparison files. Select Create Comparison Files under Reliability on the Menu bar on the Interview Information Screen. (See figure 9.1-1)

Step 2: A dialog screen is opened which asks the user to specify how many respondents the user wants EZ-Text to randomly select and copy to the two comparison files. This number is constrained by the number of records in the database. After entering the desired number, click Next. (See figure 9.1-2)

Step 3: Specify the name and path for each of the comparison files. To make file management easier, the two files can be named after each coder and iteration number. For example, the comparison file names "mary_no1.mdb" and "john_no1.mdb" might indicate the names of the coders and also show that the files are to be used for conducting the first inter-coder reliability test. When the names and paths for the files have been selected, click on Finish. (See figure 9.1-3)

Step 4: A screen will appear informing the user that the two comparison files have been successfully created. Click on OK, and then on Cancel to return to the Interview Information Screen.

Step 5: Each coder should then independently code his or her files using the same codebook.

9.2: Running the Reliability Comparison

After the two coders have independently coded their files, the files can be compared to assess the degree of agreement. EZ-text provides two different statistical methods for assessing reliability.

9.2(a): Method 1: Comparison of Sets of Codes Assigned by Both Coders to the Same Set of Responses

This method pinpoints which questions or respondents in the database have poor coding reliability. To do this, EZ-Text compares the sets of codes that each coder has assigned to each open-ended question response in the comparison files (see Carey et al. 1996 for an example of an application using this technique). For example, suppose that a coder named Mary assigned CODE1, CODE2, and CODE3 to the response for open-ended question 1 for respondent ID 1000. A coder named John also assigned CODE1, CODE 2, and CODE3 to the same response for the same respondent. EZ-Text counts this situation as an "agreement" . However, if John had only assigned CODE1 and CODE3, or has assigned an additional CODE4 to this response, EZ-Text would count these situations as disagreements. For the purposes of computing reliability statistics, EZ-Text treats any discrepancy in how the two coders assign sets of codes to a response - no matter how small - as a disagreement. This procedure is intended to yield a statistically conservative estimate of reliability.

The comparison process is repeated for all the responses selected by the user (i.e., the number of selected respondents multiplied by the number of selected questions). EZ-Text generates an output summarized by each respondent ID number and each open-ended question number. In addition, the EZ-Text reliability output provides an estimate of overall reliability by combining the results from all the selected questions and respondents.

The output can be used to identify which open-ended questions and which respondents have the lowest coding reliability levels. Subsequent coding improvement efforts can focus attention on the data from these difficult-to-code questions and respondents. After developing a final version of the codebook, both coders should recode their files; generation of another reliability report will provide the final level of agreement achieved in this study.

Step 1: Select the Run Reliability Comparison item from the Reliability Menu on the Interview Information Screen. (See figure 9.2-1)

Step 2: The user is asked to provide the names of the comparison files. Type the name and path of the desired files, or browse other choices by clicking the Browse button. Once the desired files have been selected, click Next. (See figure 9.2-2)

Step 3: The user is then asked to select a reliability method. Click the selection button to the left of the Compare set of codes assigned by the two coders to responses field. Click Next. (See figure 9.2-3)

Step 4: On the next screen, select which questions to compare.

Step 5: Click on Finish once the desired questions have been selected. See section 9.3 for instructions on how to view and print the reliability reports.

9.2(b): Comparison of How Two Coders Use Each Code Across the Same Set of Responses

The second reliability method compares how each coder has used each code across the responses. EZ-Text constructs a 2 x 2 contingency table containing four cells showing whether each coder assigned or did not assign a specific code to the set of responses. For example, suppose that a user wants to see how coders Mary and John used CODE1 across 200 response passages (e.g., responses to 20 open-ended questions collected from 10 respondents). There are two agreement, or concordant cells: the upper left cell shows the number of times both coders assigned the code, and the lower right cell contains the number of times both coders did not assign CODE1 to the responses. The two discordant cells indicate the number of times CODE1 was assigned by one coder but not the other. The sum of all four cells is equal to the total number of response passages (in this example, the sum is 200).

A deceptively simple measure of agreement for how CODE1 is assigned would be the proportion of times the two coders agreed (i.e., in this example, the sum of the two concordant or agreement cells divided by 200). However, simple proportions do not take into account that some of this agreement may occur due to chance (Gorden 1992: 184-185). Therefore, the proportion of agreement is a biased overestimate of the agreement of how two coders have assigned CODE1.

To correct for this problem, the kappa statistic is a more appropriate measure (see Carey et al. 1996 for an example of an application using this technique). In general, kappa can be thought of as the amount of agreement between two coders after statistically adjusting for agreement due to chance. A kappa value of 1.00 indicates perfect agreement. A value of 0.00 indicates no agreement beyond chance, and kappa takes on negative values as low as -1.00 when there is less agreement than would be expected by chance (Fleiss 1981).

Ideally, researchers will want the kappa scores calculated for all the codes in their final codebook to be as close as possible to 1.00. In the study conducted by Carey et al. (1996), final inter-coder reliability kappa score values for 135 out of 152 codes present in the final codebook fell between 0.90 and 1.00. Of these 135 codes, 126 had a kappa value exactly equal to 1.00, which indicated complete agreement between the two coders.

Users may obtain kappa statistics for any combination of codes present in the codebook in the two comparison files. After instructing EZ-Text which codes, respondents, and questions to include in the calculations, EZ-Text generates an on-screen report that shows the following results for each code: the simple percent agreement between the two coders, the amount of their agreement estimated to be due to chance, the kappa statistic, and the standard error for kappa. If desired, the 2 x 2 table can be viewed on screen. A report summarizing these results can be printed.

The results can be used to identify which codes have the poorest reliability. In many cases, simple changes in the codebook can ameliorate the problems (e.g., clarify ambiguous code definitions, remove duplicate or overlapping codes, clarify coder training misunderstandings, etc.).

Step 1: Select the Run Reliability Comparison item from the Reliability Menu on the Interview Information Screen.

Step 2: The user is asked to provide the names of the comparison files. Type the name and path of the desired files, or browse other choices by clicking the Browse button. Once the desired files have been selected, click Next.

Step 3: The user is then asked to select a reliability method. Click the selection button to the left of the Compare how the two coders assigned each code across responses field. (See figure 9.2-4)

Step 4: On the next screen specify which questions to compare, then click Next.

Step 5: The next screen asks the user to specify which codes are to be compared. Select the codes in the same manner as questions are selected in Step 4 above.

Step 6: Click on Finish once the desired questions have been selected. See section 9.3 for instructions on how to view and print the reliability reports.

Step 7: To view the 2 x 2 contingency table (see figure 9.2-5) discussed at the beginning of the section for a particular code, double-click on the row containing the code to be viewed. Click Return to return to the Kappa Summary Screen.

Note: It sometimes happens that the distribution of cell values in the 2 x 2 contingency table fall into one cell. When this occurs kappa cannot be calculated using the formula. Instead, since this is a boundary condition, the occurrence must be handled by definition of the particular condition. Specifically,

If all the cell values fall into either one of the two concordant cells (i.e. all in the "Yes-Yes" or the "No-No" cells in figure 9.2-5), the two coders are in complete agreement. Kappa is 1.0 by definition. (Fleiss, 1981)

If all the cell values fall into either one of the two disconcordant cells (i.e. all in the "Yes-No" or the "No-Yes" cells in figure 9.2-5, the two coders are in complete disagreement. Kappa is -1.0 by definition. (Fleiss, 1981)

In other circumstances, all of the cell values may fall into the two cells within a single row of the 2 x 2 table or into the two cells in a single column of the 2 x 2 table. When this occurs, the value of kappa is zero (0) and the standard error is zero (0).

9.3: Viewing and Printing Reliability Reports

After running the reliability comparison, EZ-Text computes the reliability statistics and displays them automatically. The reliability statistics are presented differently for each method.

Note: WARNING Computation of these reliability statistics and reports places considerable demands on a computer's processing capabilities. Keep in mind that many EZ-Text database files can be very large, and heavy statistical demands may exceed resources available on less powerful computers. Users with weak computer hardware may experience difficulties or lengthy delays. If problems occur, users should try dividing the task into smaller less demanding tasks (e.g., instead of asking the computer to generate kappa statistics for 300 codes across two large comparison files, request the same results in three groups of 100 codes).

9.3(a): Viewing and Printing Reliability Reports for Method 1

Step 1: Click Finish after question numbers to be compared have been selected. EZ-Text will display the Reliability Report Print Preview Screen automatically. (See figure 9.3-1)

Step 2: Use the scroll-down bar to view the rest of the first page of the reliability report. To view subsequent pages, click Next Page.

Step 3: To print the current page, click Print Page. To print the entire report, click Print All.

Step 4: Click Cancel to go back to the previous screen (Step 3: Select Open-Ended Questions). Click Cancel again to return to the Interview Information Screen.

9.3(b): Viewing and Printing Kappa Summary Reports for Method 2

Step 1: Click Finish after codes numbers to be compared have been selected. EZ-Text will display the Kappa Summary report automatically. (See figure 9.3-2)

Step 2: To print the Kappa Summary Report, click Print Report.

Step 3: To return to the previous screen (Step 4: Select Codes to Compare), click Cancel. Click Cancel again to return to the Interview Information Screen.

Go to References