TPC Journal-Vol 10- Issue 3-FULL ISSUE

344 The Professional Counselor | Volume 10, Issue 3 reliability (McHugh, 2012). Table 1 illustrates that the κ free for the pretest was .36 while the percent of overall agreement was 57.6%. The posttest for the κ free was .56 and the percent of overall agreement was 70.4%. After examining the change in pretest to posttest calculations from both the κ free and the percent of overall agreement, both offer additional support for and provide evidence that training improved the agreement of dispositional ratings on the PDCA-RA. Table 1 Pre and Post Statistics: Percent Overall Agreement and Free-Marginal Fleiss’ Kappa Time of Rating Percent Overall Agreement Free-Marginal Kappa 95% CI for Free- Marginal Kappa Before Training: Pre 57.6 .36 [.23, .49] After Training: Post 70.4 .56 [.31, .80] The overall κ , κ free and percent of agreement results were promising, but a comparison of the percent of correct responses (the response intended by the research team) by disposition showed that the ratings of correct responses decreased by more than 2% from pre- to posttesting for three dispositions (Openness, Cooperativeness, and Moral Reasoning). Because this was an unexpected finding, the research team analyzed the ratings for incorrect responses and learned that the raters appeared to be better able to discern the difference between a rating of 1 ( developing ) and 3 ( meets expectation ) than between 3 and 5 ( above expectation ). As a post-hoc analysis, we calculated the percent of agreement with the correct score, collapsing the 3 and 5 ratings. The percent of correct responses with dichotomous categories of 1 and a collapsed category for 3 and 5 are shown in Table 2. As is evident in Table 2, using the collapsed category, the percent of correct responses for eight of the nine dispositions improved from pretest to posttest. The percent of correct responses for one disposition, Cooperativeness, decreased by more than 2% from pretest to posttest. Table 2 Pre and Post Percent of Correct Responses by Disposition Disposition Pre Percent Overall Agreement 1, 3, 5 Ratings Post Percent Overall Agreement 1, 3, 5 Ratings Pre Percent Overall Agreement 1, 3 & 5 Collapsed Post Percent Overall Agreement 1, 3 & 5 Collapsed 1. Conscientiousness 62.0 97.1 77.1 98.6 2. Coping & Self-Care 59.9 94.4 22.9 97.1 3. Openness 51.0 49.4 94.3 100.0 4. Cooperativeness 47.3 39.0 94.3 87.1 5. Moral Reasoning 84.1 68.8 91.4 98.6 6. Interpersonal Skills 48.0 94.4 98.6 97.1 7. Cultural Sensitivity 69.3 94.4 100.0 100.0 8. Self-Awareness 40.7 40.0 54.3 64.3 9. Emotional Stability 56.3 56.5 67.1 95.7