TPC Journal-Vol 10- Issue 3-FULL ISSUE

346 The Professional Counselor | Volume 10, Issue 3 complexity of the video clips should be conducted. It may be desirable to revise the training protocol to utilize less complex responses for Part 1 training, followed by equivalent complex interviews for Part 2 training, and more complex interview responses for Part 3. More complex responses, meaning the responses are partially descriptive of two categories on the rubric, are realistic to actual admissions interviews in the field. In conducting trainings for the PDCA-RA, a potentially interesting observation was that raters appeared predisposed to using their own subjective experience to rate the video interviews instead of applying the item descriptions in the rubric. Often the trainers observed that the disposition title, such as Self-Awareness, triggered an automatic response of high rater confidence in their ability to rate self- awareness without carefully reading the rubric descriptions. The tendency of raters to believe they are “right” rather than applying a rubric description is a potential barrier for any dispositional measure. Implications of the Study The implications of this study relate primarily to counselor education programs. As evident from the review of literature, careful admissions processes are critical to prevent or diminish the number of gatekeeping and remediation situations that occur in academic programs after admission. In addition to the importance of fair admissions procedures from a legal perspective, the effort required of applicants to engage in the application process justifies the importance of developing fair processes in which acceptance or denial decisions are not based solely upon the subjectivity of faculty. For those academic programs utilizing admissions interviews, one important implication of the study is that the results suggest that without training, raters will have high variability in their ratings of admissions applicants, as illustrated by the variability of the pretest scores in this study. Structuring the rating of admissions interviews by using an assessment is one method of mitigating the variability of faculty ratings of applicants. A holistic (global) rubric such as the PDCA-RA is unlikely to ever garner the almost perfect interrater reliability associated with analytic rubrics, but the PDCA-RA is available as one practical, field-tested tool with promising reliability to help facilitate transparent and fair admissions interview rating processes. Limitations and Future Research In light of the lack of an established list of professional dispositions, the PDCA-RA’s utility may be limited, as the selected dispositions may not align with the values of all counselor education programs. A second limiting factor is that the sample included both field site supervisors and faculty, and all participants were from the rural Western United States. The reliability of the tool is limited by the demographics of the sample. Another limitation was that the study’s pretest and posttest video clips, although similar, were different from one another. The initial decision to use different pretest and posttest video clips was based on an attempt to reduce the influence of testing as a threat to internal validity. However, this also introduced the possibility that either of the sets of video clips was inherently easier or more difficult to rate than the other. Further research would include randomly juxtaposing pretest and posttest video clips, or perhaps using the same video clips pre- and posttest to eliminate the possibility that differences in pretest and posttest video clips were responsible for the improvements in score reliability rather than the intended independent variable, the training. Another potential limitation to the results is that it is possible that some of the graduate students who were filmed in the vignettes may have been known by six of the faculty members from one of the institutions. The impact of this possibility was reduced by the use of multiple student actors, but prior knowledge of the student could have influenced raters’ scores.