Chad M. Yates, Courtney M. Holmes, Jane C. Coe Smith, Tiffany Nielson
Implementing continuous feedback loops between clients and counselors has been found to have significant impact on the effectiveness of counseling (Shimokawa, Lambert, & Smart, 2010). Feedback informed treatment (FIT) systems are beneficial to counselors and clients as they provide clinicians with a wide array of client information such as which clients are plateauing in treatment, deteriorating or at risk for dropping out (Lambert, 2010; Lambert, Hansen, & Finch, 2001). Access to this type of information is imperative because counselors have been shown to have poor predictive validity in determining if clients are deteriorating during the counseling process (Hannan et al., 2005). Furthermore, recent efforts by researchers show that FIT systems based inside university counseling centers have beneficial training features that positively impact the professional development of counseling students (Reese, Norsworthy, & Rowlands, 2009; Yates, 2012). To date, however, few resources exist on how to infuse FIT systems into counselor education curriculum and training programs.
This article addresses the current lack of information regarding the implementation of a FIT system within counselor education curricula by discussing: (1) an overview and implementation of a FIT system; (2) a comprehensive review of the psychometric properties of three main FIT systems; (3) benefits that the use of FIT systems hold for counselors-in-training; and (4) how the infusion of FIT systems within a counseling curriculum can help assess student learning outcomes.
Overview and Implementation of a FIT System
FIT systems are continual assessment procedures that include weekly feedback about a client’s current symptomology and perceptions of the therapeutic process in relation to previous counseling session scores. These systems also can include other information such as self-reported suicidal ideation, reported substance use, or other specific responses (e.g., current rating of depressive symptomology). FIT systems compare clients’ current session scores to previous session scores and provide a recovery trajectory, often graphed, that can help counselors track the progress made through the course of treatment (Lambert, 2010). Some examples of a FIT system include the Outcome Questionnaire (OQ-45.2; Lambert et al., 1996), Session Rating Scale (SRS; Miller, Duncan, & Johnson, 2000), Outcome Rating Scale (ORS; Miller & Duncan, 2000), and the Counseling Center Assessment of Psychological Symptoms (CCAPS; Locke et al., 2011), all of which are described in this article.
Variety exists regarding how FIT systems are used within the counseling field. These variations include the selected measure or test, frequency of measurement, type of feedback given to counselors and whether or not feedback is shared with clients on a routine basis. Although some deviations exist, all feedback systems contain consistent procedures that are commonly employed when utilizing a system during practice (Lambert, Hansen, & Harmon, 2010). The first procedure in a FIT system includes the routine measurement of a client’s symptomology or distress during each session. This frequency of once-per-session is important as it allows counselors to receive direct, continuous feedback on how the client is progressing or regressing throughout treatment. Research has demonstrated that counselors who receive regular client feedback have clients that stay in treatment longer (Shimokawa et al., 2010); thus, the feedback loop provided by a FIT system is crucial in supporting clients through the therapeutic process.
The second procedure of a FIT system includes showcasing the results of the client’s symptomology or distress level in a concise and usable way. Counselors who treat several clients benefit from accessible and comprehensive feedback forms. This ease of access is important because counselors may be more likely to buy in to the use of feedback systems if they can use them in a time-effective manner.
The last procedure of FIT systems includes the adjustment of counseling approaches based upon the results of the feedback. Although research in this area is limited, some studies have observed that feedback systems do alter the progression of treatment. Lambert (2010) suggested that receiving feedback on what is working is apt to positively influence a counselor to continue these behaviors. Yates (2012) found that continuous feedback sets benchmarks of performance for both the client and the counselor, which slowly alters treatment approaches. If the goal of counseling is to decrease symptomology or increase functioning, frequently observing objective progress toward these goals using a FIT system can help increase the potential for clients to achieve these goals through targeted intervention.
Description of Three FIT Systems
Several well-validated, reliable, repeated feedback instruments exist. These instruments vary by length and scope of assessment, but all are engineered to deliver routine feedback to counselors regarding client progress. Below is a review of three of the most common FIT systems utilized in clinical practice.
The OQ Measures System
The OQ Measures System uses the Outcome Questionnaire 45.2 (OQ-45.2; Lambert et al., 1996), a popular symptomology measure that gauges a client’s current distress levels over three domains: symptomatic distress, interpersonal relations and social roles. Hatfield and Ogles (2004) listed the OQ 45.2 as the third most frequently used self-report outcome measure for adults in the United States. The OQ 45.2 has 45 items and is rated on a 5-point Likert scale. Scores range between 0 and 180; higher scores suggest higher rates of disturbance. The OQ 45.2 takes approximately 5–6 minutes to complete and the results are analyzed using the OQ Analyst software provided by the test developers. The OQ 45.2 can be delivered by paper and pencil versions or computer assisted administration via laptop, kiosk, or personal digital assistant (PDA). Electronic administration of the OQ 45.2 allows for seamless administration, scoring and feedback to both counselor and client.
Internal consistency for the OQ 45.2 is α = 0.93 and test-retest reliability is r = 0.84. The OQ 45.2 demonstrated convergent validity with the General Severity Index (GSI) of the Symptom Checklist 90-Revised (SCL-90-R; Derogatis, 1983; r = .78, n = 115). The Outcome Questionnaire System has five additional outcome measures: (1) the Outcome Questionnaire 30 (OQ-30); (2) the Severe Outcome Questionnaire (SOQ), which captures outcome data for more severe presenting concerns, such as bipolar disorder and schizophrenia; (3) the Youth Outcome Questionnaire (YOQ), which assesses outcomes in children between 13 and 18 years of age; (4) the Youth Outcome Questionnaire 30, which is a brief version of the full YOQ; and (5) the Outcome Questionnaire 10 (OQ-10), which is used as a brief screening instrument for psychological symptoms (Lambert et al., 2010).
The Partners for Change Outcome Management System (PCOMS)
The Partners for Change Outcome Management System (PCOMS) uses two instruments, the Outcome Rating Scale (ORS; Miller & Duncan, 2000) that measures the client’s session outcome, and the Session Rating Scale (SRS; Miller et al., 2000) that measures the client’s perception of the therapeutic alliance. The ORS and SRS were designed to be brief in response to the heavy time demands placed upon counselors. Administration of the ORS includes handing the client a copy of the ORS on a sheet of letter sized paper; the client then draws a hash mark on four distinct 10-centimeter lines that indicate how he or she felt over the last week on the following scales: individually (personal well-being), interpersonally (family and close relationships), socially (work, school and friendships), and overall (general sense of well-being).
The administration of the SRS includes four similar 10-centimeter lines that evaluate the relationship between the client and counselor. The four lines represent relationship, goals and topics, approach or methods, and overall (the sense that the session went all right for me today; Miller et al., 2000). Scoring of both instruments includes measuring the location of the client’s hash mark and assigning a numerical value based on its location along the 10-centimeter line. Measurement flows from left to right, indicating higher-level responses the further right the hash mark is placed. A total score is computed by adding each subscale together. Total scores are graphed along a line plot. Miller and Duncan (2000) used the reliable change index formula (RCI) to establish a clinical cut-off score of 25 and a reliable change index score of 5 points for the ORS. The SRS has a cut-off score of 36, which suggests that total scores below 36 indicate ruptures in the working alliance.
The ORS demonstrated strong internal reliability estimates (α = 0.87-.096), a test-retest score of r = 0.60, and moderate convergent validity with measures like the OQ 45.2 (r = 0.59), which it was created to resemble (Miller & Duncan, 2000; Miller, Duncan, Brown, Sparks, & Claud, 2003). The SRS had an internal reliability estimate of α = 0.88, test-retest reliability of r = 0.74, and showed convergent validity when correlated with similar measures of the working alliance such as the Helping Alliance Questionnaire–II (HAQ–II; Duncan et al., 2003; Luborsky et al., 1996). The developers of the ORS and SRS have also created Web-based administration features that allow clients to use both instruments online using a pointer instead of a pencil or pen. The Web-based administration also calculates the totals for the instruments and graphs them.
The Counseling Center Assessment of Psychological Symptoms (CCAPS)
The CCAPS was designed as a semi-brief continuous measure that assesses symptomology unique to college-aged adults (Locke et al., 2011). When developed, the CCAPS was designed to be effective in assessing college students’ concerns across a diverse range of college campuses. The CCAPS has two separate versions, the CCAPS-62 and a shorter version, the CCAPS-34. The CCAPS-62 has 62 test items across eight subscales that measure: depression, generalized anxiety, social anxiety, academic distress, eating concerns, family distress, hostility and substance abuse. The CCAPS-34 has 34 test items across seven of the scales found on the CCAPS-62, excluding family distress. Additionally, the substance use scale on the CCAPS-62 is renamed the Alcohol Use Scale on the CCAPS-32 (Locke et al., 2011). Clients respond on a 5-point Likert scale with responses that range from not at all like me to extremely like me. On both measures clients are instructed to answer each question based upon their functioning over the last 2 weeks. The CCAPS measures include a total score scale titled the Distress Index that measures the amount of general distress experienced over the previous 2 weeks (Center for Collegiate Mental Health, 2012). The measures were designed so that repeated administration would allow counselors to compare each session’s scores to previous scores, and to a large norm group (N = 59,606) of clients completing the CCAPS at university counseling centers across the United States (Center for Collegiate Mental Health, 2012).
The CCAPS norming works by comparing clients’ scores to a percentile score of other clients who have taken the measure. For instance, a client’s score of 80 on the depressive symptoms scale indicates that he or she falls within the 80th percentile of the norm population’s depressive symptoms score range. Because the CCAPS measures utilize such a large norm base, the developers have integrated the instruments into the Titanium Schedule ™, an Electronic Medical Records (EMR) system. The developers also offer the instruments for use in an Excel scoring format, along with other counseling scheduling software programs. The developers of the CCAPS use RCI formulas to provide upward and downward arrows next to the reported score on each scale. Downward arrows indicate the client’s current score is significantly different than previous sessions’ scores and suggests progress during counseling. An upward arrow would suggest a worsening of symptomology. Cut-off scores vary across scales and can be referenced in the CCAPS 2012 Technical Manual (Center for Collegiate Mental Health, 2012).
Test-retest estimates at 2 weeks for the CCAPS-62 and CCAPS-34 scales range between r = 0.75–0.91 (Center for Collegiate Mental Health, 2012). The CCAPS-34 also demonstrated a good internal consistency that ranged between α = 0.76–0.89 (Locke et al., 2012). The measures also demonstrated adequate convergent validity compared to similar measures. A full illustration of the measures’ convergent validity can be found in the CCAPS 2012 Technical Manual (Center for Collegiate Mental Health, 2012).
Benefits for Counselors-in-Training
The benefits of FIT systems are multifaceted and can positively impact the growth and development of student counselors (Reese, Norsworthy, et al., 2009; Schmidt, 2014; Yates, 2012). Within counselor training laboratories, feedback systems have shown promise in facilitating the growth and development of beginning counselors (Reese, Usher, et al., 2009), and the incorporation of FIT systems into supervision and training experiences has been widely supported (Schmidt, 2014; Worthen & Lambert, 2007; Yates, 2012).
One such benefit is that counseling students’ self-efficacy improved when they saw evidence of their clients’ improvement (Reese, Usher, et al., 2009). A FIT system allows for the documentation of a client’s progress and when counseling students observed their clients making such progress, their self-efficacy improved regarding their skill and ability as counselors. Additionally, the FIT system allowed the counselor trainees to observe their effectiveness during session, and more importantly, helped them alter their interventions when clients deteriorated or plateaued during treatment. Counselor education practicum students who implemented a FIT system through client treatment reported that having weekly observations of their client’s progress helped them to isolate effective and non-effective techniques they had used during session (Yates, 2012). Additionally, practicum counseling students have indicated several components of FIT feedback forms were useful, including the visual orientation (e.g., graphs) to clients’ shifts in symptomology. This visual attenuation to client change allowed counselors-in-training to be more alert to how clients are actually faring in between sessions and how they could tailor their approach, particularly regarding crisis situations (Yates, 2012).
Another benefit discovered from the above study was that counseling students felt as if consistent use of a FIT system lowered their anxiety and relieved some uncertainty regarding their work with clients (Yates, 2012). It is developmentally appropriate for beginning counselors to struggle with low tolerance for ambiguity and the need for a highly structured learning environment when they begin their experiential practicums and internships (Bernard & Goodyear, 2013). The FIT system allows for a structured format to use within the counseling session that helps to ease new counselors’ anxiety and discomfort with ambiguity.
Additionally, by bringing the weekly feedback into counseling sessions, practicum students were able to clarify instances when the feedback was discrepant from how the client presented during session (Yates, 2012). This discrepancy between what the client reported on the measure and how they presented in session was often fertile ground for discussion. Counseling students believed bringing these discrepancies to a client’s attention deepened the therapeutic alliance because the counselor was taking time to fully understand the client (Yates, 2012).
Several positive benefits are added to the clinical supervision of counseling students. One such benefit is that clinical supervisors found weekly objective reports of their supervisees helpful in providing evidence of a client’s progress during session that was not solely based upon their supervisees’ self-report. This is crucial because relying on self-report as a sole method of supervision can be an insufficient way to gain information about the complexities of the therapeutic process (Bernard & Goodyear, 2013). Supervisors and practicum students both reported that the FIT system frequently brought to their attention potential concerns with clients that they had missed (Yates, 2012). A final benefit is that supervisees who utilized a FIT system during supervision had significantly higher satisfaction levels of supervision and stronger supervisory alliances than students who did not utilize a FIT system (Grossl, Reese, Norsworthy, & Hopkins, 2014; Reese, Usher, et al., 2009).
Benefits for Clients
Several benefits exist for counseling clients when FIT systems are utilized in the therapeutic process. The sharing of objective progress information with clients has been found to be perceived as helpful and a generally positive experience by clients (Martin, Hess, Ain, Nelson, & Locke, 2012). Surveying clients using a FIT system, Martin et al. (2012) found that 74.5% of clients found it “convenient” to complete the instrument during each session. Approximately 46% of the clients endorsed that they had a “somewhat positive” experience using the feedback system, while 20% of clients reported a “very positive” experience. Hawkins, Lambert, Vermeersch, Slade, and Tuttle (2004) found that providing feedback to both clients and counselors significantly increased the clients’ therapeutic improvement in the counseling process when compared to counselors who received feedback independently. A meta-analysis of several research studies, including Hawkins et al. (2004), found effect sizes of clinical efficacy related to providing per-session feedback ranged from 0.34 to 0.92 (Shimokawa et al., 2010). These investigations found more substantial improvement in clients whose counselors received consistent client feedback when compared with counselors who received no client feedback regarding the therapeutic process and symptomology. These data also showed that consistent feedback provision to clients resulted in an overall prevention of premature treatment termination (Lambert, 2010).
Utilization of FIT Systems for Counseling Curriculum and Student Learning Outcome Assessment
The formal assessment of graduate counseling student learning has increased over the past decade. The most recent update of the national standards from the Council for Accreditation of Counseling and Related Educational Programs (CACREP) included the requirement for all accredited programs to systematically track students at multiple points with multiple measures of student learning (CACREP, 2015, Section 4, A, B, C, D, E). Specifically, “counselor education programs conduct formative and summative evaluations of the student’s counseling performance and ability to integrate and apply knowledge throughout the practicum and internship” (CACREP, 2015, Section 4.E). The use of continuous client feedback within counselor education is one way to address such assessment requirements (Schmidt, 2014).
Counseling master’s programs impact students on both personal and professional levels (Warden & Benshoff, 2012), and part of this impact stems from ongoing and meaningful evaluation of student development. The development of counselors-in-training during experiential courses entails assessment of a myriad of counseling competencies (e.g., counseling microskills, case conceptualization, understanding of theory, ethical decision-making and ability to form a therapeutic relationship with clients; Haberstroh, Duffey, Marble, & Ivers, 2014). As per CACREP standards, counseling students will receive feedback during and after their practicum and internship experiences. This feedback typically comes from both the supervising counselor on site, as well as the academic department supervisor.
Additionally, “supervisors need to help their supervisees develop the ability to make effective decisions regarding the most appropriate clinical treatment” (Owen, Tao, & Rodolfa, 2005, p. 68). One suggested avenue for developing such skills is client feedback using FIT systems. The benefit of direct client feedback on the counseling process has been well documented (Minami et al., 2009), and this process can also be useful to student practice and training. Counseling students can greatly benefit from the use of client feedback throughout their training programs (Reese, Usher, et al., 2009). In this way, counselors-in-training learn to acknowledge client feedback as an important part of the counseling process, allowing them to adjust their practice to help each client on an individual basis. Allowing for a multi-layered feedback model wherein the counselor-in-training can receive feedback from the client, site supervisor and academic department supervisor has the potential to maximize student learning and growth.
Providing students feedback for growth through formal supervision is one of the hallmarks of counseling programs (Bernard & Goodyear, 2013). However, a more recent focus throughout higher education is the necessity of assessment of student learning outcomes (CACREP, 2015). This assessment can include “systematic evaluation of students’ academic, clinical, and interpersonal progress as guideposts for program improvement” (Haberstroh et al., 2014, p. 28). As such, evaluating student work within the experiential courses (e.g., practicum and internship) is becoming increasingly important.
FIT systems provide specific and detailed client feedback regarding clients’ experiences within therapy. Having access to documented client outcomes and progress throughout the counseling relationship can provide an additional layer of information regarding student growth and skill development. For instance, if a student consistently has clients who drop out or show no improvement over time, those outcomes could represent a problem or unaddressed issue for the counselor-in-training. Conversely, if a student has clients who report positive outcomes over time, that data could show clinical understanding and positive skill development.
Student learning outcomes can be assessed in a myriad of ways (e.g., FIT systems, supervisor evaluations, student self-assessment and exams; Haberstroh et al., 2014). Incorporating multiple layers of feedback for counseling students allows for maximization of learning through practicum and internships and offers a concrete way to document and measure student outcomes.
An Example: Case Study
Students grow and develop through a wide variety of methods, including feedback from professors, supervisors and clients (Bernard & Goodyear, 2013). Implementing a FIT system into experiential classes in counseling programs allows for the incorporation of structured, consistent and reliable feedback. We use a case example here to illustrate the benefits of such implementation. Within the case study, each CACREP Student Learning Outcome that is met through the implementation of the FIT system is documented.
A counselor educator is the instructor of an internship class where students have a variety of internship placements. This instructor decides to have students implement a FIT system that will allow them to track client progress and the strength of the working alliance. The OQ 45.2 and the SRS measures were chosen because they allow students to track client outcomes and the counseling relationship and are easy to administer, score and interpret. In the beginning of the semester, the instructor provides a syllabus to the students where the following expectations are listed: (1) students will have their clients fill out the OQ 45.2 and the SRS during every session with each client; (2) students will learn to discuss and process the results from the OQ 45.2 and SRS in each session with the client; and (3) students will bring all compiled information from the measures to weekly supervision. By incorporating two FIT systems and the subsequent requirements, the course is meeting over 10 CACREP (2015) learning outcome assessment components within Sections 2 and 3, Professional Counseling Identity (Counseling and Helping Relationships, Assessment and Testing), and Professional Practice.
A student, Sara, begins seeing a client at an outpatient mental health clinic who has been diagnosed with major depressive disorder; the client’s symptoms include suicidal ideation, anhedonia and extreme hopelessness. Sara’s initial response includes anxiety due to the fact that she has never worked with someone who has active suicidal ideation or such an extreme presentation of depressed affect. Sara’s supervisor spends time discussing how she will use the FIT systems in her work with the client and reminds her about the necessities of safety assessment.
In her initial sessions with her client, Sara incorporates the OQ 45.2 and the SRS into her sessions as discussed with her supervisor (CACREP Section 2.8.E; 2.8.K). However, after a few sessions, she does not yet feel confident in her work with this client. Sara feels constantly overwhelmed by the depth of her client’s depression and is worried about addressing the suicidal ideation. Her instructor is able to use the weekly OQ 45.2 and SRS forms as a consistent baseline and guide for her work with this client and to help Sara develop a treatment plan that is specifically tailored for her client based upon the client’s symptomology (CACREP Section 2.5.H, 2.8.L). Using the visual outputs and compiled graphs of weekly data, Sara is able to see small changes that may or may not be taking place for the client regarding his depressive symptoms and overall feelings and experiences in his life. Sara’s instructor guides her to discuss these changes with the client and explore in more detail the client’s experiences within these symptoms (CACREP Section 2.5.G). By using this data with the client, Sara will be better able to help the client develop appropriate and measureable goals and outcomes for the therapeutic process (CACREP Section 2.5.I). Additionally, as a new counselor, such an assessment tool provides Sara with structure and guidance as to the important topics to explore with clients throughout sessions. For example, by using some of the specific content on the OQ 45.2 (e.g., I have thoughts of ending my life, I feel no interest in things, I feel annoyed by people who criticize my drinking, and I feel worthless), she can train herself to assess for suicidal ideation and overall diagnostic criteria (CACREP Section 2.7.C).
Additionally, Sara is receiving feedback from the client by using the SRS measure within session. In using this additional FIT measure, Sara can begin to gauge her personal approach to counseling with this client and receive imperative feedback that will help her grow as a counselor (CACREP, Section 2.5.F). This avenue provides an active dialogue between client and counselor about the work they are doing together and if they are working on the pieces that are important to the client. Her instructor is able to provide both formative and summative feedback on her overall process with the client using his outcomes as a guide to her effectiveness as a clinician (CACREP, Section 3.C). Implementing a FIT system allows for the process of feedback provision to have concrete markers and structure, ultimately allowing for a student counselor to grow in his or her ability to become self-reflective about his or her own practice.
Implications for Counselor Education
The main implications of the integration of FIT systems into counselor education are threefold: (1) developmentally appropriate interventions to support supervisee/trainee clinical growth; (2) intentional measurement of CACREP Student Learning Outcomes; and (3) specific attention to client care and therapeutic outcomes. There are a variety of FIT systems being utilized, and while they vary in scope, length, and targets of assessment, each has a brief administration time and can be repeated frequently for current client status and treatment outcome measurement. With intentionality and dedication, counselor education programs can work to implement the utilization of these types of assessment throughout counselor trainee coursework (Schmidt, 2014).
FIT systems lend themselves to positive benefits for training competent emerging counselors. Evaluating a beginning counselor’s clinical understanding and skills are a key component of assessing overall learning outcomes. When counselors-in-training receive frequent feedback on their clients’ current functioning or session outcomes, they are given the opportunity to bring concrete information to supervision, decide on treatment modifications as indicated, and openly discuss the report with clients as part of treatment. Gathering data on a client’s experience in treatment brings valuable information to the training process. Indications of challenges or strengths with regard to facilitating a therapeutic relationship can be addressed and positive change supported through supervision and skill development. Additionally, by learning the process of ongoing assessment and therapeutic process management, counselor trainees are meeting many of the CACREP Student Learning Outcomes. The integration of FIT systems into client care supports a wide variety of clinical skill sets such as understanding of clinical assessment, managing a therapeutic relationship and treatment planning/altering based on client needs.
Finally, therapy clients also benefit through the use of FIT. Clinicians who receive weekly feedback on per-session client progress consistently show improved effectiveness and have clients who prematurely terminate counseling less often (Lambert, 2010; Shimokawa et al., 2010). In addition to client and counselor benefit, supervisors also have been shown to utilize FIT systems to their advantage. One of the most important responsibilities of a clinical supervisor is to manage and maintain a high level of client care (Bernard & Goodyear, 2013). Incorporation of a structured, validated assessment, such as a FIT system, allows for intentional oversight of the client–counselor relationship and clinical process that is taking place between supervisees and their clients. Overall, the integration of FIT systems into counselor education would provide programs with a myriad of benefits including the ability to meet student, client and educator needs simultaneously.
Conclusion
FIT systems provide initial and ongoing data related to a client’s psychological and behavioral functioning across a variety of concerns. They have been developed and used as a continual assessment procedure to provide a frequent and continuous self-report by clients. FIT systems have been used effectively to provide vital mental health information within a counseling session. The unique features of FIT systems include the potential for recurrent, routine measure of a client’s symptomatology, easily accessible and usable data for counselor and client, and assistance in setting benchmarks and altering treatment strategies to improve a client’s functioning. With intentionality, counselor educator programs can use FIT systems to meet multiple needs across their curriculums including more advanced supervision practices, CACREP Student Learning Outcome Measurement, and better overall client care.
Conflict of Interest and Funding Disclosure
The author reported no conflict of interest
or funding contributions for the development
of this manuscript.
References
Bernard, J. M., & Goodyear, R. K. (2013). Fundamentals of clinical supervision (5th ed.). Boston, MA: Merrill.
Center for Collegiate Mental Health. (2012). CCAPS 2012 technical manual. University Park: Pennsylvania State
University.
The Council for Accreditation of Counseling Related Academic Programs (CACREP). (2015). 2016 accreditation standards. Retrieved from http://www.cacrep.org/for-programs/2016-cacrep-standards
Derogatis, L. R. (1983). The SCL-90: Administration, scoring, and procedures for the SCL-90. Baltimore, MD: Clinical
Psychometric Research.
Duncan, B. L., Miller, S. D., Sparks, J. A., Claud, D. A., Reynolds, L. R., Brown, J., & Johnson, L. D. (2003). The Session Rating Scale: Preliminary psychometric properties of a “working” alliance measure. Journal of Brief Therapy, 3, 3–12.
Grossl, A. B., Reese, R. J., Norsworthy, L. A., & Hopkins, N. B. (2014). Client feedback data in supervision: Effects on supervision and outcome. Training and Education in Professional Psychology, 8, 182–188.
Haberstroh, S., Duffey, T., Marble, E., & Ivers, N. N. (2014). Assessing student-learning outcomes within a counselor education program: Philosophy, policy, and praxis. Counseling Outcome Research and Evaluation, 5, 28–38. doi:10.1177/2150137814527756
Hannan, C., Lambert, M. J., Harmon, C., Nielsen, S. L., Smart, D. W., Shimokawa, K., & Sutton, S. W. (2005). A lab test and algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology, 61, 155–163.
Hatfield, D., & Ogles, B. M. (2004). The use of outcome measures by psychologists in clinical practice.
Professional Psychology: Research & Practice, 35, 485–491. doi:10.1037/0735-7028.35.5.485
Hawkins, E. J., Lambert, M. J., Vermeersch, D. A., Slade, K. L., & Tuttle, K. C. (2004). The therapeutic effects of providing patient progress information to therapists and patients. Psychotherapy Research, 14, 308–327. doi:10.1093/ptr/kph027
Lambert, M. J. (2010). Prevention of treatment failure: The use of measuring, monitoring, & feedback in clinical practice.
Washington, DC: American Psychological Association.
Lambert, M. J., Hansen, N. B., & Finch, A. E. (2001). Patient-focused research: Using patient outcome data to enhance treatment effects. Journal of Consulting and Clinical Psychology, 69, 159–172.
Lambert, M. J., Hansen, N. B., & Harmon, S. C. (2010). Outcome Questionnaire system (The OQ system): Development and practical applications in healthcare settings. In M. Barkham, G. Hardy, & J. Mellor-Clark (Eds.), Developing and delivering practice-based evidence: A guide for the psychological therapies (pp. 141–154). New York, NY: Wiley-Blackwell.
Lambert, M. J., Hansen, N. B., Umphress, V., Lunnen, K., Okiishi, J., Burlingame, G. M., & Reisinger, C. (1996). Administration and scoring manual for the OQ 45.2. Stevenson, MD: American Professional Credentialing Services.
Locke, B. D., Buzolitz, J. S., Lei, P. W., Boswell, J. F., McAleavey, A. A., Sevig, T. D., Dowis, J. D. & Hayes, J.
(2011). Development of the Counseling Center Assessment of Psychological Symptoms-62 (CCAPS-62).
Journal of Counseling Psychology, 58, 97–109.
Locke, B. D., McAleavey, A. A., Zhao, Y., Lei, P., Hayes, J. A., Castonguay, L. G., Li, H., Tate, R., & Lin, Y. (2012). Development and initial validation of the Counseling Center Assessment of Psychological Symptoms-34 (CCAPS-34). Measurement and Evaluation in Counseling and Development, 45, 151–169. doi:10.1177/0748175611432642
Luborsky, L., Barber, J. P., Siqueland, L., Johnson, S., Najavits, L. M., Frank, A., & Daley, D. (1996). The Helping
Alliance Questionnaire (HAQ–II): Psychometric properties. The Journal of Psychotherapy Practice and
Research, 5, 260–271.
Martin, J. L., Hess, T. R., Ain, S. C., Nelson, D. L., & Locke, B. D. (2012). Collecting multidimensional client data using repeated measures: Experiences of clients and counselors using the CCAPS-34. Journal of College Counseling, 15, 247–261. doi:10.1002/j.2161-1882.2012.00019.x
Miller, S., & Duncan, B. (2000). The outcome rating scale. Chicago, IL: International Center for Clinical Excellence.
Miller, S., Duncan, B., & Johnson, L. (2000). The session rating scale. Chicago, IL: International Center for Clinical
Excellence.
Miller, S. D., Duncan, B. L., Brown, J., Sparks, J. A., & Claud, D. A. (2003). The Outcome Rating Scale: A
preliminary study of the reliability, validity, and feasibility of a brief visual analog measure. Journal of
Brief Therapy, 2, 91–100.
Minami, T., Davies, D. R., Tierney, S. C., Bettmann, J. E., McAward, S. M., Averill, L. A., & Wampold, B. E. (2009). Preliminary evidence on the effectiveness of psychological treatments delivered at a university counseling center. Journal of Counseling Psychology, 56, 309–320.
Owen, J., Tao, K. W., & Rodolfa, E. R. (2005). Supervising counseling center trainees in the era of evidence-based practice. Journal of College Student Psychotherapy, 20, 66–77.
Reese, R. J., Norsworthy, L. A., & Rowlands, S. R. (2009). Does a continuous feedback system improve psychotherapy outcome? Psychotherapy: Theory, Research, Practice, Training, 46, 418–431.
doi:10.1037/a0017901
Reese, R. J., Usher, E. L., Bowman, D. C., Norsworthy, L. A., Halstead, J. L., Rowlands, S. R., & Chisolm, R.
R. (2009). Using client feedback in psychotherapy training: An analysis of its influence on supervision
and counselor self-efficacy. Training and Education in Professional Psychology, 3, 157–168.
doi:10.1037/a0015673
Schmidt, C. D. (2014). Integrating continuous client feedback into counselor education. The Journal of Counselor Preparation and Supervision, 6, 60–71. doi:10.7729/62.1094
Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010). Enhancing treatment outcome of patients at risk of treatment failure: Meta-analytic and mega-analytic review of a psychotherapy quality assurance system. Journal of Consulting and Clinical Psychology, 78, 298–311. doi:10.1037/a0019247
Warden, S. P., & Benshoff, J. M. (2012). Testing the engagement theory of program quality in CACREP-accredited counselor education programs. Counselor Education and Supervision, 51, 127–140.
doi:10.1002/j.1556-6978.2012.00009.x
Worthen, V. E., & Lambert, M. J. (2007). Outcome oriented supervision: Advantages of adding systematic
client tracking to supportive consultations. Counselling & Psychotherapy Research, 7, 48 –53.
doi:10.1080/14733140601140873
Yates, C. M. (2012). The use of per session clinical assessment with clients in a mental health delivery system: An
investigation into how clinical mental health counseling practicum students and practicum instructors use
routine client progress feedback (Unpublished doctoral dissertation). Kent State University, Kent, Ohio.
Chad M. Yates is an Assistant Professor at Idaho State University. Courtney M. Holmes, NCC, is an Assistant Professor at Virginia Commonwealth University. Jane C. Coe Smith is an Assistant Professor at Idaho State University. Tiffany Nielson is an Assistant Professor at the University of Illinois at Springfield. Correspondence can be addressed to Chad M. Yates, 921 South 8th Ave, Stop 8120, Pocatello, Idaho, 83201, yatechad@isu.edu.
