Centering Social Justice in Counselor Education: How Student Perspectives Can Help

Sapna B. Chopra, Rebekah Smart, Yuying Tsong, Olga L. Mejía, Eric W. Price

This mixed methods program evaluation study was designed to assist faculty in better understanding students’ multicultural and social justice training experiences, with the goal of improving program curriculum and instruction. It also offers a model for counselor educators to assess student experiences and to make changes that center social justice. A total of 139 first-semester students and advanced practicum students responded to an online survey. The Consensual Qualitative Research-Modified (CQR-M) method was used to analyze brief written narratives. The Multicultural Counseling Competence and Training Survey (MCCTS) and the Advocacy Competencies Self-Assessment Survey (ACSA) were used to triangulate the qualitative data. Qualitative findings revealed student growth in awareness, knowledge, skills, and action, particularly for advanced students, with many students reporting a desire for more social justice instruction. Some students of color reported microaggressions and concerns that training centers White students. Quantitative analyses generally supported the qualitative findings and showed advanced students reporting higher multicultural and advocacy competencies compared to beginning students. Implications for counselor education are discussed.

Keywords: social justice, program evaluation, training, multicultural counseling, counselor education

In the midst of the COVID-19 pandemic and the long-standing inequities it brought to light, many universities began examining the ways that injustice unfolds within their institutions (Mull, 2020). Arredondo et al. (2020) noted that counseling and counselor education continue to uphold white supremacy and center the experiences of White people within theories, training, and research. White supremacy culture promotes Whiteness as the norm and standard, intersects with and reinforces other forms of oppression, and shows up in institutions in both overt and covert ways, such as emphasis on individualism, avoidance of conflict, and prioritizing White comfort (Okun, 2021). Arredondo et al. (2020) called for counselor educators to engage in social justice advocacy and to unpack covert White supremacy in training programs. The present study investigated the multicultural and social justice training experiences of students in a Western United States counseling program so that counseling faculty can be empowered to uncover biases and better integrate social justice in the curriculum.

Counselor education programs are products of the larger sociopolitical environment and dominant patriarchal, cis-heteronormative, Eurocentric culture that often fails to “challenge the hegemonic views that marginalize groups of people” which “perpetuate deficit-based ideologies” (Goodman et al., 2015, p. 148). For example, the focus on the individual in traditional counseling theories can reinforce oppression by failing to address the role of systemic oppression in a client’s distress (Singh et al., 2020). Counseling theory textbooks usually provide an ancillary section at the end of each chapter focusing on multicultural issues (Cross & Reinhardt, 2017). White supremacy culture is so ubiquitous that it is typically invisible to those immersed within it (DiAngelo, 2018). It is not surprising then that counseling is often viewed as a White, middle-class endeavor, and BIPOC (Black, Indigenous, and People of Color) clients frequently perceive that they should leave their cultural identities and experiences outside the counseling session (Turner, 2018). Counselor educators have been encouraged to reflect on how Eurocentric curricula and pedagogy may marginalize students and seek liberatory teaching practices that promote critical consciousness (Sharma & Hipolito-Delgado, 2021).

Students’ Perceptions of Their Growth, Learning Process, and Critiques of Their Training
     Studies of mostly White graduate students show gains in expanding awareness of their own biases and privilege, knowledge about other cultures and experiences of oppression, as well as the importance of empowering and advocating for clients (Beer et al., 2012; Collins et al., 2015; Sanabria & DeLorenzi, 2019; Singh et al., 2010). Others indicated the benefits of integrating feminist principles in treatment (Hoover & Morrow, 2016; Singh et al., 2010). Consciousness-raising and self-reflection were key parts of multicultural and social justice learning (Collins et al., 2015; Hoover & Morrow, 2016), and could be emotionally challenging. Indeed, Goodman et al. (2018) identified a theme of internal grappling reflecting students’ experiences of intellectual and emotional struggle; others noted students’ experiences of overwhelm and isolation (Singh et al., 2010), as well as resistance, such as withdrawing or dismissing information that challenged their existing belief system (Seward, 2019). Researchers have also documented student complaints about their social justice training; for example, that social justice is not well integrated or that there was inadequate coverage of skills and action (Collins et al., 2015). Kozan and Blustein (2018) found that even among programs that espouse social justice, there was a lack of training in macro level advocacy skills. Barriers to engaging in advocacy included: lack of time (Field et al., 2019; Singh et al., 2010), emotional exhaustion stemming from observations of the harms caused by systemic inequities (Sanabria & DeLorenzi, 2019), and ill-informed supervisors (Sanabria & DeLorenzi, 2019).

The studies reviewed thus relied on samples of mainly White, cisgender, heterosexual women. Some noted that education on social justice is often centered on helping White students expand their awareness (Haskins & Singh, 2015). In one study focused on challenges faced by students of color, participants expressed frustration with the lack of diversity among their professors, classmates, and curriculum (Seward, 2019). Participants also experienced marginalization and disconnection when professors and students made offensive or culturally uninformed comments and when course content focused on teaching students with privileged identities. Students from marginalized communities also face isolation in academic settings and sometimes question the multicultural competence of their professors (Haskins & Singh, 2015), which in turn contributes to the underrepresentation of students of color in counseling and psychology (Arney et al., 2019).

The Present Study

Counselor educators must critically examine their curriculum, course materials, and overall learning climate for students (Haskins & Singh, 2015). Listening to students’ experiences and perceptions of their training offers faculty an opportunity to model cultural humility, gain useful feedback, and make necessary changes. Given the increased recognition of racial trauma and societal inequities, it is critical that counseling programs engage with students of diverse backgrounds as they seek to shift their pedagogy. Historically, academic institutions have responded to student demands with performative action rather than meaningful change (Zetzer, 2021). This mixed methods study is part of a larger process of counseling faculty working to invite student feedback and question internalized assumptions and biases in order to implement real change. The goal of program evaluation is to investigate strengths and weaknesses in order to improve the program (Royse et al., 2010). According to the 2024 Council for Accreditation of Counseling and Related Educational Programs (CACREP) standards, program evaluation is essential to assess and improve the program (CACREP, 2023). Thus, the purpose of this program evaluation study was to understand students’ self-assessment and experiences with the counseling program’s curriculum in the area of multicultural and social justice advocacy, with the overarching goal of program curriculum and instruction improvement. This article offers counselor educators a model of how to assess program effectiveness in multicultural and social justice teaching and practical suggestions based on the findings. The research questions were: What are beginning and advanced students’ self-perceptions regarding their multicultural and social justice advocacy competencies? What are beginning and advanced students’ perceptions of the multicultural and social justice advocacy competencies training they are receiving in their program?

Method

We employed a mixed method, embedded design in which the quantitative data offered a supportive and secondary role to the qualitative results (Creswell et al., 2003). Qualitative and mixed methods research designs are particularly useful in program evaluation (Royse et al., 2010). Mixed method approaches also offer value in research that centers social justice advocacy, as the integration of diverse methodological techniques within a single study fosters the understanding of multiple perspectives and facilitates a deeper comprehension of intricate issues (Ponterotto et al., 2013). We used an online survey to collect written narratives (qualitative) and survey data (quantitative) from two counseling courses: a beginning counseling course in the first semester (beginning students), and an advanced practicum course, taken by those who had completed at least part of their year-long practicum (advanced students).

Participants
     Participants were counseling students enrolled in a CACREP-accredited program at a large West Coast public university in the United States that is both a federally designated Hispanic-serving institution and an Asian American and Native American Pacific Islander–serving institution. Responses were collected from two courses, which included 94 beginning students (84% response rate) and 62 advanced students (71% response rate). Twelve percent of the advanced practicum students also completed the survey when they were first-semester (beginning) students. The mean age of the 139 participants was 27.7 (SD = 7.11), ranging from 20 to 58 years. Racial identifications were 40.3% White, 33.1% Latinx, 14.4% Asian, 7.2% Biracial or Multiracial, 2.9% Black, 0.7% Middle Eastern, 0.7% American Indian/Alaska Native, and 0.7% Native Hawaiian/Pacific Islander. The majority identified as women (82.0%), followed by 14.4% as men, and 2.9% as nonbinary/queer. Students self-identified as heterosexual (71.2%), bisexual (11.5%), lesbian/gay (6.5%), queer (4.3%), pansexual (1.4%), and about 1% each as asexual, heteroflexible, and unsure. About 19.4% of students were enrolled in a bilingual/bicultural (Spanish/Latinx) emphasis within the program.

Procedure
     After receiving university IRB approval, graduate students enrolled in the first-semester beginning counseling course (fall 2018 and 2019) or the advanced practicum course (summer 2019 and 2020) were asked to complete an online survey through Qualtrics with both quantitative measures and open-ended questions as part of their preparation for class discussion. Students were informed that this homework would not be graded and was not intended to “test” their knowledge but rather would serve as an opportunity to reflect on their experience of the program’s multicultural and social justice training. Students were also given the option to participate in the current study by giving permission for their answers to be used. Those who consented were asked to continue to complete the demographic questionnaire. In accordance with the American Counseling Association Code of Ethics (2014), students were informed that there would be no repercussions for not participating. A faculty member outside the counseling program managed the collection of and access to the raw data in order to protect the identities of the students and ensure that their participation or lack of participation in the study could not affect their grade for the course or standing in the program. All students, regardless of participation status, were given the option to enter an opportunity drawing for a small cash prize ($20 for data collection in 2018 and 2019, $25 for 2020) through a separate link not connected to their survey responses.

Data Collection
     We collected brief written qualitative data and responses to two quantitative measures from both beginning and advanced students.

Qualitative Data
     The faculty developed open-ended questions that would elicit student feedback on their multicultural and social justice training. Prior to beginning the counseling program, first-semester students were asked two questions about their experiences and impressions: How would you describe your knowledge about and interest in multiculturalism/diversity and social justice from a personal and/or academic perspective? and How would you describe your initial impressions or experience of the focus on multicultural and social justice in the program so far? They were also asked, if it was relevant, to include their experience in the Latinx counseling emphasis program component. Advanced students, who were seeing clients, were asked the same questions and also asked to: Consider/describe how this experience of multiculturalism and social justice in the program may impact you personally and professionally (particularly in work with clients) in the future.

Quantitative Data
     Two instruments were selected to quantitatively assess students’ perceptions of their own multicultural and advocacy competencies. The Multicultural Counseling Competence and Training Survey (MCCTS; Holcomb-McCoy & Myers, 1999) is designed to assess counselors’ perceptions of their multicultural competence and the effectiveness of their training. The survey contains 32 statements for which participants answer on a 4-point Likert scale (not competent, somewhat competent, competent, extremely competent). Sample items include: “I can discuss family therapy from a cultural/ethnic perspective” and “I am able to discuss how my culture has influenced the way I think.” The reliability coefficients for each of the five components of the MCCTS ranged from .66 to .92: Multicultural Knowledge (.92), Multicultural Awareness (.92), Definitions of Terms (.79), Knowledge of Racial Identity Development Theories (.66), and Multicultural Skills (.91; Holcomb-McCoy & Myers, 1999). In this study, the Cronbach’s alpha coefficients ranged from .75 to .96.

The Advocacy Competencies Self-Assessment Survey (ACSA; Ratts & Ford, 2010) assesses for competency and effectiveness across six domains: (a) client/student empowerment, (b) community collaboration, (c) public information, (d) client/student advocacy, (e) systems advocacy, and (f) social/political advocacy. It contains 30 statements that ask participants to respond with “almost always,” “sometimes,” or “almost never.” Sample questions include “I help clients identify external barriers that affect their development” and “I lobby legislators and policy makers to create social change.” Although Ratts and Ford (2010) did not provide psychometrics of the original ACSA, it was validated with mental health counselors (Bvunzawabaya, 2012), suggesting an adequate internal consistency for the overall measure, but not the specific domains. In this study, the Cronbach’s alpha coefficients ranged from .69 to.79 for the six domains, and .94 for the overall scale. For the purposes of this study, we were not interested in specific domains and used the overall scale to assess students’ overall social justice/advocacy competencies.

Data Analysis
Qualitative Data Analysis
     To analyze the qualitative data, we used Consensual Qualitative Research-Modified (CQR-M; Spangler et al., 2012), which was based on Hill et al.’s (2005) CQR but modified for larger numbers of participants with briefer responses. In contrast to the in-depth analysis of a small number of interviews, CQR-M was ideal for our data, which consisted of brief written responses from 139 participants. CQR-M involves a consensus process rather than interrater reliability among judges, who discuss and code the narratives, and relies on a bottom-up approach, in which categories
(i.e., themes) are derived directly from the data rather than using a pre-existing thematic structure. Frequencies (i.e., how many participants were represented in each category) are then calculated. We analyzed the beginning and advanced students’ responses separately, as the questions were adjusted for their time spent in the program.

After immersing themselves in the data, the first two authors, Sapna B. Chopra and Rebekah Smart, met to outline a preliminary coding structure, then met repeatedly to revise the coding into more abstract categories and subcategories. The computer program NVivo was used to organize the coding process and determine frequencies. After all data were coded, the fifth author, Eric W. Price, served as auditor and provided feedback on the overall coding structure. Both the consensus process and use of an auditor are helpful in countering biases and preconceptions. Brief quantitative data, as used in this study, can be used effectively as a means of triangulation (Spangler et al., 2012).

Quantitative Data Analysis
     To examine for significant differences in the self-perceptions of multicultural competencies and advocacy competencies between White and BIPOC students as well as between beginning and advanced students, a two-way (2×2) ANOVA was conducted with the overall MCCT as the criterion variable and student levels (beginning, advanced) and race (White, BIPOC) as the two independent variables. In addition, two (5×2) multivariate analyses of variances (MANOVAs) were conducted with the five factors of multicultural competencies (knowledge, awareness, definition of terms, racial identity, and skills) as criterion variables and with student levels (beginning, advanced) and student races (White, BIPOC) as independent variables in each analysis. Data for beginning and advanced students were analyzed separately to assess whether time in the counseling program helped to expand their interest and commitment to social justice.

Research Team
     We were intentional in examining our own social identities and potential biases throughout the research process. Chopra is a second-generation South Asian American, heterosexual, cisgender woman. Smart is a White European American, heterosexual, cisgender woman. Yuying Tsong identifies as a genderqueer first-generation Taiwanese and Chinese American immigrant. Olga L. Mejía is an Indigenous-identified Mexican immigrant, bisexual, cisgender woman. Price is a White, gay, cisgender male. All have experience as counselor educators and in qualitative research methods, and all have been actively engaged in decolonizing their syllabi and incorporating multicultural and social justice into their pedagogy.

Results

The research process was guided by the overarching question: What are beginning and advanced counseling students’ perceptions of their multicultural and social justice competencies and training and how can their feedback be used to improve their counselor education program? We explore the qualitative findings first, as the primary data for the study, followed by the quantitative data.

Qualitative Findings for Beginning Counseling Students
     Two higher-order categories emerged from the beginning students’ narratives: developing competencies and learning process so far.

Developing Competencies
     Students’ descriptions of the competencies they were developing included themes of awareness, knowledge, and skills and action. Some students entered the program with an already heightened awareness, while others were making new discoveries. Awareness included subthemes of humility (24.5%), awareness of own privilege (6.4%), and awareness of bias (3.2%). “There’s a lot to learn” was a typical sentiment, particularly from White students. One White female student wrote: “I definitely need more and I believe that open discussions, even hard ones would be some of the best ways to go about this.” A large group expressed knowledge of oppression and systemic inequities (33%); a smaller group referenced intersectionality (3.2%). Within skills and action, some students expressed specific intentions in allyship (11.7%); a number of students expressed commitment to social action but felt unsure how to engage in social justice (11.7%).

Learning Process So Far
     Central themes in this category were support for growth, concerns in training, and internal challenges. Some students felt excited and supported, while some were cautiously optimistic or concerned. Support for growth was a strong theme that reflected excited and enthusiastic to learn (22.3%); appreciation for the Latinx emphasis (18.1%); and receiving support from professors and program (17.0%). For example, one Mexican student in the Latinx emphasis who noted that mental health was rarely discussed in her family shared: “For me to see that there is a program that teaches students how to communicate to individuals who are unsure of what counseling is about, gave me a sense of happiness and relief.”

A few students were adopting a wait-and-see attitude and expressed some concerns about their training. Although the percentage for these subthemes is low, they provide an important experience that we want to amplify. This theme had multiple subthemes. The subtheme concerns from students of color included centering White students (3.2%), microaggressions (3.2%), and lack of representation (1.1%). A student who identified as a Mexican immigrant shared experiences of microaggressions, including classmates using a hurtful derogatory phrase referring to immigrants with no comment from the professor until the student raised the issue. Concerns in training also included the subtheme concerns with how material is presented in classes (7.0%). For some, the concern related to the potential for harm in classes in which White and BIPOC students were encouraged to process issues of privilege and oppression. For example, one Asian Pacific Islander student wrote that although they appreciated the emphasis on social justice, “Time always runs out and I believe it’s careless and dangerous to cut off these types of conversations in a rushed manner.” A small minority seemed to suggest a backlash to the emphasis on social justice, stating that the content was presented in ways that were too “politically correct,” “biased,” or “repetitive.”

     Multiple subthemes emerged from the theme of internal challenges. Both BIPOC and White students shared feeling afraid to speak up (5.3%). BIPOC students expressed struggling with confidence or wanting to avoid conflict, while White students’ fear of speaking up was also connected to discomfort and uncertainty as a White person (2.1%). A small minority of White students did not express explicit discomfort but seemed to engage in a color-blind strategy, as indicated in the theme of people are people (2.1%): “I find people are people, regardless of any differences, and love hearing the good and bad about everybody’s experiences.” Some students of color expressed limited knowledge about cultures other than one’s own (4.3%). For example, an Asian American student stated that they had gravitated to “those who were most similar to me” growing up. Lastly, a few students shared feeling overwhelmed and exhausted (3.2%).

Qualitative Findings for Advanced Counseling Students
     Four higher-order themes emerged: competencies in process, multiculturalism and diversity in the program, social justice in the program, and the learning process.

Competencies in Process
     Similar to beginning students, advanced students described growing self-awareness, knowledge and awareness of others, skills, and action. Their disclosures often related to clinical work, now that they had been seeing clients. Self-awareness included strong subthemes of: humility and desire to keep learning (25.8%); increased open-mindedness, acceptance of others, and compassion (22.6%); awareness of personal privilege and oppression (17.7%); awareness of personal bias and value systems (17.7%); and awareness of personal cultural identity (14.5%). One Mexican American student wrote: “I have also gained an increased awareness of how my prejudices can impact my work with clients and learned about how to check-in with myself.”

     Knowledge and awareness of others had subthemes of privilege and oppression (19.4%) and increased knowledge of culture (14.5%), with awareness of the potential impact on clients. The advanced students also had more to say about skills, which included subthemes of diversity considerations in conceptualization (29%), and in treatment (12.9%), and cultural conversations in the therapy room (21%). One White student wrote: “I have been able to have difficult conversations that once were unheard of. I have also been able to bring culture, ethnicity, and oppression into the room so that my clients can feel understood and safe.” Within the theme of action, 52% wrote about their commitment to social justice and intention to advocate. Although this strongest subtheme suggested action was still more aspirational than currently enacted, a smaller group also wrote about the experiences that they have already had with client advocacy (12.9%), community and/or political action (12.9%), and unspecified action (11.3%).

Multiculturalism and Diversity in the Program
     Many students (44%) indicated that they appreciated that multicultural issues were integrated or addressed well within the program. However, with more time spent in the program, 26% felt that there was more nuance, depth, or scope needed. Some wanted more attention to specific issues, such as disability, gender identity, and religion/spirituality. One Asian American student wrote that the focus had been “basic and surface-level,” adding “I feel like it has also generally catered to the protection of White feelings and voices, which is inherently complicit in the system of White supremacy, especially in higher ed.” Others (9.7%) said more training in clinical application was needed.

Social Justice in the Program
     Students expressed a variety of opinions. The largest number (29%) were satisfied that social justice issues were well integrated into the program. Although more students were satisfied than not, many (24%) noted that social justice is addressed but not demonstrated. Similarly, 24% noted minimal attention, specifically that social justice was not addressed much beyond the one course focused on culture, and 24% noted a desire for more opportunities within the program to engage in advocacy. Some suggested requiring social justice work rather than leaving it as an optional activity. Others (13%), mostly from 2020, noted the relevance of current events and sociopolitical climate. One White student shared about a presentation on Black Lives Matter: “This project opened my eyes to my limited knowledge of systemic oppression in the U.S. and impacted me in ways that I will NEVER be the same.” A small number of students (3%) reported that there was no need or room for more training in social justice. One White student wrote that they felt “frustrated” and that the social justice “agenda is so in my face all the time,” adding “sometimes I feel like I am being trained to be an advocate and an activist, which is/are a different job.”

The Learning Process
     Three central themes emerged: enrichment experienced, challenges, and suggestions for change. Many students were appreciative of their experience. A strong subtheme within enrichment experienced was professors’ encouragement and modeling (24%). Others commented on how much came from learning from peers (21%). Some shared feeling personally empowered (14.5%). For example, a student who identified as coming from an Asian culture wrote about the hesitancy to be an activist, stating, “There is an underlying belief that our voices will not really ever be heard which is strongly tied to systemic oppression and racism throughout history. Consequently, I appreciate this challenge to grow more in social justice issues.” Others shared ways that the program prompted them to engage in social justice outside the classroom (11.3%). For example, one student wrote: “This program gave me the knowledge and education I needed to make sure that when I did speak out I wasn’t just talking to talk. I would actually have facts, stats, evidence-based research to back up my argument.” A number of students noted the unique benefits of the Latinx program (9.7%). One Mexican American student reflected that they had learned about diversity within Latinx cultures, and that, “As a result, I feel more confident in being able to serve clients from various Latinx cultures or at least know where to obtain relevant information when needed.” Many students expressed a sense of belonging (8.1%).

     Challenges. Nearly 10% wrote about struggling to make time [for social justice] and 6.5% noted the emotional impact. For example, one White student wrote: “It was a rude and brutal awakening, to say the least. It was riddled with emotion and heartache but was worth the process.” A few had conflicted or mixed feelings (8.1%)they felt appreciative but wanted more. A few noted possible harm to marginalized students (6.5%). One Asian American student wrote that faculty should be “calling out microaggressions . . . otherwise, their stance on social justice feels more performative and about protecting their own liability rather than caring for their students of color.” A smaller number (4.8%) struggled with peers and colleagues who seemed uninformed.

     Suggestions for Change. Students offered suggestions for improvement, with a strong theme to develop more diverse representation (16.1%), including more representation in faculty, students, case examples, and class discussions. Some comments were specifically about needed attention to Black experiences; one concerned teaching about resiliencies and strengths in the face of oppression. Almost 15% suggested making changes to courses or curriculum. One White student wrote: “If it were me running the program (lol) I would . . . remove the culture class and have all those topics embedded into the fabric of each class because culture and diversity are in all those topics.” A few suggested that faculty require social justice assignments (8.1%), adding that many students will not act unless required. A few also suggested that the program provide more education of White students (8.1%).

Quantitative Findings
Quantitative analyses were conducted to provide triangulation for the qualitative findings and a different view of the data, including possible differences between BIPOC and White students and beginning and advanced students. Table 1 includes descriptive statistics providing an overview of beginning and advanced students’ self-perception of their multicultural and social justice competencies.

Table 1
Descriptive Statistics of Competencies

Multicultural Social Justice/Advocacy
N Mean SD Mean SD
White Beginning 35 2.58 .50 62.97 24.23
Advanced 27 3.09 .38 76.07 19.11
Total 62 2.80 .52 68.68 22.93
BIPOC Beginning 59 2.66 .56 63.05 29.30
Advanced 35 3.01 .30 77.14 20.71
Total 94 2.79 .51 68.30 27.19
Total Beginning 94 2.63 .54 63.02 27.39
Advanced 62 3.05 .34 76.68 19.87
Total 156 2.80 .51 68.45 25.51

 

To examine if there were discernable differences between the beginning and advanced students’ perceptions of their competencies, and if there were differences between White and BIPOC students, a two-way (2×2) ANOVA was conducted with the overall MCCT as the criterion variable and student levels (beginning, advanced) and race (White, BIPOC) as the two independent variables. Results indicated that although there were no interaction effects between race and student levels, there were significant differences in overall multicultural competencies between beginning and advanced students, F(1, 152) = 30.54, p < .001, indicating that advanced practicum students reported significantly higher overall multicultural competencies than beginning students. There were no statistically significant differences between White and BIPOC students in their overall multicultural competencies. Two (5×2) MANOVAs were conducted with the five factors of multicultural competencies as criterion variables (knowledge, awareness, definition of terms, racial identity, and skills). Student levels (beginning, advanced) and student race (White, BIPOC) were independent variables. Results indicated that there were significant differences between beginning and advanced students in at least one of the multicultural competencies components, Wilks’ Lambda = .72, F(5, 150) = 11.97, p < .001. More specifically, follow-up univariate ANOVAs indicated that advanced students reported significantly higher multicultural competencies in their knowledge, F(1, 154) = 43.74, p < .001, µ2 = .22; awareness, F(1, 154) = 6.20, p = .014, µ2 = .04; and racial identity, F(1, 154) = 43.17, p < .001, µ2 = .21. However, there were no significant differences in definitions of terms or skills. Even though there were no significant differences between White and BIPOC students in their overall multicultural competencies, the results of the 5×2 MANOVA indicated that there were significant differences in at least one of the components, Wilks’ Lambda = .87, F(5, 150) = 4.49, p = .001. Follow-up univariate ANOVAs indicated that White students reported higher multicultural competencies in racial identity than BIPOC students in this study, F(1, 154) = 4.51, p = .035, µ2 = .03. There were no differences in the other areas.

A two-way (2×2) ANOVA was conducted with the overall ACSA as the criterion variable and student levels (beginning, advanced) and race (White, BIPOC) as the two independent variables. Results indicated that while there were no interaction effects between race and student levels, there were significant differences in overall advocacy competencies between beginning and advanced students, F(1, 152) = 10.78, p = .001, indicating that advanced students reported significantly higher overall advocacy competencies (M = 76.68) than beginning students (M = 63.02). There were no statistically significant differences between White and BIPOC students in their overall advocacy competencies.

Discussion

This study was designed to examine students’ experiences of their multicultural and social justice training as an aspect of program evaluation, specifically to assist faculty in improving curriculum and instruction with regard to multicultural and advocacy competencies; the study also offers a unique contribution to existing literature by including a more racially diverse (60% BIPOC) sample. Students reported growth in the core areas of multicultural and social justice competency as outlined by Ratts et al. (2016): awareness, knowledge, skills, and action. Consistent with Field et al.’s (2019) findings, students reported more growth in awareness and knowledge than in social justice action, with some differences as students moved through the program. Although beginning students identified personal biases, the theme of self-awareness was more complex for them later in the program. This suggests that a longer time spent in the program contributed to personal growth; although this seems expected, these outcomes have not necessarily been examined before and confirm that the programs’ increasing effort on multiculturalism and social justice are showing gains. The advanced students wrote about clinical application as well and made overt statements of their commitment to social justice. The quantitative results supported these qualitative findings, with advanced students reporting higher multicultural competencies in knowledge, awareness, and racial identity and higher overall advocacy competencies compared to beginning students. With one exception, there were no significant differences between White and BIPOC students in their self-assessment of multicultural or advocacy competencies. Across racial groups, students expressed humility and desire to learn more.

Although students expressed mixed opinions about their experience of the multicultural and social justice training, a greater number of advanced students reported that they thought multicultural (44%) and social justice issues (30%) were well integrated into the program compared to the number of students with critiques. Students reported that support from faculty and peers facilitated their growth and learning, consistent with previous research (e.g., Beer et al., 2012; Keum & Miller, 2020). Some students noted a sense of belonging, particularly those in the Latinx emphasis.

Similar to other researchers, we found that many students wanted social justice issues to be integrated across the curriculum rather than into one course (Beer et al., 2012; Collins et al., 2015); they also wanted more focus on skills and action (Collins et al., 2015; Kozan & Blustein, 2018). Students’ scores on the ACSA advocacy competencies scale reflect this gap in training as well. Though fewer students offered critiques of their training, these responses are important to amplify because some of these concerns are rarely solicited or acknowledged. For example, BIPOC students echoed the challenges faced by students in Seward’s (2019) study, including lack of representation in their faculty, classmates, and curriculum as well as feelings of marginalization when microaggressions in the classroom went unchecked and when instruction centered the needs of White students. Additionally, a few advanced students from 2020, during a time of significant racial-sociopolitical uprising in the United States, expressed concern that class discussions potentially caused harm to students from marginalized communities. Though more students expressed a desire for greater in-depth training, a small minority of mostly White students indicated that they did not want more social justice training and would rather focus solely on traditional counseling skills. These different student perspectives point to the challenges of teaching social justice amidst diverse political and ideological backgrounds and the need to increase community and collaboration.

Listening to Student Feedback and Implications for Decolonizing Program Curriculum
     This study’s findings support the benefits of listening to students’ voices related to multicultural and social justice to inform counselor educators on program strengths and areas for growth. Although student feedback was not the sole impetus for making program changes, accessing this more detailed response was helpful in refining our purpose and direction, as well as highlighting weaknesses. Perhaps more important was the faculty’s willingness to engage in this self-reflective process and to take necessary actions. Rather than waiting for exit interview feedback from graduating students, counselor educators can conduct ongoing program evaluations through anonymous online surveys as well as town hall meetings that invite students to share their process of learning, perceptions of the cultural climate, and experiences of microaggressions. We have a growing understanding that during such evaluations great care needs to be taken for building safety, so as not to retraumatize students from marginalized communities. Based on the results and a series of Zoom town hall meetings, we have implemented changes, such as more consistent integration of social justice across the curriculum; training and day-long retreats focused on increasing faculty competence; faculty participation in Academics for Black Survival and Wellness, an intensive training led by Dr. Della Mosely and Pearis Bellamy; accountability support groups in social justice work; and decolonizing syllabi and class content (e.g., including BIPOC voices and non–APA-style writing assignments). Faculty have also made significant modifications to course materials. For example, beginning students complete weekly modules that include readings and exercises from The Racial Healing Handbook (Singh, 2019), and students study Liberation Psychology during the first week of theories class so they can consider ways to decolonize more traditional models throughout the semester. These strategies have been helpful in preparing students for more difficult conversations surrounding anti-racism in more advanced courses throughout the program. Forming faculty accountability partners or small groups is helpful so that faculty can support each other as a part of their ongoing development in addressing internalized White supremacy and avoiding harm to students.

Student feedback also called attention to the need for self-care, which our program continues to explore. Consistent with previous research (Collins et al., 2015; L. A. Goodman et al., 2018; Hoover & Morrow, 2016; Singh et al., 2010), students reported that their multicultural and social justice learning was often accompanied by moments of overwhelm, hopelessness, and despair. Without tools to manage these emotions, some students may retreat into defensiveness and withdrawal (Seward, 2019), and some may experience activist burnout (Gorski, 2019). Sustainability is necessary for effective social change efforts (Toporek & Ahluwalia, 2021). Counseling programs can offer resources and guidance for students to practice self-care with counselor educators modeling self-care behavior. For example, the Psychology of Radical Healing Collective (Chen et al., 2019) offered strategies to practice radical self-care, including making space for one’s own healing, finding joy and a sense of belonging, and engaging in advocacy at the local community level. Mindfulness practices can be integrated into social justice education to help students and counselor educators manage difficult emotions, increase their ability to be present, and strengthen compassion and curiosity (Berila, 2016). In addition to individual self-care practices, counselor educators can advocate for community care by tending to the community’s needs and drawing on collective experience and wisdom (Gorski, 2019).

The findings point to the need for counselor educators to better address Whiteness and White supremacy, as well as to center the experiences of students from marginalized communities. Counselor educators may be able to mobilize and direct White students’ feelings of guilt into racial consciousness and action by helping them explore Whiteness, White privilege, and what it means to them while allowing and confronting feelings that arise (Grzanka et al., 2019). It may be helpful for educators to read and assign books on White fragility and ways to address it (DiAngelo, 2018; Helms, 2020; Saad, 2020), so that they can assist White students in managing these emotions. It is important that educators explicitly name and recognize White supremacy as it shows up in counseling theory and practice, and to include a shift from the primary focus on the individual to understanding and dismantling oppressive systems. Counselor educators must also attend to the ways in which they center the comfort of White students over the needs of BIPOC students, so that they do not perpetuate harm and trauma (Galán et al., 2021). Although students with privileged identities may learn powerful lessons about oppression from their classmates, it is important that such learning does not occur at the expense of students with marginalized identities. Offering spaces for White students, especially those who are new to conversations about race and racism, to process their feelings may be helpful to avoid harm to BIPOC students who have experienced racial trauma. Similarly, BIPOC students may benefit from spaces in which they can talk freely and support each other as they unpack their own experiences of microaggressions and trauma (Galán et al., 2021).

Based on the finding that support from faculty was important in facilitating student growth and learning, counselor educators may benefit from implementing strategies informed by relational pedagogy and relational–cultural theory (Dorn-Medeiros et al., 2020). Relational pedagogy centers the relationship between teachers and students and posits that all learning takes place in relationships. Relational–cultural theory emphasizes mutual empathy and empowerment and is rooted in feminist multicultural principles. Practices grounded in these approaches include professors’ use of self-disclosure to model openness, vulnerability, and self-reflection; and their work to reduce power imbalances and invite student feedback at multiple points in time through anonymous surveys and one-on-one meetings. Counselor educators can uplift students as the experts of their experience (Sharma & Hipolito-Delgado, 2021).

Limitations and Future Research
     The results of this study must be considered in light of a number of limitations. The use of the online survey meant that we were not able to follow up with students for further discussion or clarification of their responses. Adding focus groups or interviews to this methodology would likely provide a more thorough picture. In spite of assurances to the contrary, some students may have been hesitant to be honest out of concern that their own professors would be reading their feedback. It is possible that different themes would have emerged if all students had participated. In addition, 12% of the advanced students had participated as beginning students and therefore were previously exposed to the survey materials. Although this could have impacted their later responses, we suspect that given the nearly 2-year time lapse this may not have been meaningful. Nevertheless, future research and program evaluation would be strengthened with longitudinal analyses. Lastly, the reliability for the ACSA was relatively low, so conclusions are tentative; however, the results support the qualitative data. Despite these limitations, this study offers a model for assessing students’ learning and experiences with the goal of program improvement. The process of counselor educators humbling themselves and inviting and integrating student feedback is an important step in decolonizing counselor education and better serving students and the clients and communities that they will serve.

 

Conflict of Interest and Funding Disclosure
The authors reported no conflict of interest
or funding contributions for the development
of this manuscript.

 

References

American Counseling Association. (2014). ACA code of ethics. https://www.counseling.org/docs/default-source/default-document-library/ethics/2014-aca-code-of-ethics.pdf?sfvrsn=55ab73d0_1

Arney, E. N., Lee, S. Y., Printz, D. M. B., Stewart, C. E., & Shuttleworth, S. P. (2019). Strategies for increased racial diversity and inclusion in graduate psychology programs. In M. T. Williams, D. C. Rosen, & J. W. Kanter (Eds.), Eliminating race-based mental health disparities: Promoting equity and culturally responsive care across settings (pp. 227–242). Context Press.

Arredondo, P., D’Andrea, M., & Lee, C. (2020). Unmasking White supremacy and racism in the counseling profession. Counseling Today. https://ctarchive.counseling.org/2020/09/unmasking-white-supremacy-and-racism-in-the-counseling-profession

Beer, A. M., Spanierman, L. B., Greene, J. C., & Todd, N. R. (2012). Counseling psychology trainees’ perceptions of training and commitments to social justice. Journal of Counseling Psychology, 59(1), 120–133. https://doi.org/10.1037/a0026325

Berila, B. (2016). Integrating mindfulness into anti-oppression pedagogy: Social justice in higher education. Routledge.

Bvunzawabaya, B. (2012). Social justice counseling: Establishing psychometric properties for the Advocacy Competencies Self-Assessment Survey (Doctoral dissertation, Auburn University). https://etd.auburn.edu/handle/10415/3369

Chen, G. A., Neville, H. A., Lewis, J. A., Adames, H. Y., Chavez-Dueñas, N. Y., Mosley, D. V., & French, B. H. (2019). Radical self-care in the face of mounting racial stress: Cultivating hope through acts of affirmation. Psychology Today. https://www.psychologytoday.com/us/blog/healing-through-social-justice/201911/radical-self-care-in-the-face-mounting-racial-stress

Collins, S., Arthur, N., Brown, C., & Kennedy, B. (2015). Student perspectives: Graduate education facilitation of multicultural counseling and social justice competency. Training and Education in Professional Psychology, 9(2), 153–160. https://doi.org/10.1037/tep0000070

Council for the Accreditation of Counseling and Related Educational Programs. (2023). 2024 CACREP standards.
https://www.cacrep.org/wp-content/uploads/2023/06/2024-Standards-Combined-Version-6.27.23.pdf

Creswell, J. W., Plano Clark, V. L., Gutmann, M. L., & Hanson, W. E. (2003). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), SAGE handbook of mixed methods in social and behavioral research (1st ed.; pp. 209–240). SAGE.

Cross, W. E., Jr., & Reinhardt, J. S. (2017). Whiteness and serendipity. The Counseling Psychologist, 45(5), 697–705. https://doi.org/10.1177/0011000017719551

DiAngelo, R. (2018). White fragility: Why it’s so hard for White people to talk about racism. Beacon Press.

Dorn-Medeiros, C. M., Christensen, J. K., Lértora, I. M., & Croffie, A. L. (2020). Relational strategies for teaching multicultural courses in counselor education. Journal of Multicultural Counseling and Development, 48(3), 149–160. https://doi.org/10.1002/jmcd.12174

Field, T. A., Ghoston, M. R., Grimes, T. O., Sturm, D. C., Kaur, M., Aninditya, A., & Toomey, M. (2019). Trainee counselor development of social justice counseling competencies. Journal for Social Action in Counseling and Psychology, 11(1), 33–50. https://doi.org/10.33043/JSACP.11.1.33-50

Galán, C. A., Bekele, B., Boness, C., Bowdring, M., Call, C., Hails, K., McPhee, J., Mendes, S. H., Moses, J., Northrup, J., Rupert, P., Savell, S., Sequeira, S., Tervo-Clemmens, B., Tung, I., Vanwoerden, S., Womack, S., & Yilmaz, B. (2021). Editorial: A call to action for an antiracist clinical science. Journal of Clinical Child and Adolescent Psychology50(1), 12–57. https://doi.org/10.1080/15374416.2020.1860066

Goodman, L. A., Wilson, J. M., Helms, J. E., Greenstein, N., & Medzhitova, J. (2018). Becoming an advocate: Processes and outcomes of a relationship-centered advocacy training model. The Counseling Psychologist, 46(2), 122–153. https://doi.org/10.1177/0011000018757168

Goodman, R. D., Williams, J. M., Chung, R. C.-Y., Talleyrand, R. M., Douglass, A. M., McMahon, H. G., & Bemak, F. (2015). Decolonizing traditional pedagogies and practices in counseling and psychology education: A move towards social justice and action. In R. D. Goodman & P. C. Gorski (Eds.), Decolonizing “multicultural” counseling through social justice (pp. 147–164). Springer.

Gorski, P. C. (2019). Fighting racism, battling burnout: Causes of activist burnout in US racial justice activists. Ethnic and Racial Studies, 42(5), 667–687. https://doi.org/10.1080/01419870.2018.1439981

Grzanka, P. R., Gonzalez, K. A., & Spanierman, L. B. (2019). White supremacy and counseling psychology: A critical–conceptual framework. The Counseling Psychologist, 47(4), 478–529. https://doi.org/10.1177/0011000019880843

Haskins, N. H., & Singh, A. (2015). Critical race theory and counselor education pedagogy: Creating equitable training. Counselor Education and Supervision, 54(4), 288–301. https://doi.org/10.1002/ceas.12027

Helms, J. E. (2020). A race is a nice thing to have: A guide to being a White person or understanding the White persons in your life (3rd ed.). Cognella.

Hill, C. E., Knox, S., Thompson, B. J., Williams, E. N., Hess, S. A., & Ladany, N. (2005). Consensual qualitative research: An update. Journal of Counseling Psychology, 52(2), 196–205. https://doi.org/10.1037/0022-0167.52.2.196

Holcomb-McCoy, C. C., & Myers, J. E. (1999). Multicultural competence and counselor training: A national survey. Journal of Counseling & Development, 77(3), 294–302. https://doi.org/10.1002/j.1556-6676.1999.tb02452.x

Hoover, S. M., & Morrow, S. L. (2016). A qualitative study of feminist multicultural trainees’ social justice development. Journal of Counseling & Development, 94(3), 306–318. https://doi.org/10.1002/jcad.12087

Keum, B. T., & Miller, M. J. (2020). Social justice interdependence among students in counseling psychology training programs: Group actor-partner interdependence model of social justice attitudes, training program norms, advocacy intentions, and peer relationships. Journal of Counseling Psychology, 67(2), 141–155. https://doi.org/10.1037/cou0000390

Kozan, S., & Blustein, D. L. (2018). Implementing social change: A qualitative analysis of counseling psychologists’ engagement in advocacy. The Counseling Psychologist, 46(2), 154–189. https://doi.org/10.1177/0011000018756882

Mull, R. C. (2020). Colleges are in for a racial reckoning. The Chronicle of Higher Education. https://www.chronicle.com/article/colleges-are-in-for-a-racial-reckoning

Okun, T. (2021). White supremacy culture – Still here. https://drive.google.com/file/d/1XR_7M_9qa64zZ00_JyFVTAjmjVU-uSz8/view

Ponterotto, J. G., Mathew, J. T., & Raughley, B. (2013). The value of mixed methods designs to social justice research in counseling and psychology. Journal for Social Action in Counseling and Psychology, 5(2), 42–68. https://doi.org/10.33043/JSACP.5.2.42-68

Ratts, M. J., & Ford, A. (2010). Advocacy Competencies Self-Assessment (ACSA) Survey©: A tool for measuring advocacy competence. In M. J. Ratts, R. L. Toporek, & J. A. Lewis (Eds.), ACA advocacy competencies: A social justice framework for counselors (pp. 21–26). American Counseling Association.

Ratts, M. J., Singh, A. A., Nassar-McMillan, S., Butler, S. K., & McCullough, J. R. (2016). Multicultural and social justice counseling competencies: Guidelines for the counseling profession. Journal of Multicultural Counseling and Development, 44(1), 28–48. https://doi.org/10.1002/jmcd.12035

Royse, D., Thyer, B. A., & Padgett, D. K. (2010). Program evaluation: An introduction (5th ed.). Cengage.

Saad, L. F. (2020). Me and White supremacy: Combat racism, change the world, and become a good ancestor. Sourcebooks.

Sanabria, S., & DeLorenzi, L. (2019). Social justice pre-practicum: Enhancing social justice identity through experiential learning. Journal for Social Action in Counseling and Psychology, 11(2), 35–53. https://doi.org/10.33043/JSACP.11.2.35-53

Seward, D. X. (2019). Multicultural training resistances: Critical incidents for students of color. Counselor Education and Supervision, 58(1), 33–48. https://doi.org/10.1002/ceas.12122

Sharma, J., & Hipolito-Delgado, C. P. (2021). Promoting anti-racism and critical consciousness through a critical counseling theories course. Teaching and Supervision in Counseling, 3(2), 15–25. https://doi.org/10.7290/tsc030203

Singh, A. A. (2019). The racial healing handbook: Practical activities to help you challenge privilege, confront systemic racism, and engage in collective healing. New Harbinger.

Singh, A. A., Appling, B., & Trepal, H. (2020). Using the multicultural and social justice counseling competencies to decolonize counseling practice: The important roles of theory, power, and action. Journal of Counseling & Development, 98(3), 261–271. https://doi.org/10.1002/jcad.12321

Singh, A. A., Hofsess, C. D., Boyer, E. M., Kwong, A., Lau, A. S. M., McLain, M., & Haggins, K. L. (2010). Social justice and counseling psychology: Listening to the voices of doctoral trainees. The Counseling Psychologist, 38(6), 766–795. https://doi.org/10.1177/0011000010362559

Spangler, P. T., Liu, J., & Hill, C. E. (2012). Consensual qualitative research for simple qualitative data: An introduction to CQR-M. In C. E. Hill (Ed.), Consensual qualitative research: A practical resource for investigating social science phenomena (pp. 269–283). American Psychological Association.

Toporek, R. L., & Ahluwalia, M. K. (2021). Taking action: Creating social change through strength, solidarity, strategy, and sustainability. Cognella Press.

Turner, D. (2018). “You shall not replace us!” White supremacy, psychotherapy and decolonisation. Journal of Critical Psychology, Counselling and Psychotherapy, 18(1), 1–12. https://cris.brighton.ac.uk/ws/portalfiles/portal/495044/JCPCP+18-1+-+Turner+article.pdf

Zetzer, H. A. (2021). Decolonizing the curriculum in health service psychology. Association of Psychology Training Clinics Bulletin: Practicum Education & Training, 20–22. https://aptc.org/images/File/newsletter/APTC_Bulletin_PET_2021_Spring%20FINAL.pdf

 

Sapna B. Chopra, PhD, is an associate professor at California State University, Fullerton. Rebekah Smart, PhD, is a professor at California State University, Fullerton. Yuying Tsong, PhD, is a professor and Associate Vice President for Student Academic Support at California State University, Fullerton. Olga L. Mejía, PhD, is a licensed psychologist and an associate professor at California State University, Fullerton. Eric W. Price, PhD, is an associate professor at California State University, Fullerton. Correspondence may be addressed to Sapna B. Chopra, Department of Human Services, California State University, Fullerton, P.O. Box 6868, Fullerton, CA 92834-6868, sapnachopra@fullerton.edu.

Development of a Logic Model to Guide Evaluations of the ASCA National Model for School Counseling Programs

Ian Martin, John Carey

A logic model was developed based on an analysis of the 2012 American School Counselor Association (ASCA) National Model in order to provide direction for program evaluation initiatives. The logic model identified three outcomes (increased student achievement/gap reduction, increased school counseling program resources, and systemic change and school improvement), seven outputs (student change, parent involvement, teacher competence, school policies and processes, competence of the school counselors, improvements in the school counseling program, and administrator support), six major clusters of activities (direct services, indirect services, school counselor personnel evaluation, program management processes, program evaluation processes and program advocacy) and two inputs (foundational elements and program resources). The identification of these logic model components and linkages among these components was used to identify a number of necessary and important evaluation studies of the ASCA National Model.

 

Keywords: ASCA National Model, school counseling, logic model, program evaluation, evaluation studies

 

 

Since its initial publication in 2003, The ASCA National Model: A Framework for School Counseling Programs has had a dramatic impact on the practice of school counseling (American School Counselor Association [ASCA], 2003). Many states have revised their model of school counseling to make it consistent with this model (Martin, Carey, & DeCoster, 2009), and many schools across the country have implemented 3the ASCA National Model. The ASCA Web site, for example, currently lists over 400 schools from 33 states that have won a Recognized ASCA Model Program (RAMP) award since 2003 as recognition for exemplary implementation of the model (ASCA, 2013).

 

While the ASCA National Model has had a profound impact on the practice of school counseling, very few studies have been published that evaluate the model itself. Evaluation is necessary to determine if the implementation of the model results in the model’s anticipated benefits and to determine how the model can be improved. The key studies typically cited (see ASCA, 2005) as supporting the effectiveness of the ASCA National Model (e.g., Lapan, Gysbers, & Petroski, 2001; Lapan, Gysbers, & Sun, 1997) were actually conducted before the model was developed and were designed as evaluations of Comprehensive Developmental Guidance, which is an important precursor and component of the ASCA National Model, but not the model itself.

 

Two recent statewide evaluations of school counseling programs focused on the relationships between the level of implementation of the ASCA National Model and student outcomes. In a statewide evaluation of school counseling programs in Nebraska, Carey, Harrington, Martin, and Hoffman (2012) found that the extent to which a school counseling program had a well-implemented, differentiated delivery system consistent with practices advocated by the ASCA National Model was associated with lower suspension rates, lower discipline incident rates, higher attendance rates, higher math proficiency and higher reading proficiency. These results suggest that model implementation is associated with increased student engagement, fewer disciplinary problems and higher student achievement. In a similar statewide evaluation study in Utah, Carey, Harrington, Martin, and Stevens (2012) found that the extent to which the school counseling program had a programmatic orientation, similar to that advocated in the ASCA National Model, was associated with both higher average ACT scores and a higher number of students taking the ACT. This suggests that model implementation is associated with both increased achievement and a broadening of student interest in college. While these studies suggest that benefits to students are associated with the implementation of the ASCA National Model, additional evaluations are necessary that use stronger (e.g., quasi-experimental and longitudinal) designs and investigate specific components of the model in order to determine their effectiveness or how they can be improved.

 

There are several possible reasons why the ASCA National Model has not been evaluated extensively. The school counseling field as a whole has struggled with general evaluation issues. For example, questions have been raised regarding the effectiveness of practitioner training in evaluation (Astramovich, Coker, & Hoskins, 2005; Heppner, Kivlighan, & Wampold, 1999; Sexton, Whiston, Bleuer, & Walz, 1997; Trevisan, 2000); practitioners have cited lack of time, evaluation resources and administrative support as major barriers to evaluation (Loesch, 2001; Lusky & Hayes, 2001); and some practitioners have feared that poor evaluation results may negatively impact their program credibility (Isaacs, 2003; Schmidt, 1995). Another contributing factor is that while the importance of evaluation is stressed in the literature, few actual examples of program evaluations and program evaluation results have been published (Astramovich & Coker, 2007; Martin & Carey, 2012; Martin et al., 2009; Trevisan, 2002).

 

In addition, there are several features of the ASCA National Model that make evaluations difficult. First, the model is complex, containing many components grouped into four interrelated, functional subsystems referred to as the foundation, delivery system, management system and accountability system. Second, ASCA created the National Model by combining elements of existing models that were developed by different individuals and groups. For example, the principle influences of the model (ASCA, 2012) are cited as Gysbers and Henderson (2000), Johnson and Johnson (2001) and Myrick (2003). Furthermore, principles and concepts derived from important movements such as the Transforming School Counseling Initiative (Martin, 2002) and evidence-based school counseling (Dimmitt, Carey, & Hatch, 2007) also were incorporated into the model during its development. While these preexisting models and movements share some common features, they differ in important ways. Elements of these approaches were combined and incorporated into the ASCA National Model without a full integration of their philosophical and theoretical perspectives and principles. Consequently, the ASCA National Model does not reflect a single cohesive approach to program organization and management. Instead, it reflects a collection of presumably effective principles and practices that have been applied in school counseling programs. Third, instruments for measuring important aspects of model implementation are lacking (Clemens, Carey, & Harrington, 2010). Fourth, the theory of action of the ASCA National Model has not been fully explicated, so it is difficult to determine what specific benefits are intended to result from the implementation of specific elements of the model. For example, it is not entirely clear how changing the performance evaluation of counselors is related to the desired benefits of the model.

 

In this article, the authors present the results of their work in developing a logic model for the ASCA National Model. Logic modeling is a systematic approach to enabling high-quality program evaluation through processes designed to result in pictorial representations of the theory of action of a program (Frechtling, 2007). Logic modeling surfaces and summarizes the explicit and implicit logic of how a program operates to produce its desired benefits and results. By applying logic modeling to an analysis of the ASCA National Model, the authors intended to fully explicate the relationships between structures and activities advocated by the model and their anticipated benefits so that these relationships can be tested in future evaluations of the model.

 

The purpose of this study, therefore, was to develop a useful logic model that describes the workings of the ASCA National Model in order to promote its evaluation. More specifically, the purpose was to mine the logic elements, program outcomes and implicit (unstated) assumptions about the relationships between program elements and outcomes. In developing this logic model, the authors followed the processes suggested by the W. K. Kellogg Foundation (2004) and Frechtling (2007). Several different frameworks exist for logic models, but the authors elected to use Frechtling’s framework because it focuses specifically on promoting evaluation of an existing program (as opposed to other possible uses such as program planning). This framework identifies the relationships among program inputs, activities, outputs and outcomes. Inputs refer to the resources needed to deliver the program as intended. Activities refer to the actual program components that are expected to be related to a desired outcome. Outputs refer to the immediate products or results of activities that can be observed as evidence that the activity was actually completed. Outcomes refer to the desired benefits of the program that are expected to occur as a consequence of program activities. The authors’ logic model development was guided by four questions:

 

What are the essential desired outcomes of the ASCA National Model?

What are the essential activities of the ASCA National Model and how do these activities relate to its outputs?

What are the essential outputs of the ASCA National Model and how do these outputs relate to its desired outcomes?

What are the essential inputs of the ASCA National Model and how do these inputs relate to its activities?

 

Methods

All analyses in this study were based on the latest edition of the ASCA National Model (ASCA, 2012). In these analyses, every attempt was made to base inferences on the actual language of the model. In some instances (for example, when it was unclear which outputs were expected to be related to a given activity) the professional literature about the ASCA National Model was consulted.

Because the authors intended to develop a logic model from an existing program blueprint (rather than designing a new program), they began, according to recommended procedures (W. K. Kellogg Foundation, 2004), by first identifying outcomes and then working backward to identify activities, then outputs associated with activities and finally, inputs.

 

Identification of Outcomes

The authors independently reviewed the ASCA National Model (2012) and identified all elements in the model. The two authors’ lists of elements (e.g., vision statement, annual agreement with school leaders, indirect service delivery and curriculum results reports) were merged to create a common list of elements. The authors then independently created a series of if, then statements for each element of the model that traced the logical connections explicitly stated in the model (or in rare instances, stated in the professional literature about the model) between the element and a program outcome. In this way, both the desired outcomes of the ASCA National Model and the desired logical linkages between elements and outcomes were identified.

 

During this process, some ASCA National Model elements were included in the same logic sequence because they were causally related to each other. For example, both the vision statement and the mission statement were included in the same logic sequence because a strong vision statement was described as a necessary prerequisite for the development of a strong mission statement. Some ASCA National Model elements also were included in more than one logical sequence when it was clear that two different outcomes were intended to occur related to the same element. For example, it was evident that closing-the-gap reports were intended to result in intervention improvements, leading to better student outcomes and also to apprising key stakeholders of school counseling program results, in order to increase support and resources for the program.

 

Identification of Activities

Frechtling (2007) noted that the choice of the right amount of complexity in portraying the activities in a logic model is a critically important factor in a model’s utility. If activities are portrayed in their most differentiated form, the model can be too complex to be useful. If activities are portrayed in their most compact form, the model can lack enough detail to guide evaluation. Therefore, in the present study, the authors decided to construct several different logic models with different sets of activities that ranged from including all the previously identified ASCA National Model elements as activities to including only the four sections of the ASCA National Model (i.e., foundation, management system, delivery system and accountability system) as activities. As neither of the two extreme options proved to be feasible, the authors began clustering ASCA National Model elements and developed six activities, each of which represented a cluster of program elements.

 

Identification of Outputs Related to Activities

Outputs are the observable immediate products or deliverables of the logic model’s inputs and activities (Frechtling, 2007). After the authors identified an appropriate level for representing model activities, they generated the same level of program outputs. Reexamining the logic sequences, clustering products of identified activities and then creating general output categories from the clustered products accomplished this task. For example, the activity known as direct services contained several ASCA National Model products, such as the curriculum results report, the small-group results report and the closing-the-gap results report (among others), and the resulting output was finally categorized as student change. Ultimately, seven logic model outputs were identified through this process to help describe the outputs created by ASCA National Model activities.

 

Identifying the Connections Between Outputs and Outcomes

Creating connections between model outputs and outcomes was accomplished by linking the original logic sequences to determine how the ASCA National Model would conceive of outputs as being linked to outcomes. Returning to the above example, the output known as student change, which included such products as results reports, was connected to the outcome known as student achievement and gap reduction in several logic sequences. At the conclusion of this process, each output had straightforward links to one or multiple proposed model outcomes. Not only was this process useful in identifying links between outputs and outcomes, but it also functioned as an opportunity to test the output categories for conceptual clarity.

 

Identification of Inputs and Connections Between Inputs and Activities

The authors reviewed the ASCA National Model to determine which inputs were necessary to include in the logic model. They identified two essential types of inputs: foundational elements (conceptual underpinnings described in the foundation section of the ASCA National Model) and program resources (described throughout the ASCA National Model). The authors determined that these two types of inputs were necessary for the effective operation of all six activities.

 

Identifying Other Connections Within the Logic Model

     After the inputs, activities, outputs, outcomes and the connections between these levels were mapped, the authors again reviewed the logic sequences and the ASCA National Model to determine if any additional linkages needed to be included in the logic model (see Frechtling, 2007). They evaluated the need for within-level linkages (e.g., between two activities) and feedback loops (i.e., where a subsequent component influences the nature of preceding components). The authors determined that two within-level and one recursive linkage were needed.

 

Results

 

Outcomes

A total of 65 logic sequences were identified for the ASCA National Model sections: foundation (n = 7), management system (n = 30), delivery system (n = 7) and accountability system (n = 21). Table 1 contains sample logic sequences.

 

Table 1

 

Examples of Logic Sequences Relating ASCA National Model Elements to Outcomes

 

National Model

Section

Logic Sequence

Foundation a. If counselors go through the process of creating a set of shared beliefs, then they will establish a level of mutual understanding.b. If counselors establish a level of mutual understanding, then they will be more successful in developing a shared vision for the program.c. If counselors develop a shared vision for the program, then they can develop an effective vision statement.d. If counselors create a vision statement, then they will have the clarity of purpose that is needed to develop a mission statement.e. If counselors create a mission statement, then the program will be more focused.f. If the program is better focused, counselors will create a set of program goals, which will enable counselors to specify how the attainment of the goals should be measured.

g. If counselors specify how the attainment of goals should be measured, then effective program evaluation will be conducted.

h. If effective program evaluation is conducted, then the program will be continuously improved.

i. If the program will be continuously improved, then improved student achievement will result.

Management System a. If school counselors create annual agreements with the leader in charge of the school, then the goals and activities of the counseling program will be more aligned with the goals of the school.b. If the goals and activities of the counseling program are more aligned with the goals of the school, then school leaders will recognize the value of the school counseling program.c. If school leaders recognize the value of the school counseling program, then they will commit resources to support the program.
Delivery System a. If school counselors engage in indirect services (e.g., consultation and advocacy), then school policies and processes will improve.b. If school policies and processes improve, then teachers will develop more competency, and systemic change and school improvement will occur.
Accountability System a. If counselors complete curriculum results reports, then they will have the information they need to demonstrate the effectiveness of developmental and preventative curricular activities.b. If counselors have the information they need to demonstrate the effectiveness of developmental and preventative curricular activities, then they can communicate their impact to school leaders.c. If school leaders are aware of the impact of developmental and preventative curricular activities, then they will recognize their value.d. If school leaders recognize the value of developmental and preventative curricular activities, then they will commit resources to support them.

 

 

 

Forty of these logic sequences terminated with an outcome related to increased student achievement or (relatedly) to a reduction in the achievement gap. Twenty-two sequences terminated with an outcome related to an increase in program resources. Only three sequences terminated with an outcome related to systemic change in the school. From this analysis, the authors concluded that the primary desired outcomes of the ASCA National Model are increased student achievement/gap reduction and increased school counseling program resources. They also concluded that systemic change and school improvement is another desired outcome of the ASCA National Model.

 

Activities

Based on a clustering of ASCA National Model elements identified previously, six activities were developed for the logic model. These activities included the following: direct services, indirect services, school counselor personnel evaluation, program management processes, program evaluation processes and program advocacy processes. Each of these activities represents a cluster of elements within the ASCA National Model. For example, the activity known as direct services includes the school counseling core curriculum, individual student planning and responsive services. Consequently, the direct services activity represents the spectrum of services that would be delivered to students in an ASCA National Model school counseling program.

 

Activities Related to Outputs

Based on the clustering of the ASCA National Model products or deliverables around the related logic model activities, seven outputs were identified. These outputs included the following: student change, parent involvement, teacher competence, school policies and processes, school counselor competence, school counseling program improvements, and administrator support. The outputs represent all of the ASCA National Model products generated by model activities and help to collect evidence and determine to what degree an activity was successfully accomplished. In essence, for evaluation purposes, these outputs represent the intermediate outcomes (Dimmitt et al., 2007) of an ASCA National Model program. Activities should result in measurable changes in outputs, which in turn should result in measurable changes in outcomes. For example, the output known as student change reflects student changes such as increased academic motivation, increased problem-solving skills, enhanced emotional regulation and better interpersonal problem-solving skills; these changes lead to the longer-term outcome of student achievement and gap reduction.

 

Connections Between Outputs and Outcomes

Connecting the seven ASCA National Model outputs to its outcomes strengthens the logic model by identifying the hypothesized relationships between the more immediate changes that result from school counseling program activities (i.e., outputs) and the more distal changes that result from the operation of the program (i.e., outcomes). As described earlier, two primary outcomes (student achievement and gap reduction and increased program resources) and one secondary outcome (systemic change and school improvement) were identified within the ASCA National Model. Three of the seven outputs (student change, parent involvement and administrator support) were connected to only one outcome. Three other outputs (teacher competence, school policies and processes, and school counselor competence) were connected to two outcomes. One output (administrator support) was connected to all three outcomes. Interpreting these linkages is useful in understanding the implicit theory of change of the ASCA National Model and consequently in designing appropriate evaluation studies. The authors’ logic model, for example, indicates that student changes (related to both direct and indirect services of an ASCA National Model program) are expected to result in measurable increases in student achievement and a reduction in the achievement gap.

 

It also is helpful to scan backward in the logic model to identify how changes in outcomes are expected to occur. For example, student achievement and gap reduction is linked to six model outputs (student change, parent involvement, teacher competence, school policies and processes, school counselor competence, and school counseling program improvements). Student achievement and gap reduction is multiply determined and is the major focus of the ASCA National Model. Increased program resources are connected to three model outputs (school counselor competence, school counseling program improvements and administrator support). Systemic change and school improvement also can be connected to three outputs (teacher competence, school policies and processes, and school counseling program improvements).

 

Inputs and Connections Between Inputs and Activities

Based on an analysis of the ASCA National Model, two inputs were identified for inclusion in the logic model: foundational elements (which include the elements in the ASCA National Model’s foundation section considered important for program planning and operation) and program resources (which include elements essential for effective program implementation such as counselor caseload, counselor expertise, counselor professional development support, counselor time-use and program budget). Both of these inputs were identified as being important in the delivery of all six activities.

 

Additional Connections Within the Logic Model

Based on a final review of the logical sequences and another review of the ASCA National Model, three additional linkages were added to the authors’ logic model. The first linkage was a unidirectional arrow leading from management processes to program evaluation in the activities column. This arrow was intended to represent the tight connection between management processes and evaluation activities that is evident in the ASCA National Model. Relatedly, a unidirectional arrow leading from the school counseling program evaluation activity to the program advocacy activity was added. This arrow was intended to represent the many instances of the ASCA National Model suggesting that program evaluation activities should be used to generate essential information for program advocacy. The final additional link was a recursive arrow leading from the increased program resources outcome to the program resources input. This linkage was intended to represent the ASCA National Model’s concept that investment of additional resources resulting from successful implementation and operation of an ASCA National Model program will result in even higher levels of program effectiveness and eventually even better outcomes.

 

The Logic Model

Figure 1 contains the final logic model for the ASCA National Model for School Counseling Programs. Logic models portray the implicit theory of change underlying a program and consequently facilitate the evaluation of the program (Frechtling, 2007). Overall, the theory of change for an ASCA national program could be described as follows: If school counselors use the foundational elements of the ASCA National Model and have sufficient program resources, they will be able to develop and implement a comprehensive program characterized by activities related to direct services, indirect services, school counselor personnel evaluation, management processes, program evaluation and (relatedly) program advocacy. If these activities are put in place, several outputs will be observed, including the following: student changes in academic behavior, increased parent involvement, increases in teacher competence in working with students, better school policies and processes, increased competence of the school counselors themselves, demonstrable improvements in the school counseling program, and increased administrator support for the school counseling program. If these outputs occur, then the following outcomes should result: increased student achievement and a related reduction in the achievement gap, notable systemic improvement in the school in which the program is being implemented, and increased program support and resources. If these additional resources are reinvested in the school counseling program, the effectiveness of the program will increase.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 1. Logic Model for ASCA National Model for School Counseling Programs

 

 

Discussion

 

Logic models can be used for a number of purposes including the following: enhancing communication among program team members, managing the program, documenting how the program is intended to operate and developing an approach to evaluation and related evaluation questions (Frechtling, 2007). The present study was conducted in order to develop a logic model for ASCA National Model programs so that these programs could be more readily evaluated, and based on the results of these evaluations, the ASCA National Model could then be improved.

 

Evaluations can focus on the question of whether or not a program or components of a program actually result in intended changes. At the most global level, an evaluation can focus on discovering the extent to which the program as a whole achieves its desired outcomes. At a more detailed level, an evaluation can focus on discovering the extent to which the components (i.e., activities) of the program achieve their desired outputs (with the assumption that achievement of the outputs is a necessary precursor to achievement of the outcomes).

 

In both types of evaluations, it is important to use a design that allows some form of comparison. In the simplest case, it would be possible to compare outputs and outcomes before and after implementation of the ASCA National Model. In more complex cases, it would be possible to compare outputs and outcomes of programs that have implemented the ASCA National Model with programs that have not. In these cases, it is essential to control for the confounding effects of extraneous variables (e.g., the affluence of students in the school) by the use of matching or covariates. If the level of implementation of the ASCA National Model program as a whole can be measured, it is even possible to use multivariate correlation approaches to examine whether the level of implementation of the program is related to desired outcomes while simultaneously controlling statistically for potential confounding variables. These same correlational procedures can be used to examine the relationships between the more discrete activities of the program and their corresponding outputs.

 

At the most global level, it is important to evaluate the extent to which the implementation of the ASCA National Model results in the following: increases in student achievement (and associated reductions in the achievement gap), measurable systemic change and school improvements, and increases in resources for the school counseling program. At present, there is some evidence that implementation of the ASCA National Model is related to achievement gains (Carey, Harrington, Martin, & Hoffman, 2012; Carey, Harrington, Martin, & Stevens, 2012). No evaluations to date have examined whether ASCA National Model implementation results in systemic change and school improvement or in an increase in program resources.

 

It also is important to evaluate the extent to which specific program activities achieve their desired outputs. Table 2 contains a list of sample evaluation questions for each activity. Within these questions, evaluation is focused on whether or not components of the program result in overall benefits. No evaluation study to date has evaluated the impact of ASCA National Model implementation on these factors.

 

Table 2

 

Sample Evaluation Questions for ASCA National Model Activities

 

Activities

Evaluation Questions

Direct Services Does organizing and delivering school counseling direct services in accordance with ASCA National Model principles result in an increase in important aspects of students’ school behavior that are related to academic achievement?
Indirect Services Does organizing and delivering school counseling indirect services in accordance with ASCA National Model principles result in an increase in parent involvement?
Does organizing and delivering school counseling indirect services in accordance with ASCA National Model principles result in an increase in teachers’ abilities to work effectively with students?
Does organizing and delivering school counseling indirect services in accordance with ASCA National Model principles result in improvements in school policies and procedures that support student achievement?
School Counselor Personnel Evaluation Does the implementation of personnel and processes recommended by the ASCA National Model result in increases in the professional competence of school counselors?
Management Processes Does the implementation of the management processes recommended by the ASCA National Model result in demonstrable improvements in the school counseling program?
Program Evaluation Does the implementation of program evaluation processes recommended by the ASCA National Model result in demonstrable improvements in the school counseling program?
Program Advocacy Does the implementation of the program advocacy practices recommended by the ASCA National Model result in increases in administrator support for the program?

 

 

 

In addition to examining program-related change, it is important to evaluate whether a basic assumption of the ASCA National Model bears out in reality. The major assumption is that school counselors who use the foundational elements of the ASCA National Model (e.g., vision statement, mission statement) and have access to typical levels of program resources can develop and implement all the activities associated with an ASCA National Model program (e.g., direct services, indirect services, school counselor personnel evaluation, management processes, program evaluation and program advocacy). Qualitative evaluations of the relationships between inputs and quality of the activities are necessary to determine what levels of inputs are necessary for full implementation. While full evaluation studies of this type have yet to be undertaken, Martin and Carey (2012) have recently reported the results of a two-state qualitative comparison of how statewide capacity-building activities to promote school counselors’ competence in evaluation were used to promote the widespread implementation of ASCA National Model school counseling programs. More studies of this type that focus on the relationships between a broader range of program inputs and school counselors’ ability to fully implement ASCA National Model program activities are needed.

 

Limitations and Future Directions

 

Constructing a logic model retrospectively is inherently challenging and complex. This is especially true when the program for which the logic model is being created was not initially developed with reference to an explicit, coherent theory of action. In the present study, the authors approached the work systematically and are confident that others following similar procedures would generate similar results. With that said, a limitation of this work is that the logic model was created based on the authors’ analyses of the written description of the ASCA National Model (2012) and literature surrounding the ASCA National Model. Engaging individuals who were involved in the development and implementation of the ASCA National Model in dialogue might have resulted in a richer logic model with even more utility in directing evaluation of the ASCA National Model. As a follow-up to the present study, the authors intend to continue this inquiry by asking key individuals involved with the ASCA National Model to evaluate the present logic model and to suggest revisions and extensions. Even given this limitation, the current study has potential immediate implications for improving practice that go beyond its role in providing focus and direction for ASCA National Model evaluation.

 

A potentially fertile testing ground for the implementation of the logic model is present within the RAMP Award process. As aforementioned, RAMP awards are given to exemplary schools that have successfully implemented the ASCA National Model. Currently, schools provide evidence (data) and create narratives regarding how they have successfully met RAMP criteria. Twelve independent rubrics are scored and totaled to determine whether a school receives a RAMP Award. At least two contributions of the logic model for improving the RAMP process seem feasible. First, practitioners can use the logic model to help construct narratives that better articulate how ASCA National Model activities/outputs relate to model outcomes. Second, the logic model may also help improve the RAMP process by highlighting clearer links between activities, outputs and outcomes. In future revisions of the RAMP process, more attention could be paid to the documentation of benefits achieved by the program in terms of both outputs (i.e., the immediate measurable positive consequences of program activities) and outcomes (i.e., the longer-term positive consequences of program operation). In this vein, the authors hope that the logic model developed in this study will help to improve the RAMP process for both practitioners and RAMP evaluators.

 

Retrospective logic models map a program as it is. In that sense, they are very useful in directing the evaluation of existing programs. Prospective logic models are used to design new programs. Using logic models in program design (or redesign) has some distinct advantages. “Logic models help identify the factors that will impact your program and enable you to anticipate the data and resources you will need to achieve success” (W. K. Kellogg Foundation, 2004, p. 65). When programs are planned with the use of a logic model, greater opportunities exist to explore foundational theories of change, to explore issues or problems addressed by the program, to surface community needs and assets related to the program, to consider desired program results, to identify influential program factors (e.g., barriers or supports), to consider program strategies (e.g., best practices), and to elucidate program assumptions (e.g., the beliefs behind how and why the strategies will work; W. K. Kellogg Foundation, 2004). The authors hope that logic modeling will be incorporated prospectively into the next revision process of the ASCA National Model. Basing future editions of the ASCA National Model on a logic model that comprehensively describes its theory of action should result in a more elegant ASCA National Model with a clearer articulation between its components and its desired results. Such a model would be easier to articulate, implement and evaluate. The authors hope that the development of a retrospective logic model in the present study will facilitate the prospective use of a logic model in subsequent ASCA National Model revisions. The present logic model provides a map of the current state of the ASCA National Model. It is a good starting point for reconsidering such questions as how the model should operate, whether the outcomes are the right outcomes, whether the activities are sufficient and comprehensive enough to lead to the desired outcomes, and whether the available program resources are sufficient to support implementation of program activities.

 

 

 

Conflict of Interest and Funding Disclosure

The authors reported no conflict of

interest or funding contributions for

the development of this manuscript.

 

 

 

References

 

American School Counselor Association. (2003). The ASCA national model: A framework for school counseling programs. Alexandria, VA: Author.

American School Counselor Association. (2005). The ASCA national model: A framework for school counseling programs. (2nd ed.). Alexandria, VA: Author.

American School Counselor Association. (2012). The ASCA national model: A framework for school counseling programs (3rd ed.). Alexandria, VA: Author.

American School Counselor Association. (2013). Past RAMP Recipients. Retrieved from http://www.ascanationalmodel.org/learn-about-ramp/past-ramp-recipients

Astramovich, R. L., & Coker, J. K. (2007). Program evaluation: The accountability bridge model for counselors. Journal of Counseling & Development, 85, 162–172. doi:10.1002/j.1556-6678.2007.tb00459.x

Astramovich, R. L., Coker, J. K., & Hoskins, W. J. (2005). Training school counselors in program evaluation. Professional School Counseling, 9, 49–54.

Carey, J., Harrington, K., Martin, I., & Hoffman, D. (2012). A statewide evaluation of the outcomes of ASCA National Model school counseling programs in rural and suburban Nebraska high schools. Professional School Counseling, 16, 100–107.

Carey, J. C., Harrington, K., Martin, I., & Stevens, D. (2012). A statewide evaluation of the outcomes of the implementation of ASCA National Model school counseling programs in high schools in Utah. Professional School Counseling, 16, 89–99.

Clemens, E. V., Carey, J. C., & Harrington, K. M. (2010). The school counseling program implementation survey: Initial instrument development and exploratory factor analysis. Professional School Counseling, 14, 125–134.

Dimmitt, C., Carey, J. C., & Hatch, T. A. (2007). Evidence-based school counseling: Making a difference with data-driven practices. New York, NY: Corwin Press.

Frechtling, J. A. (2007). Logic modeling methods in program evaluation. New York, NY: Wiley & Sons.

Gysbers, N. C., & Henderson, P. (2000). Developing and managing your school guidance program (3rd ed.). Alexandria, VA: American Counseling Association.

Heppner, P. P., Kivlighan, D. M., Jr., & Wampold, B. E. (1999). Research design in counseling (2nd ed.). Belmont, CA: Wadsworth.

Isaacs, M. L. (2003). Data-driven decision making: The engine of accountability. Professional School Counseling, 6, 288–295.

Johnson, C. D., & Johnson, S. K. (2001). Results-based student support programs: Leadership academy workbook. San Juan Capistrano, CA: Professional Update.

Lapan, R. T., Gysbers, N. C., & Petroski, G. F. (2001). Helping seventh graders be safe and successful: A statewide study of the impact of comprehensive guidance and counseling programs. Journal of Counseling & Development, 79, 320–330. doi:10.1002/j.1556-6676.2001.tb01977.x

Lapan, R. T., Gysbers, N. C., & Sun, Y. (1997). The impact of more fully implemented guidance programs on the school experiences of high school students: A statewide evaluation study. Journal of Counseling & Development, 75, 292–302. doi:10.1002/j.1556-6676.1997.tb02344.x

Loesch, L. C. (2001). Counseling program evaluation: Inside and outside the box. In D. C. Locke, J. E. Myers, & E. L. Herr (Eds.), The handbook of counseling (pp. 513–525). Thousand Oaks, CA: Sage.

Lusky, M. B., & Hayes, R. L. (2001). Collaborative consultation and program evaluation. Journal of Counseling & Development, 79, 26–38. doi:10.1002/j.1556-6676.2001.tb01940.x

Martin, I., & Carey, J. C. (2012). Evaluation capacity within state-level school counseling programs: A cross-case analysis. Professional School Counseling, 15, 132–143.

Martin, I., Carey, J. C., & DeCoster, K. (2009). A national study of the current status of state school counseling models. Professional School Counseling, 12, 378–386.

Martin, P. J. (2002). Transforming school counseling: A national perspective. Theory Into Practice, 41, 148–153. doi:10.1207/s15430421tip4103_2

Myrick, R. D. (2003). Developmental guidance and counseling: A practical approach (4th ed.). Minneapolis, MN: Educational Media Corporation.

Schmidt, J. J. (1995). Assessing school counseling programs through external interviews. School Counselor, 43, 114–123.

Sexton, T. L., Whiston, S. C., Bleuer, J. C., & Walz, G. R. (1997). Integrating outcome research into counseling practice and training. Alexandria, VA: American Counseling Association.

Trevisan, M. S. (2000). The status of program evaluation expectations in state school counselor certification requirements. American Journal of Evaluation, 21, 81–94. doi:10.1177/109821400002100107

Trevisan, M. S. (2002). Evaluation capacity in K-12 school counseling programs. American Journal of Evaluation, 23, 291–305. doi:10.1016/S1098-2140(02)00207-2

W. K. Kellogg Foundation. (2004). Logic model development guide. Battle Creek, MI: Author.

 

Ian Martin is an assistant professor at the University of San Diego. John Carey is a professor at the University of Massachusetts, Amherst, and the Director of the Ronald H. Fredrickson Center for School Counseling Outcome Research and Evaluation. Correspondence can be addressed to: Ian Martin, 5998 Alcala Park, San Diego, CA 92110, imartin@sandiego.edu.