This article provides an overview of a grant project designed to create a district-wide elementary school counseling program with a strong data-based decision-making process. Project goals included building data literacy skills among school counselors and developing the infrastructure to efficiently collect important social-emotional indicators through a revised system for recording disciplinary infractions and a new research-based behavioral component for the district’s standards-based report cards. This enhanced system for accessing and analyzing social-emotional indicators resulted in broad systemic changes in the district, including extending a number of grant initiatives to the middle and high school levels, restructuring data teams to adopt a multi-tiered system of supports, and establishing school counselors as leaders in data-driven discussions about student success.
Keywords: school counseling, data-based decision making, multi-tiered system of supports, social–emotional, elementary school
This article reports on an Elementary and Secondary School Counseling Program (ESSCP) grant project designed to build an elementary school counseling program in a district that previously had not employed school counselors at that level. The new school counseling program was organized around an innovative shift in the district’s multi-tiered system of supports (MTSS) model that expanded to integrate social-emotional and behavioral data with academic indicators. School counselors used the new social-emotional data to help answer the question of why students were struggling academically when scholastic deficiencies were not the primary cause. The grant project also focused on developing strong data literacy skills among elementary school counselors so they could serve as leaders in data-based discussions. These complementary grant goals transformed the data team process as school counselors, teachers and administrators began to use data to better understand the complex relationship between social-emotional factors and academic achievement. These practices resulted in systemic changes throughout the district as data-driven elements of the elementary school counseling program were adopted at the secondary level. The purpose of this article is to: (a) highlight the importance of engaging in data-based decision making regarding students’ social-emotional needs in schools, (b) provide an overview of the specific elements that comprised the new MTSS model in the school district as a part of this grant-funded project, and (c) underscore the importance of building human capacity to enable school-based data teams to meaningfully integrate academic and social-emotional data to promote improved student outcomes. Limitations of this project, directions for future research and implications for school counselors also are discussed.
School Counselors and Social-Emotional Data
School counselors are often advised to adopt a data-based decision-making model as part of their practice (American School Counselor Association [ASCA], 2012; Dimmitt, Carey, & Hatch, 2007). Accountability mandates require school counselors to use data to demonstrate the impact of their work and to link their interventions to academic achievement (Dahir & Stone, 2009: Isaacs, 2003; Sink & Stroh, 2003.) Moreover, data use also is central to the transformed model of school counseling, which positions school counselors as advocates in educational reform efforts such as closing the achievement gap and carrying out school improvement initiatives (Dahir, 2004; Hayes, Nelson, Tabin, Pearson, & Worthy, 2002; House & Hayes, 2002). However, institutional factors can limit the role of the school counselor in data-based decision making. Typically, data teams primarily (or even exclusively) consider academic indicators, and schools often lack the infrastructure to systematically collect the social-emotional data that more directly aligns with the work of the school counselor.
Accountability requirements of the No Child Left Behind Act of 2001 (NCLB; 2002) have strongly influenced schools’ approaches to data-based decision making (Mandinach, Honey, & Light, 2006; Marsh, Pane, & Hamilton, 2006). The pressure to demonstrate adequate yearly progress (AYP) has prioritized state standardized tests scores and other academic benchmark assessments in data-driven discussions. A tremendous amount of achievement data were routinely collected and housed by school districts to fulfill reporting demands of NCLB; these data will continue to be gathered under the new Every Student Succeeds Act (ESSA; 2015). School staff can access these data to guide instructional practices and measure student progress. However, these data are more directly linked to teachers’ work with students and primarily measure academic achievement and cognitive ability (Heckman & Rubinstein, 2001).
The role of the school counselor encompasses not only students’ academic achievement but also their social-emotional development (ASCA, 2012). Social-emotional and behavioral data are typically not collected in the same robust manner as academic achievement data and are often limited to office discipline referrals and attendance rates. These behaviors are poor proxies of student engagement and reveal little information about underlying issues that need to be addressed. Measures of motivation, perseverance, self-regulation and other factors that impact students’ ability to achieve are not present in most school districts’ data collection systems, rendering them absent also from data-driven discussions about student outcomes.
In addition, while NCLB articulated which data are considered the critical measures of academic achievement, a corresponding set of social-emotional data has not been clearly delineated. Despite growing recognition of the impact of non-cognitive factors on student achievement (Farrington et al., 2012), educators are often uncertain about which specific behaviors, attitudes and dispositions link to success in school and throughout life. Educational organizations such as The Partnership for 21st Century Skills; Collaboration for Academic, Social, and Emotional Learning (CASEL); and ASCA (2014) have suggested promoting specific mindsets, college and career-readiness skills, and prosocial behaviors, but consensus is lacking about which social-emotional or non-cognitive factors are integral to students’ academic and social skill development.
The process of data-based decision making in schools has been shaped both by a prevailing belief concerning which data are important to examine and an existing infrastructure that constrains what data are routinely collected to those of a primarily academic nature. These factors also limit the role of the school counselor in data-based discussions about student achievement. With the end of the NCLB era and the ushering in of ESSA, all educators are being asked to address non-cognitive factors and be accountable for showing gains in these areas in addition to academic areas.
A construct-based approach to school counseling. Squier, Nailor, and Carey (2014) extensively reviewed the educational and developmental psychology literature to determine what capabilities are strongly related to students’ academic achievement and later success in life. The authors intentionally chose lines of research connected to student competencies in the academic, personal/social and career domains that comprise the school counseling ASCA (2012) National Model. Squier and colleagues (2014) established four overarching constructs that explicitly link to student success: (a) motivation, the forces that compel action and direct the behavior of individuals; (b) self-knowledge, the understanding that people have about their own abilities, values, preferences and skills and a necessary precondition for effective self-regulation; (c) self-direction, being able to identify one’s own life directions, to make academic choices consistent with these directions and to connect classroom learning to life goals; and (d) relationships, the ability to establish and maintain productive, collaborative, social relationships with teachers and peers. These four constructs have been shown to be strongly associated with students’ academic achievement and well-being; they also are considered to be malleable, receptive to intervention and within the range of expertise of school counselors (Bass, Lee, Wells, Carey, & Lee, 2015).
Multi-Tiered System of Supports
Use of MTSS is the recommended process for assessing and potentially intervening with an array of academic, behavioral and social-emotional issues while promoting schoolwide systems change (Lane, Menzies, Ennis, & Bezdek, 2013). An MTSS approach aligns closely with the ASCA (2012) National Standards and the work of school counselors in implementing prevention-based initiatives at a schoolwide level while providing more targeted intervention-based supports for students in need. It should be noted that MTSS is neither overly prescriptive nor rigid and has varying implementations and utility based on school districts’ needs.
Schools use MTSS to approach issues within the student population in tiers and place students in such tiers in order to appropriately address their needs. For example, the primary tier refers to a universal intervention geared toward the general student body, whose members may not be faced with distinct difficulty, thereby focusing on prevention to reduce potential problems (Horner, Sugai, & Anderson, 2010). The secondary tier refers to interventions for at-risk students, which typically involve more small group-based and individual interventions for those students still demonstrating difficulty after receiving primary intervention and support (Horner et al., 2010). The tertiary tier refers to working with students who are faced with identified difficulties and have not responded efficiently to primary or secondary levels and are subsequently in need of significant school- and community-based supports (Horner et al., 2010).
An MTSS approach can be conceptualized as incorporating elements of Response to Intervention (RTI) and Positive Behavioral Interventions and Supports (PBIS; Sugai & Horner, 2009). While RTI brings forth opportunities for preventative approaches and early intervention for students struggling with academic skills (Sandomierski, Kincaid, & Algozzine, 2007), MTSS incorporates a broader focus on both academic and social-emotional matters. Within the PBIS framework, the primary focus is on promoting consistent behavior expectations and systems of support to incentivize behaviors of all students within a school (Bohanon, Fenning, Eber, & Flannery, 2007). Both RTI and PBIS utilize MTSS, and specifically tiered intervention delivery, to accommodate the range of student needs. These frameworks are closely aligned in regards to their prevention foci, problem solving, implementation fidelity and data-based decision making (Sugai & Horner, 2009).
Elementary and Secondary School Counseling Program Grant
The ESSCP grant was established by the U.S. Department of Education (USDOE) to provide funding for school districts that demonstrate “the greatest need for counseling services, propose the most innovative and promising approaches, and show the greatest potential for their approach to be replicated and disseminated” (Rentner & Price, 2014, p. 28). To be eligible, proposed projects must incorporate a preventative approach, and effectiveness must at least in part be measured by: (a) the reduction in school counselor-to-student ratios in the district, and (b) decreases in student discipline referrals (USDOE, 2015). Selected projects also must involve the collection, examination, and use of high-quality and timely data, including data on program participant outcomes, and improving instructional practices, policies, and student outcomes (Rentner & Price, 2014).
The current grant project was considered trailblazing in its approach to expanding the data-based decision-making process in the district through a number of initiatives, including the following: (a) identifying research-based social-emotional indicators that link to academic and behavioral school success; (b) creating a user-friendly system for routinely collecting data on these critical areas of student development; and (c) developing the data literacy skills of school counselors in order to ensure that this social-emotional data would continue to be gathered, analyzed and included in data-based discussions long after the grant project had concluded. The funds provided by the ESSCP grant to support these initiatives enhanced the existing RTI model enacted by the school district by integrating a wide range of data related to student development and thus allowed data team members to examine the relationship between social-emotional factors and academic achievement, conducive to a more effective and comprehensive MTSS approach. Through a sophisticated new data collection infrastructure, as well as school counselors’ service in a leadership role, a nuanced and more targeted system of tiered supports emerged that allows the district to respond to a wide range of non-cognitive as well as cognitive issues.
Method
The grant project, formally entitled “An Asset Building Culture,” consisted of four primary initiatives: (a) hiring school counselors in order to create more favorable counselor-to-student ratios, (b) reducing the number of disciplinary incidents, (c) establishing a robust system of strengths-based social-emotional data collection grounded in sound theory, and (d) building human capacity and the technological means to incorporate new social-emotional information in a formal data-based decision-making process. These initiatives would subsequently inform a continuum of cognitive and non-cognitive supports and services within an MTSS model. Ultimately, the goal was to create positive systemic change within the district in which school counselors serve as leaders in using data as a tool for supporting students’ social-emotional, academic and behavioral development.
Setting and Participants
The project was conducted in an urban suburb with a population of approximately 30,000, located in the Northeast region of the United States. The district served nearly 3,000 students and had four elementary schools. More than half of the students were considered low-income and 43% did not speak English as their first language, with 52% identifying as Black/African American, 17% Asian-American, 15% White/Caucasian, 12% Hispanic/Latino/a, and 4% as Multiracial. The racial diversity represented in students was not reflected in its school staff, as more than 80% identified as White/Caucasian.
The school district was awarded the ESSCP grant in 2012. The grant team, comprised of school district leadership, Unique Potential Consulting (UPC), the Ronald H. Fredrickson Center for School Counseling Outcome Research and Evaluation (CSCORE), and Sebastian Management oversaw the grant project’s objectives. UPC served as coordinator of the day-to-day operations of the grant project and provided coaching and professional development to the district’s superintendent, elementary school principals and four grant school counselors. By allocating grant resources to this coordinator position, the project had an advocate for transformed school counseling practices who kept grant priorities in focus amidst other district initiatives. As evaluator of the grant, CSCORE collected quantitative and qualitative data to measure project outcomes and provided training in evidence-based practice to school counselors and district administrators.
Improving School Counselor-to-Student Ratios
The ASCA (2012) National Standards recommend a ratio of one school counselor to every 250 students, though the national average is actually well above these recommendations at nearly 1:500 (Carey & Dimmitt, 2012). Ample research suggests that school counselors have a positive impact on students’ academic, social-emotional and behavioral outcomes (Lapan, Gysbers, & Petroski, 2001; Lapan, Gysbers, & Sun, 1997; Sink & Stroh, 2003; Webb, Brigman, & Campbell, 2005), with further research suggesting that these ratios matter a great deal in a school counseling program’s overall effectiveness (Carrell & Carrell, 2006; Lapan, Whitcomb, & Aleman, 2012). Improving these ratios is especially impactful in high-poverty school districts (Lapan, Gysbers, Stanley, & Pierce, 2012).
Prior to the ESSCP grant, the district’s elementary school staff did not include school counselors at all, resulting in very high mental health provider-to-student ratios. Hiring four school counselors at the beginning of the grant period brought the counselor caseload ratios down to 1:369. Because the district experienced economies of hiring, the grant team added a half-time school counselor in the 2013–2014 school year, further reducing the ratio of school counselor to student to 1:340 despite an increase in enrollment. Grant monies continued to fund each of the 4.5 school counseling positions in the subsequent two school years, strengthening the district’s capacity to provide a broad range of services to students and maintain ratios more closely aligned with ASCA recommendations.
Office Discipline Referral Data
Office discipline referrals (ODR) offer a measure of both individual student behavior and school climate (Clonan, McDougal, Clark, & Davison, 2007; McIntosh, Frank, & Spaulding, 2010) and convey valuable information about students’ social-emotional competencies. A primary requirement of the ESSCP grant was to reduce the number of disciplinary infractions in the district and to demonstrate this improvement through ODR data. The process of determining baseline discipline data revealed great variability in how these incidents were both defined and recorded across different schools. Collecting and using valid discipline data is essential for creating safe schools conducive to teaching and learning (USDOE, 2015), and systematic data collection offers useful information for “understanding and ameliorating individual student and school-wide disruptive behavior problems” (Rusby, Taylor, & Foster, 2007, p. 333). The grant team therefore established new protocols for collecting discipline data in the district’s elementary schools, including creating a standardized ODR form that provided detailed information about the nature and frequency of disciplinary infractions. In addition, the district moved from a paper to an electronic system of recording these data.
The revised ODR form included a comprehensive list of disciplinary infractions that teachers considered high incidence behaviors in the elementary schools. The form was divided into three tiers to delineate progressive levels of severity. Level 1 infractions, such as “failure to obey classroom rules/procedures,” were regarded as problematic behaviors to be managed within the classroom. Documenting Level 1 infractions provided a data-based mechanism for teachers to record a student’s behavioral challenges in the classroom, and this information could be used within an MTSS model to justify the need for additional support or special education services. Level 2 infractions were considered more serious and included behaviors such as “using obscene language/gestures or a repeated offense of the same Level 1 behavior.” Teachers involved the assistance of other staff, such as another teacher or the school counselor, in handling Level 2 infractions. A list of classroom management and behavioral strategies also were listed on the ODR form, and teachers were asked to indicate any strategy they employed in addressing Level 1 or Level 2 problem behaviors. Infractions at Level 3 were recognized as major offenses and warranted involvement of the building principal. Level 3 infractions were further divided into two categories so that crisis incidents demanding immediate action and state reporting, such as “possession of a weapon” or “physical attack on a student or staff,” were recorded separately. The ODR form also included name of staff making the referral, grade of student, date and time of disciplinary incident, location where infraction took place and administrative action taken. In addition, space was provided for teachers to write a brief narrative about events as they occurred, including possible motivation for observed behaviors. The ODR form was revised multiple times based on feedback from principals, teachers and school counselors and piloted during the second year of the grant project.
The Protective Factors Index
The ESSCP grant was launched at a time when district leadership was considering introducing a standards-based student report card. Standards-based report cards list specific skills and knowledge linked to learning standards in each academic subject, and classroom teachers assess a student’s proficiency in each of these areas using a rating scale instead of traditional grades (Swan, Guskey, & Jung, 2014). This shift in practice for measuring academic performance provided an opportunity to create a district-wide system for assessing students’ social-emotional development to inform a more elaborate MTSS framework. While most elementary-level report cards contain a section for behavior or deportment, these indicators may not systematically align with research on personal, social and emotional factors related to achievement and success. In addition, teachers are often asked to rate student behavior without reference to a rubric that would ensure the reliability and validity of these ratings (Squier et al., 2014). To ground the new behavioral component of the report card in the research base, the grant team used the aforementioned Construct-Based Approach to School Counseling (CBA; Squier et al., 2014).
Incorporation of CBA included the identification of four social-emotional constructs that correlate with academic achievement. The grant team broke these constructs down into 15 indicators, which they deemed protective factors. The Protective Factors Index (PFI) was created as the assessment instrument for systematically collecting social-emotional data. Furthermore, the grant team developed a number of specific and measurable competency indicators related to each construct (see Table 1). In addition to being informed by a strong research base, the grant team wanted to ensure that each indicator reflected competencies considered relevant by staff and families in the grant school district. A representative group of school counselors, teachers from each grade level, a teacher of English Language Learners, a special education teacher and the principals from each school reviewed the 15 original PFI items for developmental appropriateness and cultural sensitivity. The group expressed misgivings about two standards under the self-knowledge construct (i.e., “identifies personal feelings,” and “identifies personal strengths and abilities”). There was concern that these behaviors involved attributes valued more by the dominant culture and that benchmarking students against what families might view as culturally specific standards was not fair. These items were therefore omitted from the pilot version, leaving a total of 13 items.
Once the final version was complete, teachers assessed students’ social-emotional development on each of the PFI’s indicators when grading report cards three times a year. In order to expand the consistency of the PFI and subsequently improve inter-rater reliability in data analysis, the grant team also created a scoring rubric to assist teachers in more accurately assigning ratings to these social-emotional indicators.
Creating a scoring rubric. In order to assist teachers in assessing the behaviors and attitudes that comprise the PFI within a developmental lens, the rubric was organized into three levels (K–1st, 2nd–3rd, and 4th–5th grades) to delineate the expected progression for each PFI indicator. The rubric lists specific, observable behaviors to help teachers determine whether a student was demonstrating age-appropriate skills in each domain. For example, descriptors to assess whether a kindergarten or first grade student “works collaboratively in groups of various sizes” included the descriptor “interacts appropriately with peers in group activities,” and “contributes ideas in a group.” Descriptors for second- and third-grade students included the same two behaviors as the earlier grades as well as “shows respect for others by listening to their ideas and opinions.” For fourth- and fifth-grade students “agrees or disagrees with others in a respectful manner” was added to the rubric descriptors. The rubric helped to ensure greater accuracy and consistency in scoring behaviors across classrooms and to reduce subjectivity in teachers’ ratings.
During the first year of the project, teachers requested a simple dichotomous response set for assessing PFI indicators (i.e., “struggling” or “on target”). After a successful year of piloting the new report card and accompanying rubric, teachers requested to move to a four-item response format: meets standard, progressing toward standard, emerging, and not meeting standard. The grant team expanded the original rubric, anchoring responses in degrees of support needed for a student to successfully demonstrate a behavior. Teachers were again provided concrete examples of student behavior within the rubric and were asked to assess if a student consistently and independently displayed the behavior or whether the student needed occasional, frequent or ongoing support to meet the standard.
Table 1
Summary of Primary Constructs and Indicators in the PFI
| Primary Construct | Indicators |
| Motivation | Engages in class activities |
| Demonstrates an eagerness to learn | |
| Demonstrates perseverance in completing tasks | |
| Self-Knowledge | Identifies academic strengths and abilities |
| Identifies things he/she is interested in learning | |
| Self-Direction | Demonstrates the ability to self-regulate actions and emotions |
| Demonstrates resilience after setbacks | |
| Makes productive use of classroom time | |
| Relationships | Works collaboratively in groups of various sizes |
| Seeks assistance when necessary | |
| Respects and accepts authority | |
| Forms respectful, equitable relationships with peers |
Building Technological and Human Capacity
Developing a more comprehensive approach to using data requires that educators have access to meaningful and useful data (Poynton & Carey, 2006). Technology is a key component to establishing effective data use, and research has demonstrated that the state of computer systems can hinder this process in schools (Mandinach, 2012; Wayman, Jimerson, & Cho, 2012) and that easy, integrated and timely access to data facilitates the data-based decision-making process (Ikemoto & Marsh, 2007; Wayman, 2005). Staff at the grant site could readily access classroom grades, state test scores and other achievement data through the district’s Student Information System (SIS). A primary objective of the grant project was to develop the infrastructure to support the same ease of access to important social-emotional indicators. The grant’s technology consultant worked with the district to interface the PFI data recorded on the new report card with the district’s SIS. Teachers, counselors and administrators could then view information about a student’s engagement in class activities or perseverance in completing tasks in the same way they could examine a student’s academic data. The technology consultant also wrote queries to extract PFI data from the SIS into user-friendly Excel reports so that school counselors could disaggregate the data by demographic variables such as gender, grade level or subsidized lunch status. Data also were aggregated at the classroom, grade or building level. The consultant then trained the school counselors to use Excel to illustrate on graphs the number of students struggling with specific PFI indicators (e.g., self-regulation, cooperation, motivation). These graphs could be organized by grade level, school site and individual students. Building strong technological capacity and functionality provides an essential foundation for effective data use. However, translating the wealth of data collected by schools into meaningful actions to support student success within an MTSS framework also requires building human capacity in data literacy skills (Ikemoto & Marsh, 2007; Mandinach, 2012; Wayman, 2005; Wayman & Stringfield, 2006). To build these competencies among school counselors, the grant team organized monthly professional development workshops in evidence-based practice, tiered interventions, data-based decision making, data analysis, and Excel charting and graphing. Counselors learned to extract the PFI data from the SIS, conduct simple analyses to determine what issues existed at various levels within the building, and create graphs to share with teachers and other educators at building-based data team meetings (see Figure 1).
Figure 1. Sample of PFI data aggregated by a Single Indicator, Grade Level, and School Site
Results
The district’s elementary schools had previously stored hard copies of disciplinary incident forms in the principal’s office. This system did not support easy analysis of disciplinary data or examination of behavioral issues in the building. In the revised process, an administrative assistant electronically entered all information from the new ODR form into the school’s SIS database. The electronic system allowed staff to quickly determine the total number of disciplinary infractions in the building over a given period, identify patterns in the data such as a spike in infractions immediately before vacations, and disaggregate the data to determine the frequency of different problem behaviors among various subgroups of students. This streamlined method of data collection also enabled staff to identify possible trends in disciplinary infractions. If data revealed issues such as disproportionality in the district, school counselors served as advocates in establishing more equitable protocols around discipline policies. Notably, the number of disciplinary infractions dropped significantly throughout the 3-year grant program.
Data collected from the PFI provided valuable information to all stakeholders about students’ social-emotional competency development. Because teachers observe behavior and peer interactions every day, their perspective provides a keen understanding of whether a student is able to put into practice each of the indicators listed. In addition, since teachers rate students on the PFI multiple times each year through the district’s electronic report cards, educators throughout the building had access to real-time data about behavioral issues impacting individuals or groups of students. The school counseling program, which prior to this grant project had not been established, consistently reviewed these data, generated charts to determine where gaps existed in social-emotional or academic skill areas and focused their weekly classroom guidance lessons on teaching these competencies. Subsequent report card data were also analyzed to evaluate the impact of counseling lessons on students’ skill development.
Data Teams and a Multi-Tiered System of Supports
Prior to the district’s ESSCP award, data teams were operating at each elementary school and were led by the building principal. Student names were only considered for data team discussion if a teacher completed a referral form indicating a student was struggling academically in the classroom. These forms, often inconsistently completed and comprised largely of teachers’ perceptions about academic performance, served as the principal mechanism for identifying at-risk students. The only other information frequently reviewed by data teams were standardized test scores, classroom grades and serious behavioral infractions. Interventions to support students were almost exclusively academic in nature.
The grant team collaborated with staff to restructure data teams to include social-emotional data analysis. Data teams were then able to expand their RTI approach to a more expansive MTSS framework to include multi-tiered counseling interventions in addition to existing academic interventions. School counselors created graphs and charts of PFI, ODR and attendance data to illustrate such trends as common behavioral issues across grade levels or attendance patterns during certain days of the week or times of year. Data team members reviewed these graphs to identify gaps in social-emotional, behavioral or academic skill areas. Meetings shifted from an almost exclusive focus on academic data to considering multiple sources of achievement, demographic, behavioral and social-emotional variables. As teams explored the relationship across different types of data, a greater understanding began to emerge about how social-emotional factors, such as those included in the PFI, impact academic achievement. The charge of the data teams became deciding which tiered interventions (universal, targeted and intensive) were indicated to promote the development of academic competencies as well as of the protective factors to support school success for every student.
School Counselors’ Contributions to a Multi-Tiered System of Supports
Access to accurate and real-time data about student behaviors enabled school counselors to more effectively develop tiered interventions for students and environments in need of support. The PFI data were collected three times a year at the close of each marking period. Behavioral data gathered through the revised ODR form were updated in the SIS weekly. Attendance data at the elementary school sites were available daily. Access to these real-time data allowed school counselors to continuously monitor students’ social-emotional and academic progress. It also enabled counselors to easily evaluate whether their interventions were creating the desired impact. In this continuous process of data-based decision-making, the same set of data indicators, examined at different points throughout the school year, informed school counselors’ decisions about which interventions were needed and also served as outcome data to evaluate interventions at each tier.
Schoolwide, Tier 1 interventions included delivery of success classes to all students. School counselors developed a developmental guidance curriculum with 10 lessons per grade grounded in the evidence-based programs zones of regulation (Kuypers, n.d.) and second step (Low, Cook, Smolkowski, & Buntain-Ricklefs, 2015), with weekly lesson content guided by areas of improvement demonstrated in the PFI data and behavioral data represented in discipline referrals. In addition, a school counseling program “Expo” was held at the end of each year, and parents and guardians were invited to the school to see artifacts generated by students in success class. Additional schoolwide interventions included the character trait of the month project, focused on the development of positive qualities such as respect, honesty and courage, and a parent newsletter sent out by the counseling department explaining what could be done at home to enhance the development of social-emotional competencies (i.e., informing parents and guardians of the character trait of the month, suggesting a “conversation starter” about current classroom activities, and recommending related books to read with their children).
Students who were struggling academically and for whom PFI and ODR data indicated a need for additional behavioral support and social-emotional competency instruction received Tier 2 services through small group counseling sessions. School counselors facilitated groups on topics related to PFI indicators such as self-regulation, resilience and motivation throughout the year. The school counselors used discipline data, often in combination with report card indicators reflecting students’ social-emotional competencies, to determine membership in targeted small group counseling sessions and continued participation in this targeted intervention. Subsequent ODR data was reviewed to evaluate changes in students pre- to post-intervention, as these data have been demonstrated to be sensitive measures of the impact of schoolwide interventions (Irvin, Tobin, Sprague, Sugai, & Vincent, 2004; Rusby et al., 2007). School counselors also created progress monitoring tools to assess social skill development during a group cycle. As with academically focused tiered instruction, teachers were asked to briefly rate student growth so that small group instruction could be modified in a continuous formative assessment process.
The continuum of counseling services also included development of a Summer Boot Camp Transition Program. School counselors collected quantitative and qualitative survey data from sixth graders in the district about their experience in moving from elementary to middle school, which indicated that some students were anxious about this transition and wanted more support and information about the process. To proactively address these common issues, the school counselors created a series of four week-long summer boot camps that were free of charge and open to all district fifth graders. Classroom lessons and group activities for the camp were drawn from the evidence-based curricula Student Success Skills (Webb & Brigman, 2006), WhyTry (Bird, 2010) and The Real Game (Barry, n.d.) and covered topics critical to success in middle school such as perseverance, organizational skills and study strategies.
Finally, PFI, ODR and standards-based report card data also guided decisions about Tier 3 interventions. School counselors developed Behavior Improvement Plans (BIPs) for students in need of intensive behavioral support in the classroom. They also coordinated with special education or other mental health professionals when referrals were warranted.
Positive Systemic Change
The grant initiatives resulted in definitive progress and positive systemic changes throughout the district. A new policy was established which mandated that counseling groups be formed based on issues identified in the data and no longer simply by teacher request or anecdotal evidence. This more objective approach to determining which students were in need of Tier 2 social-emotional interventions ensured that students with a documented need for additional assistance received these services.
At the beginning of the grant period, the district had been declared “underperforming” by state rankings and was mandated to write an annual Accelerated Improvement Plan (AIP). Throughout the 3-year grant cycle, a number of elements from the grant project were embedded in the AIP including: (a) revising K–5 report cards to use a standards-based system, (b) integration of the PFI within the new report cards, (c) designing and delivering a developmental guidance curriculum for grades K–5, (d) collaborating with building principals to incorporate social-emotional data into data team meetings, and (e) developing tiered strategies to better address the social-emotional needs of struggling students. Officials from the State Department of Education who monitored the AIP expressed their belief that these initiatives contributed to the district’s overall improvement and began to send other struggling school systems to the grant district to learn specifically about their data-based MTSS approach and the school counselors’ role in it.
Ultimately, the success of the grant within the district can perhaps best be measured by two key administrative decisions made when grant funding ended: (a) the decision to retain the school counselors, as teachers and administrators now saw these professionals—who had not been employed at the district before the grant—as indispensable to student success; and (b) the decision to hire UPC (who had worked as project coordinator for the grant) to work to support the expansion of the grant initiatives to the middle school and high school over the next several years. At the time of this article’s publication, work was underway to identify means to collect social-emotional data at the middle and high school levels so that their multi-tiered system of supports can be as robust as that at the elementary level.
Discussion and Implications for School Counselors
Data-based decision making has become an essential component of educational practice (Mandinach, 2012). The implementation of NCLB and standards-based education have created strong pressure for schools to demonstrate improved student performance through state test scores (Ikemoto & Marsh, 2007; Marsh et al., 2006). These data often become the primary consideration of data-driven discussions as schools strive to meet state and federal requirements. Data use has the potential, however, to be more than simply a response to meeting accountability demands. The data-based decision-making process can be transformed when multiple forms of data are viewed from different professional perspectives to better describe the factors and contexts that influence student success (Mandinach, 2012). Fortunately, the new ESSA legislation stresses the importance of considering non-academic data to foster a broader vision of student success. Clearly describing what is happening for an individual or to groups of students requires “a body of relevant data, with each individual data element imparting a complementary piece of the puzzle” (National Forum on Education Statistics, 2012, p. 9).
An integrative approach to data-based decision making requires the technological capacity to organize data into user-friendly formats. It also may necessitate the collection of data beyond the scope of what is traditionally stored in district’s information systems (Poynton & Carey, 2006). Behavior in the classroom occurs within the broader context of a student’s life and developing interventions to support student success requires collecting data that reflect this context (National Forum on Education Statistics, 2012). Creating a data collection infrastructure that allows those who observe students on a daily basis (e.g., teachers) to rate social-emotional competency attainment in addition to academic competency attainment on a regular basis is a complex undertaking, but one that has very promising potential. When educators triangulate data by using multiple types and sources of data, the relationship between academic outcomes and social-emotional factors is better understood and reliance on a single data point, such as academic scores, is reduced (Marsh et al., 2006).
The grant team developed a number of initiatives designed not only to fulfill requirements of the ESSCP award, but also to create systemic changes around the culture of data use and continuum of tiered supports in the district. Each individual grant initiative aimed to improve a particular aspect of data-based decision making: incorporating research-based social-emotional indicators into the elementary school report cards, creating the infrastructure for easy and timely access to these data, developing new protocols for collecting discipline data, and building the data literacy skills of school counselors. The combined effect of each of these initiatives was a restructuring of building-based data teams that operated from a strong MTSS; these included the following: (a) coordination of schoolwide prevention efforts and systems, (b) universal screening and progress monitoring, (c) selection and use of evidence-based practices, (d) professional development that targets evidence-based practice, (e) evaluating outcomes using data-based decision making, and (f) leadership commitment from administrators and school-based teams that supports schoolwide implementation (Harn, Basaraba, Chard, & Fritz, 2015; Kame’enui, Good, & Harn, 2005; Sugai & Horner, 2009).
Notably, the grant project integrated an academic, behavioral, and social-emotional focus in the gathering of data, examined how specific behaviors and social-emotional skills impacted student achievement, and subsequently selected targeted interventions to build the competencies needed for school success. Although the majority of research and scholarly discussion has focused on using data-based decision-making models for academic concerns, researchers have proposed a similar model for social-emotional and behavioral problems (Eber, Sugai, Smith, & Scott, 2002; Fairbanks, Sugai, Guardino, & Lathrop, 2007; Gresham, 1991; Sugai, Horner, & Lewis, 2009). Though currently the majority of schools are operating these schoolwide efforts independently (McIntosh, Bohanan, & Goodman, 2010), there is a growing call for the holistic approach MTSS offers due to the known interaction of academic, behavioral and social-emotional issues in students who struggle (Mclntosh, Horner, Chard, Boland, & Good, 2006).
The grant project’s approach to adopting MTSS was also unique in the pivotal role of school counselors in the data-based decision-making process. The role of the school counselor is infrequently defined in the RTI literature (Gruman & Hoelzen, 2011) or in educational reform agendas (Dahir, 2004). School counselors have sometimes been seen as resistant to using data (Young & Kaffenberger, 2011). However, school counselors work at the intersection of the academic and social-emotional domains (ASCA, 2012) and support student development across these areas. School counselors, previously not represented on the building data teams, have now become data leaders in these schools. Because data-based decision making has focused largely on academic achievement, data use may have been seen as the charge of the classroom teacher. Through grant-based professional development workshops, the counselors developed competencies in organizing, analyzing and graphing data. These new skills have enabled the school counselors to lead data-based conversations, develop progress monitoring tools and create results reports for administrators and the school committee. Using data routinely collected through the SIS provides an efficient and timely access to not only determine which interventions are needed, but also to evaluate the impact of the schoolwide counseling curriculum, targeted small groups and other activities.
This mode of data collection represents a change from the pre/posttest method commonly employed by school counselors. Pre/posttests may provide information about whether students learned the content of a specific lesson but do not show whether students are applying these skills, attitudes or beliefs in their lives. School counselors can contribute unique insights to the data team process by going a step further and helping to determine the underlying causes for a student’s misbehavior or poor academic performance. Incorporating social-emotional indicators into data-based discussions may make the process feel more relevant to the work of the school counselor. In fact, many of the words used to describe this more comprehensive approach to data (e.g., relationships, linking, connecting, inclusion and contextualizing) sound more from the counseling lexicon than from a statistics textbook.
The overarching goal of this pilot project was to create a meaningful data-based decision-making process to promote an MTSS model based on academic and social-emotional data. Therefore, the success of this project contributes ideas as to not only what non-academic data can be analyzed, but also how to go about collecting, analyzing and incorporating findings into the planning around a continuum of supports to foster student success. Using research-based constructs, redesigning report cards, developing rubrics, identifying professional development needs, and developing human technological capacity to manage and interpret data are feasible and effective strategies to support achievement. Ultimately, discussions shifted from examining symptoms of an issue—such as disciplinary infractions, low grades and test scores, or poor attendance—to trying to unearth the underlying causes for student issues and how the school could support growth with a variety of academic and social-emotional tiered supports.
Limitations and Directions for Future Research
The grant project was not designed or implemented as an experimental study; therefore, we cannot know with certainty whether the implementation of the grant initiatives and subsequent positive outcomes share a causal relationship. Furthermore, we cannot yet know which specific elements of the grant project brought about the most positive change, or whether some elements may have been superfluous, as outcomes have been viewed as a comprehensive result of all grant-related activities. Future research involving an experimental study in which: (a) outcomes are compared to similar schools that did not received grant-funded resources; and (b) there are outcomes measures in place for each grant initiative, is recommended. Moreover, additional studies that expand these efforts to students and schools in different regions, grade levels and with a higher number of participants also is suggested.
Although the PFI is a promising new instrument for the measurement of positive social-emotional behaviors in the classroom, further research is necessary to validate its use as a universal brief screener. Bass and colleagues (2015) conducted a confirmatory factor analysis with the PFI using data gathered during the present grant project, which resulted in a three factor measurement model rather than four as hypothesized. These findings warrant further exploration with additional populations of students to determine whether they will be replicated. The PFI also relies on teacher observation, which occurs consistently at the elementary school level; therefore, it would be valuable to study its use in upper grades (i.e., middle school and high school) to verify whether the PFI is still a reliable and valid instrument in settings where teachers experience less face-to-face time with each individual student throughout the school day.
Finally, it bears noting that the research base is still emerging around social-emotional learning and which competencies best link to school success. There is not even consensus within the scholarly community on how to refer to these constructs (e.g., non-cognitive factors, non-academic skills, soft skills, grit). Further research will be necessary to determine which social-emotional learning theory or theories exhibit applicability in school settings, and the development of assessment instrumentation based on a CBA in particular is still in its early stages.
Conclusion
The ESSCP grant offered by the USDOE provides funding to establish and improve school counseling programs in high-needs school districts. The current grant project was implemented at four elementary sites in a diverse school district in an urban suburb of the Northeastern United States. Specific grant initiatives included the hiring of four full-time and one part-time school counselor in order to reduce the student-to-counselor ratio. The office discipline referral process was restructured to include greater specificity and objectivity, and the PFI was developed in order to provide an assessment tool of social-emotional competencies in the classroom. School counselors also were provided training in how to collect, analyze and include social-emotional data in the data-based decision-making process. Subsequently, the combination of a new school counseling program and data on discipline and social-emotional competencies along with existing academic data resulted in a much-improved MTSS model in the district, providing a continuum of supports for students’ needs. The study sheds light on the value of providing school counseling at the elementary level and the importance of data literacy and advocacy as a major tenet of these positions. As ESSCP grants are awarded based on their potential for replication and dissemination, the initiatives described in this manuscript represent innovative practices that hold tremendous promise at a national level.
Conflict of Interest and Funding Disclosure
The authors reported no conflict of interest or funding contributions for the development of this manuscript.
References
American School Counselor Association. (2012). The ASCA national model: A framework for school counseling programs (3rd ed.). Alexandria, VA: Author.
American School Counselor Association. (2014). Mindsets & behaviors for student success: K-12 college and career-readiness standards for every student. Alexandria, VA: Author.
Barry, B. (n.d.). The Real Game design process. Retrieved from http://www.realgame.org/process.html
Bass, G., Lee, J. H., Wells, C. W., Carey, J. C., & Lee, S. (2015). Development and factor analysis of the Protective Factors Index: A report card section related to the work of school counselors. The Professional Counselor, 5, 516–528. doi:10.15241/gb.5.4.516
Bird, B. (2010). WhyTry evaluation report 2006–2010. Churchill County Probation Report, Field Research. Retrieved from https://www.whytry.org/index.php?option=com_content&view=article&id=109
Bohanon, H., Fenning, P., Eber, L., & Flannery, B. (2007). Identifying a roadmap of support for secondary students in school-wide positive behavior support applications. International Journal of Special Education, 22, 39–52.
Carey, J., & Dimmitt, C. (2012). School counseling and student outcomes: Summary of six statewide studies. Professional School Counseling, 16, 146–153.
Carrell, S. E., & Carrell, S. A. (2006). Do lower student to counselor ratios reduce school disciplinary problems? The BE Journal of Economic Analysis & Policy, 5, 1–24.
Clonan, S. M., McDougal, J. L., Clark, K., & Davison, S. (2007). Use of office discipline referrals in school-wide decision making: A practical example. Psychology in the Schools, 44, 19–27.
Dahir, C. A. (2004). Supporting a nation of learners: The role of school counseling in educational reform. Journal of Counseling & Development, 82, 344–353. doi:10.1002/j.1556-6678.2004.tb00320.x
Dahir, C. A., & Stone, C. B. (2009). School counselor accountability: The path to social justice and systemic change. Journal of Counseling & Development, 87, 12–20. doi:10.1002/j.1556-6678.2009.tb00544.x
Dimmitt, C., Carey, J. C., & Hatch, T. (Eds.) (2007). Evidence-based school counseling: Making a difference with data-driven practices. Thousand Oaks, CA: Corwin.
Eber, L., Sugai, G., Smith, C. R., & Scott, T. M. (2002). Wraparound and positive behavioral interventions and supports in the schools. Journal of Emotional and Behavioral Disorders, 10, 171–180. doi:10.1177/10634266020100030501
Every Student Succeeds Act (2015) Pub. L. No. 114–95. Retrieved from https://www.congress.gov/bill/114th-congress/senate-bill/1177
Fairbanks, S., Sugai, G., Guardino, D., & Lathrop, M. (2007). Response to intervention: Examining classroom behavior support in second grade. Exceptional Children, 73, 288–310. doi:10.1177/001440290707300302
Farrington, C. A., Roderick, M., Allensworth, E., Nagaoka, J., Keyes, T. S., Johnson, D. W., & Beechum, N. O. (2012). Teaching adolescents to become learners – the role of noncognitive factors in shaping school performance: A critical literature review. Chicago, IL: University of Chicago Consortium on Chicago School Research.
Gresham, F. M. (1991). Conceptualizing behavior disorders in terms of resistance to intervention. School Psychology Review, 20, 23–36.
Gruman, D. H., & Hoelzen, B. (2011). Determining responsiveness to school counseling interventions using behavioral observations. Professional School Counseling, 14, 183–190.
Harn, B., Basaraba, D., Chard, D., & Fritz, R. (2015). The impact of schoolwide prevention efforts: Lessons learned from implementing independent academic and behavior support systems. Learning Disabilities: A Contemporary Journal, 13, 3–20.
Hayes, R. L., Nelson, J.-L., Tabin, M., Pearson, G., & Worthy, C. (2002). Using school-wide data to advocate for student success. Professional School Counseling, 6, 86–94.
Heckman, J. J., & Rubinstein, Y. (2001). The importance of noncognitive skills: Lessons from the GED testing program. The American Economic Review, 91, 145–149.
Horner, R. H., Sugai, G. M., & Anderson, C. M. (2010). Examining the evidence base for school-wide positive behavior support. Focus on Exceptional Children, 42(8), 1–14.
House, R. M., & Hayes, R. L. (2002). School counselors: Becoming key players in school reform. Professional School Counseling, 5, 249–256.
Ikemoto, G. S., & Marsh, J. A. (2007). Cutting through the “data-driven” mantra: Different conceptions of data-driven decision making. YSSE Yearbook of the National Society for the Study of Education, 106, 105–131. doi:10.1111/j.1744-7984.2007.00099.x
Irvin, L. K., Tobin, T. J., Sprague, J. R., Sugai, G., & Vincent, C. G. (2004). Validity of office discipline referral measures as indices of school-wide behavioral status and effects of school-wide behavioral interventions. Journal of Positive Behavior Interventions, 6, 131–147.
Isaacs, M. L. (2003). Data-driven decision making: The engine of accountability. Professional School Counseling, 6, 288–295.
Kame’enui, E. J., Good, R. G., & Harn, B. A. (2005). Beginning reading failure and the quantification of risk: Reading behavior as the supreme index. In W. L. Heward et al. (Eds.), Focus on behavior analysis in education: Achievements, challenges, and opportunities (pp. 69–89).Upper Saddle River, NJ: Prentice Hall.
Kuypers, L. (n.d.). Evidence on the Zones. Retrieved from http://www.zonesofregulation.com/research.html
Lane, K. L., Menzies, H. M., Ennis, R. P., & Bezdek, J. (2013). School-wide systems to promote positive behaviors and facilitate instruction. Journal of Curriculum and Instruction, 7, 6–31. doi:10.3776/joci.2013.v7n1pp6-31
Lapan, R. T., Gysbers, N. C., & Petroski, G. F. (2001). Helping seventh graders be safe and successful: A statewide study of the impact of comprehensive guidance and counseling programs. Journal of Counseling & Development, 79, 320–330.
Lapan, R. T., Gysbers, N. C., Stanley, B., & Pierce, M. E. (2012). Missouri professional school counselors: Ratios
matter, especially in high-poverty schools. Professional School Counseling, 16, 108–116.
Lapan, R. T., Gysbers, N. C., & Sun, Y. (1997). The impact of more fully implemented guidance programs on the school experiences of high school students: A statewide evaluation study. Journal of Counseling & Development, 75, 292–302.
Lapan, R. T., Whitcomb, S. A., & Aleman, N. M. (2012). Connecticut professional school counselors: College and career counseling services and smaller ratios benefit students. Professional School Counseling, 16, 117–124.
Low, S., Cook, C. R., Smolkowski, K., & Buntain-Ricklefs, J. (2015). Promoting social-emotional competence: An evaluation of the elementary version of Second Step. Journal of School Psychology, 53, 463–477.
Mandinach, E. B. (2012). A perfect time for data use: Using data-driven decision making to inform practice. Educational Psychologist, 47, 71–85.
Mandinach, E. B., Honey, M. & Light, D. (2006). A theoretical framework for data driven decision making. Paper presented at the annual meeting of American Educational Research Association, San Francisco, CA. Retrieved from http://cct.edc.org/sites/cct.edc.org/files/publications/DataFrame_AERA06.pdf
Marsh, J. A., Pane, J. F. & Hamilton, L. S. (2006). Making sense of data-driven decision making in education: Evidence from recent RAND research. Retrieved from http://www.rand.org/content/dam/rand/pubs/occasional_
papers/2006/RAND_OP170.pdf
McIntosh, K., Bohanan, H., & Goodman, S. (2010). Toward true integration of academic and behavior response to intervention systems: Part two—Tier 2 support. Communique, 39(3), 4–6.
McIntosh, K., Frank, J. L., & Spaulding, S. A. (2010). Establishing research-based trajectories of office discipline referrals for individual students. School Psychology Review, 39, 380–394.
McIntosh, K., Horner, R. H., Chard, D. J., Boland, J. B., & Good, R. H., III. (2006). The use of reading and behavior screening measures to predict nonresponse to school-wide positive behavior support: A longitudinal analysis. School Psychology Review, 35, 275–291.
National Forum on Education Statistics. (2012). Forum guide to taking action with education data. Washington, DC: Author.
No Child Left Behind Act of 2001. (2002). Pub. L. No. 107-110 § Title V. Retrieved from http://www2.ed.gov/
policy/elsec/leg/esea02/index.html
Poynton, T. A., & Carey, J. C., (2006). An integrative model of data-based decision making for school counseling. Professional School Counseling, 10, 121–130.
Rentner, D. S., & Price, O. A. (2014). A guide to federal education programs that can fund K-12 universal prevention and social and emotional learning activities. Center on Education Policy, 1–40.
Rusby, J. C., Taylor, T. K., & Foster, E. M. (2007). A descriptive study of school discipline referrals in first grade. Psychology in the Schools, 44, 333–350. doi:10.1002/pits.20226
Sandomierski, T., Kincaid, D., & Algozzine, B. (2007). Response to intervention and positive behavior support: Brothers from different mothers or sisters with different misters. Positive Behavioral Interventions and Supports Newsletter, 4(2), 1–4.
Sink, C. A., & Stroh, H. R. (2003). Raising achievement test scores of early elementary school students through comprehensive school counseling programs. Professional School Counseling, 6, 350–364.
Squier, K. L., Nailor, P., & Carey, J. C. (2014). Achieving excellence in school counseling through motivation, self-direction, self- knowledge, and relationships. Thousand Oaks, CA: Corwin.
Sugai, G., & Horner, R. H. (2009). Responsiveness-to-intervention and school-wide positive behavior supports: Integration of multi-tiered system approaches. Exceptionality, 17, 223–237.
Sugai, G., Horner, R., & Lewis, T. (2009). School-wide positive behavior support implementers’ blueprint and self-assessment. Eugene, OR: OSEP TA-Center on Positive Behavioral Interventions and Supports.
Swan, G. M., Guskey, T. R., & Jung, L. A. (2014). Parents’ and teachers’ perceptions of standards-based and traditional report cards. Educational Assessment, Evaluation and Accountability, 26, 289–299.
doi:10.1007/s11092-014-9191-4
Wayman, J. C. (2005). Involving teachers in data-based decision making: Using computer data systems to support teacher inquiry and reflection. Journal of Education for Students Placed at Risk, 10, 295–308. doi:10.1207/s15327671espr1003_5
Wayman, J. C, Jimerson, J. B., & Cho, V. (2012). Organizational considerations in establishing the data-informed district. School Effectiveness and School Improvement, 23, 159–178.
Wayman, J. C., & Stringfield, S. (2006). Technology-supported involvement of entire faculties in examination of student data for instructional improvement. American Journal of Education, 112, 549–571.
Webb, L. D., & Brigman, G. A. (2006). Student Success Skills: Tools and strategies for improved academic and social outcomes. Professional School Counseling, 10, 112–120.
Webb, L. D., Brigman, G. A., & Campbell, C. (2005). Linking school counselors and student success: A replication of the Student Success Skills approach targeting the academic and social competence of students. Professional School Counseling, 8, 407–413.
U.S. Department of Education. (2015). Elementary and secondary school counseling programs. Retrieved from http://www2.ed.gov/programs/elseccounseling/index.html
Young, A., & Kaffenberger, C. J. (2011). The beliefs and practices of school counselors who use data to implement comprehensive school counseling programs. Professional School Counseling, 15, 67–76.
Karen Harrington is the Assistant Director at the Center for Youth Engagement at the University of Massachusetts Amherst. Catherine Griffith is Associate Director at the Ronald H. Fredrickson Center for School Counseling Outcome Research and Evaluation and Assistant Professor at the University of Massachusetts Amherst. Katharine Gray is a leadership coach at Unique Potential Consulting and Leadership Coaching in Hopkinton, MA. Scott Greenspan is a doctoral student at the University of Massachusetts Amherst. Correspondence can be addressed to Karen Harrington, Furcolo Hall, University of Massachusetts, Amherst, MA 01003, Karen.harrington07@gmail.com.
