Development and Validation of the College Mental Health Perceived Competency Scale

Jun 28, 2018 | Volume 8 - Issue 2

Michael T. Kalkbrenner, Christopher A. Sink

College counselors provide training to their campus constituents on various mental health issues, including the identification of warning signs and the referral of students to appropriate resources. Though extensive information on these topics is available in the counseling literature, college counselors lack a psychometrically sound screening instrument to support some of these educational efforts. To meet this need, the present researchers developed and validated the College Mental Health Perceived Competency Scale (CMHPCS). Based largely on self-determination theory, the measure appraises college student and faculty members’ perceived competence for supporting student mental health. Reliability and construct validity of the CMHPCS are demonstrated through exploratory and confirmatory factor analyses. Hierarchical logistic regression procedures yielded sufficient evidence of the CMHPCS’s predictive validity. Specific applications to assist college counselors with outreach and consultation are discussed.

Keywords: College Mental Health Perceived Competency Scale, college counselors, confirmatory factor analysis, hierarchical logistic regression, screening instrument

The prevalence and complexity of mental health disorders remain a serious concern for mental health professionals working in university and college settings in the United States and internationally (Lee, Ju, & Park, 2017). Another distressing trend is the incongruity between the relatively high frequency of students living with mental health disorders and the small number of students who receive needed treatment (Eisenberg, Hunt, Speer, & Zivin, 2011). Preliminary evidence shows that faculty members, staff, and college student peers might serve as helpful counseling referral agents for individuals at risk for mental health disorders (Kalkbrenner, 2016; White, Park, Israel, & Cordero, 2009). Identifying and training counseling referral agents (e.g., student peers and faculty members) to recognize and refer students to the counseling center is a key role of college counselors (Brunner, Wallace, Reymann, Sellers, & McCabe, 2014; Sharkin, 2012).

The purpose of the present study was to develop and validate a scale for appraising student and faculty members’ perceived competence for supporting college student mental health. Throughout the present study, “perceived competence for supporting college student mental health” refers to the extent to which university community members are confident in their ability to promote a campus climate that is supportive, accepting, and facilitative toward mental wellness. The College Mental Health Perceived Competency Scale (CMHPCS) has potential to aid college counselors with identifying and training university community members (e.g., student peers and faculty) to recognize issues and refer their peers and students to campus counseling services. In the following section, we provide an overview of the pertinent literature.

Undergraduates in Western countries are typically in late adolescence, a period when mental disorders are most likely to emerge, and college students report more frequent mental health concerns than other age groups (de Lijster et al., 2017; Eisenberg et al., 2011). Despite this reality, Eisenberg et al. (2011) indicated that only 20% of college students with mental health disorders were actively seeking treatment. Research suggests that there are common factors contributing to students’ underutilization of counseling services, including: stigma, gender, culture, experience and knowledge (mental health literacy), fear, and accessibility (Brunner et al., 2014; Marsh & Wilcoxon, 2015). For example, many undergraduates are simply unaware of the campus counseling services provided by their universities (Dobmeier, Kalkbrenner, Hill, & Hernández, 2013). Relatedly, college students’ general knowledge of mental health issues varies substantially. Kalkbrenner, James, and Pérez-Rojas (2018) found that students who attended at least one session of personal counseling reported a significantly higher awareness of warning signs for mental distress when compared to students who had not attended counseling. Other evidence suggests that the perceived stigma associated with obtaining mental health support can be a barrier to treatment (Rosenthal & Wilson, 2016) for college students.

Demographic differences exist in college students’ counselor-seeking behavior, with female students reporting a greater willingness to pursue counseling and to refer peers to resources for mental distress when compared to male students (Kalkbrenner & Hernández, 2017; Yorgason, Linville, & Zitzman, 2008). Students from ethnic minority groups also underutilize counseling centers’ mental health services (Han & Pong, 2015; Li, Marbley, Bradley, & Lan, 2016). In addition, Eisenberg, Goldrick-Rabe, Lipson, and Broton (2016) identified differences in college students’ utilization of resources for mental distress by age, with younger students (under 25) being particularly vulnerable to living with untreated mental issues. To enhance access and usage of counseling services by all college students, these variables must be seriously considered by campus policymakers and mental health practitioners.

Given this situation, college counselors must not only address the increased demand for counseling services, they may need to enhance prevention services as well. These latter activities include outreach, consultation, and education of university community members (e.g., student peers and faculty members). For instance, counselors educate students and faculty members on recognizing the warning signs of mental health distress in themselves and others (Brunner et al., 2014). Training also is commonly provided to campus members on the referral process. Participants learn the skills needed to guide others (e.g., students at risk for mental health disorders) to appropriate counseling and related services (Brunner et al., 2014; Sharkin, 2012). Preliminary investigations support these efforts, and faculty members, staff, and college student peers have been found to be helpful referral agents (Kalkbrenner, 2016; White et al., 2009).

Although research shows that students and faculty members are viable referral sources (Kalkbrenner, 2016; White et al., 2009), Albright and Schwartz’s (2017) national survey of these groups found that approximately half of their respondents felt unprepared to recognize the warning signs of mental distress in others. Based on these findings, as suggested above, college counselors may need to revise the content and delivery of their mental health–related training. Moreover, the literature appears to be lacking a psychometrically sound screening tool to assist with this effort. To help fill this instrumentation gap, the authors developed a brief questionnaire for college counselors to appraise student and faculty members’ perceived competence for supporting college student mental health.

Theoretical Foundation for Measurement Instrument

The first step in designing a measurement instrument involves the use of theory to guide the item development process (DeVellis, 2016). In recent years, self-determination theory (SDT), a psychological orientation to human motivation, is increasingly deployed by counseling researchers as an orienting conceptual framework (Adams, Little, & Ryan, 2017; Ryan & Deci, 2000; Ryan, Lynch, Vansteenkiste, & Deci, 2011). Aligned with this trend, SDT guided the item development for the CMHPCS. This perspective conceptualizes motivation in terms of the extent to which one’s behaviors are autonomous (self-motivated) contrasted with the extent to which behaviors are coerced or pressured (Patrick & Williams, 2012). Leading SDT proponents contend that the satisfaction of people’s needs is essential to foster their intrinsic motivation (i.e., a person’s autonomous or self-generated behaviors; Patrick & Williams, 2012; Ryan & Deci, 2000). Key elements of this approach include one’s perceptions of self-competence, autonomy, and relatedness to others (Ryan & Deci, 2000). Evidence suggests that increases in the extent to which individuals feel competent that they can perform an action or behavior are associated with increases in their motivation to participate in that action or behavior (Adams et al., 2017; Jeno & Diseth, 2014).

Elements of SDT are utilized in various helping professions, including psychiatry (Piltch, 2016), medicine (Mancini, 2008), and college counseling (A. E. Williams & Greene, 2016). Research suggests that SDT is a valuable framework for various mental health practices. For instance, Patrick and Williams (2012) demonstrated that perceived competence, a key dimension of SDT, was a significant predictor of clients’ medication adherence. Other investigators demonstrated the utility of SDT for promoting college student mental health (Emery, Heath, & Mills, 2016; A. E. Williams & Green, 2016). In one study, college students’ level of motivation and perceived competence were found to be important factors associated with their mental and physical well-being (Adams et al., 2017). Jeno and Diseth (2014) indicated that a college student’s sense of autonomy and perceived competence were significant predictors of improved academic performance. Another investigation found that group therapy based on SDT and motivational interviewing reduced college women’s susceptibility to high-risk alcohol use (A. E. Williams & Green, 2016). Moreover, university students’ sense of perceived competence and emotional regulation were associated with reductions in non-suicidal self-injury (Emery et al., 2016). Emery et al. (2016) concluded that SDT and college students’ need for perceived competence were salient notions for conceptualizing non-suicidal self-injury and supporting college student mental health.

Self-Determination Theory and Psychometric Instruments

SDT is a widely used theoretical framework to develop measurement instruments in the social sciences. Multiple educational scales have been founded on constructs aligned with SDT, including the Learning Climate Questionnaire (G. C. Williams & Deci, 1996), the Basic Psychological Need Scale (Ntoumanis, 2005), the Academic Self-Regulation Questionnaire (Ryan & Connell, 1989), and the Perceived Competence scale (G. C. Williams & Deci, 1996). Each instrument appraises latent variables related to students’ level of perceived competence and intrinsic motivation toward academic success (Jeno & Diseth, 2014). Given the promising implications of SDT for informing the development of clinical and educational interventions and appraisal instruments, college counselors might benefit from a scale that assesses student and faculty members’ perceived competence related to supporting college student mental health. Such a measure has potential to aid in the early identification of college students at risk for mental health issues and support general campus mental health services. Research indicates that effective screening generally leads to more college students seeking meaningful treatment and support (Hill, Yaroslavsky, & Pettit, 2015).

In an extensive review of the measurement literature with no restrictions on participants or locations, Wei, McGrath, Hayden, and Kutcher (2015) identified 215 measurement instruments for appraising three major components of mental health literacy, including help-seeking, knowledge, and stigma. While these instruments have utility within the screening process, a measure designed to appraise one’s sense of perceived competence toward promoting mental health support on college campuses is absent. The characteristic of perceived competency has potential to act as a protective factor against mental distress (A. E. Williams & Green, 2016). Therefore, the authors incorporated the perceived self-competence dimension of SDT to formulate CMHPCS items.

To summarize, the purpose of the present study was to develop and validate a measurement instrument for appraising student and faculty members’ perceived competence for supporting college student mental health through recognizing and referring student peers to resources for mental wellness. The following research questions were posed: (1) What is the underlying factor structure of the CMHPCS using a large sample of college faculty and are the emergent scales reliable? (2) Is the emergent factor structure from the CMHPCS confirmed in a new sample of undergraduate students? and (3) To what extent do participants’ CMHPCS scores have predictive validity for whether or not they have made a student referral to the counseling center?

Method

Participants and Procedures

Data were collected from students and faculty members at a large mid-Atlantic public university. G*Power was used to conduct a priori power analysis for the hierarchical logistic regression analyses described below (Faul, Erdfelder, Lang, & Buchner, 2007). A minimum sample size of 264 (132 in each sample) would provide a 95% power estimate, α = .05 (two tailed), with an odds ratio of 2.0. Based on the recommendations of Mvududu and Sink (2013), the researchers ensured that the ratio of respondents to each estimated parameter for the student sample (26:1) and for the faculty sample (11:1) was sufficient for factor analysis. The CMHPCS was administered to 513 university community members, including a sample of 201 faculty members and 312 undergraduate students. The sampling procedures and demographic profiles of the two samples are described in the following subsections.

Faculty. Potential faculty participants (N = 1,000) were solicited via an email list provided by the university’s Office of Institutional Research. The measure was administered to this sample using a well-known e-survey platform, Qualtrics (2017). Overall, the response rate was 21%, consistent with the response rates of previous survey research with faculty members (e.g., Brockelman & Scheyett, 2015). Of faculty respondents, 59% (n = 118) identified as female, 40% (n = 81) identified as male, 0.5% (n = 1) identified as “other gender,” and 0.5% (n = 1) did not specify their gender. The majority of participants, 81% (n = 162), identified as Caucasian or White, followed by African American, 4% (n = 8); Hispanic or Latinx, 4% (n = 8); Asian, 3% (n = 6); and multiethnic, 2% (n = 3); while 8% (n = 14) did not specify their ethnic background. Faculty members comprised a variety of different ranks, including adjunct instructor (29%, n = 59), lecturer (19%, n = 39), assistant professor (17%, n = 35), associate professor (18%, n = 37), and full professor (8%, n = 16), while 7.5% (n = 15) did not specify their rank.

Students. Data were collected from 312 undergraduate college students using a nonprobability sampling procedure. Over 34 days (four data collection sessions lasting 2.5 hours), the questionnaire was administered to students in the student union. These respondents ranged in ages from 18–51 (M = 21, SD = 5), with 95% of participants under the age of 29 at the time of data collection. Furthermore, 64% (n = 201) were females, 34% (n = 107) were males, 1% (n = 3) identified as “other gender,” and 0.3% (n = 1) did not specify their gender. The college generational status of these respondents was 37% (n = 116) first, 40% (n = 124) second, and 23% (n = 72) third and beyond. Ethnicities were distributed as follows: 48% (n = 150) African American, 30% (n = 95) Caucasian or White, 10% (n = 30) multiethnic, 6% (n = 19) Hispanic or Latinx, 4% (n = 12) Asian, 1% (n = 3) Native Hawaiian or Pacific Islander, and 0.3% (n = 1) American Indian or Alaska Native, while 0.6% (n = 2) did not report their ethnic identity.

Instrumentation and Procedures

The authors followed the instrument development guidelines discussed by experts in psychometrics and questionnaire design (DeVellis, 2016; Fowler, 2014). An initial set of 18 items was created on a Likert-type scale, ranging from 1 (strongly disagree) to 5 (strongly agree). As discussed above, the original theoretical framework of SDT (Ryan & Deci, 2000) and its contemporary extensions (Adams et al., 2017) guided the development of item content. Item content was also derived from major themes identified in the literature review (comfort, stigma, referrals, prevalence, and complexity), particularly those related to student and faculty members’ connection to college student mental health support (Bishop, 2016; Eisenberg et al., 2011; Lee et al., 2017). The following CMHPCS items, for example, reflect SDT (the positive association between one’s sense of competency and action) and the research findings that one’s sense of comfort with mental health disorders is associated with increased referrals to resources for mental health disorders: “I am comfortable talking to students about mental health”; “I am comfortable referring college students with mental health issues to the health center on campus”; “I am aware of the university resources for mental health”; and “Mental health issues are increasing among college students.” Negatively worded items were recoded so that higher scores would indicate higher perceived competence.

To obtain background information on the respondents, 11 demographic items were added to the questionnaire. These were developed in light of previous college counseling research that showed group differences (e.g., gender, ethnicity, previous attendance in counseling) on various mental health–related variables (Eisenberg et al., 2016; Kalkbrenner & Hernández, 2017). Sample items included the following: (1) Please select your gender; (2) Please specify your age (in years); and
(3) Indicate your ethnic identity.

The initial item pool was subjected to expert review and pilot testing to establish content validity. The items were sent to three expert reviewers with advanced training in clinical psychology, mental health counseling, and psychometrics. Their recommendations informed slight modifications to 15 items, improving their clarity and readability. A few additional items and formatting revisions were made based on pertinent feedback from pilot study participants (22 graduate students). For example, we clarified the meaning of “referred another student to counseling services” to “referred (recommended) that another student seek counseling services.”

Statistical Analyses

A series of statistical analyses were computed to answer the research questions, including exploratory factor analysis (EFA), confirmatory factor analysis (CFA), and hierarchical logistic regression (HLR). During phase 1 of the study using the faculty sample, a principal factor analysis (PFA) was conducted to determine the underlying latent factor structure of the CMHPCS (Mvududu & Sink, 2013). Given that the constructs related to SDT are generally correlated (Adams et al., 2017), the researchers used an oblique rotation (direct oblimin, ∆ = 0). The Kaiser criterion (eigenvalues [Λ] > 1), meaningful variance accounted for by each factor (≥ 5%), a review of the scree plot, and parallel analysis results guided the factor extraction process. Factor retention criteria were used based on the recommendations of Mvududu and Sink (2013): factor loadings > .40, commonalities (h²) > .30, and cross-loadings < .30. The content of items that loaded on each factor were reviewed for redundancy, as it is an accepted practice to remove an item that is highly correlated and conceptually similar to at least one other item (Byrne, 2016).

To cross-validate these initial factor analytic results, a CFA using a maximum likelihood estimation method was conducted to test the validity of the factor solution that emerged in the EFA with a sample of undergraduate students (research question 2). Using the recommendations of Byrne (2016), the following goodness-of-fit indices were reported: chi-square absolute fit index (CMIN), comparative fit index (CFI), root mean square error of approximation (RMSEA), standardized root mean square residual (SRMR), goodness-of-fit-index (GFI), and normed fit index (NFI).

Two HLR analyses were computed to examine the predictive validity of the CMHPCS for both faculty member and student participants (research question 3). Previous investigators found group demographic differences in college students’ willingness to utilize mental health services by age (Eisenberg et al., 2016) and their willingness to make peer-to-peer referrals to resources by gender (Kalkbrenner & Hernández, 2017). Based on these findings, gender and age were entered into the first regression model as predictor variables. Participants’ composite scores on the knowledge, fear, and engagement scales of the CMHPCS were entered into the second regression model as predictor variables. The criterion variable was participants’ referrals to the counseling center (1 = has not made a referral to the counseling center, or 2 = has made referrals to the counseling center).

Results

After screening the data, descriptive statistics were computed on the faculty and student samples to examine unusual or problematic response patterns, missing data, and the parametric nature of the item distributions. Missing values analyses revealed that less than 2% of data was absent from faculty participants and less than 1% of data was absent from student participants. Both data sets were winsorized and missing values were replaced with the series mean (Field, 2018). Skewness and kurtosis values for items were largely within the acceptable range of a normal distribution (absolute value < 1) for the sample of faculty members and the sample of students (see Table 1). The findings are presented in three phases of analyses that correspond to the three research questions, respectively.

Phase 1: Exploratory Factor Analysis

A PFA was conducted using the sample of faculty members to establish the initial dimensionality of the CMHPCS (research question 1). The inter-item correlation matrix revealed low-to-moderate correlations among items (r = .17 to r = .69). The Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO = .81) and Bartlett’s Test of Sphericity (B [153] = 1375.91, p < 0.001) provided further evidence that the data set was factorable. The oblique rotated PFA (direct oblimin, ∆ = 0) revealed a 5-factor solution based on the Kaiser criterion (Λ > 1.00). Seventy percent of the total variance in the correlation matrix was explained by these five factors. The scree plot, parallel analysis, and meaningful variance explained (at least 5% for each factor) that a 3-factor solution was the most parsimonious with the least evidence of cross-loadings (see Table 2). Five items displayed commonalities < .30 and were consequently removed from the analysis. The first factor accounted for 31.6% of the variance (Λ = 4.74), the second factor comprised 12.5% of the variance (Λ = 1.89), and the third factor accounted for 11.8% of the variance (Λ = 1.78).

Redundant items that were highly correlated, and thus conceptually interrelated, were deleted. The inter-item correlation matrix was reproduced and indicated that item 8 (“I am aware of resources in the community for mental health”) and item 15 (“I am aware of the university resources for mental health”) were statistically and conceptually similar, suggesting that these items were measuring the same construct. Item 8 was subsequently removed, as the content of item 15 was more closely related to mental health services on campus. The PFA was recomputed and a final 3-factor solution (see Table 2) comprised of 12 items was retained. These 12 items were renumbered in chronological order.

Table 1

Descriptive Statistics for Final Items

Note. Windsorized values (z-scores) are reported; faculty: SE_Kurtosis = 0.34, SE_Skewness = 0.17; students: SE_Kurtosis = 0.13, SE_Skewness = 0.20. Spinets of item content are provided based on the guidelines from the Publication Manual of the American Psychological Association, 6th edition. To access the full version of the scale, please contact the corresponding author.

The three emergent factors were named engagement, fear, and knowledge, respectively (see Table 2). The first factor, engagement, was comprised of items 3, 7, 8, 10 and 12. It estimates the degree to which a faculty member is involved with interacting, supporting, and working with students who are struggling with mental health disorders (e.g., item 7 [“I am comfortable referring college students with mental health issues to the health center on campus”] and item 8 [“I am comfortable talking to students about mental health”]). The second factor, fear, was comprised of items 4 and 9 and appraises one’s anxiety or concern surrounding mental health issues on college campuses (e.g., item 4 [“Students with mental health issues are dangerous”]). The last factor, knowledge, was marked by items 1, 2, 5, 6, and 11. These items reflect the extent to which the respondent was familiar with mental health issues on college campuses (e.g., item 4 [“Mental health issues are becoming more complex among college students”] and item 10 [“Mental health issues are increasing among college students”]).

Table 2

Principal Factor Analysis Results Using Oblique Rotation: Faculty Members (N = 201)

Note. Factor loadings over 0.40 appear in bold and mark the particular factor. Blank cells indicate factor loadings ≤ 0.10.
E = Engagement; F = Fear; K = Knowledge.

Item and internal consistency reliability analyses were computed for the three derived factors to partially answer research question 1. Adequate reliability coefficients were found for the overall measure (α = .81) and for each dimension: engagement (α = .84), fear (α = .83), and knowledge (α = .75). The low correlations between factors (engagement and fear, r = 0.09; engagement and knowledge,
r = 0.37; and fear and knowledge, r = 0.11) supported the discriminant validity of the measure.

Phase 2: Confirmatory Factor Analysis

To cross-validate the CMHPCS with a sample of undergraduate students, a CFA was computed (research question 2). The assumptions necessary for conducting a CFA were met (Byrne, 2016). Multicollinearity was not present, as bivariate correlations did not exceed an absolute value of 0.36. In addition, Mahalanobis d² indices revealed no extreme multivariate outliers. The standardized path model is depicted in Figure 1. It was not surprising that the CMIN absolute fit index was statistically significant due to the large sample size: χ²(51) = 1.97, p = .007. However, fit indices that are more appropriate for sample sizes larger than 200 revealed an adequate model fit. For example, the CFI = .96, RMSEA = .05, 90% CI [.04, .07], SRMR = .04, and GFI = .95. The path coefficients (see Figure 1) between the engagement and knowledge scales (.48) indicated a stronger relationship than the engagement and fear (.05) or fear and knowledge scales (.07). (These path coefficients are interpreted in the discussion section). Taken together, the CFA results produced a moderate-to-strong fit based on the guidelines from structural equation modeling researchers (Byrne, 2016). Reliability of the dimensions was re-examined with the student sample, yielding similar estimates to those found with faculty respondents. Internal consistency indices for the overall measure (α = .78) as well as for the three scales (engagement, α = .82; knowledge, α = .75; fear scale, α = .74) were adequate for an attitudinal questionnaire.

Phase 3: Hierarchical Logistic Regression Analyses

The guidelines for HLR assumption checking were followed (Field, 2018). Items were winsorized to remove extreme outliers. Skewness and kurtosis values (see Table 1) were largely within the acceptable range (± 1.00) for both samples. Pearson product correlations were computed between the independent variable scores, revealing no multicollinearity. Box and Tidwell’s (1962) procedure revealed that the assumption of linearity was met for both samples (i.e., the logit of the criterion variable was linearly related to all continuous predictor variables).

Figure 1. Confirmatory Factor Analysis Path Model for Undergraduate Student Sample (N = 312)

Faculty members. HLR analyses were computed to investigate the predictive validity of the CMHPCS (research question 3). Specifically, researchers aimed to determine the extent to which respondents’ scores on the CMHPCS predicted if they had made a referral to the counseling center. Among the sample of faculty members, the correct classification rate of the null model was 56%. The first model of gender and age was significant (χ² = 15.80, p < 0.001) and explained 11% (Nagelkerke R²) of the variance in participants’ referrals to the counseling center. There was a statistically significant increase in the odds (Exp(B) = 1.30) of female faculty members making a student referral to the counseling center. The second LR model revealed that adding the knowledge, fear, and engagement scales significantly improved the predictability of model (χ² = 46.61, p < 0.001) and explained 30% (Nagelkerke R²) of the variance in participants’ referrals to the counseling center. The engagement scale was a significant predictor of referrals to the counseling center. The odds ratio, Exp(B), revealed that an increase in one unit on the engagement scale was associated with an increase in the odds of making a referral to the counseling center by a factor of 3.47. The correct classification rate of this model was 71.2%.

 Undergraduate students. For the sample of undergraduate students, the correct classification rate of the null model was 58%. Gender and age were entered as predictor variables in the first regression block and revealed statistical significance (χ²(1) = 9.35, p = 0.01) and explained 4.2% (Nagelkerke R²) of the variance in participants’ referrals to the counseling center. A statistically significant increase in the odds emerged (Exp(B) = 1.78) for female students having made a peer-referral to the counseling center. In the second block, the knowledge, fear, and engagement subscales of the CMHPCS were added to the regression model. The addition of the CMHPCS scales as predictor variables significantly improved the model (χ²(1) = 29.82, p < 0.001) and explained 13% (Nagelkerke R2) of the variance in participants’ referrals to the counseling center. Similar to faculty members, the engagement scale was a significant predictor of students’ referrals to the counseling center. The odds ratio, Exp(B), revealed that an increase in one unit on the engagement scale was associated with an increase in the odds of having made a referral to the counseling center by a factor of 2.10.

Discussion

The results of three major analyses provided evidence that the construct—perceived competence for promoting college student mental health—and its dimensions were estimated adequately by the CMHPCS. Feedback from expert reviewers and pilot study participants showed initial support for the content validity of the measure. The findings from the PFA and CFA provided evidence for the factorial validity of the measure. The low correlations between factors provided further support for the relative distinctiveness (discriminant validity) of each dimension. Tests of internal consistency revealed adequate support for the reliability of the measure with college students and with faculty members.

The results of the HLR models demonstrated a moderate level of predictive validity of the CMHPCS. Similar to previous investigations (e.g., Kalkbrenner & Hernández, 2017), female students in the present study were more likely to make peer-to-peer referrals to the counseling center when compared to male students. Extending previous findings, the addition of participants’ scores on the CMHPCS scale as predictor variables significantly improved the logistic regression model’s capacity to predict the odds of making a referral to the counseling center. The CMHPCS appears to be measuring a construct that is associated with greater odds of both students and faculty members supporting college student mental health (i.e., making a referral to the counseling center). In particular, higher scores on the engagement scale emerged as a significant predictor of an increase in the odds of having made a student referral to the counseling center among both faculty members and undergraduate students.

This study introduced a new theoretical dimension, perceived competence for promoting college student mental health, to the growing body of literature on the utility of SDT for supporting college student mental health. The emergent factor structure of the CMHPCS was largely consistent with key elements of SDT (Adams et al., 2017). According to the theory, individuals’ motivation for engaging in an action or behavior will be enhanced when they feel a sense of competence or self-efficacy for the activity (Adams et al., 2017; Ryan & Deci, 2000). Similarly, the emergent factor of knowledge on the CMHPCS (i.e., the extent to which one is familiar or knowledgeable with mental health issues on campus) is consistent with research on the personal competency component of SDT. Weber and Koehler (2017), for example, found a moderate, positive association between respondents’ knowledge and perceived competence. Similarly, in the present study, knowledge emerged as a factor of perceived competence (i.e., one who is more knowledgeable about college student mental health has a higher level of perceived competence for supporting college student mental health). Autonomy and relatedness also are central components of SDT, as individuals’ intrinsic motivation is enhanced when their behaviors are active and self-determined (Adams et al., 2017; Jeno & Diseth, 2014). Finally, the engagement scale on the CMHPCS reflects the extent to which one is actively involved with supporting college student mental health. One who is more engaged with supporting college student mental health has a higher level of perceived competence for supporting college student mental health.

The relationship between the path coefficients (see Figure 1) provided further support that the CMHPCS is largely consistent with SDT. The path coefficients were stronger between the engagement and knowledge scales (0.48) than they were with the fear scale—0.05 and 0.07, respectively. According to the theory, intrinsic motivation toward wellness generally increases when individuals are competent (knowledgeable) and related (engaged) to a person or activity (Patrick & Williams, 2012). Thus, it was not surprising that the strongest association between the three factors (knowledge, fear, and engagement) emerged between the knowledge and engagement subscales. There are complex associations between fear and one’s level of motivation (Halkjelsvik & Rise, 2015). Some researchers demonstrated that higher levels of respondent fear were associated with higher levels of motivation (e.g., motivation for smoking cessation; Farrelly et al., 2012). However, in other investigations, anxiety elicited the opposite response in participants, substantially decreasing their motivation (Halkjelsvik & Rise, 2015). Considering the complex connection between motivation and fear, it is possible in the present study that participants’ fear of mental health issues on college campuses was associated with ambivalence in their engagement. Fear may motivate students to support a peer experiencing mental distress. In other situations, fear might lead to students avoiding a peer in mental distress. While future research is needed to investigate these issues, there is sufficient statistical (EFA and CFA) and conceptual evidence to retain the fear scale.

To summarize, the theoretical construct underlying CMHPCS, which was designed to measure  perceived competence toward promoting college student mental health, reflects aspects of SDT. Individuals with high levels of perceived competence for promoting college student mental health appear to be knowledgeable about, unfearful of, and engaged with supporting students who are living with mental health issues. At this stage of development, the CMHPCS has potential to enhance the practice of college counseling.

Implications for the Profession

Considering the rise in college counselors’ roles and responsibilities with outreach and consultation (Brunner et al., 2014; Sharkin, 2012), the CMHPCS can assist college counselors with these activities. Specifically, the CMHPCS can be used by college counselors to provide a baseline measure of perceived competence for promoting mental health on campus among students and faculty members. The questionnaire can be administered and scored as a holistic measure (total score), as an overall measure, or as three separate dimensions (subscales) of students and/or faculty members’ perceived competence for promoting mental health on campus. On a practical level, the CMHPCS has utility for college counselors when participating in new student and new faculty orientations due to the brevity (12 items) and versatility (use with faculty and student populations) of the measure. The results might provide college counselors with valuable baseline information on new students and faculty members’ perceived competence toward supporting college student mental health and aid in structuring the content of educational sessions for recognizing and referring students to the counseling center.

Brunner et al. (2014) identified supporting referral agents through consultation as another key aspect in the practice of college counseling. The findings presented above demonstrated that higher scores on the engagement scale predicted a greater likelihood in the odds of student referrals to the counseling center among both students and faculty members. This outcome can inform college counselors’ outreach and consultation efforts. Specifically, it is recommended that college counselors focus on increasing university community members’ knowledge and engagement with supporting college student mental health. Advocacy efforts can be directed toward implementing training sessions for faculty members and students for recognizing warning signs of mental health disorders in college students and connecting trainees to resources for mental health disorders. The CMHPCS can be used as a pretest/posttest measure to provide information about the extent to which trainings and mental health support resources are useful for promoting perceived competence for supporting college student mental health. For example, the REDFLAGS Model, an acronym of common warning signs of mental health disorders in college students (Kalkbrenner, 2016), and the National Suicide Prevention Lifeline’s wallet cards (National Suicide Prevention Lifeline, 2008) are resources for increasing university community members’ awareness of warning signs of mental health disorders in college students. The CMHPCS could be implemented to assess the value of these resources.

Limitations and Future Research

Although results of the current study were promising, the research caveats should be considered. First, self-report measures can sometimes generate response biases influenced by the respondent’s need for social desirability. Second, the 2-item fear scale is not ideal. Although dimensions composed of few items often generate lower reliability coefficients, there is no absolute threshold for the minimum number of items necessary to comprise a scale (Fowler, 2014). Given the CMHPCS’s stage of development, the researchers chose to retain the dimension. The strong reliability coefficient of the fear subscale (α = .83, student sample and α = .80, faculty sample) exceeded the threshold for acceptable internal consistency reliability. The overall scale is also stronger with the fear scale items included. Finally, it should be noted that other validated instruments in social sciences research have scales comprised of two items (Luecht, Madsen, Taugher, & Petterson, 1990), suggesting that the fear scale may be useful.

The demographic profile of faculty in our sample was consistent with the ethnic identities of the larger university and with a national sample of faculty members (Myers, 2016). However, the homogeneity of ethnicity among faculty participants still might have affected the generalizability of our findings. Most faculty participants (81%, n = 162) identified as Caucasian or White. It is recommended that future researchers confirm the factor structure of the CMHPCS with an ethnically diverse sample of faculty members. Subsequent investigation should examine the goodness-of-fit of the CMHPCS with different populations of college students and faculty members. Specifically, the following sub-groups of college students appear to be especially susceptible to mental health disorders: first-generation college students, community college students, students enrolled in Greek life organizations, international students, and male students (Dobmeier et al., 2013; Eisenberg et al., 2011).

Conclusion

The professional identity of college counselors has grown to include outreach and consultation with counseling referral agents as key components in the contemporary practice of college counseling (Brunner et al., 2014; Sharkin, 2012). The multidimensional aim of the present study was to establish the validity and reliability of the CMHPCS, a newly developed questionnaire designed to measure college student and faculty members’ perceived competence for promoting college student mental health. To do so, the measure was subjected to rigorous psychometric testing (EFA and CFA). A 3-factor model (knowledge, fear, and engagement) emerged from the data. Initial support for the reliability and factorial validity of the instrument was reported. A series of two HLR analyses reinforced, in part, the predictive validity of the measure. The brief nature of the CMHPCS coupled with its adequate reliability and coherent factor structure suggests the measure might have utility for supporting and enhancing the consultation and outreach activities of college counseling practitioners. For instance, the CMHPCS can be carefully utilized as a screening measure for students to enhance the practice (outreach, education, and consultation) of college counselors. The instrument also is perhaps useful as a pretest/posttest measure in outcome research aimed at assessing mental health support interventions among college students.

Conflict of Interest and Funding Disclosure

The authors reported no conflict of interest or funding contributions for the development of this manuscript.

References

Adams, N., Little T. D., & Ryan, R. M. (2017). Self-determination theory. In M. L. Wehmeyer, K. A. Shogren, T. D. Little, & S. J. Lopez (Eds.), Development of self-determination through the life-course (pp. 47–54). Dordrecht, Netherlands: Springer.

Albright, G., & Schwartz, V. (2017). Are campuses ready to support students in distress? Retrieved from https://www.jedfoundation.org/wp-content/uploads/2017/10/Kognito-JED-Are-Campuses-Ready-to-Support-Students-in-Distress.pdf

Bishop, K. K. (2016). The relationship between retention and college counseling for high-risk students. Journal of College Counseling, 19, 205–217. doi:10.1002/jocc.12044

Box, G. E. P., & Tidwell, P. W. (1962). Transformation of the independent variables. Technometrics, 4, 531–550. doi:10.2307/1266288

Brockelman, K. F., & Scheyett, A. M. (2015). Faculty perceptions of accommodations, strategies, and psychiatric advance directives for university students with mental illnesses. Psychiatric Rehabilitation Journal, 38, 342–348. doi:10.1037/prj0000143

Brunner, J. L., Wallace, D. L., Reymann, L. S., Sellers, J.-J., & McCabe, A. G. (2014). College counseling today: Contemporary students and how counseling centers meet their needs. Journal of College Student Psychotherapy, 28, 257–324. doi:10.1080/87568225.2014.948770

Byrne, B. M. (2016). Structural equation modeling with AMOS: Basic concepts, applications, and programming (3rd ed.). New York, NY: Routledge.

de Lijster, J. M., Dierckx, B., Utens, E. M. W. J., Verhulst, F. C., Zieldorff, C., Dieleman, G. C., & Legerstee,
J. S. (2017). The age of onset of anxiety disorders: A meta-analysis. The Canadian Journal of Psychiatry / La Revue Canadienne De Psychiatrie, 62, 237–246. doi:10.1177/0706743716640757

DeVellis, R. F. (2016). Scale development: Theory and applications (4th ed.). Thousand Oaks, CA: Sage.

Dobmeier, R. A., Kalkbrenner, M. T., Hill, T. T., & Hernández, T. J. (2013). Residential community college student awareness of mental health problems and resources. CSPA-NYS Journal, 13(2), 15–28. Retrieved from http://journals.canisius.edu/index.php/CSPANY/article/viewFile/331/500

Emery, A. A., Heath, N. L., & Mills, D. J. (2016). Basic psychological need satisfaction, emotion dysregulation, and non-suicidal self-injury engagement in young adults: An application of self-determination theory. Journal of Youth & Adolescence, 45, 612–623. doi:10.1007/s10964-015-0405

Eisenberg, D., Goldrick-Rabe, S., Lipson, S. K., & Broton, K. (2016). Too distressed to learn? Mental health among community college students. Retrieved from http://www.wihopelab.com/publications/Wisconsin_HOPE_Lab-Too_Distressed_To_Learn.pdf

Eisenberg, D., Hunt, J., Speer, N., & Zivin, K. (2011). Mental health service utilization among college students in the United States. Journal of Nervous and Mental Disease, 199, 301–308.
doi:10.1097/NMD.0b013e318217512

Farrelly, M. C., Duke, J. C., Davis, K. C., Nonnemaker, J. M., Kamyab, K., Willett, J. G., & Juster, H. R. (2012). Promotion of smoking cessation with emotional and/or graphic antismoking advertising. American Journal of Preventive Medicine, 43, 475–482. doi:10.1016/j.amepre.2012.07.023

Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191. doi:10.3758/BF03193146

Field, A. P. (2018). Discovering statistics using IBM SPSS Statistics (5th ed.). Thousand Oaks, CA: Sage.

Fowler, F. J. (2014). Survey research methods (5th ed.). Thousand Oaks, CA: Sage.

Halkjelsvik, T., & Rise, J. (2015). Disgust in fear appeal anti-smoking advertisements: The effects on attitudes and abstinence motivation. Drugs: Education, Prevention and Policy, 22, 362–369.
doi:10.3109/09687637.2015.1015491

Han, M., & Pong, H. (2015). Mental health help-seeking behaviors among Asian American community college
students: The effect of stigma, cultural barriers, and acculturation. Journal of College Student
Development, 56, 1–14. doi:10.1353/csd.2015.0001

Hill, R. M., Yaroslavsky, I., & Pettit, J. W. (2015). Enhancing depression screening to identify college students at
risk for persistent depressive symptoms. Journal of Affective Disorders, 174, 1–6.
doi:10.1016/j.jad.2014.11.025

Jeno, L. M., & Diseth, Å. (2014). A self-determination theory perspective on autonomy support, autonomous self-regulation, and perceived school performance. Reflecting Education, 9, 1–20.

Kalkbrenner, M. T. (2016). Recognizing and supporting students with mental disorders: The REDFLAGS
Model. Journal of Education and Training, 3, 1–13. doi:10.5296/jet.v3i1.8141

Kalkbrenner, M. T., & Hernández, T. J. (2017). Community college students’ awareness of risk factors for
mental health problems and referrals to facilitative and debilitative resources. The Community College
Journal of Research and Practice, 41, 56–64. doi:10.1080/10668926.2016.1179603

Kalkbrenner, M. T., James, C., & Pérez-Rojas, A. E. (2018). College students’ awareness of mental disorders and counselor seeking behaviors: Comparison across academic disciplines. Journal of College Student Psychotherapy. Manuscript submitted for publication.

Lee, H. J., Ju, Y. J., & Park, E.-C. (2017). Utilization of professional mental health services according to recognition rate of mental health centers. Psychiatry Research, 250, 204–209.
doi:10.1016/j.psychres.2017.01.051

Li, J., Marbley, A. F., Bradley, L. J., & Lan, W. (2016). Attitudes toward seeking professional counseling services among Chinese international students: Acculturation, ethnic identity, and English proficiency. Journal of Multicultural Counseling and Development, 44, 65–76. doi:10.1002/jmcd.12037

Luecht, R. M., Madsen, M. K., Taugher, M. P., & Petterson, B. J. (1990). Assessing professional perceptions: Design and validation of an interdisciplinary education perception scale. Journal of Allied Health, 19(2), 181–191.

Mancini, A. D. (2008). Self-determination theory: A framework for the recovery paradigm. Advances in Psychiatric Treatment, 14, 358–365. doi:10.1192/apt.bp.107.004036

Marsh, C. N., & Wilcoxon, S. A. (2015). Underutilization of mental health services among college students: An examination of system-related barriers. Journal of College Student Psychotherapy, 29, 227–243.
doi:10.1080/87568225.2015.1045783

Myers, B. (2016). Where are the minority professors? The Chronical of Higher Education. Retrieved from https://www.chronicle.com/interactives/where-are-the-minority-professors

Mvududu, N. H., & Sink, C. A. (2013). Factor analysis in counseling research and practice. Counseling Outcome Research and Evaluation, 4(2), 75–98. doi:10.1177/2150137813494766

National Suicide Prevention Lifeline. (2008). Having trouble coping? With help comes hope. Retrieved from https://store.samhsa.gov/product/National-Suicide-Prevention-Lifeline-Wallet-Card-Having-Trouble-Coping-With-Help-Comes-Hope-/SVP13-0155R

Ntoumanis, N. (2005). A prospective study of participation in optional school physical education using a self-determination theory framework. Journal of Educational Psychology, 97, 444–453.
doi:10.1037/0022-0663.97.3.444

Patrick, H., & Williams, G. C. (2012). Self-determination theory: Its application to health behavior and complementarity with motivational interviewing. The International Journal of Behavioral Nutrition and Physical Activity, 9, 1–12. doi:10.1186/1479-5868-9-18

Piltch, C. A. (2016). The role of self-determination in mental health recovery. Psychiatric Rehabilitation Journal, 39, 77–80. doi:10.1037/prj0000176

Qualtrics [Online survey platform software]. (2017). Provo, UT, USA. Retrieved from https://www.qualtrics.com/

Rosenthal, B. S., & Wilson, W. C. (2016). Psychosocial dynamics of college students’ use of mental health services. Journal of College Counseling, 19, 194–204. doi:10.1002/jocc.12043

Ryan, R. M., & Connell, J. P. (1989). Perceived locus of causality and internalization: Examining reasons for acting in two domains. Journal of Personality and Social Psychology, 57, 749–761.

Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55, 68–78. doi:10.1037/0003-066X.55.1.68

Ryan, R. M., Lynch, M. F., Vansteenkiste, M., & Deci, E. L. (2011). Motivation and autonomy in counseling, psychotherapy, and behavior change: A look at theory and practice. The Counseling Psychologist, 39(2), 193–260. doi: 10.1177/0011000009359313

Sharkin, B. S. (2012). Being a college counselor on today’s campus: Roles, contributions, and special challenges. New York, NY: Routledge Taylor & Francis Group.

Weber, M., & Koehler, C. (2017). Illusions of knowledge: Media exposure and citizens’ perceived political competence. International Journal of Communication, 11(2017), 2387–2410.

Wei, Y., McGrath, P. J., Hayden, J., & Kutcher, S. (2015). Mental health literacy measures evaluating knowledge, attitudes and help-seeking: A scoping review. BMC Psychiatry, 15, 1–20. doi:10.1186/s12888-015-0681-9

White, S., Park, Y. S., Israel, T., & Cordero, E. D. (2009). Longitudinal evaluation of peer health education on a college campus: Impact on health behaviors. Journal of American College Health, 57, 497–505.
doi:10.3200/JACH.57.5.497-506

Williams, A. E., & Greene, C. A. (2016). Creating change through connections: A group for college women experiencing alcohol-related consequences. Journal of Creativity in Mental Health, 11, 90–104.
doi:10.1080/15401383.2015.1110509

Williams, G. C., & Deci, E. L. (1996). Internalization of biopsychosocial values by medical students: A test of self-determination theory. Journal of Personality and Social Psychology, 70, 767–779.

Yorgason, J. B., Linville, D., & Zitzman, B. (2008). Mental health among college students: Do those who need services know about and use them? Journal of American College Health, 57(2), 173–182.
doi:10.3200/JACH.57.2.173-182

Michael T. Kalkbrenner, NCC, is an assistant professor at New Mexico State University. Christopher A. Sink, NCC, is a professor and Batten Chair at Old Dominion University. Correspondence can be addressed to Michael Kalkbrenner, 1780 E. University Ave., Las Cruces, NM 88003, mkalk001@nmsu.edu.