We evaluated intervention processes in the CLP during the 2019-2020 school year. The purpose of this study was to identify meaningful and important session interactions and events from the perspectives of the youth who experience them. In other words, we wanted to address the question, what intervention processes mattered to these young people? Specifically, we aimed to identify the categories of meaning and importance that the mentees used to construct their experiences of what happened in the CLP intervention.
Participants and Procedure
Participants were from the second year of CLP implementation. They were low-income, culturally diverse, and at-risk adolescents (14 to 18 years old) aspiring to be first in their families to go to college. As noted, CLP-M participants met in small groups with a college student mentor 45 minutes to one hour each week for two school semesters. After each session, participants assessed the impact of the session using the Session Evaluation Form (Bussell & Kurtines, 1999). The Session Evaluation Form, which was adapted for an adolescent population from Elliott and Wexler’s (1994) Session Impact Scale consists of ten items and four subscales: Group Impact (four items referring to felt group cohesion and connectedness), Facilitator Impact (two items referring to felt connectedness with the facilitator), Skills Impact (two items referring to learning useful and relevant skills), and Exploration Impact (two items referring to talking about alternative points of view). All items were rated on a 5-point Likert scale, from 1 (not atall) to 5 (very much).
Participants were from the second year of CLP implementation. They were low-income, culturally diverse, and at-risk adolescents (14 to 18 years old) aspiring to be first in their families to go to college. As noted, CLP-M participants met in small groups with a college student mentor 45 minutes to one hour each week for two school semesters. After each session, participants assessed the impact of the session using the Session Evaluation Form (Bussell & Kurtines, 1999). The Session Evaluation Form, which was adapted for an adolescent population from Elliott and Wexler’s (1994) Session Impact Scale consists of ten items and four subscales: Group Impact (four items referring to felt group cohesion and connectedness), Facilitator Impact (two items referring to felt connectedness with the facilitator), Skills Impact (two items referring to learning useful and relevant skills), and Exploration Impact (two items referring to talking about alternative points of view). All items were rated on a 5-point Likert scale, from 1 (not atall) to 5 (very much).
A multilevel (454 sessions nested within 97 mentees) confirmatory factor analysis of the Session Evaluation Form provided support for a three-factor model at the session level and a one-factor model at the mentee level. At the session level, Group Impact and Facilitator Impact were combined into a single factor: Group and Facilitator Impact. Composite reliabilities for the session factors ranged from ω = .90 to ω = .97. However, the factor correlations ranged from r = .89 to r = .93, suggesting a lack of differentiation among these session factors.
At the end of each session, participants also completed the Session Evaluation Form-Qualitative Extension which asked, “What was meaningful or important about today’s session?” This open-ended question elicits from the mentees a narrative description of their experience in the mentoring session. We analyzed a subset of 55 randomly selected responses using Relational Data Analysis, a practical approach to developing program theory—and updating it—from what participants tell us (Kurtines et al., 2008b).
What is Relational Data Analysis?
Relational Data Analysis uses methods associated with grounded theory but extends these methods to include the systematic alternation between theory-neutral and theory-laden panels of coders as analysis moves from open coding to classification coding. In open coding, coders work together to create categories from the raw response data. In classification coding, coders work independently to assign raw response data to an existing system of response categories. The difference between theory-neutral and theory-laden panels of coders is that theory-neutral coders do not share a commitment to a theoretical framework and instead use ordinary (non-specialized) language to create categories, whereas theory-laden coders do share a commitment to a theoretical framework and use a shared theoretical language to create categories.
Relational Data Analysis uses methods associated with grounded theory but extends these methods to include the systematic alternation between theory-neutral and theory-laden panels of coders as analysis moves from open coding to classification coding. In open coding, coders work together to create categories from the raw response data. In classification coding, coders work independently to assign raw response data to an existing system of response categories. The difference between theory-neutral and theory-laden panels of coders is that theory-neutral coders do not share a commitment to a theoretical framework and instead use ordinary (non-specialized) language to create categories, whereas theory-laden coders do share a commitment to a theoretical framework and use a shared theoretical language to create categories.
In the initial Conceptual Open Coding phase, a panel of theory-neutral coders apply the method of constant comparison to the raw response data to create the largest possible set of ordinary language categories. Coders compare each response to all other responses and consensually group responses that share a property that they do not share with other responses. Their task is to work together to break the data down into the largest possible set of basic properties that most people would perceive as distinct. They then use these properties to create ordinary language categories and category descriptions. The result of Conceptual Open Coding is that each identified ordinary language category represents the most basic conceptually meaningful unit of content from an ordinary language perspective.
In the Theoretical Open Coding phase, a panel of theory-laden coders apply the method of constant comparison to the ordinary language categories and re-organize them into theoretical categories. They compare each category to all other categories. They consensually group together categories that share a common theoretically meaningful property and break apart categories that do not share a common theoretically meaningful property. In other words, they reverse the Conceptual Open Coding process by recombining the ordinary language categories into the smallest number of theoretically meaningful categories in the dataset that users of their theoretical language would perceive as distinct. Coders also specify subcategories that are nested within categories, categories that have non-nested (flat) relations, and theorized processes of categorical change. The result of Theoretical Open Coding is a theorized structural organization model of response categories and their temporal relations.
In the Theoretical Classification Coding phase, the theorized structural organizational model is used to assign response data to categories. A new panel of theory-neutral coders independently classify the response data according to the structural organizational model. The result of the Theoretical Classification Coding phase is a set of independently coded data from the theory-laden coders and another set of independently coded data from the theory-neutral coders. These sets of coded response data will then be used in the psychometric evaluation of the structural organization model. Specifically, reliability is estimated by the interrater reliability among theory-neutral coders, and validity is estimated by the correlation between classifications by theory-neutral and theory-laden coders.
Results
Conceptual Open Coding
In the Conceptual Open Coding phase, coders identified 12 ordinary language categories. These categories, category descriptions, and example excerpts are presented in Table 9. A second conceptual open coding with a new group of theory-neutral coders identified highly similar content categories, suggesting that the identified content categories were not idiosyncratic to the initial group of coders. Category labels provided by the second group of coders is in parentheses.
Conceptual Open Coding
In the Conceptual Open Coding phase, coders identified 12 ordinary language categories. These categories, category descriptions, and example excerpts are presented in Table 9. A second conceptual open coding with a new group of theory-neutral coders identified highly similar content categories, suggesting that the identified content categories were not idiosyncratic to the initial group of coders. Category labels provided by the second group of coders is in parentheses.
▪️
Table 9. Conceptual Open Coding: Content Category Descriptions
- Activities (Activities). The responses in this category share a description of participating in an activity with the group and or mentor. The described activities include but not limited to card games, sports, and art related. The responses in this category differ from the other responses because they specifically mention participating in a game or activity. these responses specifically reference uncertainty. Excerpt: “I liked drawing.”
- Connection (Relationship Building). The responses in this category share a description of bonding and or connecting with a group. The responses in this category differ from the other responses because the specially mention bonding/connecting with the group during their sessions. Excerpt: “the most meaningful & important thing about today is how we connected.”
- Getting Acquainted (Got to Know Each Other). The responses in this category share a description of getting to know the other members of their group. The responses in this category differ from the other responses because the responses specially mention getting to know the members through talking or an activity. Further, these responses are different from the responses in the connection group because the responses fail to explicitly mention a connection or bond was being made. Excerpt: “We got to know and understand each other.”
- Having Fun (Fun). The responses in this category share a description of having fun and enjoyment. The responses in this category differ from the other responses because the responses specifically mention having fun and laughing out of enjoyment during the session. Excerpt: “We just had fun and talked.”
- Communication (Talking and Communication). The responses in this category share a description being able to communicate with the mentor or group members. Specifically, the responses describe being about to talk about personal matters. responses in this category differ from the other responses because the responses mention being able to communicate with the mentor and or group members. Excerpt: “Good I had someone to talk to about certain things.”
- Active Listening (Guidance). The responses in this category share a description of how the mentor was actively listening. Specifically, the responses describe feeling encouraged, understood, and being heard because of how the mentor was listening. The responses in this category differ from the other responses because the responses described the mentor listening. Excerpt: “The most meaningful thing about today’s session is that ms. [Mentor] understood everything that I talked about, she told me that I am on the right track with my mindset. Also she encouraged me to continue doing what I am doing because it is working.”
- Mentor’s Likability (Mentor). The responses in this category share a description of what the mentor is like. Specifically, the responses describe something about the mentor such as their personality, kindness, and likability. The responses in this category differ from the other responses because the focus is on the mentor. Excerpt: “I really enjoyed today’s session because from the get-go, my mentor was nice & kind she gave the group a comfortable feeling that allowed for us to have an open discussion. Although this is just the beginning, I am hopeful for the potential in this mentorship/relationship.”
- Tutoring (Worked on Something). The responses in this category share a description of how mentor helped or attempted to help with mentees class assignments. The responses in this category differ from the other responses because the responses mention receiving academic help from the mentor. Excerpt: “Helping me with pre cal test I have 8th period. Learning to understand the concept.”
- Learning Skills (Learning). The responses in this category share a description of learning about or working on something. Specifically, the responses describe learning and or working on a skill such as self-worth, avoiding procrastination, sticking to a career, etc. The responses in this category differ from the other responses because the responses only mention learning or working on a skill. Excerpt: “We learned about a fun topic and useful for our relationship life.”
- Receiving Advice (Advice). The responses in this category share a description of receiving advice for various reason. The advice described was for future education and mental health resources. The responses in this category differ from the other responses because these responses specifically mention receiving advice from mentor. Excerpt: “I liked the application info and scholarship advice, which I will use in the future.”
- Relief (Decompression). The responses in this category share a description of feeling better after the session. The descriptions mention appreciation for sessions being helpful in providing relief from a situation. The responses in this category differ from the other responses because the responses express feeling relieved after the session. Excerpt: “It was very helpful, venting wise.”
- Lack of Elaboration (Uncodable). The responses in this category share a lack of description that makes them uncodable.
Theoretical Open Coding
In the Theoretical Open Coding phase, theory-laden coders used the theoretically evolving activities in mentoring (TEAM) framework (Karcher & Nakkula, 2010) to re-organize the 12 content categories into theoretical categories. The TEAM framework identifies two distinct types of mentoring interactions: goal-directed and relational. Goal-directed interactions focus on outcomes related to school performance, behavior, and peer relationships, whereas relational interactions focus on building and sustaining the bond between mentor and mentee.
In the Theoretical Open Coding phase, theory-laden coders used the theoretically evolving activities in mentoring (TEAM) framework (Karcher & Nakkula, 2010) to re-organize the 12 content categories into theoretical categories. The TEAM framework identifies two distinct types of mentoring interactions: goal-directed and relational. Goal-directed interactions focus on outcomes related to school performance, behavior, and peer relationships, whereas relational interactions focus on building and sustaining the bond between mentor and mentee.
Using the TEAM framework to re-organize the content categories resulted in two superordinate theoretical categories: Unfocused and Focused. Two subcategories of Focused responses were identified. Responses with a Relational Focus described the meaning and importance of relational processes, whereas those with a Goal-Oriented Focus described the meaning and importance of the content covered in the sessions. Unfocused responses did not refer to either relational processes or content.
Among the responses with a Relational Focus, three sub-subcategories of responses were identified: those that described the context provided by the mentor (Person-Centered Support), those that described the nature of the interaction among group members (Mutuality in Communication), and those that described the growing connection with others (Relational Bonding). Three sub-subcategories were also identified among the responses with a Goal-Oriented Focus: Receiving Advice, Learning (e.g., new skills), and Accomplishing Tasks (e.g., schoolwork). Overall, most responses (56%) had a Relational Focus. Smaller percentages had a Goal-Oriented Focus (22%) or were Unfocused (22%). Descriptions of these theoretical categories are provided in Table 10.
▪️
Table 10. Theoretical Open Coding: Theoretical Category Descriptions
-
Relational Focus: The key property of the Relational Focus category is that the responses refer to how the mentee related to their mentor and/or group within a single session. The coders agreed that this property made it distinct from the Goal-Oriented Focus category because the responses exclusively go into “how” this occurred rather than “what” occurred. The Relational Focus category includes four subcategories: Person-Centered Support, Mutuality in Communication,Relational Bonding, and Mixed Relational.
- Person-Centered Support: Responses in this category show evidence of the mentor’s authenticity, understanding, and/or acceptance of the mentee. Specifically, responses describe characteristics of or techniques used by the mentor that elicited feelings of ease, comfort, being heard, understood, or encouraged. They have the unique property of referring to the creation of conditions that give rise to mutuality in communication.
- Mutuality in Communication: Responses in this category demonstrate two-way, interpersonal communication between the mentee and the mentor or the mentee and the group. Specifically, responses indicate an invitation on behalf of the mentor for the mentee to “talk with” him or her, or each other, as opposed to being “talked to.” They have the unique property of referring to two-way, “talking with” communication in the session, which induces relational bonding.
- Relational Bonding: Responses in this category mention a growing connection or bond between the mentee and the mentor or the mentee and the group. The unique property of these responses is a reference to a connection or bond among two or more people as the result of shared experience in mentoring.
- Mixed Relational: Responses in this category are a combination of any two or all three Relational Focus subcategories.
-
Goal-Oriented Focus: The key property of the Goal-Oriented Focus category is that the responses refer to something the mentee acquired or achieved within a single session. The coders agreed that this property made it distinct from the Unfocused category because the Goal-Oriented responses mention an outcome with a practical or conventional purpose, whereas Unfocused responses do not. Goal-Oriented responses are distinct from Relational responses because Goal-Oriented responses do not meet any Relational Focus criteria. The Goal-Oriented Focus category includes four subcategories: Receiving Advice, Learning, Accomplishing Tasks, and Mixed Goal-Oriented.
- Receiving Advice: Responses in this category discuss the mentor’s guidance or recommendations to the mentee. Specifically, responses describe tips, information, or resources the mentor offered that the mentee acknowledged.
- Learning: Responses in this category discuss psychoeducational topics, strategies, or skills the mentee learned about or worked on. Specifically, responses indicate that the mentee gained an understanding of or developed their ability to deal effectively with challenges.
- Accomplishing Tasks: Responses in this category refer to important tasks or discuss class assignments the mentee learned about or worked on.
- Mixed Goal-Oriented: Responses in this category are a combination of any two or all three Goal-Oriented Focus subcategories.
- Unfocused: The key property of the Unfocused category is that the responses do not meet the criteria for either a Relational or Goal-Oriented Focus.
Non-content properties
-
Valences
- Positive: Reflects the valence of a single session with regard to the type of feelings or behaviors evoked; applies to all other categories. Responses in this category possess a positive valence and indicate the session was a “good” experience for the mentee.
- Negative: Reflects the valence of a single session with regard to the type of feelings or behaviors evoked; applies to all other categories. Responses in this category possess a negative valence and indicate the session was a “bad” experience for the mentee.
- Neutral: Reflects the valence of a single session with regard to the type of feelings or behaviors evoked; applies to all other categories. Responses in this category possess neither a positive nor negative valence and do not indicate that the session was either a “good” or “bad” experience for the mentee.
Figure 1 presents a tree chart depicting the hierarchical structural organization of theoretical categories. That is, the tree chart presents a theoretical model of the categories of meaning and importance that the mentees used to construct their experiences of what happened in the CLP intervention in response to the question, “What was meaningful or important about today’s session?” The root construct is “Mentoring Session” which is represented at Level 0 of the tree chart. At Level 1, there are two subcategories, “Unfocused” and “Focused.” The “T ->” signifies a temporal relation such that “Focused” sessions are theorized to be more developmentally advanced than “Unfocused” sessions. That is, change from Unfocused to Focused is theorized to constitute progressive change in the intervention process experienced by the participant. The “Focused” subcategory has two further levels of subcategories. At Level 3, the relation between “Goal-Oriented Focus” and “Relational Focus” is theorized to be flat (nonhierarchical), as are the relations among the subcategories at Level 4. No temporal relations are theorized among these subcategories.