Serious Games in Support of School Consultation: An Examination of a Novel Approach to Behavior Intervention in Middle School Classrooms

In April 2024, my students Kelly Lojinger and Abby Miller delivered a poster presentation at the Research and Creative Activities Conference at ECU and the Southeastern School Behavioral Health Conference in Myrtle Beach, SC. The data used for that presentation were from a study of ATHEMOS the Game funded by research grants to East Carolina University and Ohio University (co-PI, Steven W. Evans) from the Institute of Education Sciences (R324A180219).

INTRODUCTION

Teachers receive limited behavior intervention training prior to entering the classroom (Freeman et al., 2014). And yet externalizing behavior problems, like Attention-Deficit Hyperactivity Disorder (ADHD), are common among school-age children. ADHD alone affects 8.7% to 9.8% of all students in the U.S. (Bitsko et al., 2022).

The Challenging Horizons Program - Mentorship Model (CHP-M) is an evidence-based intervention for secondary school students with ADHD. The goal is to train students in academic enabler skills, like organization, planning, and study skills, to reduce ADHD-related impairments (DuPaul et al., 2021). Although effective long-term, the CHP-M is often difficult to initiate and can take long periods before student benefits are realized (Evans et al., 2016).

Digital technologies have emerged to augment behavioral health care (Wang et al., 2023) and could potentially support programs like the CHP-M. To that end, the serious game ATHEMOS was developed to teach academic enabler skills and provide skill rehearsal in an engaging and rewarding game environment (Schultz et al., 2023). Screenshots are shown in Figures 1 and 2.

Figure 1. Screenshot of a space battle sequence in ATHEMOS. Players are challenged to collect “intel” about mysterious extraterrestrial robots that have appeared in our solar system for an unknown reason.

Figure 2. Screenshot of an interaction with a non-player character in ATHEMOS who helps the player understand the “intel” using organization, planning, and note-taking skills.

In the present study, we compared indicators of implementation quality across the novel ATHEMOS and traditional CHP-M conditions during a small pilot trial.

MATERIALS & METHODS

Thirty-one young adolescents with ADHD were recruited from middle schools in North Carolina and Ohio and randomly assigned to either the traditional CHP-M (n = 15) or an experimental approach combining the CHP-M with ATHEMOS (n = 16). Teacher mentors were then recruited for all participants and ongoing consultation was provided by graduate students for up to 24-weeks of intervention.

We then compared the two conditions on several indicators of implementation dose, quality, and user satisfaction using Bayesian t-tests in JASP 0.18.3.0

RESULTS

Despite a rolling recruitment strategy, intervention length across conditions (ATHEMOS M = 13.5 weeks; CHP-M M = 13.8 weeks) appeared roughly equivalent, Bayes factor (BF) = 0.35.
A comparison of teacher consultation sessions between the CHP-M and ATHEMOS conditions was inconclusive, BF = 1.14. On average, teacher mentors in the CHP-M participated in 5.47 sessions and teachers in ATHEMOS participated in 6.94 sessions, as depicted in Figure 3.
Teachers in the ATHEMOS condition spent credibly more time in consultation sessions than did teachers in the CHP-M condition, BF = 37.61. Total consultation time in the ATHEMOS condition (M = 124 mins) was longer than the CHP-M (M = 70 mins), as depicted in Figure 4.
A comparison of session quality, using a 4-point Likert-type scale created for the study and completed by consultants, weakly suggested group equivalence, BF = 0.36.
Similarly, a comparison of average teacher ratings on a 5-point Likert-type satisfaction scale at the end of the study weakly suggested group equivalence, BF = 0.35, as depicted in Figure 5.

Figure 3. Raincloud plot of total consultation sessions across the two conditions.

Figure 4. Raincloud plot of total minutes spent in consultation across the two conditions.

Figure 5. Raincloud plot of teacher satisfaction with the two conditions.

DISCUSSION

Combining a serious game (ATHEMOS) with an evidence-based treatment for ADHD (CHP-M) did not affect intervention dosage (i.e., treatment length, number of sessions), session quality, or teacher satisfaction, but it did increase total time in teacher consultation. Given that the two conditions led to similar student outcomes (Schultz et al., under review) and teacher satisfaction, the extra consultation time for ATHEMOS may have benefits. The novel ATHEMOS approach may also engage teachers in the intervention process in a way that traditional school consultation does not. Future analyses are needed to qualitatively examine the content of the consultation sessions to assess such possibilities. In short, ATHEMOS implementation resembles CHP-M implementation, but with longer consultation sessions—at least during initial cases.

Limitations

Our results should be interpreted with caution for several reasons. First, our sample size was small—ATHEMOS was only piloted with 15 students and teachers. Second, session quality ratings were provided by the consultants themselves, and those results may have been biased. Third, it is possible that boys and girls responded differently to ATHEMOS, and this influenced teacher engagement and satisfaction in a way that we did not assess. And finally, our present analysis only examines five quantitative indicators of implementation quality. A mixed-method analysis of ATHEMOS consultation transcripts would provide additional insight into our present results.

REFERENCES

Bitsko, R.H., Claussen, A.H., Lichstein, J., Black, L.I., Jones, S.E., Danielson, M.L., Hoenig, J.M., Davis Jack, S.P., Brody, D.J., Gyawali, S., Maenner, M.J., Warner, M., Holland, K.M., Perou, R., Crosby, A.E., Blumberg, S.J., Avenevoli, S., Kaminski, J.W., & Ghandour, R.M. (2022). Mental Health Surveillance Among Children - United States, 2013-2019. MMWR Suppl., 71(2), 1-42. http://dx.doi.org/10.15585/mmwr.su7102a1

DuPaul, G.J., Evans, S.W., Owens, J.S., Cleminshaw, C.L., Kipperman, K., Fu, Q., & Benson, K. (2021). School-based intervention for adolescents with attention-deficit/hyperactivity disorder: Effects on academic functioning. Journal of School Psychology, 87, 48-63. https://doi.org/10.1016/j.jsp.2021.07.001

Evans, S.W., Langberg, J.M., Schultz, B.K., Vaughn, A., Altaye, M., Marshall, S.A. & Zoromski, A.K., (2016). Evaluation of a school-based treatment program for young adolescents with ADHD. Journal of Consulting and Clinical Psychology, 84, 15-30. https://psycnet.apa.org/doi/10.1037/ccp0000057

Freeman, J., Simonsen, B., Briere, D. E., & MacSuga-Gage, A. S., (2014). Pre-service teacher training in classroom management: A review of state accreditation policy and teacher preparation programs. Teacher Educational and Special Education, 37, 106–120. https://doi.org/10.1177/0888406413507002

Schultz, B.K., Evans, S.W., Bowditch, J., Carter, K., Rogers, E.E., Donelan, J., & Dembowski, A. (2023). Acceptability and playability of an organization training videogame for young adolescents with ADHD: The development of ATHEMOS. PLOS Digital Health, 2(11), e0000374. https://doi.org/10.1371/journal.pdig.0000374

Schultz, B.K., Evans, S.W., Schoemann, A.M., & Rogers, E. (under review). A Pilot Study of a Game-supported Organization and Planning Skills Intervention for Young Adolescents with ADHD. Journal of School Psychology.

Wang, C., Lee, C., & Shin, H. (2023). Digital therapeutics from bench to bedside. NPJ Digital Medicine, 6(38). https://doi.org/10.1038/s41746-023-00777-z