Examining the Psychometric Properties of a Wellness Behavior Rating Scale

My student, Karlie Mirabelli, presented the results of her psychometric analysis of the Physical Activity, Nutrition, and Technology Survey (PANT), created by the designers of the MATCH Wellness Program. Special thanks to Tim Hardison, Suzanne Lazorick, and Kelsey Ross Dew of the MATCH Wellness research team and Alexander M. Schoemann of ECU for their support and guidance on this project.

Introduction

The risks of obesity are well documented (Nguyen, Nguyen, Lane, & Wang, 2011), with an increasing recognition of obesity-related psychosocial implications, including depression, low quality of life, and low self-esteem (Moreno, Johnson-Shelton, & Boles, 2013). To prevent such outcomes, various school-wide obesity prevention programs have been initiated across the nation. A central challenge is outcome monitoring, which can be intrusive and costly. In response, program evaluators often rely on child self-report measures, but there are few options with acceptable psychometric properties (Glasgow et al., 2005).

The purpose of this study is to provide the first psychometric analysis of the Physical Activity, Nutrition, and Technology Survey (PANT; Education Wellness Consulting, 2010), a self-report measure of weight management strategies across the domains of physical activity, nutritional choices, and technology use. The PANT was developed for use in the Motivating Adolescents with Technology to CHOOSE Health wellness program (MATCH Wellness)—a 14- to 16-week universal intervention for seventh grade students (Lazorick, et al., 2014). The current study (a) examines the construct validity of the PANT survey using exploratory and confirmatory factor analysis; (b) establishes the reliability of the resulting factor structure; and (c) examines construct validity by estimating the relationship between instrument scores and body mass index (BMI).

Participants

Participants in the current study were a subset of MATCH Wellness participants in North Carolina who completed the PANT survey online (n = 2,283). All participants were in seventh grade, with an average age of 12-years, 8-months (SD = 6.2 months). There was a nearly equal distribution of girls (49.4%) and boys (46.9%), with data missing in some instances (3.7%).1 The sample was roughly representative of the racial diversity of the region, with 48.1% of the sample identifying as white, 28.8% as African American, and 13.7% identifying as Hispanic. Based on average weight in pounds (M = 129.4; SD = 40.5) and height in inches (M = 62.5; SD = 3.6), the average BMI of the sample was 23.0 (SD = 6.1).

Measures

Physical Activity, Nutrition, and Technology Survey (PANT; Education Wellness Consulting, 2010). The PANT Survey is a 20-item scale that is intended to assess weight management strategies across three domains: (a) physical activity; (b) nutritional choices; and (c) technology usage. Each item on the PANT survey uses a 5-point Likert-type response format, with a response of 1 indicating a Poor representation, and a response of 5 indicating a Great representation of that behavior. Body Mass Index (BMI). BMI was calculated from the height and weight measurements. The school nurse or a trained researcher collected height and weight measures. A Schorr stadiometer was used for all height measures. The formula for BMI was: Weight (lbs) / Height2 (in.) X 703.

Method

Exploratory Factor Analysis (EFA). An EFA was conducted on a randomly selected half of the data (n = 1,107). Using SPSS, the Kaiser–Meyer–Olkin (KMO) and Bartlett’s test of sphericity (BTS) were conducted a priori. An Ordinary Least Squares (OLS) estimation for the EFA was conducted with an oblique rotation using the psych package in R statistical software (R Core Team, 2016; Revelle, 2018). Conclusions were driven by an iterative process based on results from a scree plot, parallel analysis, the MAP criterion, and the interpretability of results. Confirmatory Factor Analysis (CFA). The CFA was conducted on the remaining half of the data (n = 1,176) using the lavaan package in R statistical software (R Core Team, 2016; Rosseel, 2012). Results were interpreted based on the magnitude of factor loadings and several fit indices including the Root mean square error of approximation (RMSEA), the Comparative fit index (CFI) the Tucker-Lewis index (TLI), and the standardized root mean-square residual (SRMR). Reliability. Reliability was computed for the selected factors and their items. For the CFA dataset, alpha and omega were computed through the reliability function using the lavaan package. In the EFA dataset, alpha was computed using SPSS, and omega was computed through the omega function using the psych package (R Core Team, 2016; Revelle, 2018; Rosseel, 2012). Criterion-related Validity. The correlation between the PANT survey factors and participant BMI was conducted using the Pearson Product Moment Correlation Coefficient (two-tailed) in SPSS, where missing data was accounted for using a pairwise deletion method. Correlations were also conducted on BMI and those items removed from the final factor structure. A meaningful proportion (9.4%) of the Pre-BMI data were missing; there was no missing data on either of the factor composites. Results

EFA. Based on the MAP analysis and scree plot, a two-factor solution emerged as the most interpretable solution. Factor one consists of six items, which appear to target physical activity. Factor two consists of 12 items measuring healthy choices. Two items failed to load on either factor due to either falling below the cut off (0.32) or failing to load on the conceptually consistent factor (Table 1). CFA. Results of the standardized estimates for the CFA indicate that the comparative fit index (CFI = 0.92) demonstrates good fit. The Tucker-Lewis index (TLI = 0.91) shows that the model meets criterion to be considered as having good fit. The Root mean square error of approximation (RMSEA) is an indication of how well unknown parameter estimates of the model would fit the covariance matrix of the population. Results show that the RMSEA (RMSEA = 0.06, 90% CI: 0.06 – 0.07, p < 0.001) meets general criteria to be considered as having a close fit (West, Taylor, & Wu, 2012). The standardized root mean-square residual (SRMR = 0.05) also demonstrated good fit. The standardized factor loadings ranged from 0.59 to 0.95, indicating sufficient factor loadings (Table 2). The correlation between the factors was 0.62, indicating a moderate positive relationship. Reliability. Based on the results of the CFA, factor one has an acceptable reliability, α = 0.78 (N = 1176), 𝜔 = 0.86. Factor two has a high reliability, α = 0.86 (N = 1176), 𝜔 = 0.86. Criterion-related Validity. Results of a two-tailed correlation show that factor one (M = 3.19, SD = 0.95) had a significant negative correlation with pre-BMI (M = 23.03, SD = 6.05), r(2040) = -0.12, p < 0.001. However, this is a weak relationship. The relationship between the factor two composite (M = 2.85, SD = 0.83) and pre-BMI was inconclusive, r(2040) = 0.01, p = 0.52. Table 1. Exploratory Factor Analysis Results for a Two-Factor Solution (n = 1,107)

Table 2. Confirmatory Factor Analysis Standardized Factor Loadings for a Two-Factor Solution (n = 1,176)

Discussion

Our results suggest that the PANT survey is comprised of two factors measuring healthy choices and physical activity, resulting in a 18-item total scale. Results were replicated in a CFA, which demonstrated good fit on several indices. Both factors demonstrated acceptable reliability. With some proposed revisions, the PANT survey shows promise for being used as a two-factor outcomes tracking tool. However, some items appear problematic (e.g., jargon). It would be beneficial to rewrite these items to improve specificity within the intended domain, as well as improving conciseness and clarity. Moreover, the scale does not include a set timeframe of reference in the directions (e.g., weeks, months) and there is no conceptual anchor for the middle of the response format—both issues can introduce ambiguity (Nadler, Weston, & Voyles, 2015). It is also noteworthy that we did not find criterion-related validity in terms of BMI; however, BMI is not the only indicator that may benefit the clinical utility of this scale. Future studies might consider examining the relationship to other established self-report wellness surveys or physical fitness indicators such as mile times or pacer test times. There are some limitations to our present analysis. First, we did not specifically examine socioeconomic status (SES), which can influence access to high-quality nutritious food and physical activity opportunities (Pfingst, 2010). Second, our analysis was conducted on only a subset of the MATCH participants who took the PANT survey through an online system, which might bias the data toward participants with relatively high socioeconomic status. Future studies of the PANT would benefit by accounting for the socioeconomic status of respondents.

Conclusion

Obesity prevention is a crucial area of research and intervention due to the various health-related and psychosocial implications. But evaluating and tracking intervention outcomes remains a critical area for development. The current study evaluated the psychometric properties of the PANT survey, a previously unevaluated survey aimed at tracking and monitoring the progress of middle school-aged children across the areas of physical activity, nutrition, and technology usage. Based on our analysis, the PANT survey appears to be comprised of two factors—healthy choices and physical activity—that appear acceptably reliable. Despite demonstrating promise, the PANT will need to undergo meaningful revisions to improve item clarity and criterion-related validity.

References

Education Wellness Consulting. (2010). MATCH wellness: Motivating adolescents with technology to CHOOSE health. Jamesville, NC: Author.

Glasgow, R. E., Ory, M. G., Klesges, L. M., Cifuentes, M., Fernald, D. H., & Green, L. A. (2005). Practical and relevant self-report measures of patient health behaviors for primary care research. Annals of Family Medicine, 3, 73–81. doi: 10.1370/afm.261

Lazorick, S., Crawford, Y., Gilbird, A., Fang, X., Burr, V., Moore, V., & Hardison, G. T. (2014). Long-term obesity prevention and the motivating adolescents with technology to CHOOSE health™ program. Childhood Obesity, 10, 25-33. doi:10.1089/chi.2013.0049

Moreno, G., Johnson‐Shelton, D., & Boles, S. (2013). Prevalence and prediction of overweight and obesity among elementary school students. Journal of School Health, 83, 157-163. doi:10.1111/josh.12011

Nadler, T., Weston, R., & Voyles, C. (2015). Stuck in the middle: The use and interpretation of mid-points in items on questionnaires. The Journal of General Psychology, 142, 71–89.

Nguyen, N. T., Nguyen, X. T., Lane, J., & Wang, P. (2011). Relationship between obesity and diabetes in a US adult population: Findings from the national health and nutrition examination survey, 1999–2006. Obesity Surgery, 21, 351-355. doi:10.1007/s11695-010-0335-4

Pfingst, L. (2010). Weighing in on inequality: Obesity among low and high SES children (Doctoral Dissertation) Retrieved from ProQuest Dissertations & Theses Global (Order No. 3431655).

R Core Team (2016). R: A language and environment for statistical computing. R Foundation for Statistical Computing: Vienna, Austria.

Revelle, W. (2018) Psych: Procedures for personality and psychological research. Northwestern University: Evanston, Illinois. Retrieved from: https://CRAN.R-project.org/package=psych Version = 1.8.4.

Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48, 1-36. Retrieved from: http://www.jstatsoft.org/v48/i02/

West, S. G., Taylor, A. B., & Wu, W. (2012). Model fit and model selection in structural equation modeling. In R. H. Hoyle (Ed.), Handbook of Structural Equation Modeling. New York: Guilford.