App-Based Studies of Physical Activity May Lack Real-World Efficacy

Existing studies of app-based interventions to promote physical activity contain important limitations, according to a systematic review and meta-analysis recently published in JMIR mHealth and uHealth.

A potential reason is that research generally centers on the reliability and accuracy of specific outcomes, the authors explained. This approach, called an explanatory study, explores whether an intervention is beneficial under ideal, thoroughly controlled circumstances. Conversely, pragmatic studiesdetermine the effect of an intervention under more realistic, real-world conditions.

Even so, studies are not strictly dichotomous in their design, the authors wrote. Rather, they are “situated along the explanatory-pragmatic continuum.”

“The challenge is to strike a balance between a highly effective program and whether it can be integrated into practice settings,” they said.

The primary aim of this systematic review and meta-analysis was to analyze the degree to which app-based studies reported characteristics necessary to support their generalizability and applicability in real-world settings and to assess the explanatory vs pragmatic nature of existing studies. A secondary aim considered the association between study design characteristics and the observed effect sizes on participants’ physical activity.

Studies were limited to app-based physical activity interventions published in a peer-reviewed journal between January 2012 and April 2020. Eligibility was limited to studies using device-based physical activity measures to collect data. Seventeen studies from 10 countries met inclusion criteria. Ninety-five percent of the studies used a randomized controlled study design.

The study population included persons of all ages who participated in a physical activity program conducted in a health promotion or preventive care setting. Mean (SD) age was 39.6 (6.5) years, and 48.8% were male patients. Most studies used a randomized controlled study design. Duration varied from 2 weeks to 6 months.

The researchers used Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM), a framework that reports study characteristics across 5 dimensions. Afterwards, they applied the Pragmatic-Explanatory Continuum Indicator Summery (PRECIS-2) framework to explore the pragmatic vs explanatory nature of each study.

In accordance with previous research, PRECIS-2 scores above 3.5 were considered primarily pragmatic; values between 2.5 and 0.5 were deemed equally pragmatic and explanatory; and scores below 2.5 were rated as primarily explanatory.

The researchers determined that 64% of studies were equally pragmatic and explanatory (range, 2.56-3.44), 23% of studies were identified as (range, 2.00-2.044), and 14% of studies were primarily pragmatic (range, 3.56-4.44).

Importantly, they observed a significant degree of underreporting on key RE-AIM dimensions, which, according to the authors, limits researchers’ and policymakers’ ability to assess the generalizability of the research results. For example:

• The overall rating of sufficiently reported individual RE-AIM items across all studies was 18%
• Effectiveness (52%) and reach (45%) were the most commonly reported dimensions
• Reported data within the maintenance categories were observed in 12% of the studies
• Reporting of items in the adoption and implementation dimensions were found in just 4% and 10% of the interventions, respectively

The authors emphasized that underreporting in the adoption and maintenance categories limits information on clinicians’ ability to adopt app-based physical activity intervention tools. In turn, this could significantly limit the willingness of organizations to implement new intervention approaches.

The meta-analysis was not without limitations, they noted. Studies varied significantly in terms of design and methodological parameters, and most of the studies included other intervention components, precluding their ability to isolate the individual effect of the app on physical activity. Further, 39% of the studies were classified as having a high risk of bias.

Despite these limitations, the authors’ aggregate study results did show a small but significant improvement in participants’ physical activity. However, treatment effect sizes varied according to the PRECIS-2 classification, where the more pragmatic studies were associated with smaller increases in physical activity.

Results of their review suggest that app-based physical activity interventions would have limited efficacy in promoting physical activity if more widely scaled and adopted among the general population, the authors concluded.

“The interventions in this literature, in general, had more explanatory rather than pragmatic designs,” the authors said. “More pragmatic study designs are needed to increase the transferability from research to practice.”

Reference

Stecher C, Pfisterer B, Harden SM, Epstein D, et. al. Assessing the pragmatic nature of mobile health interventions promoting physical activity: systematic review and meta-analysis. JMIR mHealth and uHealth. 2023;11:e43162. doi:10.2196/43162