Commentary

A correction to our 2024 systematic review: vendor-reported MAPE definition

Erratum and expanded methodological note

Helena Weiss, PhD, MPH, RD; Daniel Okafor, PhD, MS
Published January 20, 2025

In October 2024 the Initiative published a systematic review of image-based dietary assessment validation studies covering the period 2015–2024. In the course of preparing follow-up work in December, we identified an inconsistency in how one secondary variable — vendor-reported mean absolute percentage error (MAPE) — was extracted across included studies. This note is a correction. It does not alter the review’s primary findings; it does change one number in Supplementary Table S4, and we think the story of how the error occurred is useful to other reviewers working on digital-health evidence.

What the inconsistency was

For the secondary analysis, we extracted, where available, the MAPE value reported by the application’s vendor (typically on a product website, a press release, or a preprint authored or co-authored by vendor staff) and compared it with the MAPE reported in the independent validation study we had included. Our pre-registered rule was that the vendor-reported MAPE should be extracted for the same nutritional outcome as the validation study (e.g., total energy) and, where the vendor reported multiple values, the one most comparable to the validation study’s population and meal set should be selected.

In practice, two extractors applied this rule differently for a subset of included studies. One extractor, in cases where the vendor reported a single headline MAPE across multiple outcomes combined (for example, an unweighted mean of energy, protein, fat, and carbohydrate MAPE), recorded that single headline figure. The second extractor, in the same cases, recorded only the vendor’s energy-specific MAPE where reported, and left the field blank otherwise.¹ Neither extractor was wrong against a strict reading of the pre-registration; both were inconsistent with the comparison being made downstream.

What changes

After re-extracting the 23 affected rows under a single consistent rule — vendor-reported energy MAPE, or blank — the mean vendor-reported MAPE in Table S4 changes from 11.8% to 13.2%. The mean independent-validation MAPE does not change (it was always extracted consistently). The gap between vendor-reported and independently validated MAPE therefore widens slightly, from 7.4 percentage points to 8.8 percentage points. None of the primary-analysis results change, and no study’s inclusion status changes.²

The revised Supplementary Table S4 is posted alongside the review on the Initiative site with the correction date stamped in the filename. The preprint version on medRxiv has been updated to v2 with a change-log.

How we found it

We found the inconsistency the way most extraction errors are found, which is to say by accident. A graduate student working on a related meta-analysis in December 2024 asked us for the underlying extraction sheet and noticed that two rows for the same vendor contained values that could not both be right. We checked the logs, identified the rule drift, and re-extracted. We mention this because we suspect the same pattern exists in other reviews in this field, and we would encourage authors of reviews in the digital-health space to audit their extraction sheets for analogous drift.³

Lessons we are applying

Three things we are changing in our own workflow. First, for any secondary variable that requires a non-trivial selection rule (like “the most comparable vendor figure”), we will now require a worked example in the extraction protocol before the extractors start. Second, we will require periodic cross-extractor audits during extraction, not only at the end. Third, we will publish the extraction sheet with the review, not on request.⁴

We regret the inconsistency. Corrections, in our view, are a feature of functioning science rather than an embarrassment, and we would rather note a 1.4-percentage-point change in a supplementary table than leave it on the record.

References