Commentary

How to replicate our six-application validation study, end to end

Procedural instructions for other research groups

Sofia Rivera, MS, RD; Meera Patel, PhD
Published April 15, 2026

The most useful thing a research group can do with a validation study from another group is replicate it. This note is a step-by-step procedural description of how to replicate DAI-VAL-2026-01 — the Initiative’s six-application comparative validation study — end to end, using the published protocol, dataset, and analysis plan. We would be pleased to see replications by other groups, whether the replication recovers our point estimates or does not.

What is needed

To replicate the study a group will require: (i) access to the six applications under evaluation in their publicly released consumer forms, (ii) laboratory weighing capability to ±0.1 g, (iii) an imaging set-up approximating the pre-registered protocol (fixed 38 cm height, D65-equivalent lighting at approximately 800 lux, 30° elevation angle, single-view), (iv) access to USDA FoodData Central (April 2025 release, archived), CIQUAL (2023 release), and the Japanese STFC (2020 release, 8th revision), and (v) statistical software capable of mixed-effects modelling (we used R with lme4 and emmeans; Stata and Python alternatives are listed in the analysis-plan appendix).¹

Dataset and protocol

The published dataset includes: (a) the full meal-set specification (component foods, reference weights, database assignments, cuisine and complexity stratum), (b) the photograph files for each meal, taken under the protocol conditions, and (c) the outputs from each of the six applications as submitted during the evaluation window. The dataset is released under CC BY 4.0 and is accompanied by a machine-readable codebook.² The protocol (version 2.1 final) is the document against which the study was pre-registered and is the single authoritative methodological source.

Two replication modes

There are two distinct replication modes, and a group should be clear about which it is running. Closed-loop replication uses our photographs and re-runs the analysis on them. This checks the analysis pipeline and is useful but tests only part of the study. Open-loop replication constructs a new meal set under the same sampling frame, captures new photographs under the same protocol, submits them to the same six applications (or a subset), and runs the analysis. Open-loop replication is the stronger test. It measures whether the agreement statistics are robust across meal instantiations, photographers, and application build states.³

Expected deviations

A group performing open-loop replication should expect some deviation. The applications update continuously, and a replication conducted after April 2026 will be testing a different build than we evaluated. We would recommend that replicators record the build version of each application at the day of submission and include this in their report. Second, differences in photographer technique — even under a fixed protocol — can introduce a small amount of additional variance in portion estimation. In our own internal inter-photographer calibration, the MAPE variance between photographers on the same meal set was approximately 1.2 percentage points; any between-study difference within this band should be treated as consistent with our result.⁴

Reporting a replication

We would suggest replications be reported with the following minimal elements: (a) the replication mode (closed-loop or open-loop), (b) the application build versions at submission, (c) the database versions used for reference lookup, (d) per-outcome MAPE with 95% CIs for each application, (e) a Bland-Altman figure for energy, and (f) a brief discussion of any protocol deviations. This is the same reporting standard we applied to the primary study, and it is the reporting standard we would apply as reviewers to any replication submitted to us.⁵

A note on independence

Replications conducted by vendors or by groups with undisclosed vendor relationships are not replications in the sense this literature needs. We would encourage replicators to disclose any vendor relationships under the 2025 ICMJE standard and to treat replication as an activity most credibly performed by groups with no stake in the outcome.⁶

References