Commentary

Responding to vendor feedback on our six-application validation study

Substantive technical comments and Initiative responses

Helena Weiss, PhD, MPH, RD; Daniel Okafor, PhD, MS
Published April 10, 2026

Our six-application validation study (DAI-VAL-2026-01) was published on 2 April 2026. Between then and 9 April 2026, three of the six vendors whose applications were evaluated submitted technical comments through the Initiative’s post-publication comment channel. Two additional non-technical messages were received; these are acknowledged but not reproduced here, in keeping with editorial policy that distinguishes substantive technical comments from general correspondence.¹

What follows is a lightly edited version of each substantive comment, paired with the Initiative’s response. Comments are reproduced with the submitting vendor’s permission; we have truncated for length where the original comment is long and we have indicated truncation clearly.

Comment 1 — database version

Vendor A comment (abridged). “The database version your study associates with our application is listed in Supplementary Table S2 as our 2024-Q3 release. In fact, the production build your team evaluated on 12 February 2026 was running against our 2025-Q2 release, which was rolled out to our US region on 28 January 2026. The 2024-Q3 release contains roughly 41,000 fewer entries than 2025-Q2 and is materially different on mixed-plate items. We believe the MAPE figures for our application would be different under the correct database attribution.”

Initiative response. The vendor is correct that the database-version attribution in Supplementary Table S2 is in error. The build version of the application was recorded correctly (§2.4 of the primary analysis), but the underlying database version was inferred from the application’s public changelog at the pre-registration date, which in this case lagged the actual production rollout by approximately three weeks. We have issued a correction to Supplementary Table S2 (version 2026-04-11, stamped with the correction date) and have updated the narrative reference in §3.2 accordingly. The MAPE point estimates and confidence intervals for this application are unchanged — the analysis used the application’s actual outputs, which reflect the database version the application was running against at submission time — but the attribution was misstated, and we regret the error.²

Comment 2 — meal selection

Vendor B comment (abridged). “The meal-set construction described in §2.2 over-weights mixed-plate meals with ≥5 components relative to the real-world use of our application. Our user base predominantly logs single-item meals and mixed plates with ≤3 components. We believe the headline MAPE for our application is biased upward relative to the performance a user would experience in practice.”

Initiative response. The meal-set construction is a stratified frame, not an attempt to reproduce any particular vendor’s usage distribution. The strata were pre-registered (§2.2, protocol version 2.1) on 15 January 2026, before the evaluation window opened, and were designed to provide balanced coverage across complexity strata rather than to reflect the marketing distribution of any individual application. The vendor’s comment describes a legitimate secondary question — how each application would perform on a usage-weighted distribution reflecting its own user base — but this was explicitly not the primary analysis, and the pre-registration lodged three months ago commits us to the stratified frame we used.³ The secondary stratum-specific MAPE values are reported in Supplementary Table S6 and permit readers to construct a reweighted estimate should they wish to. We will not, however, revise the primary analysis to adopt a different frame post-publication.

Comment 3 — pipeline release

Vendor C comment. “The developer notes that the photo pipeline released in 2026-04 differs from the 2025-Q4 build evaluated during our pre-registration period. We plan to include the 2026-04 build in the next snapshot.”

Initiative response. Acknowledged. The 2026-04 build was released after our evaluation window closed on 30 March 2026 and is therefore out of scope for the primary analysis, consistent with the pre-registration. We plan to include the current publicly available build of each of the six applications in any follow-up snapshot of DAI-VAL-2026-01 at the next scheduled evaluation period. No change to the current paper is indicated.⁴

Editorial note

We publish these exchanges because we think the post-publication correction of a supplementary table, an unresolved but honest methodological disagreement, and a neutral build-version clarification are, taken together, approximately what a healthy post-publication process should look like. Two of the three comments did not change any primary result; one changed a supplementary attribution that should have been correct and now is. Readers are invited to submit additional technical comments through the standard channel.⁵

References