Commentary

The 95% confidence interval problem in mobile app marketing claims

On point estimates without uncertainty in consumer-facing accuracy claims

Daniel Okafor, PhD, MS
Published February 9, 2025

A point estimate without a confidence interval is a number, not a finding. This is a commonplace of applied statistics that the marketing departments of consumer dietary-assessment applications have, as a group, not yet internalized. A brief tour of the product webpages of the ten most-downloaded nutrition-tracking applications (as of January 2025, by App Store and Google Play rankings combined) finds that only one reports any measure of uncertainty alongside its headline accuracy figure.¹

Why this matters more than it looks

A research reader seeing “93% accurate” on a product page ought to treat that figure as having substantial implicit uncertainty. The natural question — a 95% CI of [86, 97]? [68, 99]? [48, 99]? — materially changes what the number tells us. The difference between a narrow and a wide interval on the same point estimate is the difference between a product that performs in the low 90s reliably and a product that might perform anywhere from “not much better than chance” to “close to ceiling.”²

In the research literature on validation studies of these products, where confidence intervals are usually reported, the intervals on accuracy-type outcomes are often wide. We have seen 95% CIs on MAPE values that span 4 percentage points in either direction on studies with n ≈ 40 participants. Translating to accuracy-style point estimates, a headline figure of 93% frequently carries an interval that reaches down into the mid-80s and up into the high 90s. A reader who is not shown that interval is being asked to assume the wrong thing.³

The underlying structure

There are two distinct structural reasons marketing copy omits uncertainty measures. The first is aesthetic: CIs do not read well in a product hero section, and consumer marketing conventions favour single-number claims. The second is incentive-driven: a CI that includes uncomfortable values invites readers to ask questions the marketing copy does not want to answer. Between them, these pressures produce a literature-adjacent genre of writing in which point estimates are detached from the statistical context that produced them.

What a responsible presentation looks like

In our view, a responsible consumer-facing accuracy claim includes four elements: the point estimate, the 95% confidence interval, the reference method, and the sample size. This is what we require in our own publications and in any preprint the Initiative endorses, and it is what we would argue regulators in the digital-health space should eventually require of consumer-facing claims.⁴ None of these elements is technically difficult to report; the marketing choice to omit them is exactly that — a marketing choice.

A modest proposal

We do not expect voluntary change at scale. Individual vendors who compete on rigor will adopt fuller reporting; the rest will not. A more realistic path is for the relevant standards bodies — the FDA for regulated digital-health applications, and consumer-protection regulators for unregulated nutrition-tracking apps — to treat the omission of uncertainty measures as a form of incomplete disclosure. There is precedent: drug advertising in most jurisdictions must report not only efficacy point estimates but also confidence information and the population in which the estimate was derived.⁵ There is no first-principles reason consumer digital-health accuracy claims should be held to a weaker standard.

References