Online advertising tests can produce misleading conclusions about which message works best if a platform delivers competing ads to different kinds of people, marketing researchers warn.
A campaign may appear to compare two headlines or images, but the delivery system can also influence who sees each one. If the audiences differ in ways that affect their likelihood of clicking or buying, the observed result combines the effect of the creative with the effect of the platform’s targeting.
Michael Braun and Eric M. Schwartz examine the problem in a Journal of Marketing research account published by the American Marketing Association in 2025. Their paper calls the issue divergent delivery.
What an A/B test is supposed to compare
An A/B test compares two versions of a message, page or feature. A retailer might show one group an ad emphasizing delivery speed and another an ad emphasizing a wider selection, then compare the response.
When people are assigned randomly to otherwise identical conditions, differences in outcomes can provide evidence about the effect of the changed message. In practice, an advertising platform must also decide which opportunities to buy and which users should receive each ad.
That delivery process can optimize toward different audiences for the two ads. Even if the marketer starts with comparable ad sets, people who actually see one message may differ from those who see the other.
The researchers illustrate the problem with two hypothetical ads from a landscaping business. A platform might show a sustainability message to people with different interests from those who receive a design-focused message. A better response to one ad could then reflect the audience reached, the message itself or both.
Two questions that need different evidence
A business may want to know which ad and delivery combination performs best on a specific platform. That is a different question from whether a message would persuade the same audience more effectively in another channel.
Platform results can still help a marketer choose between campaigns within the conditions tested. But the authors caution against treating those comparisons as clean evidence that one creative message is inherently more persuasive across audiences or platforms.
A related Journal of Consumer Research article by Braun, Bart de Langhe, Stefano Puntoni and Schwartz reviews several ways researchers collect advertising-platform data. It explains that the conclusions supported depend on the design, including whether a holdout group is used.
A holdout is a comparison group that does not receive the advertising being evaluated. It can help estimate the combined effect of an ad and the delivery system under the tested conditions. It does not automatically isolate the effect of one creative element from every other difference.
What marketers can check
Start by writing down the decision the test is meant to inform. Choosing a campaign for the same platform calls for a different interpretation from deciding what language to use in a store, email or television campaign.
Record the target audience, dates, budget, objective, delivery settings and creative variations. Compare the characteristics of people reached where the platform provides them, while recognizing that unobserved differences may remain. Keep the campaign’s cost and the eventual business outcome alongside click or engagement data.
Where a platform offers a suitable randomized holdout, the business can use it to measure the effect of running a campaign under that platform’s delivery conditions. To isolate a landing-page change, a company could send visitors to versions assigned at random after arrival, provided its measurement and consent arrangements allow it.
Small differences deserve particular care. A headline that wins in one short test may not remain the winner when the audience, season, budget or platform objective changes. Repeated tests can help reveal whether a result holds under the conditions that matter to the business.
Our earlier reporting on retail app advertising also found that an increase in one form of activity did not establish an increase in sales. The question being measured matters as much as the size of the reported effect.