Two people use smartphones over coffee. Photo: RDNE Stock project / Pexels.

Why an ad A/B test can compare audiences as well as creative

Written by Joseph Nordqvist

Published: 17:00, November 19, 2026

Online advertising tests can produce misleading conclusions about which message works best if a platform delivers competing ads to different kinds of people, marketing researchers warn.

A campaign may appear to compare two headlines or images, but the delivery system can also influence who sees each one. If the audiences differ in ways that affect their likelihood of clicking or buying, the observed result combines the effect of the creative with the effect of the platform’s targeting.

Michael Braun and Eric M. Schwartz examine the problem in a Journal of Marketing research account published by the American Marketing Association in 2025. Their paper calls the issue divergent delivery.

What an A/B test is supposed to compare

An A/B test compares two versions of a message, page or feature. A retailer might show one group an ad emphasizing delivery speed and another an ad emphasizing a wider selection, then compare the response.

When people are assigned randomly to otherwise identical conditions, differences in outcomes can provide evidence about the effect of the changed message. In practice, an advertising platform must also decide which opportunities to buy and which users should receive each ad.

That delivery process can optimize toward different audiences for the two ads. Even if the marketer starts with comparable ad sets, people who actually see one message may differ from those who see the other.

The researchers illustrate the problem with two hypothetical ads from a landscaping business. A platform might show a sustainability message to people with different interests from those who receive a design-focused message. A better response to one ad could then reflect the audience reached, the message itself or both.

Two questions that need different evidence

A business may want to know which ad and delivery combination performs best on a specific platform. That is a different question from whether a message would persuade the same audience more effectively in another channel.

Platform results can still help a marketer choose between campaigns within the conditions tested. But the authors caution against treating those comparisons as clean evidence that one creative message is inherently more persuasive across audiences or platforms.

A related Journal of Consumer Research article by Braun, Bart de Langhe, Stefano Puntoni and Schwartz reviews several ways researchers collect advertising-platform data. It explains that the conclusions supported depend on the design, including whether a holdout group is used.

A holdout is a comparison group that does not receive the advertising being evaluated. It can help estimate the combined effect of an ad and the delivery system under the tested conditions. It does not automatically isolate the effect of one creative element from every other difference.

What marketers can check

Start by writing down the decision the test is meant to inform. Choosing a campaign for the same platform calls for a different interpretation from deciding what language to use in a store, email or television campaign.

Record the target audience, dates, budget, objective, delivery settings and creative variations. Compare the characteristics of people reached where the platform provides them, while recognizing that unobserved differences may remain. Keep the campaign’s cost and the eventual business outcome alongside click or engagement data.

Where a platform offers a suitable randomized holdout, the business can use it to measure the effect of running a campaign under that platform’s delivery conditions. To isolate a landing-page change, a company could send visitors to versions assigned at random after arrival, provided its measurement and consent arrangements allow it.

Small differences deserve particular care. A headline that wins in one short test may not remain the winner when the audience, season, budget or platform objective changes. Repeated tests can help reveal whether a result holds under the conditions that matter to the business.

Our earlier reporting on retail app advertising also found that an increase in one form of activity did not establish an increase in sales. The question being measured matters as much as the size of the reported effect.

Joseph Nordqvist Avatar

Other News

Personalized AI video ads drew more clicks in a WhatsApp experiment

Nov 19, 2026

Restaurant advertising results can depend on an existing reputation

Nov 19, 2026

Search suggestions can help shoppers find less familiar products

Nov 19, 2026

AI-generated shoppers can give misleading pricing estimates, study finds

Nov 19, 2026

The design of a discount label can change how shoppers judge an offer

Nov 19, 2026

What virtual shopping tools can change before and after a purchase

Nov 19, 2026

Game-like loyalty features drew more engagement in an online grocery study

Nov 18, 2026

A delivery cue helped more shoppers move to the next checkout step

Nov 18, 2026

Small changes in a store can bring promotions back to repeat customers’ attention

Nov 18, 2026

Some sellers accept less when a buyer shares an item’s heritage

Nov 18, 2026

Changing a marketplace trust badge can change what sellers prioritize

Nov 18, 2026

Referred customers may bring in more customers of their own

Nov 18, 2026

An extra option can make an attractive menu less appealing, experiments find

Nov 17, 2026

Adding similar choices can change which products look like the middle option

Nov 17, 2026

World Bank plans e-commerce pilot for small firms in three West African countries

Nov 17, 2026

Contextual ads were the most common targeting method among EU online advertisers

Nov 17, 2026

People treat the same money differently across spending categories, study finds

Nov 17, 2026

Small-business loan applications can stall before they are formally submitted

Nov 17, 2026

Describing the parts of an experience can change how much people enjoy it

Nov 16, 2026

A prize can feel different when customers know they might have received less

Nov 16, 2026