Researchers have developed a method for screening AI-generated marketing messages using a company’s previous campaign results, with an email application covering 3.3 million observations across 34 campaigns.
The approach estimates how new content might affect customers and identifies messages that need direct testing because they differ too much from past campaigns.
The study, published online in the Journal of Marketing Research on 3 August, reports better prediction on data outside the model’s training sample and better performance in real-world deployment than standard approaches.
Its application concerns email marketing. The results do not establish equivalent performance across advertising channels or businesses with smaller campaign datasets.
“AI doesn’t automatically know what works for a particular company,” said Wreetabrata “Wreeto” Kar, assistant professor of marketing at Penn State’s Smeal College of Business, in a university account updated on 1 October.
Using previous campaigns to assess new content
The framework uses large language models, AI systems trained to process and generate language, to turn marketing content into numerical representations of its meaning.
Kar compares this to placing messages on a map. Emails offering free shipping can sit close together despite different wording. A message using “free” in an unrelated sense sits elsewhere.
The researchers connect those representations with historical customer outcomes to estimate the effects of new messages. Their method also screens out content that would require predictions beyond what the available data can reliably support.
Marketers can then concentrate experiments on unfamiliar content. A/B testing compares customer responses to different versions of a message sent to separate groups.
An unusual birthday email still performed strongly
A retailer’s birthday greeting illustrates why an uncertain prediction should not be treated as evidence that a message will fail.
In an account published on 30 September by the University of Tennessee, Knoxville, co-author James Reeder described a birthday email that differed from the retailer’s usual promotional messages.
The framework could not predict its performance reliably, yet the greeting ultimately had the strongest effect on customer spending, according to Reeder.
Rejecting a prediction therefore leaves room for a company to test a new idea. Automatically rejecting the idea itself could mean overlooking an effective campaign.
Marketers choose what reaches customers
Kar said the researchers gave an AI model elements from a retailer’s emails, including discounts, free shipping and clearance sales. When asked to select its own five best new emails, it chose messages predicted to perform poorly. That was a prediction, not a report of observed sales losses.
In the proposed process, marketers choose the inputs, decide how far content can depart from previous experience before requiring a test, and select the final message.
Customer reactions also depend on the context in which AI is used, as our earlier coverage of research into emotional AI advertising examined. That separate research concerned viewers’ responses to video ads.
“Firms should not blindly adopt what ChatGPT says to do,” Reeder said.
