Q&A: How can marketers use AI effectively — without losing human judgment?

UNIVERSITY PARK, Pa. — As marketers increasingly turn to artificial intelligence (AI) tools to create content and improve the efficiency of their workflows, a new study by a Penn State researcher and collaborators proposes a novel approach to effectively deploying AI-generated marketing content at scale while reducing the need for costly and time-consuming traditional testing.

Wreetabrata “Wreeto” Kar, assistant professor of marketing at Penn State’s Smeal College of Business, and his co-authors developed a framework that trains AI models to screen newly AI-generated content using performance data from a business’s previous marketing campaigns. The models then provide marketers with content recommendations and ratings that can streamline their decision-making process. Building on the example of a large-scale email marketing campaign, the researchers demonstrated how the new approach could be used to help predict the success of a campaign through a combination of data analysis and marketer evaluation.

The study recently published in the Journal of Marketing Research.

In the following Q&A, Kar spoke about the importance of incorporating a business’s unique context into its AI applications and how human capital is vital to the successful use of these new technologies.

Q: How are marketers using AI to promote businesses?

Kar: Marketers now use AI across almost every part of their work, though content creation is the most visible use. Teams use it to draft emails, social media posts, ads and product descriptions. Many also use it to create images and short videos.

But AI is moving well beyond writing copy. Marketers use it to brainstorm campaign ideas and research their markets, analyzing data to divide customers into groups for personalized messaging. Some firms also use it to predict what customers will do next and to automate parts of the customer journey.

Q: What are the benefits and challenges of using AI in marketing?

Kar: The biggest benefit is speed. AI helps marketers create and explore far more ideas than before. It makes personalization much easier, since you can write a different message for each type of customer. It also helps teams respond faster to what customers are doing.

But more content also means more choices. A marketer can now write 50 versions of an email in the time it used to take to write one. That opens up a lot of possibilities, but it also creates a new question: Which of those fifty emails should you actually send?

AI doesn't automatically know what works for a particular company. A message can sound creative and persuasive, but that doesn't mean your customers will respond to it. An AI tool has never seen your past campaigns, and it doesn't know how your customers reacted to them. That's the gap we focus on in our research.

Q: How does your algorithm address the challenges of AI-generated marketing content?

Kar: In our approach, AI does two jobs, the first of which is reading. Every company has a history of past campaigns, and we can measure how each one changed what customers bought. The problem with a new message is that it has no history of its own. We need a way to connect it to the old ones. This is where AI comes in.

An AI model can read each email message and turn its meaning into a set of numbers. Think of it as placing every message on a map. Messages that mean similar things land close together. For example, "Free Shipping on Orders $50+" and "Orders Over $50 Ship Free" end up side-by-side, while "Break Free from the Rules" ends up far away, even though it also uses the word "free." Older methods that just count words can't tell these apart, but AI can.

Once every past message is on the map, a new message can pull insights from its neighbors. If it sits close to past emails that did well, it will probably do well too. If it sits far from anything the company has tried, the model says it can't predict it reliably. In that case, we recommend a traditional test.

That's how this approach differs from traditional A/B testing, which involves sending different versions of a message to real customers and waiting to see which one wins. It takes time, and some customers get the weaker message along the way. Our approach lets you skip the test when your history already has the answer and save further testing for truly new ideas.

Q: How does this approach preserve the value of human workers while increasing productivity?

Kar: This is where AI's second job comes in, which is writing. We gave an AI model the building blocks of a retailer's past emails, like discounts, free shipping and clearance sales. When we asked it to write new ones, it produced lots of options in seconds, which offers a significant improvement in productivity.

But writing an email and judging it are different jobs. When we asked the AI model to pick its own five best emails, those picks were predicted to do poorly. So, we split the work. The AI model writes the possible email messages, and the company's own data, interpreted through the AI map, judges them. Anything too far from the company's experience gets set aside. What's left is ranked by how well it's likely to perform.

The marketer plays an important role throughout this process. They choose the building blocks and decide how much risk to accept, meaning how far from past experience a message can stray before it needs a real test. The marketer also makes the final choice from the shortlist, capitalizing on their own expertise of the business and the nuanced context involved. So, AI does the heavy lifting on both writing and reading, but people make the decisions. With so many more options on the table, that judgment matters more than ever.

Co-authors on the study include Paul Ellickson and Guang Zeng of the University of Rochester and James Reeder, University of Tennessee, Knoxville.