Chatty chatbots can mislead users, but verification tools help, researchers find

UNIVERSITY PARK, Pa. — As artificial intelligence (AI)-powered chatbots become more popular and more human-like in the way they communicate, users are increasingly turning to them as trusted sources of information on topics ranging from personal finance to health. That may be problematic, according to Penn State researchers, who found in a recent study that the more conversational an AI chatbot is, the more likely users will trust it — even when it shares misinformation. The team said verification options may help rebuild needed skepticism.

In their study, published in the Journal of Computer-Mediated Communication, the researchers found that the more conversational or personable the chatbot, the more users curbed their “negative machine heuristic,” which is the tendency to believe that machines lack the human touch in terms of intuition and subjective decision-making. This was the case even when the information was “ridiculously wrong,” they said.

“The conversational delivery of information by generative AI is hyper-specific to your particular question, so it seems very tailored,” said co-author S. Shyam Sundar, Evan Pugh University Professor and James P. Jimirro Professor of Media Effects at Penn State. “This is the first time in the history of technology where a machine can actually hold forth a conversation that has not been predicted by a programmer or has not been pre-scripted.”

Sundar and Maggie Liao, lead author and assistant professor at the University of Georgia, have conducted several studies on how people use and trust AI chatbots. Knowing that the technology can sometimes produce inaccurate information, they designed this study to find out if a conversational chatbot makes users more likely to trust misinformation.

“We tested whether the conversational nature of GenAI chatbots is driving users to trust misinformation coming from them,” said Liao, who completed her doctoral studies with Sundar at Penn State. “And if that is the case, the natural next question is: What can we do about it?”

The researchers conducted an online experiment with 477 participants who were told they were helping test a new AI health assistant. Each participant was asked to submit one prompt from a pool of sample prompts to the AI chatbot. During the interaction, the AI provided incorrect information. Participants were then asked to indicate how credible they thought the chatbot was. The study varied how conversational the AI — low versus high — was with the participants.

A second part of the study tested whether adding a verification option to the chatbot experience offset misplaced trust in AI-produced information. Modeled after the standard human fact-checking process, the system gave users the opportunity to confirm if the AI responses were backed by evidence. Offering a verification option, the scholars theorized, could counter a common assumption among users that the AI is always more objective and accurate than humans. The four possible verification options were: a simple cue with a small icon warning that information may be incorrect; a required check with a button that users must click on to proceed; an optional check with a button that users could click on if they chose to check the information; and no verification option.

The researchers found that users who could verify the information were more skeptical of the AI’s responses. However, just a verification cue alone wasn’t enough. Trust in the AI system only decreased for the users who actively clicked on the verification buttons, the researchers said.

“The verification option is a useful antidote to the conversational persuasiveness of GenAI,” Sundar said. “That’s what we say in the title of our study: ‘Chat, but verify.’ Enjoy the conversation by all means, but don’t be lulled into believing all that you’re getting.”

Participants who received highly conversational AI responses were more likely to find the information credible than those interacting with less conversational versions. This was the case even when the information was blatantly inaccurate.

“Some of the misinformation we used in the study was ridiculous,” Liao said. “One example was that ginger can be more effective than chemotherapy in treating cancer. It sounds ridiculous, but if it is given by a very conversational AI, people trust it more. I think that’s worrisome.”

According to the researchers, users who verified information were less likely to accept false claims as accurate. When given the option, nearly half of the participants verified the information they were given. Sundar said the results from both parts of the study can be a warning sign as well as provide a solution to AI developers.

“Hopefully, research showing that verification tools help combat misinformation will persuade AI platforms to take them seriously and develop more such checks on information generated by AI,” he said.