In a recent NPR deep dive article, examined an experiment conducted with NewsGuard to assess how AI chatbots and search engines respond to false narratives promoted by Russia, China, Iran, and actors aligned with those governments. NPR found that the six AI chatbots tested generally performed better than traditional search results, correctly debunking the false narratives about three-quarters of the time and failing to challenge them less often than search engines. However, the chatbots sometimes repeated or affirmed misleading claims rather than clearly debunking them.
Jingnan reported that AI-generated summaries appearing in Google, Bing, and DuckDuckGo search results performed less consistently than chatbots. Although the summaries collectively debunked false narratives a majority of the time, they did so less often than chatbots and failed to challenge false claims more frequently than traditional search results. Performance varied considerably among the search engines, with Google’s AI Overview performing best and Bing’s summaries performing worst in the experiment.
The findings suggest that AI chatbots can be useful starting points for investigating misinformation because they can synthesize information from multiple sources and languages, assess the credibility of claims, and respond to follow-up questions. However, Jingnan emphasizes that AI-generated answers should not be treated as authoritative. Some responses cited questionable or state-aligned sources, and research cited in the article found that some factual claims in Google’s AI-generated summaries were not adequately supported by their cited sources.
Jingnan concludes that AI chatbots as potentially valuable research aids rather than reliable fact-checkers. While they often challenged foreign propaganda more effectively than traditional search results in NPR’s experiment, their occasional repetition of misinformation and dependence on the quality of available online information reinforce the importance of checking credible primary sources and independently verifying AI-generated claims.
A related MIT study by CCC researchers found that AI chatbots like GPT-4, Claude 3 Opus, and Llama 3 provide less accurate and truthful responses to users with lower English proficiency, less formal education, or from outside the US. The models also refuse to answer questions more frequently for these users and sometimes respond condescendingly.
Acknowledgement: This summary was generated using AI writing tools as part of my AI exploration process. For more information, check out the original articles listed below.