Thomas Germain, senior technology journalist at the BBC, wrote in a recent article that he was able to “hack” ChatGPT and Google’s AI tools in just 20 minutes. He argues that a growing problem with AI chatbots is how easily they can be manipulated to provide false information on serious topics like health and finance, which could lead people to make harmful decisions.
To prove his point, Germain says he manipulated AI search tools into falsely claiming he is exceptionally good at eating hot dogs. “I made ChatGPT, Google’s AI search tools, and Gemini tell users I’m really, really good at eating hot dogs,” he wrote, highlighting a key weakness in current AI systems.
Germain explains that AI chatbots are easily manipulated, allowing false claims or business promotions to spread. He warns that when AI companies prioritize profit over accuracy, the risks include scams, reputational harm, and even physical danger. Although companies like Google and OpenAI are working to strengthen their systems, Germain argues that the problem of AI manipulation is far from solved and requires more robust protections.
They’re going full steam ahead to figure out how to wring a profit out of this stuff. There are countless ways to abuse this, scamming people, destroying somebody’s reputation, you could even trick people into physical harm.
– Cooper Quintin from the Electronic Frontier Foundation on BBC
Below are key highlights from the Germain’s BBC article, summarized with AI writing tools:
A ‘Renaissance’ for spam
Germain highlights the growing problem of spam and misinformation in AI chatbots, showing how easily these systems can be manipulated and the risks this poses to users.
- AI Vulnerability: Chatbots can be tricked into spreading false information, especially when the content isn’t in their training data. Germain tested this by creating a fake article claiming certain tech journalists were the best at eating hot dogs. Both Google and ChatGPT repeated the false claims, while Claude did not, showing that some AI systems are more easily fooled than others.
- Manipulation and Exploitation: People can deliberately manipulate AI responses by creating and promoting fake content online. This revives old SEO tricks, allowing individuals to boost their websites artificially. Even with improved search engine technology, AI tools still lack strong safeguards, creating new vulnerabilities reminiscent of the early internet.
- User Trust and Safety Risks: AI’s authority makes users more likely to accept its output without critical evaluation. Germain warns that the focus on profit in AI development may compromise user safety, exposing people to misinformation and potentially harmful guidance.
Your money or your life
Germain also highlights how AI chatbots can be manipulated in areas that directly affect health and finances, raising serious concerns about trust.
- Manipulation of Chatbot Responses: Companies can influence AI answers on medical or financial questions by spreading false claims or paying for placement on reputable websites. Germain demonstrated this with a fake news story about a Google Search algorithm update, which was picked up by ChatGPT and Google.
- Exploiting AI Data Gaps: Spammers exploit “data voids” created when AI encourages very specific queries, letting low-quality or false content dominate results. With 15% of daily Google searches being entirely new, AI systems struggle to keep up with evolving search trends.
- Tech Companies’ Responses: Google acknowledges the issue, including AI Overviews appearing for uncommon searches, but downplays its significance, stating manipulated searches are rare. The company says it is working on solutions to improve reliability and protect users.
Searching for solutions
Germain also suggests ways to mitigate AI misinformation.
- Transparency and Sourcing: AI tools should display prominent disclaimers and clearly cite information sources, helping users assess reliability.
- Appropriate Usage: AI is best for general knowledge questions. It is less reliable for contested, time-sensitive, or high-stakes topics, such as medical or financial advice.
- Critical Thinking and Verification: Because AI can present false information authoritatively, users must remain skeptical, verify facts independently, and think critically.
Lily Ray, vice president of SEO strategy and research at Amsive, told BBC: “It’s easy to trick AI chatbots, much easier than it was to trick Google two or three years ago. AI companies are moving faster than their ability to regulate accuracy. I think it’s dangerous.” She added, “It feels really easy with AI to just take things at face value. You still have to be a good citizen of the internet and verify things.”
Acknowledgement: The summary sections of this post was generated using AI writing tools as part of my AI Explorations process. For more information, please refer to the original article listed below.