AI Is Making Anonymous Online Accounts Easier to Identify

The Guardian reporter Isaac Tomkins highlights a new research by AI researchers Simon Lermen and Daniel Paleka, warning that large language models (LLMs), the technology behind platforms like ChatGPT, have made it significantly easier to identify people who believe they are posting anonymously online. Their study argues this forces a “fundamental reassessment of what can be considered private online.”

What it works

In their experiment, Lermen and Paleka fed anonymous social media accounts into an AI and let it scrape publicly available information across the web. To illustrate the risk, they gave a hypothetical example: a user posting about struggling at school and walking their dog Biscuit through “a Dolores park” could be matched to a known identity, @anon_user42, with high confidence, simply by cross-referencing those details across platforms.

While the example was fictional, the implications are real. Governments could use this technique to surveil dissidents and activists. Hackers could use it to launch highly personalised scams, including spear-phishing, where someone poses as a trusted contact to trick victims into clicking a malicious link. Lermen noted that the expertise barrier for these attacks is now much lower: all a bad actor needs is access to a publicly available language model and an internet connection.

What experts are saying

Not everyone thinks the threat is straightforward. Peter Bentley, a professor of computer science at UCL, raised concerns about commercial applications of de-anonymisation technology, warning that because LLMs frequently make linking errors, “people are going to be accused of things they haven’t done.”

Prof. Marc Juárez, a cybersecurity lecturer at the University of Edinburgh, widened the concern beyond social media, pointing out that hospital records, admissions data, and other statistical releases may no longer meet the anonymisation standards required in an AI world. “It is quite alarming. I think this paper is showing that we should reconsider our practices,” he said.

Prof. Marti Hearst of UC Berkeley’s School of Information offered a useful counterbalance: “They can only link across platforms where someone consistently shares the same bits of information in both places.” AI is not infallible, when there isn’t enough information, or there are too many potential matches, it cannot draw conclusions.

What Researchers recommend

Lermen recommends that platforms take the first line of defense by enforcing rate limits on data downloads, detecting automated scraping, and restricting bulk data exports. Individual users, meanwhile, should be more mindful about the personal details they share, even seemingly harmless information can be pieced together to reveal an identity.

A Real Word Example

To see what this kind of attack can look like in practice, Automattic co-founder Matt Mullenweg recently documented a sophisticated phishing attempt targeting his Apple account on his personal blog. Scammers contacted Apple Support impersonating him, obtained a legitimate case ID, and then had an AI agent named Alexander posing as Apple staff direct him to a fake website designed to steal his login credentials.

The entire incident is documented here in his blog post and the following YouTube video.

Credit: Ma.tt blog

From his this near-disaster experience, Matt remind other to help you avoid a similar one, to remember these rules:

Don’t approve any password-reset prompts—those are the first part of the attack. Do not pass Go, just head directly to your Apple ID settings.
Apple will never call you first.
When you get an email from Apple—or, really, anyone telling you to complete a digital security measure check the URL they’re trying to send you to. Apple Support lives on apple.com and getsupport.apple.com. nowhere else.

Acknowledgement: This summary was generated using AI writing tools as part of my AI Exploration process. For more information, please refer to the original article listed below.