Hallucinations: Why AI Makes Stuff Up


AI hallucination is “synthetically generated data,” or “fake data that is statistically indistinguishable from actual factually correct data,” Stefano Soatto, vice president and distinguished scientist at Amazon Web Services, told CNET.

“An AI model that can generate text and was trained on Wikipedia. Its purpose is to generate text that looks and sounds like the posts we already see on Wikipedia. The model is trained to generate data that is “statistically indistinguishable” from the training data, or that has the same type of generic characteristics. There’s no requirement for it to be ‘true’.”

Hallucination as a property of an AI model is unavoidable, but as a property of the system that uses the model, it is not only unavoidable, it is very avoidable and manageable.

Stefano Soatto from Amazon Web Services

Discussing why AI hallucinate, Swabha Swayamdipta, assistant professor of computer science at the USC Viterbi School of Engineering and leader of the Data, Interpretability, Language and Learning (DILL) lab, told CNET, “It generalizes or makes an inference based on what it knows about language, what it knows about the occurrence of words in different contexts. This is why these language models produce facts which kind of seem plausible but are not quite true because they’re not trained to just produce exactly what they have seen before.”

The model doesn’t have contextual information. It’s just saying, ‘Based on this word, I think that the right probability is this next word.’ That’s what it is. Just math in the basic sense.

– Tarun Chopra, vice president of product management at IBM Data & AI(source: CNET)

Sabato added “If users hope to download a pretrained model from the web and just run it and hope that they get factual answers to questions, that is not a wise use of the model because that model is not designed and trained to do that. But if they use services that place the model inside a bigger system where they can specify or customize their constraints … that system overall should not hallucinate.”

Subbarao Kambhampati, a computer science professor who researches artificial intelligence at Arizona State University, told American ScientistScientific American, “Today’s LLMs were never designed to be purely accurate. They were created to create—to generate. The reality is: there’s no way to guarantee the factuality of what is generated, adding that all computer-generated creativity is hallucination, to some extent.”

Ziwei Xu, Sanjay Jain and Mohan Kankanhalli, from the National University of Singapore, told Scientific American, “For any LLM, there is a part of the real world that it cannot learn, where it will inevitably hallucinate.”

Scientific American says that the future AI text generators will be tailored to specific tasks, replacing today’s one-size-fits-all models. These custom-built systems will be integrated into various applications such as customer service, news summaries, and legal counsel, in order to ensure their effectiveness. 

More generalist chatbots will continue to be versatile but they may not be reliable in providing accurate information. Rather, they’ll serve as creative partners and entertainment sources, as mentioned in Scientific American.

Related Resource Links