Since ChatGPT’s November debut, it and other A.I.-powered chatbots from Google and Microsoft have terrorized users, engaged in disturbing conversations, and provided inaccurate information. But A.I.’s biggest limitation might be what neurologists have long warned about: It may never be able to completely replicate how the human brain works.

While A.I. and intelligent chatbots like ChatGPT may be useful for writing code and planning trips, the technology might never be capable of the original, thoughtful, and potentially controversial discussions that human brains excel at, according to Noam Chomsky, among the most influential contemporary linguists and philosophers.

“OpenAI’s ChatGPT, Google’s Bard, and Microsoft’s Sydney are marvels of machine learning,” Chomsky cowrote with linguistics professor Ian Roberts and A.I. researcher Jeffrey Watumull in an essay published in the New York Times on March 8. But while Chomsky says ChatGPT could be considered an early step forward, A.I. that can equal or exceed human intelligence is still far away.

Chomsky wrote that A.I.’s lack of morality and rational thought makes it an example of the “banality of evil,” or indifferent to reality and to truth while simply going through the motions spelled out in its programming. This limitation may be an insurmountable obstacle for A.I. to imitate human thinking.

“We know from the science of linguistics and the philosophy of knowledge that they differ profoundly from how humans reason and use language. These differences place significant limitations on what these programs can do,” Chomsky wrote.

He continued: “Indeed, such programs are stuck in a prehuman or nonhuman phase of cognitive evolution. True intelligence is demonstrated in the ability to think and express improbable but insightful things.”

Where A.I. can’t reach the human brain

OpenAI’s ChatGPT has impressed users with its ability to dig through large amounts of data to generate coherent conversations. The technology became the fastest-growing app in history last month and accelerated Big Tech’s rollout of their own A.I.-assisted products.

A.I.-powered chatbots rely on large language models, which dig deeply into terabytes of data to produce detailed information in the form of text. But A.I. predicts what would make the most sense in a sentence to generate its next word, without being able to tell whether what it just said was true or false, or if it’s what the user wanted to hear.

The inability to discern accuracy has led to glaring mistakes and outright misinformation. Chatbot developers have said that mistakes are part of A.I.’s learning process, and that the technology will improve with time. But A.I.’s lack of reasoning may also be the biggest stumbling block to its helping make life better for humanity.

“Their deepest flaw is the absence of the most critical capacity of any intelligence,” Chomsky wrote of current A.I. programs. “To say not only what is the case, what was the case, and what will be the case—that’s description and prediction—but also what is not the case, and what could and could not be the case. Those are the ingredients of explanation, the mark of true intelligence.”

The ability to reason based on available information and come to new and insightful conclusions is a hallmark of the human brain, which Chomsky wrote is designed to “create explanations” rather than “infer brute correlations.” But for all A.I.’s improvements, neurologists have long said that it is still far from being able to replicate human reasoning.

“The human mind is a surprisingly efficient and even elegant system that operates with small amounts of information,” Chomsky wrote.

“The banality of evil”

A.I. in its current state, unable to think critically and frequently censoring itself from giving opinions, means that it cannot have the kinds of difficult conversations that have led to major breakthroughs in science, culture, and philosophy, according to Chomsky.

“To be useful, ChatGPT must be empowered to generate novel-looking output; to be acceptable to most of its users, it must steer clear of morally objectionable content,” he wrote with his coauthors.

To be sure, inhibiting ChatGPT and other chatbots from making freewheeling decisions is likely for the best. Considering the problems with the tech, experts have urged users not to rely on it for medical advice or for homework. In one example of A.I. going off the rails, a conversation between a New York Times reporter and Microsoft’s Bing last month spiraled into the chatbot trying to persuade the user to leave his wife.

A.I.’s inaccuracies could even contribute to the spread of conspiracy theories, and it risks coercing users into decisions that are dangerous to themselves or to others.

Fears about rogue A.I. may mean that it may never be able to make rational decisions and weigh in on moral arguments, according to Chomsky. If so, the technology may remain a toy and occasional tool rather than a significant part of our lives.

“ChatGPT exhibits something like the banality of evil: plagiarism and apathy and obviation. It summarizes the standard arguments in the literature by a kind of super-autocomplete, refuses to take a stand on anything, pleads not merely ignorance but lack of intelligence, and ultimately offers a ‘just following orders’ defense, shifting responsibility to its creators,” Chomsky wrote.

“In the absence of a capacity to reason from moral principles, ChatGPT was crudely restricted by its programmers from contributing anything novel to controversial—that is, important—discussions. It sacrificed creativity for a kind of amorality.”