ChatGPT has huge hidden costs that could stall AI development

Artificial intelligence chatbots have a problem: they lose money on every chat.

The enormous costs of operating the current large-scale language models underlying tools such as ChatGPT and Bard limit their quality and threaten to derail the global AI boom they have sparked.

Their cost and the limited availability of the computer chips needed for them also limit the range of companies that can afford to run them, and force even the world's wealthiest companies to turn chatbots into money makers sooner than they might be ready for.

"The models currently in use, as impressive as they may seem, are not really the best models available," says Tom Goldstein, a computer science professor at the University of Maryland. "As a result, the models you see have a lot of weaknesses" that could be avoided if not for the cost - such as a tendency to produce biased results or outright lies.

What happens when ChatGPT lies about real people?

Technology giants betting on the future of artificial intelligence rarely discuss the cost of the technology. OpenAI (creator of ChatGPT), Microsoft and Google declined to comment. However, according to experts, this is the most serious obstacle to generative AI sweeping all industries, reducing the number of employees and increasing efficiency.

The intensive computation that artificial intelligence requires is the reason why OpenAI did not include its powerful new GPT-4 language model in the free version of ChatGPT, which still runs on the weaker GPT-3.5 model. The underlying ChatGPT dataset was last updated in September 2021, making it useless for learning or discussing the latest developments. And even those who pay $20 a month for GPT-4 can only send 25 messages every three hours, as it's very expensive to run. (It is also much slower to respond to queries).

That cost may be one reason Google has yet to build an artificial intelligence chatbot into its flagship search engine, which processes billions of queries every day. When Google released the Bard chatbot in March, it chose not to use its largest language model. Dylan Patel, principal analyst at SemiAnalysis, a semiconductor research firm, estimated that a single conversation with ChatGPT could cost 1,000 times more than a simple Google search.

In a recent report on artificial intelligence, the Biden administration called the computational costs of generative AI a national problem. The White House writes that the technology is expected to "dramatically increase computing demands and associated environmental impacts," and that there is an "urgent need" to develop more sustainable systems.

More than other types of machine learning, generative AI requires dizzying amounts of computing power and specialized computer chips known as GPUs that only the wealthiest companies can afford. The intensifying competition for access to these chips has seen leading vendors emerge as technology giants in their own right, gaining the keys to what has become the tech industry's most valuable asset.

Silicon Valley came to dominate the Internet economy in part by offering services such as Internet search, email and social networking to the entire world for free, losing money at first but eventually making huge profits through personalized advertising. And advertising is likely to come to chatbots with artificial intelligence. However, analysts say advertising alone will likely not be enough to make advanced AI tools profitable anytime soon.

For now, companies offering AI models for consumer use must weigh their desire to gain market share against the financial losses they incur.

The quest for more robust artificial intelligence is also likely to profit primarily chipmakers and cloud computing giants, who already control much of the digital space, as well as chipmakers whose hardware they need to run the models.

It's no coincidence that the companies building the leading AI language models are among the biggest cloud computing vendors, like Google and Microsoft, or have close partnerships with them, like OpenAI with Microsoft. Companies that buy these firms' AI tools don't realize they're getting into a heavily subsidized service that costs far more than they're paying now, says Clem Delangue, CEO of Hugging Face, an open-source AI company.

OpenAI CEO Sam Altman indirectly acknowledged the problem at a Senate hearing last month, when Sen. Jon Ossoff, D-Ga. warned that if OpenAI tried to make ChatGPT addictive, which harms children, Congress would "take a very hard look" at it. Altman said Ossoff needn't worry: "We try to design systems that don't require maximum involvement. In fact, we have so few GPUs that the fewer people using our products, the better."

The costs of creating AI language models start with their development and training, which requires gigantic amounts of data and software to identify patterns in the language. In addition, AI companies typically hire star researchers whose salaries can rival those of professional athletes. This is an initial barrier for any company hoping to create its own model, although a few well-funded startups have succeeded - including Anthropic AI, which OpenAI alumni founded with funding from Google.

Each request to a chatbot like ChatGPT, Microsoft Bing or Anthropic's Claude is then sent to data centers where supercomputers pore over the models and simultaneously perform multiple high-speed computations - first interpreting the user's request, then working to predict the most plausible answer, one "token," or sequence of four letters at a time.

That kind of processing power requires graphics processing units (GPUs), which were originally designed for video games but have turned out to be the only chips capable of handling heavy computer tasks like creating large language models. Currently, only one company, Nvidia, sells the best of them, for which it charges tens of thousands of dollars. Because of the expected sales, Nvidia's value recently rose to $1 trillion. The Taiwanese company TSMC, which makes many of these chips, has similarly grown in value.

"At this point, GPUs are much harder to get than drugs," Elon Musk, who recently bought about 10,000 GPUs for his own AI startup, told the Wall Street Journal's May 23 summit.

These computational requirements also help explain why OpenAI is no longer the nonprofit organization it was founded to be.

Launched in 2015 with a stated mission to develop AI "in a way that is most likely to benefit all of humanity beyond the need for financial gain," it moved to a commercial model by 2019 to attract investors, including Microsoft, which invested $1 billion and became the exclusive computing provider for OpenAI. (Microsoft subsequently invested another $10 billion and integrated OpenAI's technology into Bing, Windows, and other products.)

The exact cost of maintaining chatbots like ChatGPT remains unchanged as companies work to improve their efficiency.

In December, shortly after launch, Altman estimated the cost of ChatGPT at "probably single-digit cents per chat." That may seem insignificant, but when you multiply that by the number of users, which exceeds analysts' estimates of 10 million per day. In February, SemiAnalysis estimated that ChatGPT was costing OpenAI about $700,000 per day just for the computational costs associated with processing the data needed to run GPT-3.5, the default model in use at the time.

If you multiply these computational costs by the 100 million people who use Microsoft's Bing search engine every day, or the more than 1 billion people who use Google, it becomes clear why the tech giants are reluctant to make the best AI models publicly available.

The new Bing told our correspondent that it "can feel or think".

"Such an equation is not good for the democratization and widespread adoption of generative AI, nor for the economy or the environment," says Sid Sheth, founder and CEO of d-Matrix, a company working to create more efficient AI chips.

In Bard's February announcement, Google said it would initially run on a "lightweight" version of the LaMDA language model because it requires "significantly less processing power, allowing us to scale to more users." In other words, even a company as rich as Google isn't willing to pay the cost of implementing its most powerful artificial intelligence technology into a free chatbot.

The cost-cutting took its toll: Bard stumbled on basic facts during the demo, leading to a $100 billion drop in Google's stock price. Bing, for its part, went off the rails early on, forcing Microsoft to reduce both its personality and the number of questions users could ask it in a conversation.

Such errors, sometimes called "hallucinations," have become a major problem with AI language models as they are increasingly relied upon by individuals and companies alike. According to experts, they are a consequence of the underlying design of the models: they are built to generate probable sequences of words rather than true statements.

Another Google chatbot called Sparrow was developed by a subsidiary of DeepMind to search for information online and cite sources to reduce falsehoods. However, Google has not released it yet.

ChatGPT is "hallucinating." Some researchers fear that this is not fixable.

Meanwhile, each of the big players is looking for ways to make AI language models cheaper.

Running a query on OpenAI's new, lightweight GPT-3.5 Turbo model costs less than one-tenth of one percent of its top-of-the-line GPT-4 model. Google is building its own artificial intelligence chips, which it claims are more efficient than Nvidia's, as are start-ups like d-Matrix. In addition, many companies are creating open-source language models, such as Meta's LLaMA, to avoid paying OpenAI or Google to use their models, although these models don't work that well yet and may not have safeguards against abuse.

According to Maryland-based Goldstein, the emergence of smaller, cheaper models is an unexpected turnaround for the industry.

"For the last four years, we've just been trying to build the biggest models we could," he says. But back then, the goal was to publish scientific papers, not to release AI chatbots to the public. "Now, just in the last few months, there's been a complete upheaval in the community, and everyone is trying to build the smallest possible model to control costs."

For consumers, this could mean that the days of unfettered access to powerful general-purpose AI models are numbered.

Microsoft is already experimenting with embedding ads in Bing search results based on artificial intelligence. At the Senate hearing, OpenAI representative Altman did not rule out doing the same, although he said he preferred a paid subscription model.

Both companies say they are confident that the economics will eventually pay off. Altman told Stratechery's tech blog in February, "There's so much value here, it's inconceivable to me that we can't figure out how to make a cash register on it."

However, critics point out that generative AI also carries a cost to society.

"All this data processing has implications for greenhouse gas emissions," says Bhaskar Chakravorty, dean of global business at Tufts University's Fletcher School of Global Business. Computing requires energy that could be used for other purposes - including other computational tasks less fashionable than AI language models. This "may even slow down the development and application of AI for other, more meaningful purposes, such as health care, drug discovery, cancer detection, etc.," Chakravorty said.

Based on estimates of ChatGPT's usage and its computing needs, Casper Gros data analyst Albin Ludvigsen calculated that it used as much electricity as 175,000 people in January, the equivalent of an average city.

According to Goldstein, for now, tech giants are willing to lose money trying to gain market share with their artificial intelligence chatbots. But if they can't make them profitable? "Eventually, you're going to come to the end of the popularity curve, and the only thing your investors are going to be looking at at that point is the bottom line."

Still, Goldstein believes that many people and companies won't be able to resist generative AI tools, even with all their drawbacks. "Even if it's expensive," he said, "it's still much cheaper than human labor.