One night in September 2017, Paris’s Eiffel Tower lit up with a dazzling multicolored light show—all to celebrate the city winning its bid to host the 2024 Olympics.

Just as joyous, perhaps, was the relief that night, among members of the International Olympics Committee. For the IOC, finding cities willing—and able—to spend billions of dollars to host a two-week sporting bonanza, for the promise of prestige and glamor, has become an Olympian task all of itself.

That fact is even clearer now, as the Olympics season unfolds amid the worst pandemic in a century.

With the Tokyo Summer Games set to open on July 23—just two months away—about 83% of Japanese say they want the event either postponed or cancelled altogether, with the two options about evenly split, according to a poll published last weekend by Asahi Shimbun newspaper. The country’s surge in COVID-19 cases, and its slow vaccine rollout, has also prompted thousands of Japanese doctors to petition the government to scrap the Olympics, which have already been postponed from last year. And dozens of Japanese cities have withdrawn their offers to host foreign athletes arriving from across the world, fearing the Games might result in an explosion of coronavirus cases. Even Japan’s biggest Olympic star, tennis champion Naomi Osaka, has voiced concern about the Games going ahead.

But for Olympics officials, the problems extend far beyond the pandemic.

More and more cities have forgone the idea of vying for the Games, after seeing hosts’ ballooning Olympics costs (not to mention hearing of a string of sex-harassment and doping scandals among Olympic officials). “Bid committees often underestimate the costs, and the costs always overrun by enormous amounts,” Heather Dichter, associate professor of sports history at De Montfort University in Leicester, U.K. and an expert on Olympics bids, tells Fortune. “People think, why is this money being spent on sporting events rather than infrastructure, or health or education?”

For a growing number of cities and countries, there is no obvious answer to that question.

The lesson of Athens

That is a drastic change from recent times. Back in the 1990s, Athens won the bid to host the 2004 Summer Games, after an fierce contest between 11 different cities. But many in the country came to regret the Olympics, which is believed to have partly led to the disastrous Greek debt crisis.

Tokyo has already spent about $15.8 billion on its Olympics, more than double what it budgeted in 2013, when it won the bid. Bidding for the Olympics, in fact, is “the highest level of risk a city can take on,” economist Bent Flyvberg, of Oxford University’s Saïd School of Business, told the A.P. last year. “The trend cannot continue."

In retrospect, Athens’s cost overrun seemed to signal the beginning of the steady decline in Olympics bids.

Ten cities competed for the 2008 Olympics, which Beijing won. Nine vied for the 2012 Games (held in London), seven for the 2016 Games (held in Rio) and just five for this year’s Tokyo Olympics.

And the Paris 2024 Olympics? Just two cities competed for that. The city won after Boston, Hamburg, and Budapest all withdrew their bids. Los Angeles was left as the sole rival, and the IOC awarded L.A. the 2028 Olympics at the same time, saving itself the headache of looking for new bidders for a while. It is negotiating with Brisbane, Australia, to host the 2032 Games, sidestepping the usual public competition.

Indeed, previous bids have exposed a deeply unflattering side of the IOC, which has long trumpeted the Games as a kumbaya moment furthering global unity and peace.

In 2014, Oslo was the clear frontrunner to host the 2022 Winter Games, since Norway has won dozens of winter-sport medals, and most venues were already built. But Oslo dropped out after Norwegian journalists revealed that the IOC had issued a mind-boggling list of demands, including private traffic lanes for the IOC members, a private audience with the Norwegian King, and hotel minibars stocked with Coca-Cola products (the company is an Olympic sponsor). “All the meeting rooms had to be exactly at 20-degrees Celsius at all times,” Dichter says, with a tone of disbelief.

Beijing was finally awarded the Games—but only after Lviv in Ukraine, Krakow in Poland, and Stockholm, Sweden, also dropped out. Four years later, voters in Calgary, Canada overwhelmingly rejected the city’s proposal to host the 2026 Winter Olympics. The vote came after three other cities withdrew their bids. In the end, the Games went to Milan and the prosperous alpine town, Cortina d'Ampezzo.

Lost swagger

As more and more cities shy away from competing, the IOC seems to have lost some of the swagger it used to have, with negotiating Olympics bids. “They cannot dictate all those demands,” Dichter says. “It has hurt a lot of bids.”

The French still support the 2024 Olympics being held in Paris, according to polls. Four years after the city won its Olympics bid, a Harris Interactive poll found last December that about 84% of French people saw the Games as good for the city’s tourism appeal, and a source of much-needed jobs.

The city’s Olympics committee says 97% of the €3.8 billion ($4.6 billion) cost for the Games will come from private sponsors, tickets and merchandise. The public, it claims, will contribute just €100 million ($122 million).

But with the Games more than three years away, there is still plenty of time for complaints to build, particularly if the city—following its predecessors—vastly overshoots its budget. Unlike the residents of Calgary, no one asked voters in Paris if the city should bid on the Olympics in the first place.