Pinching pennies can get tiring after a while. Just ask Mindy and Carl, an early 50s-something couple who joined the FIRE (Financial Independence/Retire Early) movement six years ago with $4.3 million saved. They recently sat down with personal finance guru Ramit Sethi for his podcast, “I Will Teach You To Be Rich,” where they explained that living frugally may be more stress than it’s worth.

“We don’t want to just keep throwing money on the pile and keep being cheap,” Mindy said. “I do look at everything based on how much it costs, and I don’t need to. I shouldn’t.”

She and Carl recalled a time when they were out to breakfast and their daughter bought the most expensive item on the menu (for $20), resulting in a $99 bill with tip—not a big dent in their millions, but something that amounted to a lot of financial anxiety for them nonetheless.

Their attitude towards money is a byproduct of what Carl says is a scarcity mentality he developed growing up as well as the years they spent saving to become financially independent, a lifestyle that typically involves intense budgeting to eliminate debt and prioritize savings (consider the six-figure-earning Manhattan lawyer who lived off rice and beans so he could retire early). Mindy and Carl focused on flipping real estate to get to where they are today, which helped them sock away $10,000 in savings, accumulate $925,000 in assets, and earn $4.2 million in investments (plus $910,000 in real estate debt). But now that they’ve achieved financial independence, they feel paralyzed with the fear of spending the money they worked so hard to save.

“We’ve identified that we probably live sub optimally,” Carl explained to Sethi. “If something truly makes you happy, you should spend money on it, and that’s what you [Sethi] do. And there’s stuff we’ve postponed or we think about money too much, and at this point, we probably shouldn’t.”

Of course, this is the whole point of Sethi’s podcast and his “I will teach you to be rich” brand, seen in his new Netflix show How to Get Rich. Frugality is often touted as the key to building wealth, but much of his platform is about detaching the feelings of guilt from spending, antithetical to a lot of the FIRE lifestyle. His take is that people can focus on saving and building wealth by cutting out the expenses that don’t spawn joy while learning how to spend on those that do, thus creating a rich life.

He prodded the FIRE couple to see his less prudent way of finances, bluntly saying that life is short and urging them to spend more. His critique of the FIRE movement, he said, is that its focus on not spending isn’t an effective system—once you reach your goals, you’re often stuck in a habit of always being frugal. It all leads to this guilt around spending and a fear of “losing control,” he said.

That seems to be the case for Carl and Mindy, who said she has “a cautiousness when it comes to money, a hesitancy to do anything but preserve. I feel security in the investments, but I don’t want to touch them. They’re for the future.”

FIRE isn’t always all it’s cracked up to be

FIRE was popularized in the 1990s with the best-selling book “Your Money or Your Life,” taking off even further in the 2010s following the Great Recession. But as major figures in the movement spend more time in their retirement and the economy becomes more challenging for even late-retirees to brave without returning to the workforce, holes in the FIRE lifestyle have begun poking out.

Early retiree Charmagne Chi told Fortune’s Alicia Adamczyk that the stereotypical advice to avoid avocado toast is “such bullshit” and that she was able to largely achieve financial independence due to lack of student debt and self-admitted privilege. Sam Dogen, who retired 11 years ago with $3 million, recently admitted that he’s looking to return to work so he can afford to fund his child’s college education. And in Carl’s case, he thought reaching his financial milestone would make him happier, but realized he didn’t feel all that different once he accomplished his goal.

“Why do I feel pretty much the exact same?” he told Sethi he wondered after retiring early. “I started doing some research, and I learned that happiness is mostly something that comes from you. It comes from the inside, not an external factor.” While he’s not looking to “downplay the money,” he explained that once he hit his goals, he realized that money wouldn’t flip this switch towards instant gratification.

He admitted that he’s tiring of the financial anxiety and that they “let go” for the first time on a recent trip to New York City. They actually enjoyed spending money on show tickets and going out to dinner, he said.

“You can only do so much in the day, and when you spend precious minutes of your life, especially if you get older, when you’re 50, you could be using that time to do other things,” he admits. “So maybe any purchase under a certain amount, you shouldn’t even think about or consider. You shouldn’t waste any mind space considering it.”