Though student loan debt is often portrayed as an issue holding back primarily younger generations financially, Americans ages 45 and up hold significantly higher balances than new grads on average, according to a new report.

An analysis of Credit Karma member information finds that 23% of the 21 million people carrying student loan debt are over the age of 45, with an average outstanding balance of $47,000. The 45 to 54 age bracket, who are Gen Xers, carry the most student debt: an average of $49,000. At the same time, new grads with student loan debt carry average balances of around $28,950.

The balances are high because of a confluence of factors, says Colleen McCreary, Credit Karma’s consumer financial advocate and chief people officer. Federal student loans had higher interest rates in the ’80s and early ’90s than they do now, causing balances to grow faster; some borrowers may be parents who took loans out for their kids; others may have returned to school later in life; and wages have simply not kept up with the cost of living, giving borrowers less money to put toward their loans.

“All this is likely going to do is potentially delay retirement for people for a lot longer,” says McCreary.

These loan balances and payments can delay retirement, or at least make it more stressful, says McCreary. Of 464,000 Credit Karma members over the age of 65 with student loan debt, the average balance is $44,000.

“People might think, ‘I already own my house, so why do I care?’ In reality, your wages can be garnished, your Social Security benefits can be garnished, your disability can be garnished” if you’re not making payments on time, says McCreary. “That puts your potential retirement plans into jeopardy.”

Student debt might get more stressful very soon

The statistics are particularly worrisome the closer it gets to Sept. 1, when federal student loan payments—which were put on pause at the beginning of the COVID-19 pandemic two and a half years ago—will resume.

So far, the payment pause has been extended every time the expiration date has approached. But President Joe Biden has been mum on whether borrowers will continue to be able to forgo the monthly bill.

With inflation, rising housing prices, and layoffs increasing, McCreary says it will be difficult for many households to start making their student loan payments again. Credit Karma data shows 90% of members who have federal student loans have not been making payments during the pause.

“If forbearance ends, unless they’ve been saving all this money, I think we’re in a pretty precarious situation,” she says. “The picture starts to look really bleak if forbearance doesn’t continue.”

She advises those who will struggle to make their student loan payments to apply for an income-based repayment plan. This can lower the amount you owe each month, sometimes to $0. Additionally, if you’re older and your children ask you to cosign a loan, think hard about what that additional burden will mean for you and your retirement. As the advice goes: You can borrow to pay for school, you cannot borrow to fund your retirement.