"Student loan debt has been a barrier in making key life decisions and planning for the future," Brown said. AYA and AARP polled nearly 5,000 adults, including millennials, Gen Xers and boomers, between July and August.
Here's a look at some of the long-term consequences:
Saving for retirement
Four in 10 respondents said student loan debt stopped them from saving for retirement, including 41 percent of millennials, 38 percent of Gen Xers and 31 percent of boomers.
Buying a home
About 1 in 3, or 32 percent, said college debt prevented or delayed them from buying a home, including 36 percent of millennials, 26 percent of Gen Xers and 32 percent of boomers.
Helping a family member
One-quarter of those polled said student loans stood in their way when it came to financially helping a family member, including 23 percent of millennials, 29 percent of Gen Xers and 26 percent of boomers.
Having health care
Nearly 1 in 5, or 16 percent, said their debt burden hindered them from getting the health care they need, including 17 percent of millennials, 16 percent of Gen Xers and 9 percent of boomers.
Overall student debt reached a record $1.5 trillion this year, according to the Federal Reserve. Seven in 10 seniors graduate with debt, owing about $29,650 per borrower, according to the most recent data from the Institute for College Access & Success.
To ease some of the burden, Brown recommends sending in a little more than the minimum payment each month toward principal of the loan — by even $10 or $20 — to pay off your loan faster and spend less in interest.
"The sooner you can repay your student loans, the less you have to pay," he said.
More from Personal Finance:
How to save for retirement when you have student loans
Despite record student debt, Americans still bet on college
Want more college financial aid? File a FAFSA now