These are the only three times you should pull money from your 401(k) plan

  • The overall 401(k) account balance at Fidelity hit $102,900 as of end of the first quarter.
  • You may be on the hook to repay the full balance of your loan if you leave your job.
  • Pay back the money within five years and make payments at least quarterly.

You already know you shouldn’t tap your retirement plan to fund frivolous purchases, yet in a handful of cases it just might be okay to take a loan.

Retirement plans account for a large chunk of personal wealth: The average 401(k) plan account balance at Fidelity Investments hit $102,900 as of the end of the first quarter of 2018.

In order to get the most out of your retirement plan, you should let the money accumulate over the course of your career. Time and compounding market returns are your 401(k) plan’s best friends.

But, sometimes, emergencies and long-term planning goals will call for the more drastic step of taking a plan loan.

"Plan participants understand that the money is sacrosanct, but they may find themselves in a situation where the 401(k) is the largest source of capital they have," said James A. Cox, financial advisor at Harris Financial Group in Richmond, Virginia.

Here's how to borrow from your 401(k) without ending up with a big tax bill.

Loans, withdrawals, hardship

Retirement plan loans are different from withdrawals and hardship distributions.

Depending on whether your plan permits borrowing, you're generally allowed to take up to 50 percent of your vested account balance to a max of $50,000 — whichever is less. You have five years to repay the loan.

That's different from simply withdrawing money. In that case, your plan administrator will withhold 20 percent of the amount to cover income taxes and you'll trigger a 10 percent penalty if you're under age 59½.

Finally, a "hardship distribution" is what happens when an employee pulls his or her own contributions to cover what the IRS describes as an "unforeseeable emergency."

These distributions are included in your gross annual income and may be subject to additional taxes, but they aren't repaid to the plan. This means they permanently lower your account balance at work.

When you can borrow

Jamie Grill | Getty Images

Once you pull money out of your plan, those dollars no longer benefit from long-term market returns.

If you have a pool of emergency funds, it's best to use that money first. If you're managing debt, it's even better to build that repayment into your budget.

Even your boss wants you to keep your hands off your retirement plan savings.

That said, here are three extreme cases that may warrant a 401(k) loan.

You have an immediate emergency. "Say that you need to meet the deductible on your high-deductible health-care plan, and you have no money in your health savings account," said Aaron Pottichen, president of retirement services at CLS Partners in Austin, Texas.

He is referring to the tax-advantaged health savings account that individuals may use to cover qualified medical expenses. It's also known as an HSA.

You have an urgent cash need, but your credit precludes you from obtaining a competitive interest rate. Ask yourself what you can repay in five years.

You need to pay off high-interest debt that's hampering your long-term financial goals. This is the case if the interest rate on your 401(k) is lower than what your creditor is offering you.

"If you're in 'pay down debt mode,' it's all about what's your cheapest interest rate and how fast can you get the debt down," said Pottichen.

What not to do

Thomas EyeDesign | Vetta | Getty Images

In the worst of scenarios, you'll borrow from your retirement plan, fail to repay it and end up with your finances in even worse shape.

Don't borrow if you're planning on leaving. Whether you quit your job or you're fired, you may need to repay the whole balance of your loan within 60 days or else the amount borrowed is considered a taxable distribution.

Don't ignore your debt-to-income ratio. Treat your plan loan the way you would any other extension of credit. The classic rule of thumb is that no more than 36 percent of your gross monthly income should go toward servicing debt.

This is known as the debt-to-income ratio.

Don't blow off your plan's rules for loans. A 2016 study from Aon Hewitt revealed that six in 10 employers have said they'd take steps to curtail the leakage of assets from retirement plans. Those actions include limiting the number of loans available or the amount of money that's eligible for borrowing.

Plans can also establish their own repayment and schedules, which you'll need to follow.

"When you take a 401(k) loan, it comes out of payroll and reduces your take home pay," said Cox. "Either you follow the payment schedule or you fully remit the balance due."

More from Personal Finance
What you don’t know about Medicare can cost you
These monthly banking fees could ruin your budget
Why your first job out of college really, really matters

WATCH: Cramer: To Roth or not to Roth? Untangling the IRA, 401(k) Roth mystery