You're likely pulling out thousands of dollars from the stock market and no longer will reap the benefits of a potential bull market and compounded interest. Additionally, you might be banned from making new contributions for six months (which some companies impose in that situation), and you will be using after-tax dollars to repay the loan, with interest.
Moreover, if you leave your job or are fired, the loan balance likely will be due immediately. If you fail to repay it, it will be treated as a distribution and subject to a 10 percent penalty plus regular income taxes. Additionally, your plan might charge a fee for allowing the loan in the first place.
Read MorePicking the right benefits plan
Rules for loans differ from plan to plan, so it's important to read your 401(k) plan documents before doing anything.
"Anytime you are thinking about tapping your retirement account early, be entirely sure you understand the rules that apply to your [situation], because not all retirement accounts are created equally," Siebert said.