There's one big danger with moving money out of stocks and into cash: If the market continues to rise, your portfolio won't benefit from those gains.
Of course, no one has any idea where stocks are going, and just because the experts are nervous about a hot market, it doesn't mean things will go south.
In 1996 Alan Greenspan, the then Federal Reserve chairman, talked about the "irrational exuberance" of the market and told people to be careful about stocks. Whoever sold out that day would have lost out on nearly four years of stellar returns.
"It's a very slippery slope for people," said Larry Swedroe, director of research for St. Louis-based Buckingham Asset Management. "There's plenty of evidence that shows that investors shouldn't time the market."
At the moment, just 3 percent of America's 7,500 equity funds have more than 10 percent of their assets in cash, said Todd Rosenbluth, S&P Capital IQ's director of ETF and mutual fund research, and many of those funds have underperformed the market over the last year.
The Intrepid Small Cap Fund, for example, has about 73 percent of its assets in cash. If you invested $10,000 in the fund a year ago and the same amount in the S&P 500, you'd have $730 more if you stuck with the broader index, according to Morningstar.
The same thing goes for the larger-cap Invesco Charter fund, which has had between 10 percent and 17 percent of its money in cash over the last year. Investors who put $10,000 in the fund a year ago would have made $165 more if they put that money in an S&P 500 ETF.
Of course, these and other funds that hold significant amounts of cash—13 percent of U.S. funds have increased their cash holdings by more then 2 percent over the last year, according to Lipper—could have the last laugh if markets do fall.
Read MoreQuiz: How much do you know about dividends?
"This strategy will pay off if the market pulls back and managers can get in at lower prices," said Rosenbluth. "However, if we see the market move higher, then cash is going to be an anchor for performance."
For Irwin, holding cash is less about stock market fears and more about having money to jump on new opportunities. He agrees that the market timing is risky, and he thinks that without a recession on the horizon and continued low interest rates, the market will continue to climb.
However, that doesn't mean people should just buy and hold.