Investors can exit the stock market for plenty of reasons, but the Federal Reserve's monetary policy shouldn't be one of them, Josh Brown of Fusion Analytics said Thursday.

"I wouldn't take the trial balloon the Fed put out yesterday in their minutes to think that they really have any choice in the matter right now," he said on CNBC's "Fast Money."

Stocks declined Wednesday as the FOMC meeting minutes were released, hinting at a shift in its quantitative easing with the words, "several on FOMC said Fed should be prepared to vary pace of QE."

(Read More: Fed Talk on Slowing QE Hits Stocks)

Brown pointed out that the U.S. Treasury Department was not paying much more than $150 billion per year in interest expense.

"For anyone to think that the Fed has an interest in disrupting that I think is kind of silly," he said. "If you want to get out of this market, do it because of overvaluation, a little too much momentum or you're seeing a little bit of change in terms of leadership. But don't do it because you think the Fed's about to yank the punchbowl away. It's very unrealistic."

Brown shrugged off declines in the major indices on Wednesday, with the S&P 500 posting its first down day in 2013.

"Down 1 percent after a straight-up 8 percent move with almost no down days," he said.

TheStreet CIO Stephanie Link said that the correction has been well anticipated.

"I do think that we've been waiting for a pullback," she said. "Now that we get it, go back to themes that we've talked about. I don't think housing is dead by any stretch of the imagination. "

Link said that she had added long positions in Weyerhaeuser, TJX and Cisco, among other names in aerospace and autos.

"There's a lot of names out there that, on the pullback, you can be buying," she added.

Trader disclosure: On Feb. 21, 2013, the following stocks and commodities mentioned or intended to be mentioned on CNBC's "Fast Money" were owned by the "Fast Money" traders: Josh Brown is long AAPL; Josh Brown is long BAC; Josh Brown is long GLD; Josh Brown is long XLU; Josh Brown is long TLT; Josh Brown is long XLF; Josh Brown is long WMT; Josh Brown is long TGT; Josh Brown is long GOOG; Jon Najarian is long ADP; Jon Najarian is long ADP CALLS; Jon Najarian is long GSK CALLS; Jon Najarian is long BWLD CALLS; Jon Najarian is long CSCO PUTS; Simon Baker is long C; Simon Baker is long JPM; Simon Baker is long GOOG; Simon Baker is long PAY; Stephanie Link is long AAPL; Stephanie Link is long GS; Stephanie Link is long CSCO; Stephanie Link is long SBUX; Stephanie Link is long FB; Stephanie Link is long HD; Stephanie Link is long TJX; Stephanie Link is long WY.