"The housing market surely will double dip," Whitney added. "The asset classes of MBS and Treasurys are priced for a material correction in my opinion. "
But others think the market is already prepared for such an event. The Fed let the market know months in advance that it would stop buying mortgages, a measure it initially employed to goose demand and keep mortgage rates low.
The Fed did surprise the market last month when it announced it was hiking the discount rate—a largely symbolic rate that the central bank charges banks for emergency loans—by half a percentage point. The market initially wobbled but quickly adjusted.
In fact, yields on the benchmark 10-year Treasury note have been fairly flat.
"By and large the end of the purchasing program is priced in," said Zach Pandl, economist at Nomura Securities in New York. "The Fed has signaled in its public comments that it is willing to bring the mortgage purchase program back if the economy disappoints. That limits the extent to which mortgage rates are going to rise."
To be sure, that analysis looks only at the immediate impact of the MBS program's end. The future could tell a different story.
The Fed has indicated only when it will stop buying mortgage-backed securities but has yet to disclose when it will begin to sell the billions already on its books. An influx of supply at a faster rate than the market can handle could ultimately drive up mortgage rates.
"Where the Fed may think they can get out of this for 40 or 50 basis points (0.40 to 0.50 of a percentage point), at the end of the day it may be 150 basis points," Yra Harris, of Praxis Trading, said in a CNBC interview. "That's something that nobody knows and they're depending on their models to tell them. But I think we've all learned some of their models may be suspect."
Investors who have had to accept lower yields because of artificially high demand created by Fed MBS buying may demand a better return once the mortgage sales begin, Harris said.
"The Fed has perverted the curve as they have done this and a lot of investors have had to either pay or get a a lower rate of interest than they otherwise would have, and I think they're going to exact some pound of flesh going forward," he said.