As part of its quantitative easing program, the Fed had been buying as much as $40 billion a month of mortgage-backed securities—known as MBS and essentially mortgages bundled into products for investors. However, that buying has been reduced to $10 billion a month as part of a process often referred to as "tapering."
At the same time, Congress is on a path to unwind Fannie Mae and Freddie Mac, the two government-sponsored enterprises that were bailed out during the financial crisis.
Bove credited the MBS program—which was coupled with purchases in Treasurys—with rescuing the housing market from its moribund state prior to the start-up of the third QE phase in 2012.
He similarly pointed out that Fannie and Freddie control about 61 percent of mortgages as a buyer of loans on the secondary market.
Read MoreHigh-end house flipping is soaring
Under the current congressional plan, the Fannie and Freddie GSE system would be replaced by one in which a Federal Mortgage Insurance Corporation would replace the two entities. Part of the plan would see private capital take the first 10 percent of losses in case of default, a provision that has drawn critics who say the level is too high and will discourage investors.
While banks have stepped up their mortgage buying this year, Bove noted anecdotally that those institutions are unwilling to take on the risk of 30-year, fixed-rate mortgages.
"While these banks are not willing to make public statements similar to those of the industry's leaders, they all agree that the risk in making loans to low-income households is too high," he said. "The fines, lawsuits and put-backs associated with those loans make them unprofitable."
Read MoreRefinancers act on lower rates, home buyers don't
Unless someone fills that vacuum, the prospects for housing remain troublesome.
"One or the other of these institutions must step up to provide the funds since there is no entity that can take their place in the near or even intermediate term," Bove said. "Thus, the source of my fear is clear. If the government is adamant that the GSEs must go, housing prices will fall. This will be a crisis."
—By CNBC's Jeff Cox.