No matter what the Federal Reserve announces Wednesday on tapering, it "can't change housing demographics," Toll Brothers CEO Douglas Yearley told CNBC. That's why he said he won't even hazard a guess on what the Fed will do.
"It's not something we focus on tremendously," Yearley told "Squawk Box" on Wednesday. "Obviously the decision will move our stock one way or another, because that's how Wall Street trades."
The sharp rise in bond yields since the Fed taper talk began earlier this summer has led to higher mortgage rates, but Yearley said business at homebuilder Toll Brothers hasn't really been affected.
(Read more: Mortgage apps creep off 5-year low)
"Our average house is over $600,000, so we sell to the second, third time move-up," he said. "They don't have the mortgage issues. Twenty percent of our buyers are all-cash and those that get a mortgage put 30 percent down."
He did acknowledge that the move in 30-year mortgage rates from "3.5 percent to 4.75 percent in what felt like a couple of days" was faster than anyone thought and did "shock the market short-term." First-time homebuyers may need a little time to adjust, but those rates are still historically low.
He said he'd take a 4.5 percent to 5.5 percent mortgage rate in a good economy because from "2000 to 2006, which was a roaring time for housing, rates were 5.5 percent to 8 percent."
(Read more: Home builder confidence stalls in September)
"What we have [had] is five years when very few people bought homes and we built very few homes as an industry," Yearley said. "There's a lot of catch up that's necessary just to handle household formations, immigration, obsolescence from existing homes that are being torn down."
"I think we're in the early stages of recovery," he said.
—By CNBC's Matthew J. Belvedere. Follow him on Twitter @Matt_SquawkCNBC.