Market curious about Fed's 4 trillion 'shades of gray'

As Fed Chair Janet Yellen prepares the world for the first Fed rate hike in nine years, she also may have to begin to address the $4.5 trillion elephant in the room.

That is the approximate size of the Fed balance sheet, and central bank officials at some point will be allowing the hundreds of billions in mortgage securities and Treasurys it added to its balance sheet during years of quantitative easing to roll down.

But the question is how and when the Fed will allow that to happen without ruffling financial markets and triggering a run up in long-term interest rates, as it slowly raises short-term rates by lifting the fed funds rate. The Fed is not expected to address the balance sheet in its statement, but Yellen may get a question about it in her post-meeting press briefing Wednesday afternoon.

"Markets want black or white. This is shades of gray. Four trillion plus shades of gray," said Mesirow Financial's chief economist, Diane Swonk.

"It's a quagmire. There's so many uncertainties surrounding the balance sheet. It's easier getting to zero than it is getting off the zero bound. It's easier doing quantitative easing than getting out of it."

Read MoreWall Street sees Fed hike in Q3

The Fed is widely expected to slowly reduce its balance sheet—or slowly taper it, well after it begins raising rates.

"They'll keep their balance sheet constant. My guess is they keep it more measured. It's like a reverse taper ... they started with a big purchase and stepped them down. This will start low and allow their balance sheet to drain off more aggressively. They'll want to have some control on the long end, and that's what they're worried about. And they should be," she said.

While very few Treasurys matured this year, the Fed could be an active buyer next year, as an estimated $200 billion in securities are set to mature in 2016.

"They're basically just going to try to keep things very stable right now," said Swonk. "As things mature, it's lumpy and uneven. That's not what they want to influence markets with. They could distort markets with a big runoff."

Read MoreFed running out of reasons to delay rate hike

The questions about how the Fed will handle the balance sheet have been out there since the Fed first launched quantitative easing, or its bond purchase programs. But it has taken on more importance because of a recent runup in long-term rates, as the central bank moves toward hiking short-term interest rates. Those rate rises are widely expected to begin as soon as September, and continue at a slow pace with the Fed leaving rates unchanged at some meetings and hiking at others.

Read MoreThis would be scary enough to give Fed pause

"If they were to release (details) at this meeting, this would be a surprise, and rates would rally hard," said Nomura rate strategist George Goncalves. "If they were to save it for September, it would dampen the blow of rising rates. It's not insignificant."

As part of the process of keeping the balance sheet steady, the Fed could extend its reinvestment program, where it buys securities to replace the maturing Treasurys and mortgages balance sheet, analyst say.

The reinvestment program also became a bigger topic in markets after it was discussed by New York Fed President William Dudley recently.

Read MoreFed's worst nightmare: The 'ghost of 1937'

"New York Fed President William Dudley less than two weeks ago said that he preferred the Fed wait until the federal funds rate were at between 1.0 and 1.5 percent before the Fed stops reinvesting the principle and interest payments it receives on its securities holdings," wrote Pimco strategist Tony Crescenzi in an email. "The timing will be important to investors, because the Fed has been heavily influencing the markets it participates in, including the agency mortgage-backed securities market and the U.S. Treasury market. Dudley's comments are sure to result in questions on the matter."

While there is plenty of speculation about the balance sheet based on various comments, there has been no definitive plan announced by the Fed, and markets expect more clarity once the central bank begins to raise short-term interest rates.

Fed officials, going back to Chairman Ben Bernanke, have said it was not the purchases in QE programs that held down rates, but the fact that the Fed kept the securities on its balance sheet.

Read MoreWall Street predicting Greek tragedy: CNBC Fed survey

"I think the focus is on the first rate hike. They've put out guidance in terms of continuing reinvestments until probably a few quarters after the first rate hike," said John Bellows, Western Asset Management portfolio manager. "I think they've tabled the balance sheet for the moment and we're going to hear more about the balance sheet sometime next year, after the first rate hike."

Bellows said it's not clear whether the Fed will continue purchasing Treasurys, mortgages or a combination, but it may ultimately use the balance sheet to tighten.

"When they tighten, they have two tools to tighten with. One tool is rate hikes, and the other tool is contracting the balance sheet. To some extent, they can trade off those tools. While they might use the rate hike at first, at some point in the next year they could switch and use the balance sheet," he said.

Goncalves said the market is eager for clarity sooner.

"The problem with this is such a large component to the supply and demand of Treasurys. It could create unnecessary and undue volatility if they don't clarify their intentions and wait to the last minute," said Goncavles. The "last minute" would be when $200 billion begins to mature next year.

"$200 billion that's large enough to impact the market. They're going to have say whether they are reinvensting. Are they going cold turkey or are they going to taper?" said Goncalves.

He said if the Fed did nothing, the longer-term duration Treasurys that are set to mature could affect lending rates for corporations and consumers. "Higher yields on the long end, we've already seen play a bigger role in investment and housing activity than the move from zero to slightly higher in short rates."

Goncalves said some Fed officials favor slowly moving to shrink the balance sheet, while others would be more aggressive.

"The balance sheet comes as a natural result of QE, and since markets have been obsessed with QE, why would you want to see QE unwind?. If QE is what got markets and the economy strong enough then can we afford a shrinking of the monetary base? Can we allow the balance sheet to shrink. You're basically unwinding QE," he said.

Goncalves said Dudley's speech led some investors to believe the Fed was discussing the balance sheet and the reinvestment program.

"I think the way the Fed has been discussing the adjustment in financial conditions—which is code for long-term rates, stock market and credit spreads—if one of the three is going the wrong direction, the Fed gets nervous. The fact we had a tightening in long-term rates for the first time in a long time means they've been discussing it," he said.