He told "Power Lunch" there have been times when the administration had the high ground, like when Spicer denied that the bust of Martin Luther King was removed from the Oval Office, contrary to a reporter's tweet.
"I do not think the White House had high ground to get into a meaningless argument about how many people showed up to watch the inaugural. Who cares?"
According to figures issued by Washington's transit authority, Metro ridership during Friday's inauguration was the lowest since at least 2005, The Washington Post reported. By 11 a.m. EST, approximately 193,000 trips had been taken on the Metro, compared to 513,000 trips at the comparable time in 2009.
In addition, Nielsen ratings data showed that an estimated 31 million Americans watched the transfer of power. That figure was lower than Barack Obama's first inauguration in 2009, when an estimated 38 million watched, and below the record 41 million that watched Ronald Reagan take his oath of office.
Fleischer, who served under President George W. Bush, believes Spicer was only following orders during Saturday's briefing.
"Sean was doing what the president told him to do. I can only picture the scene … when the president said to Sean, 'You need to go out and you need to fix this and you need to fix it now,'" he said. "The poor press secretary is put under tremendous pressure when that happens to quickly collect facts, gather data, make the case."
And mistakes are going to happen, Fleischer said. He thinks Spicer's best response is to own up to it, give the updated information and reiterate his broader point.
That is what Spicer did during his second briefing on Monday, when he admitted that the Metro ridership numbers he quoted on Saturday came from an outside agency and were different from the transit authority's figures.
"There are certain things that we … may not fully understand when we come out. But our intention is never to lie to you," he said.
"We're going to do our best every time we can. I'm going to come out here and tell you the facts as I know them, and if we make a mistake, we'll do our best to correct it."
When backing up his claim that the inauguration was the most watched in history, he cited online viewing, which is not included in the Nielsen ratings.
— CNBC's Javier David contributed to this report.