To make his case, Christie pointed to a Boston College study, which found that between 2004 and 2008 about $70 billion left New Jersey after the 2003-2004 tax hikes during the Jim McGreevey administration.
"They went to places that are lower taxed. This fiction that you can continue to tax people as much as you want as a state and they don't have any place else to go, these folks that are wealthier are the most mobile," he said. "They usually have multiple homes already. So it's not even a matter of them leaving New Jersey. They just have to leave New Jersey for 181 days and then we don't get their tax money anymore."
Read MoreAmerica's most expensive states in 2014
Christie also noted that a 2012 tax hike under the Obama administration led high net worth individuals to accelerate income into 2012 to avoid paying higher taxes in 2013.
"What it tells you is how skewed our tax system is. That when 1 percent of the individuals in my particular state can skew tax revenues that severely, we're relying too heavily on them," he said. "We're taxing them too high and we're forcing them as a result to play games with their accountants that wind up hurting us."
In a highly derisive political environment, Christie said coming to terms on taxes is not always easy, though. Now in his second term, Christie said he's battled tax increases throughout his tenure.
"We're already one of the two or three highest taxed states in the country and I've had to veto income tax increases four of the last five years," he said. "Folks don't seem to understand that you have to make yourself competitive from a tax perspective."
—By CNBC's Drew Sandholm.