It's not that America can't ever grow at such rates without generating inflation (as it did in the booming late 1990s)—it's just that this particular post-crisis, weak-productivity period points that way, according to Harris.
He isn't even as pessimistic as some, like Northwestern University professor Robert Gordon, who predicts growth for the quarter century after 2007 "will be much slower, particularly for the great majority of the population," with labor productivity mired at a 1.3 percent annual growth rate and overall U.S. growth per capita below 1 percent.
Harris is optimistic by comparison. Still, he sees trend growth of only about 2 percent for the U.S. economy for the next five to 10 years.
"With each decade, we're going to get slower growth in the working-age population, and that can only be made up for with faster productivity," he said in an interview last month.
That trend growth (adding population and productivity growth together) is somewhat akin to the economy's "speed limit."
Indeed, with the unemployment rate falling back toward 5 percent and signs of wages edging up there are some, like Deutsche Bank's chief international economist, Torsten Slok, who argue we've already hit the nonaccelerating inflation rate of unemployment.
A lower speed limit makes 4 percent growth more "dangerous" than if America's trend growth rate was much higher. It means less slack in the economy and less growth in the standard of living.
It could mean, yes, "double-digit inflation," or, it could manifest as a bubble in stocks, housing, or something else.
But that's only if trend growth—population plus productivity—does remain weak.
There are still plenty who credibly argue America's trend growth hasn't been damaged for good and will surprise to the upside.
Deutsche's Slok, for one, actually thinks growth has been weak simply because of the cleaning-up process after the crisis and will start to accelerate again, keeping inflationary pressures at bay.
Goldman Sachs goes further, arguing the measured slowdown in productivity in recent years is in part just a measurement error of technology
Read MoreShockingly weak productivity haunts US job gains
No wonder the Federal Reserve keeps emphasizing that it is "data dependent" when it comes to raising interest rates.
As all presidents and policymakers know, it's not just getting to 4 percent growth. It's what kind of 4 percent growth that really matters.