The U.S. Department of Agriculture census seems to back up Vilsack's complaint and his stated "need to expand the rural economy from the middle out." Large farms with over $1 million in sales account for only 4 percent of all farms, but 66 percent of all sales. That's up considerably from 1 percent of all farms and 50 percent of all sales a decade ago.
However, three quarters of all U.S. farms gross only $50,000 a year and currently account for only 4 percent of product sales. But one analyst doesn't see that as a problem.
Read MoreWhite House issues dire warning on climate change
"It's not terribly surprising that a relatively small number of farms produce the lion's share of products," said Darren Hudson, a professor of agriculture and applied economics at Texas Tech University.
"Most of the smaller farms are 'hobby' farms with significant sources of off-farm income," he said. "They are large in number but have very small output."
Hudson argued it's because of the production from bigger farms that consumers enjoy lower food prices than they would otherwise—since those farms produce large amounts of crops cheaply through modern technology.
Read MoreWanted: More young people for an old US industry
If the recent findings—which cover the years from 2007 to 2012—do one thing, said Justin Gardner, a professor of agribusiness at Middle Tennessee State University, they dispel a common misconception.
"Most farms in the U.S. are not the big corporations people think," he said. "If you look closer I suspect that you will find that the majority of the 'corporate' farms are large family farms that found it beneficial to form a corporation."