The pay disparity between CEOs and workers is far too great because few executives value physical labor, former Minnesota Gov. Jesse Ventura told CNBC on Tuesday.

"We somehow diminish physical labor. We call that 'unskilled' and therefore we can pay so much more less for someone who has to physically sweat and labor, digging a ditch, whatever it might be, " Ventura said on "Closing Bell." "But where you're educated and you push a pencil, well then you make big money."