Can the world's largest retailer afford to its employees $15 per hour? No, says investment manager Peter Schiff, CEO of Euro Pacific Capital and author of "How an Economy Grows and Why It Crashes".

Schiff recently took a video camera to a Wal-Mart and asked shoppers exiting the store if they thought the company should pay its employees more. He then asked they were willing to pay 15% higher prices to make $15 per hour wages possible and, when he tried to collect 15% of the shoppers' receipts, he was rebuffed a majority of the time.

(Read: McDonald's removes worker site after fast food flap)

The point of this exercise, according to Schiff, is to demonstrate that the actions, rather than the words, of the general public are the reason why Wal-Mart isn't able to pay higher wages. A company that has made "everyday low prices" an integral part of its business model would be abandoned by customers should prices rise to pay higher wages, argues Schiff.

"If they actually tried to raise all the hourly pay of the low-skill workers to $15 an hour, Wal-Mart would lose 80% of its profits," says Schiff. "They would have to get rid of their dividend, the stock would crash, [and] the whole business would fall apart. It doesn't work. The only way to do it would be to raise prices and they would have to raise it not a trivial amount. It would have to be a decent amount to offset the reduction in sales that would come from the higher prices themselves."

With nearly $475 billion in revenues over the last twelve reported months, Wal-Mart is the world's largest retailer. However, its operating margin is less than 5.9% compared to the 9.7% for all the companies combined in the S&P 500 index (including Wal-Mart).

(Watch: Retail margins getting squeezed this season: Expert )

"Their markups are very small [relative] to gross sales," say Schiff. "So, even if Wal-Mart customers spent all of those extra wages at Wal-Mart – which, of course, they wouldn't do – Wal-Mart might get back three or four cents for every dollar it pays. It's hardly a good deal for Wal-Mart."

The real problem for Wal-Mart employees isn't the practices of one company, argues Schiff. Instead, he sees it as a deep-rooted economic problem.

"Wal-Mart creates entry level jobs for people who don't have a lot of skills; those jobs don't pay a lot," says Schiff. "Why aren't other companies creating better jobs for Wal-Mart workers to move up to? They used to, but they're not doing it today. So, now you have people trying to raise a family on jobs that were meant for teenagers or young adults that are still living with their parents. But, that is not Wal-Mart's fault."

To see the rest of Peter Schiff's highly charged interview with Talking Numbers, watch the video above.

More from Talking Numbers:

Here's why you should worry about housing: Strategist
Waiting for NASDAQ 5,000? Don't hold your breath, say strategists
Why 2014 could be worse for gold

-----
Follow us on Twitter: @CNBCNumbers
Like us on Facebook: facebook.com/CNBCNumbers