"I'm not saying people who study psychology or environmental studies never get jobs," adds O'Leary, "it's just that I was having a hard time getting one," he laughs. "It was the first time I realized making an investment in a college is not just about where you go. It's what you take."
O'Leary's step-father, George Kanawaty, urged him to go back to school and study business.
"When I started my undergraduate degree, I had no idea that I would want to pursue an MBA," O'Leary writes in his book "Cold Hard Truth: On Business Money & Life." "But George convinced me it would give me a practical set of skills. He called it a toolbox. The more skills I put in my toolbox, the greater my chances of success."
So in 1979, O'Leary enrolled in a Master's of Business Administration program at the Ivey Business School at Western University. It was an experience that O'Leary says helped lead him to success in entrepreneurship, but for an unexpected reason.
"The great thing about an MBA is not the technical skills you've learned — because frankly, to be honest with you, I forgot all of those — it was the people I met in my class," O'Leary explains.
"Where are they now? Running banks, they're industrialists, they're venture capitalists, they're investors, they're all around the world," he says. "I can pick up the phone and say, 'Hi. Mr. Wonderful here, let's talk about a business idea.'
"The ability to just have that contact with them ... ends up being more important than the business degree that got you that relationship in the first place," O'Leary adds.
His fellow shark Daymond John feels the same way about going to college.
"The number one reason why people are successful is access to mentors," John told CNBC Make It. In the early years, those mentors "are professors, those are other students in college around you."
Attending a business school at one of the top 20 programs in the U.S. will likely cost you between $55,000 and $68,000 a year, according to a 2017 report by U.S. News & World Report. And while connections with colleagues are valuable, O'Leary says you can also be successful without paying for them.
"Here's the cold, hard truth about business degrees: They are not a prerequisite for success," O'Leary says. "Many billionaires never finished high school.
"Being an entrepreneur is not about what's taught in school, it's having street smarts and understanding opportunities in real life."
Still, if you take advantage of an MBA program — or any educational opportunity — in the right way, O'Leary says it can pay off.
"The real value of an education is who you meet while you're getting it," he explains. "Think about that if you're in college right now."
Don't miss: 'Shark Tank' investor Kevin O'Leary: Here's how to get rich
Like this story? Like CNBC Make It on Facebook!