Warren Buffett calls this trait his 'clear weak point'

David A. Grogan | CNBC

Investment firm Berkshire Hathaway is one of the world's most successful companies, thanks to the shrewd leadership of Warren Buffett.

But even the self-made billionaire has shortcomings.

Though he’s very disciplined about most aspects of the business, the legendary investor said at his company’s 2014 shareholder meeting that when it comes to hands-on management, he’s “sloppy.” Specifically, Buffett doesn't like letting go of managers at his subsidiary companies or telling them what to do.

“A clear weak point of mine would be I’m slow to make personnel changes,” the billionaire admitted. “There will be times when, what you might call, our lack of supervision over subsidiaries [means we] miss something.”

With his vice chairman Charlie Munger beside him, Buffett recounted a scenario where an ineffective manager was allowed to stay on for too long.

“Charlie and I had a wonderful friend who couldn’t have been a greater guy. And, you know, we were slow to make a change there. We loved the guy,” Buffett recalled. “We’ve waited too long on managers."

Warren Buffett explains how his late wife convinced him to raise money for a church in a rough part of San Francisco

Buffett isn't the only manager who can be squeamish about cutting underperformers loose.

After resigning from the CEO role at Panera Bread in January, Ronald Shaich said that his was not firing more people. Instead of confronting underperforming workers, Schaich wrote in an article for Entrepreneur, he covered for them — because they were akin to family.

“I was willing to do their work,” he said. “Time and time again, that hurt the organization.”

Although Buffett contends that giving subsidiaries a wide berth is a weakness, both he and Munger believe that the benefits of a looser management style far outweigh the costs.

“I think a lot of places work better when they create a culture of deserved trust,” said the vice chairman, noting that they carefully select managers who are worthy of this confidence.

And he may have a point.

3 psychological habits Buffett's partner Charlie Munger warns can hurt your career

“Working for a micromanager can be a suffocating experience,” Amanda Augustine of TopResume tells CNBC Make It. A majority of workers say it’s the a boss can have, according to a survey by Comparably.

Buffett added that while people can point to specific instances which could have been avoided if he were more of a disciplinarian, they aren't getting the full scope.

“What they won’t be able to measure is how much on the positive side we have achieved with dozens and dozens of people because we gave them that same sort of leeway,” Buffett explained.

“We think that giving our managers the degree of freedom that they enjoy will also accomplish a lot.”

Like this story? Subscribe to CNBC Make It on YouTube!

Don't miss: 3 things two men learned from the $650,000 lunch with Warren Buffett

This rule from the Marines is the secret to successful leadership