Harris, co-founder of private equity giant Apollo Management, echoed the sentiment.
"It's a public trust," Harris said of owning and investing in the 76ers and Devils.
"For me it's not about reading it in the press. It's about doing the right thing to build a high-quality organization up and down to attract the right people," Harris added. "It's like every other business, you need the right people. In order to do that, you have to have very high-quality facilities, you have to do everything in an A-plus manner, you have to have the right management, the right coaches, the right culture."
The 76ers recently announced that they would build a major new practice facility in Camden, New Jersey, with the help of tax subsidies.
"You have to think about the financial returns over a longer run basis versus saying 'OK, we put $60 million in, are we going to make $10 million more?'" Harris said. "You don't know. You're not. Not tomorrow."
Read MoreBig hedge fund money behind the Cardinals, Red Sox
Harris led a group that bought the 76ers in 2011 for $287 million. In 2013, he similarly helped a group buy the Devils and their Newark arena for $320 million.
—By CNBC's Lawrence Delevingne