After a rough week, billionaire money manager Ken Fisher is apologizing.
Fisher, who is best known to a broad audience for his distinctive TV ads, was accused in a viral tweet (well, as viral as a tweet in the financial advisor community can go) of making sexist comments while speaking at the Tiburon CEO Summit on Tuesday before more than 200 attendees.
The state of Michigan has pulled $600 million of its pension fund from his wealth management firm, Fisher Investments, in the wake of the sexist comments, according to The Washington Post.
CNBC has obtained an audio recording of those statements, as well as audio from Fisher speaking at a previous conference. Combined, they show that Fisher made flippant remarks about sex.
Clips from both were featured on CNBC Power Lunch on Friday.
Organizers of both financial conferences subsequently banned him from speaking again in the future.
"What I witnessed was Ken telling other people how to view women, employees and non-profit organizations," said conference attendee Rachel Robasciotti, principal of Robasciotti & Philipson in San Francisco.
"That's not okay," she said. "That is a lack of responsibility to the power you have when you speak publicly and when you're a leader in the industry."
Alex Chalekian, CEO of Lake Avenue Financial, attended the conference and recounted Fisher's comments in his now-viral tweet.
The Tiburon CEO Summit has a "no media" policy, and attendees are asked not to quote speakers without their approval.
"While Alex did technically violate the Tiburon CEO Summit media policy, he recognized that the issues of dignity, respect, and inclusion are more important than the Tiburon CEO Summit media policy, and he took action," said Chip Roame, managing partner of Tiburon Strategic Advisors, sponsor of the conference.
Fisher, whose Camas, Washington-based firm manages more than $100 billion in assets, apologized for his comments on Thursday in a statement from his representative.
"Some of the words and phrases I used during a recent conference to make certain points were clearly wrong and I shouldn't have made them," he said. "I realize this kind of language has no place in our company or industry. I sincerely apologize."
The comments are a reminder that the financial services industry is very much a boys club at its highest levels.
"The industry being predominately male is still unable to see the death by a thousand cuts, the part of being a minority in this industry," said Sheri Fitts, founder of Shoe Fitts Marketing, a consultancy in Portland, Oregon.
"It might just be one guy who's saying something inappropriate, and you brush it off and move on, but when you add them up over the lifetime of a career?" Fitts said. "It's debilitating and exhausting. Men don't see that."
In the audio obtained by CNBC, Fisher says at the Tiburon conference: "Money, sex, those are the two most private things for most people," so when trying to win new clients you need to be careful.
He says: "It's like going up to a girl in a bar … (inaudible) …going up to a woman in a bar and saying, hey I want to talk about what's in your pants."
In light of his remarks, Fisher will no longer be invited back to a Tiburon CEO Summit, said Roame.
This wasn't the first time Fisher has spoken crassly in public.
Indeed, when Fisher was a speaker at the Evidence-Based Investing conference in 2018 he compared marketing mutual funds to propositioning a woman for sex at a bar.
"I mean the, the most stupid thing you can do, which is what every mutual fund firm in the world always did, was to brag about performance, uh, in, in a direct mail piece, which is a little bit like walking into a bar if you're a single guy and you want to get laid and walking up to some girl and saying, 'Hey, you want to have sex?'" Fisher said, according to audio obtained by CNBC.
He also made comments about his virility during the Evidence-Based Investing conference. When asked what he would do differently if he could start his career over, he said the following:
"I don't know if I could, but if I was 30 years old and I had to do over again, I'd have more sex while I, while I could, while I could, when once you get older, you know, once you get older, you know you're like a Christmas tree, you know you're, you're firm once a year and the balls are just for decoration," he said.
He has also been banned from the Evidence-Based Investing conference, according to Andy Melvin, managing director of the Information Management Network.
"I'm really concerned that he feels totally comfortable saying these things," said Courtney Ranstrom, a certified financial planner and co-founder of Trailhead Planners in Portland, Oregon. She attended EBI West and was present for Fisher's session.
"That's the thing; you can behave this way and still be a successful male – and that's tough," she said.
People in the financial services business need to hold each other accountable and create an environment that is safe and inclusive, said Brian Ross, CEO of Flyer in New York. He was also an attendee at the Tiburon conference.
"Speak up when something seems off," he said. "Don't be idle in the face of sexism, racism and harassment."
"Our industry will get even bigger and better if good people do more of this one simple thing — stand up for what's right," Ross said.
CNBC's Leslie Picker, Scott Zamost, Caroline Skinner and Jennifer Schlesinger contributed to this report.