Sacca, who also founded Lowercase Capital, announced in April that he is retiring from venture capital and "Shark Tank."
He went on to discuss how his appearance on the Forbes Midas List changed how people perceived him.
"That was the first time...the word billionaire was associated with me, and it just flipped everything," he said. "So the only time I get challenged now is when I'm arguing with other billionaires."
He joked, "When I'm in a crowd or arguing online and stuff like that, it's just fun because, if I argue with anybody else, they defer to anyone who's called a billionaire, in a totally non-deserving way."
But he doesn't necessarily think that billionaires should be treated differently. "There's just this aspirational culture that I think has elevated successful business people," he said. "It's been elevated to this pseudo-religious deference that I think is -- it's disrupting politics obviously, it's kind of skewing our values and it's dangerous."
Watch: Why Sacca crashed meetings with Google's co-founders