Then he said, "How are you doing? When you coming back to work? I got a seat open next to me. You could be my partner. It would be great."
I told him, "No thanks, but that sounds like a great offer. I would consider it but I like what I am doing now. And I really like my hair."
Read MoreThe 12 types of people on Wall Street
Don't get me wrong, I loved working on Wall Street. I was a trader for 13 years and loved every minute of it. I do miss my friends. And the rush of making money a trade. The dinners where everyone else in the restaurant despises your whole table. But there are more than a few things I DON'T miss. Here are the top seven:
1. Getting blamed for all that's wrong with the world. There is a segment of the population that still blames Wall Street for everything. A guy came up to me after a show recently (I'm a comedian) and said, "You worked for Bank Un-America? Wall Street is manipulating the political parties to divide the Tea Partiers and Occupiers and conspiring to manipulate the whole system to make everyone else your servant. "
I told him, "You are giving Wall Street WAY too much credit. All we want is bigger bonuses."
Read MoreSmaller Wall Street bonuses mean the Russians win
And so what? Who doesn't want more money? Every person in this country wants to make more money whether it be a raise from their employer, selling their house, or winning the lottery. And if you receive more money than you want, then give it to a worthwhile charity like The Make a Wish Foundation, Kids without iPhones or 1 Percenters with Small Hamptons Homes.
2. Meetings to have meetings. These used to drive me crazy. Most of them were such a waste of time. "We are having this meeting so we can get all our ducks in a row," one manager might say. This guy I used to work with wanted to have a meeting at 6:30 am on a Friday about how to do more business with a client. My advice is: Take him out on a Thursday night and spend $500 and show him a good time. So on Friday we can come in at 7:30 like normal people and he can pay us $20,000 in commission today.
3. Excessive supervision and compliance. Due to bad Wall Street behavior, one area that is still in a huge bull market is compliance and supervision. I have no problem with supervision and compliance but it has become RIDICULOUS. My favorites were the "Know Your Customer" and "Anti-Money Laundering" courses we would have to sit through.
They'll ask you questions like, "If your client walks in with $10 million dollars in a briefcase what should you do?"
Of course, anyone with half a brain knows the answer is to take it into the other room and never come back.
Or, "If you suspect your client is an arms dealer what should you do?"
The answer is, of course, "Do anything you can not to get yourself killed!"
Read MoreHere's how a trader's brain REALLY works
I used to walk into the "Anti-Money Laundering" course and ask the instructor, "Is this the money laundering course?"
He would sternly say, "The ANTI-money laundering course."
I would always get in trouble but I couldn't help myself from replying, "Ah, OK. I already know how to launder money so I don't really need this course."
4. Always on the BlackBerry. I was ALWAYS on my BlackBerry and Bloomberg Anywhere. When I would wake up in the middle of the night to use the bathroom, the first thing I would do is check where S&P futures were trading. If I was on at date and she went to the bathroom, I would immediately grab my phone to check the futures.
It's an annoying habit to anyone you're with — more annoying than knuckle-cracking, posting gym selfies or talking about how awesome you are because you're a vegan.
5. Office politics. They say Wall Street is a performance-based business and that's true to an extent. However, it can get more political than Washington, DC. Even if you have had a great year, you have to show the proper face time, pretend you care how your boss's trip to Turks and Caicos was, and come in with shirts with blown out elbows to make sure you get your well-deserved bonus. It's never black and white.
Read MoreAnd the most attractive bankers are ...
That process was even more arbitrary for getting promoted. If you were up for a promotion, you had to write an autobiography like every presidential candidate does. Then you would have to meet with someone who doesn't necessarily know you and have him present your case like you are on trial for murder. The first time I was passed over for managing director, the "judge" determined that, while I was completely worthy, I would be passed over because I needed another year of seasoning — like a Cajun Ribeye!
6. Getting overcharged. When people know you work on Wall Street, it sets off a fire alarm. Whether it be at a fancy restaurant, the dry cleaner, or the tailor from Hong Kong, they look to take advantage and overcharge you. All wealth is relative. If you are perceived well to do, you have a bullseye on your chest. I don't mind spending top dollar on necessities like Armani suits, monogrammed underwear and Adderall. I just don't want to be overcharged because you are a playa hater.
7. Monday morning quarterbacks. When you are making money nobody ever says a word to you other than, "Keep up the good work." You could be making trading decisions because of your biorhythm calculator, following Nostradamus or your Voodoo doctor told you to. Nobody cares. Everyone loves a winner.
Read MoreHomeowners: A new class of fools
However, if you are losing money you get so many emails, you would think you are wiring money to Nigeria. You have to explain why you bought AAPL, what your reasoning was and why you thought shorting gold was a good reason, considering LeBron was going to the Cavs.
Imagine Peyton Manning having to send emails to the coach, GM, and the cheerleaders every time he threw an interception. Or Carmelo Anthony having to send emails to the owner, the team sponsors and Obama every time he missed a shot. That's what it's like when you are losing money. And then everyone has a different opinion on what to do next, so if the janitor is right and you are wrong, you will get an email from him wondering why you didn't listen to him.
And if you are 100-percent right, then you can guarantee, someone else will claim responsibility!