"These are real serious mistakes to make no matter how many gold medals he wins in Rio," Rishe said.
Phelps' representatives did not immediately respond to CNBC for comment.
Besides his history, endorsers may also shy away from the 31-year-old Phelps because his visibility is decreasing. Phelps will not get more television exposure than he's receiving now. He has publicly said Rio will be his last Olympics.
According to Apex Marketing Group, Phelps has appeared in over 67,000 broadcasts on TV and radio since opening ceremonies.
"There will really be a drop off in visibility so companies will really be cautious about engaging in contracts w/ those types of athletes," said Rishe.
Darren Marshall, executive vice president of consulting and research at rEvolution, thinks Phelps marketability still has an upside. "Fans have the memories of goldfish," Marshall said. "They don't remember something that happened two months ago let alone two weeks. All they care about is the here and now."
Ironically, Phelps' marketability could also be impacted by mere the fact that he's an Olympian.
"There is a reason that Michael Phelps is not getting let's say Rory McIlroy, Jordan Spieth money in terms of endorsements. These are pro golfers in the limelight on a regular basis. If we believe this is Michael Phelps' last Olympics, then he's not going to be in the limelight going forward."
Phelps' current endorsement portfolio consists of Under Armour, Omega and a slew of smaller companies. Rishe said Under Armour is his strongest relationship due to the home-town connection.
Baltimore-based Under Armour said it is proud to have the Towson, Maryland, native lead its Olympic roster. "His story, like the stories of many of our athletes, is one of hard work, dedication and sacrifice. … We were honored to share that with the world," the company said in a statement.
So, who is Rio's most marketable athlete? Rishe says the women's gymnastics "Final Five."
"Simone Biles, if she meets expectations — I think you are going to see her cash in."
Don't worry about Phelps' decreasing marketability. Rishe estimates his net worth is $50 million to 55 million. Plus, if he's ever looking for extra cash, he can always cash in one of his many gold medals.
UPDATED: This article was updated to include comments from Darren Marshall at rEvolution.
DISCLOSURE: CNBC parent NBC Universal owns NBC Sports and NBC Olympics. NBC Olympics is the U.S. broadcast rights holder to all Summer and Winter Games through the year 2032.