Swartz said the two biggest reasons HTC can't get ahead are Apple and Samsung, which according to Swartz, make HTC "irrelevant."
"Unless you're one of the top three vendors, you're pretty much on the outside looking in," he said.
A number of other reasons are keeping HTC from earning top spots in an already saturated market, including poorly accepted ad campaigns and long lag time between announcements of new devices to the time of sale.
HTC is hopeful its newest phone, the HTC One (M8), released a few weeks ago, will catch and keep consumer attention.
The problem isn't with the quality of the phone though, according to Swartz. "You could have the best product in the world, but if people don't buy it or don't consider it a must-have, you're in trouble, things aren't going to change," said Swartz.
In a statement in February, HTC CEO Peter Chou said: "We will continue to stay focused on making the best smartphone and building a compelling mid-range portfolio. Meanwhile, we are going to communicate better with consumers."
In response to request for comment from CNBC, HTC said it does not comment on unaudited results.