"If they want [the stock market] to be more than an alternate casino for retail punters, then they need to foreign institutions more involved because there is no local institutional investor base to speak of," Snowball said. "Increasing supply via some successful and sensibly priced IPOs and increasing access to foreign investment [is needed to] become a more serious market."
The concerns are keeping some institutional investors out of Vietnam.
"Vietnam at this point in time is still very much a private equity market. It's still not ready for institutional investors," said Jalil Rasheed, director for Southeast Asia at Invesco, citing difficulties including how long it takes to settle trades and getting approval to swap currency.
Read More Vietnam to US: Sell us some weapons, please
Like Snowball, Rasheed is also concerned that the market doesn't have enough stocks available.
"The top ten stocks make up about 80-90 percent of the index and they're largely state-owned enterprises," Rasheed said, adding corporate governance isn't up to the standards of much of Southeast Asia.
There's another factor which could weigh on Vietnam's shares: it's an oil producer. Since this summer, Brent has fallen from above $115 per barrel to around $66.08 in Asian trade Tuesday, with many oil analysts predicting prices will continue to slide.
PetroVietnam Gas, the largest company by market capitalization, has been leading the market down recently, Snowball said.
—By CNBC.Com's Leslie Shaffer; Follow her on Twitter @LeslieShaffer1