However, the stock has gotten relatively cheap and the bank hasn't suffered a material financial hit, Miller noted. He thinks any impact on earnings per share will be somewhat minimal, with about 3 to 4 cents being shaved off.
That said, he's waiting to see how bad things could get with Wells Fargo's reputation.
"If I start to see the reputation really take a hit, where people are leaving en masse, where it's not just a 3 cent hit or a 5 cent hit, that the reputation itself is going to have a meaningful material impact on earnings, that'll make me change my opinion."
He's also concerned about the banking sector in general, which he thinks is going to be hit with more regulatory concerns and costs.
Earlier Tuesday, Morgan Stanley upgraded shares of Wells Fargo to overweight from equal weight, citing among other things its attractive valuations.
"Yes, we could see more volatility ahead as headline risk persists, but Wells is rarely this inexpensive. This is an opportunity," wrote equity analyst Betsy Graseck in a research note.
— CNBC's Giovanny Moreano contributed to this report.