Shares of the automaker fell 13.9 percent on Friday, their worst day since November 2013.
The Securities and Exchange Commission sued Musk on Thursday, alleging for fraud. The complaint says Musk issued "false and misleading" statements and failed to properly notify regulators of material company events. Musk called the SEC's allegations "unjustified" and said he "never compromised" his integrity.
Barclays believes if Musk is forced to leave because of the SEC action, it will be weigh on Tesla's stock.
"The SEC civil action may lead to Musk's exit from Tesla (either permanently or temporarily) and the Musk premium in the shares dissipating," analyst Brian Johnson said in a note to clients Friday. "Tesla shares have ~$130 of Musk premium for future success that might dissipate."
Tesla's stock closed at $307.52 Thursday.
Johnson reiterated his underweight rating and $210 price target for Tesla shares.
One Wall Street firm is concerned the controversy about the lawsuit will hurt demand for Tesla's cars.
"We see the potential for negative sentiment to impact demand and employee morale," Morgan Stanley analyst Adam Jonas said in an investor note. "In our view, this is particularly a risk if the situation is not resolved relatively quickly."
Jonas reiterated his equal-weight rating and $291 price target for Tesla shares.
J.P. Morgan also thinks the news will affect the company's ability to raise financing.
"We are concerned that decreased confidence in Tesla on the part of investors may impact the company's ability to raise capital on amenable terms," analyst Ryan Brinkman said in a note to clients Friday.
Brinkman reaffirmed his underweight rating and $195 Dec. 2018 price target for the company's shares.
Citigroup also downgraded the stock to a sell rating from neutral.
"There's little question that Mr. Musk's departure would likely cause harm to Tesla's brand, stakeholder confidence and fundraising," the note said. "If Mr. Musk ends up staying on, the reputational harm from this might still prevent the stock from immediately returning to 'normal.'"