S&P drew parallels between Scotland and Iceland, where in 2008 the Icelandic government allowed its banks to collapse rather than bailing them out as they proved too big to save. That sent the country into financial meltdown, including a 90 percent plunge in the country's stock market between October and December 2008.
With the Scottish independence vote just five months away, S&P warned the willingness and ability of a future Scottish government to support its banking system is "challenging" at present.
Read MoreScotland's Salmond: blocking pound will hurt UK
If Scotland were to join a currency union with the U.K., then it would be backed by the Bank of England. However, if the region adopted the euro it would likely need to set up its own deposit scheme, the report found.
"We note a possible parallel here with Iceland, where in 2008 the national deposit insurance scheme could not honor claims when the country's outsized banking system failed," said S&P credit analysts led by Giles Edward.
S&P also referenced figures from a HM Treasury paper published on the Scottish banking system, which said banking assets in the region are currently at a high of 1,254 percent to Scottish GDP, compared to the 880 percent for Iceland just before the banking system collapsed.
Read MoreCase for Scottish independence doesn't stack up: Osborne
The S&P warning follows concerns voiced by Prime Minister David Cameron, business bosses and the governor of the Bank of England, Mark Carney.
Carney said last month that Royal Bank of Scotland may have to move its headquarters to England if Scotland votes for independence this year, referring to an EU rule that requires banks to be based in the same country as most of their business.
The CEO of oil giant Royal Dutch Shell said an independent Scotland will be bad for business, while the chief executive of Scotland-based insurer Standard Life, David Nish, said he is preparing to shift business elsewhere if voters opt for independence from the U.K.