The EU won't crumble, but it becomes a "less sound experiment" without the influence of the United Kingdom, he argued.
related investing news
The fallout of Thursday's referendum could send Britain into recession, Summers told CNBC's "Squawk Box," pegging the chances of that at about 50-50.
World central banks have little firepower to throw at potential global economic unrest created by a Brexit, the former Obama administration economic advisor said, calling the current environment the "moment of least capacity" for policymakers to make a difference.
With the European Central Bank in stimulus mode and the U.S. Federal Reserve with only one interest rate hike since 2006 under its belt, Summers said there's not much room to ease monetary policy further if necessary.