Rail System Most Improved
The engineers gave the nation's rail system the biggest upgrade from four years ago. Rails earned a "C" this year, due to increased investment and added capacity.
"Since 2009, capital investment from both freight and passenger railroads has exceeded $75 billion," the report's authors explain, "actually increasing investment during the recession when materials prices were lower and trains ran less frequently."
On the freight side, the fracking oil boom in North Dakota has spurred rail investment from the private sector. Rail shippers like BNSF Railway, Northern Plains, and Canadian Pacific have expanded capacity in order to transport landlocked oil from the Bakken shale to refiners in the Midwest and on the Gulf Coast.
(Read More: A Clog in the Nation's Pipeline Infrastructure?)
Passenger travel on Amtrak has nearly doubled since 2000, with demand expected to grow, while congestion at the nation's airports continues to mount.
More Poor Grades Than Good
Aviation, drinking water, dams, and hazardous waste systems all got "D" grades. The report gives the nation's 100,000 miles of levees the worst grade of all, a near failing "D-."
"The reliability of these levees is unknown in many cases, and the country has yet to establish a National Levee Safety Program," the study's authors wrote. The report estimates roughly $80 billion is needed to shore up the nation's levee system by the end of the decade, but only $8 billion in projects are now planned.
$1.5 Trillion Infrastructure Funding Gap
The ASCE estimates the nation should be investing close to $3.4 trillion through the end of the decade.
Yet, at a time when local governments are strapped for revenues, and Washington is focused on budget cuts, current planned infrastructure spending is more than $1.5 trillion short of what's needed. According to the report card, the biggest funding gaps are in the areas of surface transportation, the electrical grid, and schools.
ASCE President Greg DiLoreto said failure to spend more on infrastructure will mean paying a bigger price when it comes to economic growth.
(Read More: The Race to Rebuild America's Infrastructure)
"If we don't make these improvements by 2020, we will see drop in (gross domestic product) by $3.1 trillion," he said. DiLoreto calculates more than 3.5 million jobs will be lost along with the economic contraction.
"We got a little bit of improvement this time," he added. "If we invest, it will be a better quality of life for us, fewer outages, less time in traffic, more time at home than on a plane."
The big question is how to pay for it all.
—By Bertha Coombs; Follow her on Twitter: @coombscnbc