Puerto Rico has been "destroyed," and that is why it's a problem getting all the relief supplies to the island, said Michael Brown, the former FEMA head who was in charge when Hurricane Katrina struck in 2005.
Fuel and relief items such as food, medicine and water have been in short supply since Hurricane Maria struck Puerto Rico more than a week ago.
"This is truly a logistical nightmare for FEMA," Brown said in an interview with CNBC's "Power Lunch" on Friday.
With ports, railroads and airport runways damaged, a damage assessment must be done and repairs made.
"Even then you have to make certain that the docks are safe so that you can offload those materials. Before you can land a C130 or a Chinook you have to make certain that the runway is safe or the landing zone is safe," he said.
Brown said FEMA is doing everything it can to move as rapidly as possible. The agency also makes sure it addresses acute needs first, such as hospitals and fuel distribution.
"You are going to see these pockets of things being fixed when at the same time you have pockets that they haven't gotten to because it's so remote."
On Thursday, President Donald Trump removed shipping restrictions that critics have argued held back the disaster response. The roughly 100-year old Jones Act requires goods shipped between American ports to travel on U.S.-flagged ships with American crews. Critics such as Sen. John McCain, R-Ariz., argued it had unnecessarily raised costs on Puerto Ricans in need of supplies.
Meanwhile, more than 10,000 containers filled with supplies are being stored at two of the main shipping companies at the Port of San Juan.
Brown said things are starting to come back online, like hospitals and fuel. The question is how that fuel will be distributed. However, he anticipates an "exponential increase in response time" now that things are coming back up.
'Stupid' response
Meanwhile, acting Homeland Security Secretary Elaine Duke has come under fire for comments she made on Thursday.
She called the relief effort under control, saying, "It is really a good news story, in terms of our ability to reach people."
That prompted the San Juan mayor to tell CNN the island's recovery is "not a good news story."
"This is a 'people are dying' story," Carmen Yulin Cruz said.
Brown said he guarantees FEMA Administrator Brock Long understands exactly how dire things are in Puerto Rico. However, as you go up the food chain in any bureaucracy, people are going to say stupid things, he said.
"That was a stupid thing for Elaine Duke to say. You are in the middle of a crisis. You have a humanitarian disaster. You don't talk about good news stories. You talk about what you're doing."
— CNBC's Jacob Pramuk contributed to this report.