The disaster, believed to have been triggered when the building's electricity generators were started up during a blackout, put the spotlight on Western retailers who use the impoverished South Asian nation as a source of cheap goods.
About 4 million people work in Bangladesh's garment industry, making it the world's second-largest apparel exporter after China. Some earn as little as $38 a month, conditions Pope Francis has compared to "slave labor."
Prime Minister Sheikh Hasina has blamed the factory owners for the disaster, saying they ignored warnings about cracks in the walls of the building.
(Read More: Some Retailers Rethink Roles In Bangladesh)
Duty-free access offered by Western countries and low wages have helped turn Bangladesh's garment exports into a $19 billion-a-year industry, with 60 percent of clothes going to Europe.
"If the EU or any other buyers impose any harsh trade conditions on Bangladesh it will hurt the country's economy ... millions of workers will lose their jobs," Mahbub Ahmed, the top civil servant in Bangladesh's Commerce Ministry, told Reuters.
The government has not received any formal notification of punitive action from the EU or any other country over the deaths, he said.
Separately, a group of government officials and textile factory employers and workers met officials of the International Labor Organization and agreed on steps to improve the lives of workers.
These included plans to inspect safety standards at all export-oriented garment factories, and to increase the number of safety inspectors. They also agreed to push in parliament a reform package giving workers more rights.
Accusations, Fears
Authorities have arrested nine people in connection with the collapse, including an engineer who had raised safety concerns about the eight-story complex a day before the disaster.
"The Industrial Police had asked the owners of the factories to suspend operations after cracks were noticed in the building," Hasina told a news conference late on Friday. "But they decided to operate their factories. After a power blackout when they started their generators the building caved in."
(Read More: Retailers Pressed to Compensate Bangladesh Factory Victims)
The owners of the factories have not commented publicly on the accusation that they were to blame. Four factory owners have been arrested, as has the owner of the building.
On Saturday, verses from Islam's holy book the Koran were read out for the souls of the victims, as the stench of decaying bodies hung in the air around the site.
"The bodies that are coming now cannot be identified. The clothes the victims were wearing are also damaged, the faces are decomposed," Mohammad Masum, a volunteer rescue worker at the site in Dhaka's suburbs told Reuters Television.
The collapse was the third deadly incident in six months that raised questions about worker safety and Labor conditions in Bangladesh. Human-rights groups say there has never been a case in which a factory owner was prosecuted over the deaths of workers.
"After this accident we are very scared and worried about such an accident happening at our factory," said garment worker Farida Parveen.
"We have demanded that the government take action and examine all factories so that we can all work in a good environment."
