Although qualified majority votes are acceptable under EU law, they are rarely used to force through decisions on politically sensitive topics against vocal opposition.
Hungary was supposed to be one of the beneficiaries of the scheme but has opposed it, arguing that it is not a front-line country and that it has only suffered a huge influx of migrants because Greece has failed to manage its borders.
Officials also say that it would risk turning the country into a holding pen for migrants who do not want to stay there.
French interior minister Bernard Cazeneuve criticised those countries opposed to the measures. "Europe is not Europe a la carte. If Europe wants to surmount this humanitarian challenge, it is necessary that all countries live up to their responsibilities."
The Czech Republic also refused to sign up to the proposals, saying that it would oppose efforts to introduce an automatic relocation scheme. Romania and Slovakia were also against the scheme.
More from the Financial Times:
Merkel's welcome to refugees comes back to bite her
120,000 refugee relocation target still on the table
Germany clamps down on borders as flow of migrants hits'limits'
Hungary has pushed for a summit of EU leaders, where any decision has to be taken by consensus — which requires a unanimous decision — according to two diplomats familiar with the situation.
Although migration policies can be introduced on a qualified majority basis, EU officials had hoped to reach a deal with unanimous backing. This means that the target of 120,000 refugees is likely to be introduced in some form, even if it amounts to little more than a pledge to volunteer places.
The failure to reach a deal comes as national capitals reintroduced border controls across routes used by migrants and asylum seekers to try and reach western Europe.
"To say: let's shut all the borders and keep everyone out . . . is impossible," said Frans Timmermans, first vice-president of the European Commission. "To say: let's open all the borders and let everyone in is equally unrealistic."
Read MoreFour reasons why Europe's migrant crisis matters
Britain, which has an opt-out, outlined its opposition to the plan originally put forward by the commission's president Jean-Claude Juncker.
European ministers did sign off on a scheme to share out a maximum of 40,000 refugees agreed over the summer.
Plans to allow countries to duck out of the scheme in exchange for a fee were discussed for less than one minute, according to one diplomat.
Dimitris Avramopoulos, the commission's migration chief said the policy was far from dead. "When we do not succeed, we try again," said the former Greek defence minister.