In a thinly veiled swipe at Luxembourg and other low-tax jurisdictions, they add: "This situation may lead to uncooperative behaviours between member states, which directly affects the establishment and functioning of the internal market and the benefits provided by Treaty freedoms."
The letter, a copy of which was seen by the Financial Times, comes amid a roiling controversy over Luxembourg's tax policies during Mr Juncker's long tenure as prime minister that has marred the early days of his new job as leader of the EU's executive arm.
Under pressure, Mr Juncker last week defended a series of deals that allowed multinationals to pay as little as 1 per cent tax on their earnings as a way to diversify his country's banking-dominated economy. He also admitted the rulings should have been better scrutinised.
He has agreed to present anti-tax haven legislation early in his term, putting him in the awkward position of having to oversee proposed regulations that could have a particular impact on the Grand Duchy.
While an embarrassment for Mr Juncker, the Luxembourg controversy – triggered by the release of thousands of pages of documents from consultancy PwC – has presented an opportunity for Germany and other longstanding critics of low-tax regimes to press for reform.
In their letter, they call for the EU to adopt "a set of common, binding rules on corporate taxation to curb tax competition and fight aggressive tax planning."
The ministers write that EU plans for exchange tax information between member states do not go far enough. EU law "could do more on trusts, shell companies and other non-transparent entities" by establishing compulsory ownership registers, they state.
They also urge the commission to tackle specific loopholes linked to accounting for interest payments, royalty receipts and the connections between parent and subsidiary companies.
"Transparency is not enough. We can surely not concede that situations where treaty freedoms are misused in order to avoid tax remain unaddressed. For this reason, an [EU] anti-Beps directive should set a general principle of effective taxation," they state.
If implemented, such a rule would have wide-ranging implications far beyond Luxembourg, giving authorities powers to ensure that "tax benefits are not obtained through inappropriate arrangements".
In a clear warning to tax havens the letter says: "The European Union needs to protect its internal market from tax avoidance through the use of tax havens. The anti-Beps directive will be an opportunity to fix this issue through countermeasures towards jurisdictions whose behaviour fosters non-transparency and aggressive tax planning."
Follow us on Twitter: @CNBCWorld