Here’s what worries Wall Street about Trump’s immigration ban

Joshua Roberts | Reuters

Donald Trump

Wall Streeters typically aren't the protest type. When they feel strongly about something, they don't protest it — they trade it. So, while you saw the tech industry explode with outrage over President Donald Trump's immigration ban, you haven't heard much beyond measured statements from big Wall Street firms and you don't see a lot of finance guys marching on Washington.

"Trump went too far, too fast. It seemed barbaric," a sales trader at a bulge bracket firm said. "But our firm and the industry as a whole are much better served by standing on the sidelines on this one."

The immigration ban (or "pause," as Homeland Security Secretary John Kelly likes to call it) prevents people from seven different countries — Iraq, Iran, Libya, Somalia, Sudan, Syria, and Yemen — from entering into the United States. That goes up to 120 days for refugees. However, for Syrian refugees, the ban is indefinite.

The ban hit two-hot button issues for Wall Street. The first is safety. No industry was impacted more by the Sept. 11 attacks. Some of my friends went to over 20 funerals in the months that followed. So, keeping America safe is a big concern in the industry. Second, diversity. Wall Street has worked hard to chip away at its white-guy image and diversify its staff. Sure, there's still work to be done, but this is also a big priority for the industry. So, they want to reduce the risk of a terrorist attack — but they don't want to support something that might target race or nationality.

"We can keep our citizens safe and be an openhearted country," said a hedge-fund manager who wanted to remain anonymous. 'No one will argue that it's important to know who is coming into our country. But there's a better way. And Trump just needs to find it."

By and large, the language in the official statements made by big Wall Street firms was either down the middle or slightly leaning against the executive order. Probably the strongest comment came from Goldman Sachs CEO Lloyd Blankfein who said, "It's a policy we don't support."

Beyond the official statements, what do the men and women who work on the trading floors think? I had conversations with over a hundred people who currently or recently worked in the business. More than half of them agree with the idea behind Trump's executive order. But only 14 percent agreed with how it was executed.

Finding a Wall Streeter who attended a protest was like trying to find a unicorn — maybe 1 in 10 people I spoke with attended a protest of some kind. However, many more quietly voiced their opposition with their wallets. Twenty-seven percent said they donated to the ACLU.

"Dude," a wealth management broker said. "The last time I even went near a protest is when I use to walk by Occupy Wall Street every day on my way to work. But I did write a check."

Only about a third of the people polled thought Trump's immigration action could have an impact on the stock market. Almost two-thirds of the people expect more countries to be added to the list before any ban is lifted. And the highest percentage of YES votes was if they thought some form of his executive order could be implemented and effective — nearly three-fourths of the respondents said yes to that.

The majority of concern is based on some of the language that sounds like religious testing and the execution of the order. Most feel like it could be worked to check both boxes — safety and inclusion. But the biggest concern I received from talking with industry folks was the damage it may have done for future progress. The ban or pause — whatever we're supposed to call it — created a bigger divide, more resistance and most likely will inspire any and all obstruction.

Whether you agree with it or not it has just become much harder for Trump to implement his economic agenda.

Commentary by Turney Duff, a former trader at the hedge fund Galleon Group. Duff chronicled the spectacular rise and fall of his career on Wall Street in the book, "The Buy Side." He is a commentator on CNBC's "Filthy Rich Guide" and a consultant on the Showtime show, "Billions," starring Damian Lewis and Paul Giamatti. Follow him on Twitter @turneyduff.

For more insight from CNBC contributors, follow @CNBCOpinion on Twitter.