"Right now, I feel like it's reverse discrimination," said one poll respondent, a white, 69-year-old retired teacher from Rhode Island, who was interviewed for this story and did not wish to be identified. "I did support it at first, but, gradually, because of this reverse discrimination it's gone too far."
By the fast-approaching end of its term, the Supreme Court is expected to hand down a decision in a case determining whether the University of Texas admissions program violated the Constitution by using racial preferences. It's the latest of a handful of cases the court has taken on dealing with affirmative action over the past two decades.
Not surprisingly, there is a wide divide on the issue along racial lines. Among whites polled, almost six in 10 (56 percent) oppose affirmative action. But among minorities asked, eight in 10 blacks and six in 10 Hispanics favor it.
There is also an ideological split, with 67 percent of Democrats saying the programs are still needed, compared to 22 percent of Republicans and 17 percent of Tea Party supporters. And just 39 percent of independents agree that affirmative action should be continued.
By September 1995—coinciding with the Republican takeover of the House and the welfare-reform debate of the 1990s—the number of Americans supporting affirmative action dropped to 49 percent, with 43 percent opposing it.
That's essentially where the number stood through 2010, with a brief uptick in 2000, until this June's survey. It's a change some think is attributable, in part, to the re-election of President Barack Obama.
(Read More: Corporations Find aFriend in the Supreme Court)
"On the surface, things are very positive," said Weldon Latham, a Washington-based attorney, who is black and advises corporations on diversity issues.
"When you see that we have an African-American president, African-American CEOs, African-American generals—you can mention all the names—the Colin Powells, the Barack Obamas. If you watch TV, you say, look, things have improved dramatically. When Barack Obama was elected, amazingly, everybody said it's a post-racial America, but if you look just below the surface … at the things that are very important, like jobs—African-American jobs and female jobs are still some percentage below what white males are."
Kevin Brown, a law professor at Indiana University, also spoke to the possible "Obama effect."
"Certainly, the election of Barack Obama as president has made a difference," Brown said. "I did not believe America would elect a black president in my lifetime. There's no question America is a much more tolerant, open society than 20, 25 years ago."
But Brown also stressed the ongoing need for programs to assist minorities. "My concern is underneath the veneer there is this separate story that the descendants of slaves are falling farther and farther to the bottom in a way that no one would recognize. The group most left behind is the group most affected by our history of racial discrimination."
Brown has written about the "Underrepresentation of Ascendant Blacks at Selective Educational Institutions." In other words, that American black descendants of slaves are increasingly making up fewer of those benefiting from affirmative action. Instead, he says, immigrant blacks from the Caribbean and Africa are making up bigger percentages of the blacks getting preference for elite colleges.
"In some places," Brown said, "if you go to your elite Ivy League schools, you're finding very few of your traditional African-Americans. They're maybe about one-third of blacks in those schools. Unlike the president, who grew up in the U.S., you're coming up with a lot of black immigrants. Frankly, the U.S. civil-rights struggle is not their struggle."
And even when it comes to Obama—to Latham's point of there being more black leaders like the president, former Secretary of State Powell, and Attorney General Eric Holder—Brown said, "Can I just point out—Kenyan, Caribbean, Caribbean."
There have also been campaigns against affirmative-action programs in the states over the last 20 years led by former University of California Regent Ward Connerly, points out Thomas J. Espenshade, a sociology professor at Princeton and co-author of "No Longer Separate, Not Yet Equal."
"In a broader context," Espenshade noted, "it is the case that an increasing number of states have done away with race considerations in public education or public employment either because of constitutional amendments or gubernatorial action."
(Read More: Supreme Court Strikes Down Ruling Against Sears)
Affirmative action has been banned in eight states—Arizona, California, Florida, Michigan, Nebraska, New Hampshire, Oklahoma, and Washington.
Richard Kahlenberg, a senior fellow at The Century Foundation, said many view the problems of ascendancy as having less to do with race and more to do with economic status.
"The decline in support for affirmative action based on race is not surprising, as the growing divide between rich and poor has become more important to an individual's life chances than the differences between being white and black," Kahlenberg said in an email.
"The black/white achievement gap used to be twice as large as the rich/poor achievement gap, but today the situation is reversed, and the income achievement gap is twice as large as the racial achievement gap. A number of polls find that affirmative action based on income is far more popular than affirmative action based on skin color."
But Brown said he believes that "misses the point."
"It's not that people don't have obstacles to overcome because of low socioeconomic status," Brown said, "but race provides a different set of obstacles than socioeconomic status. It's not an either-or. It's really both."
—By Domenico Montanaro of NBC News