Sitting at the corner a large conference room table in January of 2013, it felt like a conversation with an average, 74-year-old man from Queens.

It wasn't, of course. The conference room was inside the Federal Correctional Institution in Butner, North Carolina. And the friendly old man telling me about his dailylife, what books he was reading and so on was none other than the greatest con artist of all time, Bernie Madoff.

In our two-hour meeting, Madoff would express remorse for his crimes, just as he had in practically every e-mail leading up to the visit and every communication since.But he would also speak in very matter-of-fact terms about banks that had to know about the fraud but turned a blind eye, feckless regulators, and "greedy" clients who gave him no choice but to continue the fraud.

At some point it hit me: Madoff seemed so convincing because he absolutely believed every thing he was telling me.

It was a similar situation four years earlier at a law office in Houston. R. Allen Stanford,accused by the Securities and Exchange Commission of running a $7 billion Ponzi scheme, insisted to me in his first television interview that the SEC was attacking a legitimate business, and that his lavish lifestyle was merely the result of prudent investing.

"It's just unconstitutional," he said in April 2009. "They took a going, global group of great companies with the best people in the world—who I love dearly, and it's broken my heart to see what's happened to them—and they decimated this thing. They decimated it."

Stanford was confident, even imposing. He believed so fervently in what he was saying that even now—having been convicted on 13 criminal counts and sentenced to 110 years in prison—he is advancing the very same argument in his appeal currently pending before the Supreme Court.

"It takes a certain type of person to be that immoral. To be these kinds of scammers," says Steven Weisman, a senior lecturer on law, taxation and financial planning at Bentley University, in an interview with American Greed.

Another such scammer is Steven Palladino, profiled in the latest episode of American Greed, who is serving a ten-year prison sentence for running the biggest white-collar fraud in Boston in 100 years.

From a small office above his ice cream shop in the West Roxbury section of Boston,Palladino ran Viking Financial Group, which recruited investors to fund loans to local businesses.

By making himself a part of the tight-knit neighborhood, Palladino had his victims right where he wanted them. He would even hold court every morning at the local Dunkin Donuts.

"He knew everybody," says investor Ron Nasif, a retired orthopedic surgeon who often attended the sessions. "He would often speak about a loan he was going to do that day or about some upcoming gala event that he was sponsoring. It was always a way for him to make his contacts."

Weisman says it is a common tactic for con artists: playing on potential victims' common interests. Like Madoff targeting the Jewish country club set in Palm Beach orStanford going after retirees in the oil patch, Palladino played to the West Roxbury neighborhood of working class people seeking safe investments. The experts call it "affinity fraud."

"It is an innate sense of empathy. It is an innate sense of being able to mirror people when they talk with them," Weisman says. "So maybe it is someone who goes to the same house of worship. So maybe it's someone who belongs to a particular organization, a religion, race, whatever."

Palladino's Viking Financial Group turned out to be a giant Ponzi scheme in which his neighbors' investments funded his lavish lifestyle. Even after authorities tried to freeze his assets, Palladino kept spending money, landing him a separate federal conviction for criminal contempt.

In hindsight, Ron Nasif, who says he lost $1 million, tells American Greed it's no wonder Palladino got away with it for so long.

"We believed it because we wanted to believe it," he says. "Many of us had our life savings with him."

To protect yourself against affinity fraud, the Securities and Exchange Commission saysin a 2014 Investor Alert to do your own homework.

"Even if you know the person making the investment offer, be sure to research the person's background, as well as the investment itself—no matter how trustworthy the person who brings the investment opportunity to your attention seems to be,"the agency says.

Other tips include:

• Never make an investment based solely on the recommendation of a member of an organization or group to which you belong.
• Do not fall for investments that promise spectacular profits or "guaranteed" returns.
• Be skeptical of any investment opportunity that you can't get in writing.
• Don't be pressured or rushed into buying an investment before you have a chance to research the "opportunity."

"No one is too smart to be scammed,"Weisman says. "Actually studies have been done in regard to investment fraud showing that sophisticated, knowledgeable, well-educated investors can be scammed."

Or maybe you think you've got a great B.S. detector and you can tell when someone is lying. That too can be a fatal mistake.

Remember those conversations with Madoff and Stanford in which two of the biggest convicted con artists of all time clearly believed everything they were telling me? It is a whole lot harder to spot a lie when the person telling it is convinced he is telling the truth.

Think you could have spotted Steven Palladino's scam? Watch an ALL NEW episode of AmericanGreed—"The Bad Neighbors"—Thursday, August 4 at 10pm ET/PT on CNBC Prime.