Read MoreDonald Trump: This is why I'm for low interest rates
Lee said Trump was raising the flag on the right issues in a wide-ranging interview on CNBC Thursday morning. The ratio of gross U.S. government debt, including local debt, to GDP is already more than 100 percent and it is projected to keep rising.
Higher interest rates would cause servicing that rising debt to become more burdensome faster. "Markets would start questioning the sustainability of the U.S. deficit, and will start to price in potential default risk on U.S. Treasurys if that ratio continues to rise," Lee said. Trump's own plans, however, would potentially raise debt and cut taxes, and tax reform does not necessarily increase revenues.
"I think his intuition is telling him if he's going to do infrastructure spending and all these things he's talking about to keep America great, he's going to have to finance it with debt or raise taxes, and he's used a lot of debt in his past," said Lee.
Trump, in the interview, did note the hazards of debt and the issues that would emerge for U.S. debt if rates rise. "I am the king of debt. I do love debt. I love debt and I love playing with it, but of course now you're talking about something that's very, very fragile, and it has to be handled very, very carefully," he said.
The billionaire businessman also said he has no problem with Fed Chair Janet Yellen, who like himself, likes low interest rates. Trump said she is capable and doing her job. Yet, he would replace her when her term ends, noting it would be appropriate and she's not a Republican. He also said he likes low rates unless inflation is on the rise, and then higher rates would be necessary.
Read MoreHow Trump could win the presidency
"She's a low interest rate person. She's always been a low interest rate person. I must be honest, I'm a low interest rate person. If we raise interest rates and if the dollar starts getting too strong, we're going to have some very major problems," Trump said.
While the Fed and economists forecast as many as two rate hikes this year, the markets have been betting against a rate increase, in part because of the havoc caused by the firming dollar on world markets and U.S. growth.
"For us in the markets, we'd like fixed income markets to come back to life, and in order to do that, you need to have higher yields. Otherwise, we're sinking into the abyss, or getting closer to what Japan is," said Chris Rupkey, chief financial economist at MUFG Union Bank.
"We thought Republicans would think about removing Fed Chair Yellen early, but now be careful what you wish for. He doesn't seem to be favoring raising interest rates. He's a real estate guy. Since when do real estate guys like higher rates? He says he's taken on a lot of debt in his life. He knows something about the downside of higher rates."
Read MoreHow to protect your portfolio from Hillary
Rupkey said Trump's comments should be taken in context of the election. Trump just this week became the presumptive GOP candidate when he won Indiana, forcing Texas Sen. Ted Cruz and Ohio Gov. John Kasich to suspend their campaigns.
Read MoreThese stocks win regardless of who is in the White House
"This is what he says now. He doesn't seem to have a lot of people advising him on monetary policy matters," said Rupkey. "He's still kind of shooting from the hip, which has gotten him the nomination on running for the Republicans, and at some point when he studies the issues, his advisors study the issues, he won't be as against raising rates. The Republican Party generally criticized (former Fed Chairman Ben) Bernanke for keeping rates low and throwing savers under the bus. At some point, he'll hear from the savers."
The view that Trump might want to renegotiate debt was the most alarming to some, but that also makes it less likely to become a proposal. U.S. debt is held by many foreign governments and other entities, including U.S. pension funds, and taking Trump's comments about renegotiating debt on their face would not be palatable.
"Imagine the most risk-free asset suddenly becoming not a risk-free asset. By the way, S&P and Moody's would have to downgrade it if you could renegotiate the value of the debt you hold or try to force it down. What are we, Argentina?" said David Ader, chief Treasury strategist at CRT Capital.
Lee said the Treasury is already experimenting with restructuring the debt by retiring older issues and replacing them with new issues that are more liquid.