He said the United States' allies do not have the confidence the U.S. has a real strategy to prevent North Korea from continuing to provoke nations and expand its nuclear and missile programs.
"We really need to get into the diplomatic track more aggressively than we have been so far," he said in an interview with "Closing Bell."
While the U.S. has said it prefers a diplomatic solution, Trump has previously left the door open for all options. Earlier this month, the president promised "fire and fury like the world has never seen" if North Korea continued to threaten the U.S. and its allies.
Rubin said North Korea was testing the limits with its latest move.
"This is North Korea doing what North Korea often does. However, what's different this time is that they have technical capacity that many analysts did not think they were going to achieve so quickly."
He called the policy of the last 15 years "failed" and said there needs to be a broader diplomatic policy than just asking China to intervene.
John Merrill, former chief of the Northeast Asia division of the bureau of intelligence and research at the State Department, said North Korea is getting frustrated with U.S. policy, which is based entirely on sanctions.
"People sometimes forget that sanctions carried to an extreme can sometimes get us into real trouble," he told "Closing Bell," pointing to the oil and scrap metal embargo on Japan in the early 1940s.
"When they felt that their economy was being choked they responded with a military attack on Pearl Harbor. So that's the danger with sanctions, they can be carried too far," he said.
— CNBC's Jacob Pramuk contributed to this report.