"Our success won't depend on tough talk, or abandoning our values, or giving into fear — that's what groups like ISIL are hoping for," he said, using an acronym for the so-called Islamic State. "Instead we will prevail by being strong and smart, resilient and relentless, and by drawing upon every aspect of American power."
But he also called on Congress to make sure that no one on a "no-fly" list is able to purchase a gun. The no-fly list is maintained by the FBI' Terrorist Screening Center and contains people who are not allowed to board a commercial aircraft to fly in or out of the country.
Obama also renewed his calls to make it more difficult for people to purchase "powerful" assault weapons.
And he called on Congress to vote for the authorization for use of force in the war against the Islamic State group. Obama, however, emphasized that he did not seek an extended ground war against the terrorist group. He said that the current strategy of airstrikes and special forces would allow for a more sustainable victory.
Read MorePipe bombs, ammo found at Calif shooters' home: Official
The president is keen for Silicon Valley to help address the threat of militant groups that use social media and electronic communications to plan and promote attacks.
"I will urge high-tech and law enforcement leaders to make it harder for terrorists to use technology to escape from justice," Obama said. This could reignite debate about the government's digital surveillance programs.
The president also stressed that it was "the responsibility of all Americans" to reject discrimination.
"When we travel down that road, we lose," Obama said, saying that calls for religious tests and other measures against Muslims played "into the hands of groups like ISIL."
While Obama has spoken frequently about Islamic State in recent news conferences and other events, the decision to speak on prime-time television reflected concern among his advisers that his message wasn't getting through. The White House has been particularly concerned about the heated rhetoric from Republican presidential candidates about Muslims.
Read MoreRepublican presidential hopefuls chide Obama
Obama concluded his address saying that the notion of a united American people makes the U.S. stronger, and better prepared to combat terrorist organizations.
"Let's not forget that freedom is more powerful than fear; that we have always met challenges — whether war or depression; natural disasters or terrorist attacks — by coming together around our common ideals as one nation, as one people," he said. "So long as we stay true to that tradition, I have no doubt America will prevail."
— The Associated Press and Reuters contributed to this report.