"There is the potential for there to be further pressure on the market for a period of time," he said in an interview that aired Thursday on "Squawk Box."
"I think people kind of need to settle in for what is likely to be a bit of a volatile time, and I think need to settle in for what may be volatile around this level we're at."
Storage facilities in the U.S. are filling up, and that could put pressure on oil prices as producers are forced to sell.
U.S. crude stocks jumped by 10.3 million barrels last week, more than double the amount predicted by analysts, according to the U.S. Energy Information Administration. Stocks at Cushing, Oklahoma, rose by 536,000 barrels, less than anticipated but still another increase at the U.S. crude contracts delivery point.
"As this North American phenomenon has occurred over the last three or four years, [the market] has been surprised at how robust and resilient this has been, and year after year, there's another million-plus barrels coming out of North America," Tillerson said.
He would not predict where oil prices were headed. He noted that U.S. crude could dip below the $40-$50 level for a period of time because of those inventory numbers or if things calm down in spots like Libya or Iraq, which could bring more oil online. Prices could also rise if there were disruptions to supply elsewhere in the world, he added.
Read MoreWhy OPEC is dead
In the next couple of years "we're going to kinda wallow around where we are with a little bit this way, a little bit that way, until some of this sorts itself out."
While U.S. producers have cut back rigs and expenditures, it will take some time before that is evident in the market, Tillerson said. What he believes is really needed is a pickup in market demand.
"If you look at the performance of the U.S. economy, it's OK but it's not robust. Europe is still struggling with declining demand and China has actually slowed its rate of energy demand growth. So all of those are conspiring to create this imbalance," he said.
On Wednesday, U.S. crude futures closed at $51.53 a barrel.
Meanwhile, Exxon Mobil is cutting capital expenditures by about $4.5 billion, to $34 billion for 2015, Tillerson said. However, it is expecting an increase in production.
The CEO stressed that there has been no significant change to its investment philosophy or investment program.
"We live in a commodity world. We've been through cycles before," he said, noting that investment decision-making is not drive by the price, whether it is $40 a barrel or $100.
"It's all about the quality of the investment opportunity, which we then test, recognizing that we have huge uncertainties in these decisions we make."
He also said Exxon Mobil was in a good position to acquire companies if the right opportunity presented itself.
"We really are looking for great opportunities where a company has great assets, we see an opportunity because of what we can do with those to create value from that," he said. "We're pretty patient. We don't feel compelled to have to do anything."
Tillerson also responded to Warren Buffett's recent revelation that Berkshire Hathaway sold its entire stake in Exxon Mobil in the fourth quarter. Buffett also called it a "wonderful company."
"I understand he has a portfolio he has to manage," he said. "We're still a very attractive opportunity for people who want to have a piece of this part of the global economy, which will always be there in the global economy, because there's never not going to be energy demand."
Tillerson said Exxon Mobil has always considered itself to be for the long-term investor.
"People who buy our stock, they own it for generations," he said. "The stock is largely held for people who are looking to send their kids to college. We pay a lot of pension funds, dividends to people."
—Reuters contributed to this report.