"There is the potential for there to be further pressure on the market for a period of time," he said in an interview that aired Thursday on "Squawk Box."
"I think people kind of need to settle in for what is likely to be a bit of a volatile time, and I think need to settle in for what may be volatile around this level we're at."
Storage facilities in the U.S. are filling up, and that could put pressure on oil prices as producers are forced to sell.
U.S. crude stocks jumped by 10.3 million barrels last week, more than double the amount predicted by analysts, according to the U.S. Energy Information Administration. Stocks at Cushing, Oklahoma, rose by 536,000 barrels, less than anticipated but still another increase at the U.S. crude contracts delivery point.
"As this North American phenomenon has occurred over the last three or four years, [the market] has been surprised at how robust and resilient this has been, and year after year, there's another million-plus barrels coming out of North America," Tillerson said.
He would not predict where oil prices were headed. He noted that U.S. crude could dip below the $40-$50 level for a period of time because of those inventory numbers or if things calm down in spots like Libya or Iraq, which could bring more oil online. Prices could also rise if there were disruptions to supply elsewhere in the world, he added.
Read MoreWhy OPEC is dead
In the next couple of years "we're going to kinda wallow around where we are with a little bit this way, a little bit that way, until some of this sorts itself out."
While U.S. producers have cut back rigs and expenditures, it will take some time before that is evident in the market, Tillerson said. What he believes is really needed is a pickup in market demand.
"If you look at the performance of the U.S. economy, it's OK but it's not robust. Europe is still struggling with declining demand and China has actually slowed its rate of energy demand growth. So all of those are conspiring to create this imbalance," he said.
On Wednesday, U.S. crude futures closed at $51.53 a barrel.