While many commodities traders are focused on the escalating tensions in Iraq, Dennis Gartman of "The Gartman Letter" said on Monday that the biggest driver of oil prices could actually be a continent away.

On CNBC's "Fast Money" on Monday, Gartman flagged Nigeria as the country that could have the biggest impact on oil prices in the future.

OPEC states that Nigeria has more than 37 billion barrels of proven crude oil reserves and produces roughly 1.9 million barrels of crude oil per day.

According to Gartman, the real threat to Nigeria's oil production isn't necessarily from Boko Haram, the group accused of carrying out attacks in the northern part of the country, but from new separatist bands in the oil-rich south. "What bothers me is what's going on in the south, where all of Nigeria's oil is located," he said.

Gartman said that two separatist groups in the south, called the "Movement for the Emancipation of the Niger Delta" and the "Biafra Zionist Federation," present a dual threat to the country.

Watch: Gas prices rising steadily

According to Gartman, if the two separatist movements form a united front, there could be real consequences to the country's oil industry. "You're going to start seeing a problem with the exports of Nigerian crude … it's about the second best crude in the world, everyone wants to have it."

Although Nigeria is a potential powder keg in the future, Gartman said there's still plenty to be concerned about in Iraq at present. "Right now you have the military occupation going on in the middle of Iraq, and you have that probability that the oil facilities may not be hurt, but the pipelines can be," he said.

—By CNBC's Michael Newberg