Hyde-Smith is facing off against Democratic challenger Mike Espy in a run-off election on Nov. 27.
Hyde-Smith's tendency to put her foot in her mouth has made the Senate race a competitive one despite Mississippi's status as an otherwise solidly dependable Republican-voting state.
The New York Times reported Tuesday that a private Republican poll had found that Hyde-Smith holds just a 5-percentage point lead over Espy, a former U.S. Agriculture secretary under President Bill Clinton and three-term representative from Mississippi's 2nd Congressional District.
Her campaign did not return a request for comment from CNBC on Tuesday after Politico reported on the Confderate-themed photo.
Hyde-Smith had donned the rebel cap during a visit to the Biloxi, Mississippi, home of the president of the Confederate States of America, in August 2014. At the time, Hyde-Smith was serving as agriculture and commerce commissioner of the state.
"I enjoyed my tour of Beauvoir. The Jefferson Davis Home and Presidential Library located in Biloxi," Hyde-Smith wrote.
"This is a must see. Currently on display are artifacts connected to the daily life of the Confederate Soldier including weapons. Mississippi history at its best!"
That history includes Mississippi's seccession from the United States, and its war against the Union Army.
Mississippi was the second state to secede from the U.S. — in whose Senate Hyde-Smith currently serves — in January 1861.
At the time it seceded, a majority of the residents of Mississippi were black slaves. Hyde-Smith's challenger, Espy, is black.
Espy, if elected, would be the first black senator from Mississippi since Reconstruction, the period after the Civil War that ended with the defeat of the Confederacy. The two will face off in a debate Tuesday at 7 p.m.
Mississippi's declaration of secession said, "Our position is thoroughly identified with the institution of slavery — the greatest material interest of the world."
The declaration cited the risk of the abolition of slavery, and also blasted the idea of "negro equality."
After Mississippi and the other seceding states were defeated by the Union Army in 1865, the state like others in the Deep South for nearly a century harshly repressed the emancipated black population, denying them basic rights, including voting.
According to the report "Lynching in America," the state of Mississippi had 654 lynchings, or extradjudicial killings, of African-Americans from 1877 to 1950, which was both the highest tally of any single state in the United States, and also was the highest rate of lynchings in terms of overall population.
On Nov. 2, Hyde-Smith, while attending a campaign stop in Tupelo, referred to a local rancher standing next to her, and said that if he "invited me to a public hanging I'd be on the front row."
Also, on Monday, Hyde-Smith said that her campaign had returned a $2,700 donation from Peter Zieve, a businessman in Seattle, who was sued by the state of Washington in 2017 for refusing to hire Muslims at his aerospace company Electroimpact, and for expressing "hatred" of Muslims at work.
Another video recently surfaced in which Hyde-Smith was heard saying, "Maybe we want to make it just a little more difficult" for some people to vote.
"And I think that's a great idea."