The calendar may be to blame for the overall decline of travelers, the association noted. With July 4 landing on a Thursday this year, the celebration is now considered a five-day holiday, with many people taking off the day before and coming back on Sunday.
Last year, Independence Day fell on a Wednesday, making it a six-day holiday for many people and giving them more time and opportunity to travel, AAA noted.
(Read More: Want More Time Off? Some Employers Let You Buy It)
And despite Federal Reserve Chairman Ben Bernanke's more upbeat assessment on Wednesday about the U.S. economy, AAA said financial hardship continues to be a factor for travelers.
"Economic growth … is not robust enough to offset the impact of the sequester and the effect of the end of the payroll tax cut on American families," Darbelnet said.
Still, the average traveler is expected to spend $747 and travel a round-trip distance of 613 miles this Fourth of July.
(Read More: How You Can Land the Right Travel Insurance)
Gas prices won't be a major factor in travel decisions this holiday—they're up just slightly over last year, AAA noted. But weekend daily car rental rates and airfares are noticeably higher, up 29 percent and 6 percent, respectively, over 2012.
Want to avoid the crowds? Be aware that the largest share of travelers, 32 percent, will leave on Wednesday, July 3, while Sunday, July 7, is the most popular date of return, with 38 percent planning to return that day.
—By A. Pawlowski, NBC News contributor.