Gasoline at the pump was averaging $2.73 a gallon nationally Friday, 4 cents higher than last week and 26 cents more than a month ago, according to AAA. Kloza said he expects summer prices to hover around $2.50 nationally before dropping closer to $2-$2.25 in the fall.
"This weekend will be peak, and we'll see slight declines as we go through the summer," said Andrew Lipow, president of Lipow Oil Associates. "There's still a lot of crude oil out there that's going to temper any rises at all. But having said that, the demand for gasoline this summer is going to be very good, due to the lower prices."
Gasoline is expected to peak at about a dollar a gallon less nationally this year than last year, when the high was $3.71. Kloza said he thinks the futures market signaled the peak on May 6, when gasoline futures hit a high of $2.09 per gallon.
Memorial Day is viewed as the start of the summer driving season, and Kloza said drivers should spend about $10 billion less a month over last year's levels for June, July and August.
Read MoreLook who spends more gas savings
"It's a dress rehearsal for July and August. ... We're about six weeks away from the typical demand peak week which is in late July," said Kloza. "If you look at gasoline futures, there's a drop of about 30 cents that's perceived between now and December, but I think it will go down more than that."
One factor elevating the price this year is the outage at the Torrance, California, Exxon Mobil refinery, which produces about 10 percent of the gasoline for California, Lipow said. Californians were paying an average $3.80 per gallon of unleaded on Wednesday, according to AAA.
"If you throw out California, the national average is about 10 cents lower," Kloza said. "I think we're going to see a substantial give back of some of the price strength in California because the wholesale prices have dropped as ... gasoline has arrived from Korea, Canada and the U.S. Gulf Coast."
Read MoreGoldman: Oil will drop back to 2015 low
Kloza said the Torrance refinery is not expected to reopen soon enough to affect summer driving, but it has caused price dislocations on the West Coast. For instance, gasoline was trading recently in the spot market at more than $120 a barrel, double crude prices. "In the rest of the country, gasoline trades for $10 to $15 over the price of crude. California really became untethered," Kloza said.
But California should get some relief, even if it takes a while for Torrance to reopen. "We should see lower prices on the West Coast very soon because it's really dropped like a rock out there in terms of wholesale prices," said Kloza.
U.S. refineries were running at 92.4 percent of capacity last week, and gasoline production decreased slightly, according to the Energy Information Administration. Gasoline supplies declined by 2.8 million barrels, while distillates, including diesel fuel, fell by 500,000 barrels.
The EIA also pointed out that refining margins are at multiyear highs. For instance, the "crack spread," or difference between wholesale prices and crude oil, was the highest in New York Harbor since 2007, and in California it is dramatically higher.
Read MoreHas oil rout actually helped Gulf producers?
"Refiners are doing splendidly," said Kloza. "The irony is that we had all the light shale crude in the U.S. The big worry among refiners was we're running this light sweet crude, and we're going to make too much gasoline." He said what's happened is that prices rose, and the refiners are making more profits.
"These are Renaissance-level profits. They match levels some refiners thought they'd never see again," he said.
The EIA said refiners around the world have increased production as a result of better margins, and the U.S. input to refineries on the East Coast was the highest since 2010.