Passikoff said that doesn't mean the kids got out the calculator and decided to spend less on Dad. They just ended up doing so.
Read MoreChocolate? What moms really want for Mother's Day
"This is not a contest," he said.
The retail federation is expecting that total spending for Father's Day will be $12.5 billion this year, compared with an estimated $19.9 billion for Mother's Day.
The group predicts that the average person will spend $113.80 this Father's Day, down slightly from $119.84 last year. For Mother's Day, it predicted that the average American would spend $162.94, also down slightly from 2013.
Read MoreStay-at-home dads struggle to shake Mr. Mom image
More than six in 10 respondents planned to get Dad a card. About four in 10 said they plan to get him clothing such as a sweater or tie, and about four in 10 also said they plan to have a special outing such as dinner or tickets to a sporting event. Survey respondents could pick more than one answer.
Brand Keys is predicting that the average consumer will spend $145 this Father's Day, slightly higher than last year. For Mother's Day, Passikoff's firm predicted that consumers would spend $184, also slightly higher than in 2013.
The figures are in a survey conducted as part of the Brand Keys Customer Loyalty Engagement Index. It polled a national sample of 5,500 men and women 18-65 years-old, asking if and how they were planning to celebrate Father's Day.