I reached out to every company mentioned in this story. A Garmin spokeswoman reiterated my conclusion that's it's about an individual's relative gain. "Garmin activity trackers are designed to help users develop healthy habits and motivate them to beat yesterday," she said in an email. "These devices are meant as a tool to encourage people to live more active lifestyles and for the most accurate data we recommend selecting a device with added features like GPS or a chest-worn heart rate monitor."
That's in line with the conclusions: These devices are best used for relative purposes.
A representative of Fitbit stood by the company's research and product testing but cautioned against putting too much stock in the exact figures.
"Fitbit trackers are designed to provide meaningful data to our users to help them reach their health and fitness goals, and are not intended to be scientific or medical devices," she said in an email. "Overall, the success of Fitbit products comes from empowering people to see their overall health and fitness trends over time — it's these trends that matter most in achieving their goals."
A representative for Jawbone said: "We take a machine learning approach to develop an algorithm that performs work both for the counting of true steps as well as avoiding false positives."
Polar pointed out that physical activity comes in many forms, each of which can provide benefits. "Polar will credit that activity since it's offering the same kind of physical benefits," a spokesperson wrote in an email. "What's important to remember is that the reason we're tracking steps isn't just for the sake of tracking them, but ultimately because it's about achieving a better fitness result."
Another company, which declined to be named to discuss other companies' algorithms, suggested that there is always going to be variation for each statistic. For steps, some devices treat hand movements differently, so if people wave their hands a lot during the day, they might get credited differently depending on the device. Also, the company suggested that most devices are pretty good at getting steps counted while walking in a line — which is why our goal of 500 steps was mostly tracked within 4 percent.
For distance, devices have to consider stride length, as people have different leg lengths or walk at different speeds. There's also the difference between walking indoors versus outdoors, and how the inclusion of a GPS might — or might not — increase reliability, depending on the circumstance.
And for different heart rates, somebody might not get an exact measurement when putting their finger to their neck, because that action itself changes the pressure and pulse you feel.
The other companies we reached out to have not yet given a comment. We will update this article as they do.
Nicholas Wells contributed reporting to this story.
This story has been updated with a comment from Polar.