Here's what happened when I wore 10 fitness trackers at once

Fitness trackers have been all the rage these days. It seems like almost everybody has one on, and cubicle dwellers everywhere are being prodded to walk a few more steps ... walk a few more steps ... a ... few ... more ... steps.

There was news this week that people are suing Fitbit, alleging its devices aren't tracking their data accurately enough, specifically heart rate.

That got us thinking. Let's try to test some fitness trackers ourselves.

I went to a local electronics store and bought a bunch of devices, from brands like Fitbit, Garmin, Polar, Misfit, Jawbone, and Withings. I also borrowed another Fitbit and two Apple Watches from colleagues.

All those devices look like this:

Justin Solomon | CNBC

I ran the devices through three specific tests: counting steps, heart rate and total distance.

Let me just say this first: None of this would qualify as a scientific study. But I tried to be more thorough than any typical consumer would be.

The results, depending on your perspective, were either quite variable — or pretty close.

Steps

I did two different tests of step counting. In the first, I wore all the devices for a couple hours, doing a variety of tasks. The steps totals ranged widely, by more than 20 percent.

That's a lot when you consider that many people have a goal of getting to 10,000 in a day. At divergences this big, one device might say I got to 10,000 while another would say I'm still stuck under 8,000.

To be even more precise, I ran a second test, counting 500 steps out loud. In this case, I still saw variance. The total range just after 500 steps was between 446 and 513.

The most accurate device was the Fitbit chargeHR, which I purchased. That one claimed I walked 505 steps. Another Fitbit (the cheaper Flex model that we borrowed) said I came up short, at 486. So, even within the same company, I saw different numbers. To be fair, all these numbers were within 4 percent of 500 — pretty close to not mattering. The more interesting part is to see how they diverge as various other activities occur during the day.

(If you're wondering why there are 10 trackers on my wrists in the picture but only nine in the tables, it's because I tried two models from Misfit, but could only sync one to my smartphone.)

Heart rate

To test heart rate, I went on the stationary bike in the corporate gym. After a few minutes getting up to speed, I got my heart rate up to a consistent 140 beats per minute, based on me actually putting my hand up to my carotid artery and taking my pulse.

I did this same experiment two days in a row.

I found more variance — but the different devices generally undercounted the correct number.

In my brief experience, the different heart rate monitors all appeared to have their own issues. Some needed time to catch up, even though my heart was already going.

It's also completely possible that I didn't place each device "perfectly" on my wrist, but it's meant to be a test for what would happen if an average consumer just started wearing them and went for it.

Some devices would flip back and forth between a very low heart rate reading (like 90) and something more accurate (like 130). After I got off the cycle and resting for a while, some of the devices displayed a number that was too high.

The heart rate tracking was the least reliable of the three tests we did. But there is often user error with these products. I ran the tests with different versions, where I wore trackers on either wrist. In one case, I wore the trackers one at a time, while in another case I wore all the trackers at once.

The closest devices were the two Apple Watches we tried. Both of them (at 137 and 134 bpm), were pretty close to the target number.

Distance walked

To test exact distance, I got on a treadmill and walked half a mile the second day of testing. Even such a short distance was enough to see big deviations in the data.

While wearing multiple watches at the same time, I could see the distances diverge right in front of my eyes. The more I walked, the more the watches would show numbers growing further apart.

The Withings Pulse O2 most closely matched what the treadmill said. Though it could be that the real problem was the treadmill itself was wrong. If that's true, then I don't know which device was best.

In the end, I couldn't tell which device was the most accurate. They were all pretty close, but they did diverge from each other.

Maybe the best thing to do is just to use these devices for relative purposes. Buy one and use that. Whatever you did today, do a little bit more tomorrow. Be a little bit more active. In general, they get the directions right.

The more somebody moves, the faster the heart rate, the bigger the steps and distances. If somebody does more each day, that's the main goal. Trying to get something exact beyond that — well, the technology is getting pretty close. I just don't know which one it was.

What the companies said

I reached out to every company mentioned in this story. A Garmin spokeswoman reiterated my conclusion that's it's about an individual's relative gain. "Garmin activity trackers are designed to help users develop healthy habits and motivate them to beat yesterday," she said in an email. "These devices are meant as a tool to encourage people to live more active lifestyles and for the most accurate data we recommend selecting a device with added features like GPS or a chest-worn heart rate monitor."

That's in line with the conclusions: These devices are best used for relative purposes.

A representative of Fitbit stood by the company's research and product testing but cautioned against putting too much stock in the exact figures.

"Fitbit trackers are designed to provide meaningful data to our users to help them reach their health and fitness goals, and are not intended to be scientific or medical devices," she said in an email. "Overall, the success of Fitbit products comes from empowering people to see their overall health and fitness trends over time — it's these trends that matter most in achieving their goals."

A representative for Jawbone said: "We take a machine learning approach to develop an algorithm that performs work both for the counting of true steps as well as avoiding false positives."

Polar pointed out that physical activity comes in many forms, each of which can provide benefits. "Polar will credit that activity since it's offering the same kind of physical benefits," a spokesperson wrote in an email. "What's important to remember is that the reason we're tracking steps isn't just for the sake of tracking them, but ultimately because it's about achieving a better fitness result."

Another company, which declined to be named to discuss other companies' algorithms, suggested that there is always going to be variation for each statistic. For steps, some devices treat hand movements differently, so if people wave their hands a lot during the day, they might get credited differently depending on the device. Also, the company suggested that most devices are pretty good at getting steps counted while walking in a line — which is why our goal of 500 steps was mostly tracked within 4 percent.

For distance, devices have to consider stride length, as people have different leg lengths or walk at different speeds. There's also the difference between walking indoors versus outdoors, and how the inclusion of a GPS might — or might not — increase reliability, depending on the circumstance.

And for different heart rates, somebody might not get an exact measurement when putting their finger to their neck, because that action itself changes the pressure and pulse you feel.

The other companies we reached out to have not yet given a comment. We will update this article as they do.

Nicholas Wells contributed reporting to this story.

This story has been updated with a comment from Polar.

Bud Selig and Hank Aaron talk about the state of the world today