JUMP TO:

2011 Sport Device GPS Accuracy In Depth: Part II

(Note: If you’re just joining me in Part II, be sure to read Part I, which covers the first half of the tests. Also of note: A few weeks after the test was published Garmin released a new firmware update that fixes the FR610 issues, and I redid some of the tests here)

IMG_3107

As we left off in Part I, the tests thus far had been largely easy.  By that I mean that the first test was a straight out and back, the second was a 800m track test, and the third was a loop with some simple terrain and trees. None of these though really tested GPS accuracy in a satellite challenged environment or situation.

In those tests we found that the majority of the devices did quite well, with most of them being within one-half of one percent compared to the rolling tape-measure accuracy.  As you remember, all the courses are measured using a rolling tape measure, and then each course is tested with three different methods: Walking, Running and Cycling.

But that’s where the easy stops, and the tough get going.  In Part II we’ll explore forests which can degrade signal, as well as well a bit of tunnel that shows how well the devices can track despite potential drops in signal.

Test 4: The Deep Tree Adventure

Perhaps the most talked about scenario for GPS sport devices is how they handle in the trees.  Trees make very natural obstructions to GPS signal, and depending on the type of forestry being discussed – can be rather tall and dense.  While I’ve yet to find a forest or jungle anywhere in the world where I couldn’t get a GPS signal successfully, I have – like many people – seen it have an impact on tracking.  The secondary reason for inconsistencies is that most forest trails aren’t straight and narrow affairs.  Instead, they tend to wander.  With trail switchbacks it’s rare to just be going on a single compass heading for an extended period of time.

Switchbacks in particular can also cause issues if the recording interval is set such that it doesn’t catch the turns being made.  For example, if a data point is only recorded every 5 seconds, and within that 5 seconds you complete an out and back switchback on a bike – it may appear you’ve only gone 10 feet, when in reality you’ve gone 50 feet.

There’s a pretty substantial wooded park not too far from my house, complete with some running trails that wander aimlessly through the forest.  So I headed on over and setup the test.

IMG_2970

Below is an overview of the route, which was one-way along a single-track trail that did a lot of twisting and turning.  I then went back to the start and did it over again.

image

Unfortunately, park rules dictate that you’re not allowed to bike on these trails, so the tests are only walking and running…but, I ran pretty fast to make up for it. :)  Though, I believe once you see the test results, you’ll see that cycling probably wouldn’t have made things dramatically different…except perhaps made it worse.

Like always, I started with walking the rolling tape around the course, before running it.

IMG_3004IMG_3007

The course ended up being 2,157 feet long, or .41 miles:

IMG_2995

Also of note is that during my time in the forest I got a near equal number of mosquito bites as there are meters in each lap.  I was getting eaten alive!

Last year, we saw significant differences in data length – and that continued to this year as well. Check out the results:

image

You’ll notice how every single unit except the FR310XT and X8 while walking came up short (these two unit went long while walking).  This tends to be one of the key issues with using GPS for short and quick switchbacks, like those present on my route.  The units tend to miss those switchback data points and in turn present a shorter and more direct track/route than you actually ran.

Despite common misconceptions, it’s not typically GPS reception that’s the issue here in forests – but rather tracking of rapid changes in direction that are short lived and non-constant.  Meaning, had you I just made a 90* turn and kept the course – everything would have been fine.  But because I turned 90*, then back 90* a few meters later, it misses that section where I jetted out a bit – and thus gave me a short distance.

This is one reason why many trail runners still use foot pods in lieu of GPS for speed, or in the case of Garmin and Polar units – choose to select footpod within the unit’s settings for speed but keep the GPS for the map plotting and elevation.

Test 5: The Bridge Underpass Test

This last test I wanted to perform was a bridge test.  Many of us have seen that bridges and tunnels can cause our GPS reception to drop out, must frustrating during races.  But the loss of reception in and of itself is completely normal – after all, one can’t fully except the devices to track well under many feet of thick concrete and steel.  It’s the recovery that’s primarily of concern.  So I hit up a newly completed running underpass just down the street, which passes under a four lane road.

IMG_3262

Here’s a view of what it looks like from above.  Since it was just completed a few weeks ago, the satellite image is a bit older – hence all the construction going on.

image

(This little path just ‘opened’ two weekends ago, and by ‘opened’ I mean someone – not me – took down the banner closing it off.  Nonetheless it was flooded with runners/cyclists while I was there testing.)

The goal here was really to try and cause one of the GPS’s to lose satellite lock and then falsely pick it up with lower accuracy results, to see how it handles such a situation.  See, when a GPS unit loses satellite reception it goes back through an acquisition phase.  It’s during this acquisition phase that the level of accuracy is less, in other words – instead of being +/- 10ft, it’s now +/- 200ft.  Over the course of a few seconds (or longer), it then normally work it’s way back down to acceptable levels.  But sometimes, when it’s still acquiring satellites it’ll drop incorrect data points – perhaps hundreds of feet or even meters away.  In fact, for those of you that have tried swimming with your Forerunner watch (and not using the FR310XT openwater swim mode), what occurs here is the rapid loss and gain of satellites as your arm goes below the water and associated inaccurate accuracy levels upon reception during the recovery phase of the stroke, thus plotting incorrect data points.  You can actually look at your satellite accuracy level on your device, and I walk through how to do this at the end of this post.

In the below image, you can see a very tame version of this, where as I run in both directions – satellite reception drops, and then picks up closer to the other side, just not quite in the right place.  A few dozen feet later it gets back on track.

image

The above data map was from the Edge 800, while running.   But it gets more interesting if you look at the three different tracks – walk/run/bike – you can clearly see how speed affects map accuracy.

image

But you want to know the real kicker?  Despite showing vastly different tracks – the actual distances weren’t terribly different.  So while the unit showed .21 for both the cycling and running tests, it showed .20 for the walking test.  Yup, you read that right – the one test where it showed the least direct track was actually the shortest.

So I dug a bit deeper – what about the exact track lengths – in meters?

image

As you can see, the difference between the unit showing .20 and .21 in this case was only 5 meters.  Yup, that’s it.  And the reason why it was actually shorter is because while it looks wonkier, it actually cuts the corner a bit.

Now, let’s look at how the units actually performed…cause that’s what really mattered.  The wheel itself measured 1,111.4 ft round-trip – or 338.8 meters. I’ll note above that in the above running test, the Edge 800 managed to get within 6”.

IMG_3256

Here’s the full test results:

image

All in, there were no vast differences between reality and the data the units recorded – except for the FR610.  In fact, if you look at the results, you’ll see that all of them except the FR610 were within .01.

For fun, I went and dug into those data files and found that for the FR610 on these tests they were 305m, 307m, and 313m meters respectively – or about 25-30 meters off from the stated 338m.

The key takeaway here isn’t so much that GPS can recover from dropouts – but rather that speed absolutely affects how long the dropout will be, as well as track position.

Summary of Results:

All in, there were over a hundred tests files created, covering the most common scenarios.  But with so many different scenarios and data chunks, I figured it’d be more useful to consolidate them down into a single chart, showing both average and median values, as well as how each unit performed in summary.  Remember from high school math that median values work by simply eliminating the top and bottom values to find the middle ones – which is why you can have a sub-100% average and still end up with a 100% median.  I showed both metrics as a way to remove cases where a device had a single bad test (i.e. RunKeeper’s 1.15mi in the 800m test).

image

(Click the chart to expand)

As you can see, the majority of watches were within 3% on median – the only outlier being the FR610 (and if using the averages instead, the x8 and Nike+ would also fall outside).  Consistently the FR610 came in about 2% short – which is definitely interesting.  While one could note that the shorter distance results skew the percentages (and that would be true), the 1-mile test and the half-mile loop still hold being off 2%.  What’s interesting though is that it’s consistently 2%, meaning that no matter the distance its short by that same 2%, as opposed to being 2% at half a mile, and 4% at a mile.  The test that brought down the FR610 considerably in the averages was the tree test – which, to be honest, brought down everyone – hence why I also showed median values.

That said, I wouldn’t consider that a reason alone to avoid the FR610, though it’ll be interesting to see if some of the rumored firmware updates for satellite acquisition issues fix that.  I’m waiting on an outstanding answer from Garmin on that one.

Looking at the remaining of the results though – you can see that like last year, in general the watches were within the typically advertised 2.5% accuracy rates – thus continuing to cement the validity of using GPS for most training and racing scenarios in sports today.

GPS Signal Tips, Tricks and Notes:

(Last year I added these and they helped quite a few, so I copied them over again as a good way to close out the post)

There are ways you can either directly or indirectly improve your devices GPS accuracy, and most of them are quite simple and easy.

1) Wait for the signal to get fully acquired

When you first power on your GPS device, it’ll typically go through a satellite location process.  This is simply the device finding its best satellite buds and getting to know them.  On older devices (pre-Forerunner 305), this used to take a heck of along time.  On the FR305, this was improved to usually less than a minute.  On the newest class of devices with the Hotfix technology, this is typically on the order of 10-20 seconds assuming you haven’t dramatically changed locations.

With the newer devices (FR310XT/110/210, Edge 500/800, etc…), the devices essentially remember which satellites you last talked to, and aim to find those same satellites again.  By remembering where they were in the sky, they can quickly find them and then get you running in short order.

And if it’s too cold (or warm) outdoors to wait around, remember the window trick to satellite accuracy.

If however, you don’t wait for the satellites to get fully acquired, then your accuracy is less than stellar…which I’ll talk about in the next item.

2) Look at the current accuracy

Did you know that on your Garmin you can see the current satellite accuracy?  While the unit says “Acquiring Satellites” simply hit the up button to see the satellite page.  The page will look like the below:

IMG_0684

This page tells you a few things.  First off, it lets you know which satellites its connected to, and the strength is of each of those satellites.  Those are black bars you see.

Next, it tells you the current accuracy of your unit.  Meaning – exactly how far off the mark could your location be?  Typically the Garmin units need to be within 40-50ft before it’ll consider it ‘locked’.  If it shows 300+ft for example, that means that it could identify you as being almost a football field away.  This would have a direct impact on the distance of your activity, as well as speed and other items.

You may remember my In Depth Post on the Openwater Swim Mode, where I talk a bit more to how accuracy acquisition affects results.

3) Not getting correct mileage?  Try a soft reset.

I get a few e-mails a month from folks who see wildly inaccuracy distances – especially compared to friends running with GPS units.  For example, I’ll hear of someone running a 10-mile run with a group, and their unit will read 8.5 miles (that’s way off!).  In virtually all these cases a soft reset will resolve the issue.  A soft reset is kinda like rebooting your computer.  It doesn’t delete any profile or history data (a hard reset would), it simply resets a few things and most importantly – resets the satellite history.  Once you reset this satellite memory, the problem almost always resolves itself.

4) Update the firmware

You may remember my older post on the importance of updating firmware.  While many firmware fixes are addressing bugs or adding new features, there are certainly numerous instances where satellite acquisition and accuracy is being updated via firmware updates.  For example, there was a firmware update a while back that fixed a few issues with satellite accuracy on the Forerunner 310XT in certain situations.

In the case of the Timex Global Trainer, the September 2010 firmware resolved virtually all of the major satellite acquisition issues that the first firmware had – so that’s a biggie for TGT users.  However, it’s generally true that the longer distance accuracy issues are still outstanding for the TGT (being 2% short) – despite the Nov 2011 firmware update.

If you found this post interesting, then you’ll probably find some of these related posts right up your alley – all part of the larger How-To section:

- Racing the Line: Understanding how courses are measured
- Understanding GPS Sport Device Elevation
- Garmin Openwater Swim Mode (Talks to satellite acquisition & accuracy in water)
- Importance of updating GPS firmware
- Tip of the day- Avoid Doing The GPS Dance
- Sport Device GPS Accuracy In Depth- Part I (2010 Edition)
- Sport Device GPS Accuracy In Depth- Part II (2010 Edition)

Retweet 10Like 14Google +1 0

29 Comments

  1. Man, of all the watches i bought a FR610!! =( turns out it was a 400 dollar mistake! this is such BS, im upset. Hopefully the firmware updates that are in process of being worked on(from what i hear) will fix these issues. if not, then i’ll sell my FR610 on ebay and buy a more reliable 305

    Reply
  2. … i mean, look at this!! EVEN A FREE android app performed better than a 400 dollar watch!!! OMG!

    Reply
  3. “This is one reason why many trail runners still use foot pods in lieu of GPS for distance, or in the case of Garmin and Polar units – choose to select footpod within the unit’s settings for distance but keep the GPS for the map plotting and elevation.”

    Footpod for distance? You mean speed, or did I miss some setting on my garmin???

    Reply
  4. How could the FR610 possibly be that that inaccurate? I thought it was using the same chip as the 210? Getting its ass kicked by the ancient 305 is shocking. Surely Garmin will have to do something about this!?

    Reply
  5. This comment has been removed by the author.

    Reply
  6. Thanks for another great review! The only request that I (as an unpaying reader) might have is to reduce the precision of your Excel percentages – when you’re pre-rounding the distances to, say, .19-.21, its a little disconcerting to see accuracy percentages like 90.48% instead of just 90%. The .48% is far less than the rounding error in that case :) Otherwise, an amazing piece of detailed, laborious work, for which I thank you (and will continue to use your amazon links).

    Reply
  7. Ray thanks again for another great review, I appreciate the time and effort you put into these. I am a stickler for details and love the technical manner you approach each of these reviews. Keep up the good work, these definitely help in making decisions on the purchases I make.

    Reply
  8. I gotta be honest I’ve had my FR305 now for three seasons and it still runs like a top. Its not as fancy as the newer models, it may be bigger – but it has more than enough features for me. It is a pretty good steal right now at $125 or so w/ HR strap.

    Reply
  9. Accuracy has been making me nuts, but I’ve discovered that after about 2 months of regular use (and without firmware upgrades), the units all tend to become far more accurate. I’ve seen this with two 310XTs, the 210, and now the 610. And, what’s strange, is that it’s not limited to a particular location – after two months of use, no matter where I am, the unit is far more accurate.

    Reply
  10. Great review! I disagree and agree with Richard about the rounding in your results. Since it seems you’re showing distance in hundredths of miles simply b/c that’s what most watches display, I wouldn’t necessarily truncate useful information shown in the percentages. However, if the smallest unit of measurement for the devices is one meter (is that correct?), then it would make sense for a one mile test (or less) to show error down to to tenths of a percent instead of hundredths. And for a the overpass test of 338m, probably only to whole percentages.
    As is often the case, I have gotten caught up in what most would consider meaningless details.
    Oh well.
    THANK YOU! Great review.

    Reply
  11. Great review, very in depth.

    When you are using a GPS watch with a foot pod how does the data output work? Does one set of distance data override the other, or do you get both sets?

    Reply
  12. Dissapointing about the 610. I am sure they will resolve the issues with firmware, but I would not be happy if I dropped $400 on this device having a core functionality flaw that clearly should have been fettered out in development and testing. This is not the only report of possible GPS problems with the 610 that I have read. I just recently (this week) decided against the 610 because I could not justify the cost over the 305. My wife has had the 305 for 4 years and loves it. I was able to get the 305 brand new with HR strap for $100 with free shipping from Amazon with their current 20% off – glad I made that choice.

    Reply
  13. Wow, I guess I was ahead of the curve on this one then with all my complaining about the 610 accuracy and people telling me I was wrong…

    DC – you need to do the 610 test while tapping the watch to clear the Time Started/Lap windows, you will probably see higher accuracy, because if you don’t you will have the GPS accuracy shoot up to > 200ft most of the times which causes the horrible GPS errors.

    Reply
  14. Thanks for doing these tests and holding the manufacturers feet to the fire. Love my old old 305, but c’mon Garmin, let’s get a fix for the 610! Nice to see Polar back in the game too.

    Reply
  15. Another happy 305 user here. Bought it (using your link) 1 yr ago (to replace the 205). Good work on this series of posts!

    Reply
  16. Tim K

    Thanks Ray! Couple questions:

    1. I think you said in Part 1 the Edge 800 was done in smart recording rather than 1 sec. Do you think that would have helped?

    2. It seems like Garmin Connect “smooths” data when I look at it. Ex: I know I hit 35+ mph down a short 16% grade. GC says max speed was 35+ in the summary, but the graph only shows a speed of high 20s in that spot. The total ride mileage in GC matches that of the 800.

    3. Garmin Training Center seems to match up with the 800 better, but it’s so user unfriendly I never really look at it.

    4. Other on-line services end up with slightly different total mileages for rides. (Ride With GPS, Map My Ride, Running Free, etc) Would those be more accurate than the Garmin distances (either the 800 or the Garmin software)

    5. Sport Tracks allows for the “raw” distances from the 800, or the “calculated” distances from some kind of data crunching it does. They differ slightly in my experience. Not much, but on my 80+ mile MS ride for each day it’s close to 2 miles.

    Any thoughts? Are the mapping/online services that analyze the .fit file more “accurate” than the Garmin distances?

    Reply
  17. Tim K

    Forgot one comment:

    Never mind the gadget give aways. I want some of The Girl’s cupcakes.

    Reply
  18. Ihsan

    Ray,

    Thanks on the soft reset tip. On one of my bikes I keep my old computer along with the Edge 500 (so that I can track cadence from the old computer). And almost always, Edge 500 gives a total distance that is about 1 mile less compared to the other computer with the speed sensor.

    I will try the soft reset and see if it fixes the distance issue

    Ihsan

    Reply
  19. Could the difference between 210 and 610 be the medhod – you said that 610 was using 1s recording mode and 210 the “magic” recording mode?

    You should maybe replay the test with 610 also using the “magic” recording mode – that could explain the difference between those two devices?!

    Because in 1s recording mode every mistakes with gps accurancy get bigger than with “magic” mode!

    Reply
  20. I did a test between two identical 405′s. I was surprised at the differences between them, with both on the same arm in the same place. I did multiple runs and multiple cycles. They were always different, but sometimes one was longer- and sometimes the other was longer. So- it’s possible that your data would look different if you ran the same test over again with the same units, or with different serno’s of the same model.

    Secondly- it appears much of the understatement comes from the trail run. Your measurement wheel was very small, and would have picked up a lot of small undulations in the terrain that are irrelevant in running or cycling distance measurements.

    But- fun stuff Ray. thanks!

    Reply
  21. james

    Where can I get this RCX5? as a poster noted above, nice to see polar back in the game.. Not only back, but looks to be a very clear choice

    Reply
  22. This totally rocks! Thank you so much. I had decided on a Garmin 305 based on your reviews and now I’m super happy I did. We live in Columbia, MD which is full of trails. My husband has been using his Iphone 3 for walking/running and it’s been grossly inaccurate. My FR305 is always consistent. Makes me believe more that mine is definitely accurate enough to feel confident that I’m covering the distance it states. What a great little machine that is quite reasonably priced too. (it’s helped me lose 55 pounds so far too!)

    Reply
  23. Gunnar Bruun

    Thanks for a great article.

    I was wondering if you set the GPS-watches to record data as often as possible. They’re not always set up to choose pr-second-recording as default.

    Reply
  24. Magnus H

    Thanks for the test! I was thinking of setting up something similar, but also include the calculated length from software such as Sporttracks. The postprocessing software tend to get larger values then the clock internal distance when running in the forest (orinteering). Could you check the length calculated in Sporttracks from the GPS track on the GF305?

    Reply
  25. Vincent

    At last: accuracy tests!!
    Man, nobody seems to do real tests anymore and you are a wonderful exception! Thank you! Most “reviews” nowadays just ramble along about menu options and features… (like a camera, why would I need a camera on my GPSr)

    I’m still looking for accuracy tests (HDOP, VDOP) of Garmin GPSMAP 62 and Magellan Explorist 510 using survey benchmarks and under tree canopy.

    Guess what… Not a single accuracy test on the whole www. I would do them myself and plot the accuracy results in SA Watch or Visual GPS but I don’t have the funds to buy them both, nor do I know anybody who owns a unit I could borrow.

    Anyway, keep up the good work!

    regards, Vincent

    Reply
  26. Rainmaker, you are my hero!
    Amazing work, just amazing.

    Reply
  27. FWIW – and I don’t know if it will address FR610 accuracy – it IS possible to to a satellite re-sync as well as a “master reset” that dumps all satellite data. It’s on Garmin’s site in the FAQ:

    Auto-locate / satellite reset:
    link to support.garmin.com{fe4a0470-a754-11e0-d01c-000000000000}

    Master reset:
    link to support.garmin.com{0439fb90-a761-11e0-d01c-000000000000}

    Another note: thorough cleaning of the charging contacts seemed to fix the common “dis-charging while connected to charger” problem.

    Reply
  28. Moob

    Did the 610 have the latest firmware update? 2.4 I think.

    Reply
  29. Hi Moob-

    This was pre-firmware update. However, I posted afterward that firmware update (FR610) was publically released and the results were much better. Here’s those:

    link to dcrainmaker.com

    Thanks for the reminder, been meaning to update the chart.

    Reply

Post a Comment

Your email address will not be published. Required fields are marked.
If you would like a profile picture, simply register at Gravatar, which works here on DCR and across the web.

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>