Newcomers to GPS sport devices like the Garmin Forerunner or Timex Global Trainer often wonder exactly how accurate the devices are. And seasoned users occasionally notice small oddities when comparing their run totals between training partners on the same run. So that got me thinking – why not put some of the GPS devices to the test. Sure, we all know the distances our local running and cycling routes are ‘roughly’. For example, we know that in theory our local 1 mile path loop is 1 mile…but – is it exactly one mile, down to the foot? And if so – if you took a sport GPS device out there, how accurate would it be?
Would it be 100 feet off, 10 feet off, or 1 foot off?
Further, with many of the devices now using the exact same underlying GPS chip – how important is the software that runs the device when it comes to distance accuracy? And would devices made by the same manufacturer with the same chips be the same?
Well…I set to find out.
Methodology:
Over the course of a number of weekends I did four separate and distinct tests. The tests involved four different courses with varying levels of ‘difficulty’, these included:
1) The Straight Arrow: A perfectly straight out and back course with low tree coverage – exactly 1.00 miles.
2) The Rambling Loop: A half a mile loop with plenty of turns, a bit of trees and some ups and downs – basically your average run.
3) The Deep Tree Adventure: Into the woods I went, on a fairly twisty and turning route on dirt trails
4) The Quick Turn Test: A very tight .05 mile loop course done over and over again with sharp corners.
Of course, you may be wondering how I was measuring these. Well, you might remember back to some of my previous course measuring posts, where I picked up a rolling measuring tape – similar to those used to mark out distances for cross country meets.
So I took that little roller and then strapped all of the GPS devices onto it. By doing so I could guarantee they were all going the same distance. Plus, it looked way less sketchy than strapping them on my wrist.
Finally, I was curious how speed and method of travel impacted the results, so I did each test three times:
1) Walking: This was simple, I walked the route and used the roller to determine the distance. All GPS devices recorded the distance as well.
2) Running: I then ran the exact same route, holding the roller which had the devices attached. The rolling distance was already established in the walking test.
3) Cycling: I rode a bike, at speeds typically between 15MPH and 20MPH. The roller was attached to the handlebars, with the devices in turn attached to it.
In order to ensure I was using the same exact course each time I used markers (such as walking/running/cycling 1-foot from the edge of the path), or in some cases – chalk, to mark out the course so I was getting the same path each time. Also, for each test I waited for all devices to show zero speed before starting or stopping measurement, as some take a bit of time to catch up. Also, because of rounding in the display devices, I actually used the raw values within the files of each unit. These values are stored in meters (not miles as you might think).
Finally, here’s the lineup of devices I used for the test:
1) Garmin Forerunner 310XT (Running/Cycling/Triathlon)
2) Garmin Edge 500 (Cycling)
3) Garmin Edge 800 (Cycling)
4) Timex Global Trainer (Running/Cycling/Triathlon)
I picked these watches simply because they’re among the most popular (or in the case of the Edge 800 – it’s the newest).
So with that, let’s get into the first two tests that make up Part I.
Test 1: The Straight Arrow
This first test was the easiest of the bunch. Really just a way to establish baseline amongst the watches and see if I saw anything strange out of the bunch. There’s a pretty nice little running path right outside my front door that is almost perfectly straight for a significant distance. It’s got virtually no tree cover or buildings within 200+ feet. Really, it’s an ideal test case.
This was an out and back course, where I preset the test distance at exactly one mile (5,280ft).
I simply walked out half a mile, marked the turnaround, and then walked back.
When I was done, it looked like this:
Then, I repeated the same test while running. I made sure to stop and let all GPS units catch up at the turnaround.
And finally, I repeated the test once more with a bike, following the same procedure.
So, what did I find? Aside from finding out that this generates a LOT of data files to work with, I got some interesting information. Here’s the results of the first round of testing:
Now, the really important thing to note is the following:
The majority of devices were within one-half of one-percent.
Fairly impressive.
This is important, because the above color coding can skew the mind into thinking that any one unit was drastically off the mark by an insurmountable distance. If you look at the numbers above – it’s crazy how close they came. No unit was more than 10 meters in total (out of a whole mile!).
Based on everything in the first test, the results across the board pretty much the same. Some units did slightly better on one test versus the other – but largely they agreed. Don’t worry though, we’re getting to the good stuff…
Test 2: The Rambling Loop
The next step is closest to what you might typically go out and run or ride with respect to conditions. The loop was around the perimeter of a waterpark, and featured both paved and compact dirt sections, as well as some tree cover, and a bit of swerving. It also had fairly open areas, and a couple of sharp 90* corners.
The total course was determined by the route, which ended up being 2,733ft long, which converted is 833.2 meters.
Here’s the overview:
Once again, I set out with Mr. Roller and got a walkin’:
Then it was time to run:
And finally, onto the bike:
After each loop I went ahead and ensured that all the pace/speed displays showed zero before stopping the timers – just to give each one a moment to catch-up. Once done I reset the activities and got them all downloaded.
Let’s get some results up on the screen:
Once again we see pretty accurate results across the board. Though, we also are starting to see a a few patterns develop.
First, if you do an average of the accuracy for each ‘sport’, you find that they actually don’t vary much at all between sports. Check it out:
Look at especially that very last line, difference between the walk test and the bike test for example, is .025%. That’s not 2%, but rather 2/100th of a percent. Pretty small.
Secondly, remember that the sports themselves aren’t important, but rather the speeds are. The watch doesn’t really care that it’s on a bike versus being carried. What it truly cares about is to a large degree speed. In this case, my biking speed (16MPH) was about twice as fast as my running speed (8MPH), which was then also twice as fast as my walking speed (4MPH).
Part I Wrap-up
As you might have noticed above – we’re starting to see a pattern develop where one specific device (Timex Global Trainer) is lowering the average consistently, and is off by more than 1% consistently while at speed.
However, what’s really interesting though is that while that same device has lesser results at speed, it actually performs pretty well at a walk.
This is where one has to start looking at how each company has implemented data recording metrics. For example, the Timex Global Trainer records a data point every 2-seconds. If you remember that GPS accuracy varies over the course of a route, perhaps that’s coming into play.
(Above: Timex Global Trainer raw data)
Meanwhile, the Garmin’s in these test all would have used smart recording, which means it records data points ‘when it wants to’, which typically is every 4-6 seconds, but sometimes more often.
(Above: Garmin 310XT raw data)
The question starts to become how does the recording interval affect the distance – if at all? Or will we find that the type of course and speed are really the major drivers?
To me, the key takeaway thus far though is back to the accuracy numbers, which when averaged are higher than 99% accurate. Typically both Garmin and Timex quote a 2.5% accuracy rate – and in this case, we’re nowhere near that inaccurate, are instead, far more accurate.
But…things are just getting interesting – stay tuned for Part II where we get into the fun tests. Dense trees, tight turns, and high speeds…oh my! Plus, I’ll cover some of the common questions and tricks to getting the highest levels of accuracy from your device.
And finally…if you think you know what’s coming based on the above tests…well..think again – the results coming up are mindboggling.
Read onto Part II here…
FOUND THIS POST USEFUL? SUPPORT THE SITE!
Hopefully, you found this post useful. The website is really a labor of love, so please consider becoming a DC RAINMAKER Supporter. This gets you an ad-free experience, and access to our (mostly) bi-monthly behind-the-scenes video series of “Shed Talkin’”.
Support DCRainMaker - Shop on Amazon
Otherwise, perhaps consider using the below link if shopping on Amazon. As an Amazon Associate, I earn from qualifying purchases. It doesn’t cost you anything extra, but your purchases help support this website a lot. It could simply be buying toilet paper, or this pizza oven we use and love.
Fascinating stuff Ray, I’m looking forward to part 2!
My 13 and a bit mile commute via a cycle route and a canal towpath is roughly 13 and a third miles, my Garmin Edge 500 has it varying slightly in distance on every commute.
Not an issue for me, but your blog is/will no doubt tell me why!
It would be very interesting if you had a Forerunner 305 available to include in the test. The Forerunner 305 has a mode in which you can store position every second (instead of only the “smart recording modes”), giving potential for more interesting data. Orienteers typically prefer the Forerunner 305 over the newer models due to the 1 second interval, as this gives more data for the analysis.
Hi Ray,
I’m curious as to the accuracy of your “wheel”, shouldn’t you be using a Jones counter as a proper test ?
Cheers,
chilli
What I’m curious about is the accuracy of sites like mapmyride.com? I’ve compared my runs with both an FR60 and a Forerunner 305 to plots from that site, and I get slightly different distances. Not much difference (<2%), but mapmyride.com is usually just a touch "longer" than the 305, and a touch "shorter" than the FR60 (calibrated on a 1600m track).
I’m wondering if mapmyride.com incorporates elevation changes into their distance calculations, where the 305 may not? As a note, I live in very hilly Pa.
You should get yourself a Jones counter and test compared to a course you measure with that based on a steel-tape test course.
awesome stuff.. thanks for doing that for us!
Ray,
it would be interesting to see if the result of the Garmins would be different when storing position every second instead of smart recording. Accoding to Garmin manuals Edge 310XT, 500 and 800 record points every second if they are paired with a powermeter. How about adding a bike with your Cinqo to the test?
Ray, really enjoyed reading this, there are a couple of questions I always had regarding GPS accuracy.
Does having the GPS device on your wrist effect the accuracy when you are running – as your wrist is constantly moving back and forth, unlike cycling when its fairly steady. The reason I ask is because I have heard the Edge500 isn’t very accurate when running as it doesn’t have the same software to account for this as say the Forerunner310 does?
Another point about GPS accuracy I had always wondered was whether it accounts for the extra distance you travel when you go up or down hills, or do they just measure “as the crow flies” distance?
thanks
Anthony
Pretty Fascinating Stuff!! I have a Garmin 405 and 305 and notice when my boyfriend and I go out for the same run, one might say a couple feet more! interesting
This is a great article. One more test I would really enjoy seeing is a course through an area with tall(er) buildings. I live in NYC and the mapped accuracy of my Garmin 310 is very jerky through the streets, i.e. a lot of times it has my path jumping from one side to the other of a street when I ran a straight line. These are the streets of SoHo too, so only about 4-6 story buildings. I’m sure if I were in midtown, around 40+ story buildings it would be much worse.
Cheers.
yay!! an experiment!! So fun!! This was a very interesting test! Thanks for sharing :D
I would like to add my opinion about the “jones counter”… can we not buy one of those too, there’s no space left downstairs!!
I agree with Jan, can you strap the 305 for a quick run to see where is comes in.
@Bertie…Jones Counters are like jello…..There’s always room for more!!!!
I often get a .98 on a track and when looking at the map you can see the curve is just a set of straight lines and every now and then there is a tangent that cuts too much. I have also been playing with the elevation and found that the best I can get is 75% when running up a mountain for 20 minutes that is a 10% grade and about 60% when doing rolling hills bc on the track I get about 10 ft “hills” I often wonder if this triangulation also can cause a distance irregularity if the point taken is consistently above or below “ground” Thanks for the test.
Hi,
Nice work! Really enjoy your blog. I was wondering: do you know if the Garmin uses an algorithm to correct for inaccuracies? Because IMO, the raw data should always overestimate distance because only part of the inaccuracies will even out. Only inaccuracy that is in line with the direction you’re going evens out (i.e. the estimated position is “in front” or “behind” the actual position). It evens out because at one moment the estimated position is in front, and the next moment it is behind the actual position. However, if the estimated postion is to the left or right of your actual position, this will not even out, but only add up (because it’s always a detour). So, the conclusion would be that the longer the run or ride, the greater the over-estimation? Or am I mistaken here? Would love to hear your thoughts on this.
If you haven’t finished the tests yet, how about throwing on an Edge 705? It would be interesting to see how this generation of technology compares to the last. In general, I have noticed my FR310 to be quite a bit more accurate than my Edge 705 especially in windy and tree covered situations. With the 500 and 800, is there any way to force 1 sec recording? as you mention, that would be another interesting metric to compare the effects on accuracy. At least with the 705, could do that.
I think there’s another import factor: whether there are clouds or not.
I’m very interested to see how they perform in heavy tree cover such as Lake Accotink or Burke Lake. Being the owner of a 310XT, I know how mine did under tree cover, awful lol.
P.S. – As soon as I saw your map I knew where you lived. I ride into work past there a couple times a week…. on good weeks.
Great blog, keep up the good work!
RE: Foreruner 305
I had considered adding the FR305 as well, but didn’t do it primarily due to the added data tracking. I might in the future just do a stand-alone test with the FR305 vs the FR310XT for fun.
RE: Jones Counter vs. Wheel
The wheel I have is actually a certified measuring device (link above in post), and unlike a Jones Counter isn’t reliant on exact PSI to get a correct measurement (the wheel in my counter is solid-state). That’s not to say that if I was doing longer tests that a Jones would make more sense, but for anything under a mile I can walk it easily. My particular wheel measures up to 9,999ft before I’d have to reset it. I’ve measured it against a tape measure a few times and it comes out accurate each time.
RE: Sites like MapMyRide and accuracy
The challenge with comparing MapMyRide and exact GPS coordinates from a track file is that MapMyRide is simply showing you the pinpoints you enter. So if you drag your mouse a few pixels to the left/right – that may be 30-50meters, unknowingly added. In some cases, that could be short, and others long – really just depends on how conservative you are with the mouse vs the run. ;)
RE: 1s recording vs regular
Hmm, that’s something to look at – though, look for tomorrows post for a few more thoughts on that. :)
RE: Wrist movement
In all the tests I’ve done, the Edge has faired pretty well actually while running – in about half of these tests it’s actually been the most accurate, despite the arm movement.
RE: Builings and GPS accuracy
I’ll be talking about accuracy distance in Part II. Stay tuned!
RE: Raw data
All Garmin processing is done in-unit. So the raw files I get are post-processing by the unit, but prior to any particular software suite interpretting them differently (sometimes you see different software suites try and correct for errors). This way we’re working from the same page.
Thanks all, and stay tuned for Part II…just a few hours away!
Here’s an interesting article that was referenced on the USATF site about GPS accuracy and course certification.
link to hamptonrockfest.com
Really looking forward to part 2. I have a 310xt and Edge 705 and I’m often intrigued about the accuracies of both. I did a 150km mountain bike ride not long ago and took both. The final result was a difference of around 200m. The difference seemed to be that the Edge was more prone to inaccuracies in trees.
Sorry if I missed it but did you use all the devices at the same? You have to account for differences in satellite alignment and solar activity too – not to nerd out on you but those effect GPS accuracy as well when comparing different devices. Now sure how much of a difference but I know it can make a difference.
RE: Jones Counter
For me the experiment provides more information when choosing which device to use. So the use of the measuring wheel or a jones counter makes no difference as i see it.
Unless the baseline measurements are overstated, the Timex clearly has the poorest performance.
Which leads to another interesting result, why are none of the watches showing a distance greater than the baseline measurement?? They all come up short.
Great article. A lot of good data that will make the next watch purchase a challenge
1) The Forerunners (and probably the Edges too) ALWAYS samples the GPS data EVERY second. In smart mode, it just doesn’t record the data unless there is a change in the velocity vector. So, the recording (note: recording) mode, is of no import for the _unit calculated_ accuracy. It might, though, be of import for software processing the data afterwards, but that’s hardly Garmin’s problem, so to speak.
2) A strength of the GPS devices is that a 10 meter error over 1km, does not necessarily translate into a 100 meter error over 10km. The random errors might cancel out in the long run (pun certainly intended).
Great catching up on your blog today. Echo on Katz and Central Park and great summary of the latest USAT. Thanks also for the triple for the guys overseas, nice touch!
oli
Good work, well done!
I have two garmin gps devices: edge 205 (for bike) and FR 310xt. I compared them on an athletic field, running on the first track (400m) 12,5 times (5km). After several comparisons, edge 205 accuracy (in smart recording mode) is better than FR 310xt. Typically, the edge 205 error is < 1%, while FR 310 XT error is often > 2%.
What let me be astonished is the route of FR 310 XT displayed by ST or garmin connect: it is worse that edge 205’s route.
That brings me to the conclusion that FR 310 XT is worse that its predecessors 205 on athletic fields.
I’ve always been curious about this, but hadn’t investigated for myself because I knew it would involve walking a route with that tape measure thing. I’m glad you did it for me!
Oddly enough, you appear to live right around the corner from me. It was trippy to recognize the routes you used as your test cases! I frequently run along the straight one!
I have been doing track workouts with the Garmin FR 225. Today I ran 800m interval training. Two laps around the track measured between .53-.54 miles according to my watch. It’s consistently high every week. The guy I run with has a Garmin FR 220 and his watch recorded the same laps as .49 miles every time. I believe the GPS hardware is identical. Why is there a difference and are there settings I can change to make my watch more accurate on a track. PS Running in a straight line on the road is very accurate with my watch