(Note: If you’re just joining me in Part II, be sure to read Part I, which covers the first half of the tests)
As we left off in Part I, the tests thus far had been largely easy. By that I mean that the first test was a straight out and back, while the second test was a loop with some terrain and trees it had nothing that really tested GPS accuracy in satellite challenged environment or situation.
In those tests we found that the majority of the devices did quite well, with most of them being within one-half of one percent compared to the rolling tape-measure accuracy. As you remember, all the courses are measured using a rolling tape measure, and then each course is tested with three different methods: Walking, Running and Cycling.
But that’s where the easy stops, and the tough get going. In Part II we’ll explore forests which can degrade signal, as well as well a tight course that shows how well the devices can track continuous changes in direction with a very small margin of error.
Test 3: The Deep Tree Adventure
Perhaps the most talked about scenario for sport GPS is how they handle in the trees. Trees make very natural obstructions to GPS signal, and depending on the type of forestry being discussed – can be rather tall and dense. While I’ve yet to find a forest or jungle anywhere in the world where I couldn’t get a GPS signal successfully, I have – like many people – seen it have an impact on tracking. The secondary reason for this is that most forest trails aren’t straight and narrow affairs. Instead, they tend to wander. With switchbacks it’s rare to just be going on a single compass heading for extended periods of time.
Switchbacks in particular can also cause issues if the recording interval is set such that it doesn’t catch the turns being made. For example, if a data point is only recorded every 5 seconds, yet within that 5 seconds you complete an out and back switchback on a bike – it may appear you’ve only gone 5 feet, when in reality you’ve gone 25 feet.
There’s a pretty substantial wooded park not too far from my house, complete with some running trails that wander aimlessly through the forest. Here’s an overview of the route, which was one-way along a trail. I then went back to the start and did it over again.
Unfortunately, park rules dictate that you’re not allowed to bike on these trails, so the tests are only walking and running…though, I ran pretty fast to make up for it. :) Though, I believe once you see the test results, you’ll see that cycling probably wouldn’t have made things dramatically different.
First up – was the walking test.
The course ended up being 2072 feet long, or 631.7m (.392 miles). Interestingly, this was pretty close to the little park sign that had the distance for the loop noted on it (which said .40 miles).
Then, I picked up the rolling wheel and went for a rather brisk run. I was moving fairly quick here – mostly in an effort to see the effect of higher paced twists and turns.
When I finished up my run, the first thing I thought was “Holy crap”. As in “Holy crap that’s a big difference in data points!”.
See, when I finished the walk I thought it was interesting that some of the data points were off a bit more than I expected. But, it wasn’t until I did it again while running that I was blown away by how poorly one device performed. It was clear that the results were dramatically different.
Check this out:
Now, perhaps it’s time to put some numbers to what this means in miles – because I know meters can be a bit tough to conceptualize over a longer distance. The course was 2072ft, which as noted above was .392 miles. So the actual distance displayed on the devices were:
As a side note, that the Timex Global Trainer is the only device that actually displays distance to three digits beyond the decimal. Also, I was also happy to see that all the raw data points matched the actual device displays, as seen here with the TGT:
Of course, the obvious elephant in the room is how badly the Timex Global Trainer had performed. To some degree, I was expecting it to have different results than the Garmin’s – but not by the margin of error it had. If you remember back to my Timex Global Trainer In Depth Review I noticed some distance oddities of about 2%. This matched what many folks left in the comments as well – with the Timex Global Trainer shorting the distance by between 2.0% and 2.5% – almost so perfectly exact each time that many wondered if a technical bug was at play.
I talked with Timex previously about the distance, and they came back with the standard answer around being within acceptable GPS limits.
That said, I don’t believe that being off upwards of 17% should be considered ‘Acceptable’.
I went back to look at the recording interval – knowing that the Timex Global Trainer is set at every 2-seconds, and was curious to see with the Smart Recording on the Garmin’s, how often it was recording a data point. The answer? Less than I expected.
Check it out – the below snippet of the data stream was the point with the highest interval rate of the data points, compared to some time periods with even less data points. Yet, Smart Recording ended up being the most accurate compared to the 2s interval of the Timex. Time is displayed in the left column – seconds in the right digits of that column.
While we may be out of the woods from a tree standpoint, we aren’t yet out of the woods on testing – time to do one more test involving tight corners…this time, with relatively open surroundings.
Test 4: The Short Circuit Course
This last test was born out of my desire to understand the impact of tight 90* turns on GPS accuracy. We often hear of GPS units ‘cutting the corner’ from a distance measuring aspect. So – I set up a short course around two parking lot chunks. This short loop had two 90+* corners, and one ~150* sweeping turn. You can see the course in it’s entirety in the below picture with the red line:
(Funny side note, there’s a guy there in that blue car, an older gentleman. He just sat there the entire time as I walked/ran/biked 12 times around him. I’m fairly certain he thought I was staking him out or something.)
Of course, to put a bit of context to the route, here it is from space – I thought it was interesting that the Google Earth map is just slightly offset by about 4-5 feet:
Now, with a course like this being as tight and short as it was, I needed to ensure that I was walking/running/cycling the exact same route each time, so I actually chalked the ‘race line’.
The course was designed so that I’d also be able to bike it with some speed as well, so they couldn’t be perfect 90* angles, thus I built in a route to ensure I could make the turn quickly.
Finally, in order to see the net effect after a bit of time more than just one short loop, each ‘set’ actually consisted of four laps. So the walk test was four laps, the run test four laps, and the bike test four laps.
With that, off to walk it I went. Once completed, I had my total of 1,369.5 feet – or 417.5 meters:
From there I ran it…and then finally, I biked it. I was actually rather proud of how accurate I was able to hit the chalk lines on the bike in the corners.
Especially given I had that big ole rolling measuring wheel strapped onto my handlebars.
Once I was all set, I reset all the data points and got them all sorted out. Here’s the results:
Like the previous test, we continue to see the same trend with respect to which devices are performing well, and which ones are having trouble with turns.
I was actually fairly surprised to see some of the devices as accurate as they were. For example – check out the the 310XT on both the run and bike. That thing is virtually spot on, despite the crazy corners. And, in the walk-test the Edge 800 is within .1 meters – or basically a few inches of being correct. That’s the distance alone between it and the little rolling wheel on a stick. Meaning, it may indeed be exactly correct. Pretty amazing.
Given this test had virtually no tree cover, it appears that satellite lock isn’t really the issue for the TGT. Instead, it appears it’s really a software problem with either how it interrupts the data, it’s polling cycle, or it’s recording interval.
Summary of Results:
The tests results in some 44 different test files to analyze, which I’ve made available here for those that are interested in the raw files…or just need filler files for a different project. Note that the raw files are still processed by the unit itself – they just aren’t processed by any additional software. All of the distances noted throughout both posts match what the unit said on the display screen.
I was curious though – if you averaged out all the tests, how does each device perform, as well as each device by sport?
As you can see above – in general the devices were within about 1.5% in the case of the Garmin’s, with the Timex Global Trainer having it’s average brought down by the twisty/turning tests. In the straight/normal tests, the Timex Global Trainer generally was within 1-2% of the actual distance.
We see that the Garmin 310XT and the Edge 800 are remarkably close – less than .06% apart. I also found it interesting that in general, the devices in some cases actually did BETTER at speed than while walking. Note how the 310XT does better in the cycling category than the walking or running. Same with the Edge 500. Though, at the end of the day, the Garmin 310XT takes the cake for the most accurate device.
I tried to be as complete and accurate with my tests as possible. As always though, with unlimited time and ‘budget’, I could devise a test matrix that covered every conceivable test length, distance and type. But I also wanted to be realistic. I structured tests in much the same manner as my bikes/runs – going where I please on terrain that we all train with.
We don’t walk/run/bike in a perfect test lab – and thus, these devices shouldn’t either. They should be able to perform accurately every time you use them – and in the case of most of these devices, that’s certainly showing to be the case. As a general rule, in normal situations it appears the devices are coming out within about 1% of the real distance.
It would be interesting to do a similar test each year and see how GPS accuracy improves over time as new devices come out, or firmware is updated for older devices.
GPS Signal Tips, Tricks and Notes:
There are ways you can either directly or indirectly improve your devices GPS accuracy, and most of them are quite simple and easy.
1) Wait for the signal to get fully acquired
When you first power on your GPS device, it’ll typically go through a satellite location process. This is simply the device finding its best satellite buds and getting to know them. On older devices (pre-Forerunner 305), this used to take a heck of along time. On the FR305, this was improved to usually less than a minute. On the newest class of devices with the Hotfix technology, this is typically on the order of 10-20 seconds assuming you haven’t dramatically changed locations.
With the newer devices (FR310XT/110/210, Edge 500/800, etc…), the devices essentially remember which satellites you last talked to, and aim to find those same satellites again. By remembering where they were in the sky, they can quickly find them and then get you running in short order.
If however, you don’t wait for the satellites to get fully acquired, then your accuracy is less than stellar…which I’ll talk about in the next item.
2) Look at the current accuracy
Did you know that on your Garmin you can see the current satellite accuracy? While the unit says “Acquiring Satellites” simply hit the up button to see the satellite page. The page will look like the below:
This page tells you a few things. First off, it lets you know which satellites its connected to, and the strength is of each of those satellites. Those are black bars you see.
Next, it tells you the current accuracy of your unit. Meaning – exactly how far off the mark could your location be? Typically the Garmin units need to be within 40-50ft before it’ll consider it ‘locked’. If it shows 300+ft for example, that means that it could identify you as being almost a football field away. This would have a direct impact on the distance of your activity, as well as speed and other items.
You may remember my In Depth Post on the Openwater Swim Mode, where I talk a bit more to how accuracy acquisition affects results.
3) Not getting correct mileage? Try a soft reset.
I get a few e-mails a month from folks who see wildly inaccuracy distances – especially compared to friends running with GPS units. For example, I’ll hear of someone running a 10-mile run with a group, and their unit will read 8.5 miles (that’s way off!). In virtually all these cases a soft reset will resolve the issue. A soft reset is kinda like rebooting your computer. It doesn’t delete any profile or history data (a hard reset would), it simply resets a few things and most importantly – resets the satellite history. Once you reset this satellite memory, the problem almost always resolves itself.
4) Update the firmware
You may remember my older post on the importance of updating firmware. While many firmware fixes are addressing bugs or adding new features, there are certainly numerous instances where satellite acquisition and accuracy is being updated via firmware updates. For example, there was a firmware update a while back that fixed a few issues with satellite accuracy on the Forerunner 310XT in certain situations.
In the case of the Timex Global Trainer, the September 2010 firmware resolved virtually all of the major satellite acquisition issues that the first firmware had – so that’s a biggie for TGT users. Note that it doesn’t appeared to have resolved distance accuracy however, as the unit used users is on the September 2010 firmware.
Useful Related Posts
If you found this post interesting, then you’ll probably find some of these related posts right up your alley – all part of the larger How-To section:
- Racing the Line: Understanding how courses are measured
- Understanding GPS Sport Device Elevation
- Garmin Openwater Swim Mode (Talks to satellite acquisition & accuracy in water)
- Importance of updating firmware
Thanks for reading!