For those of you outside the DC area, you may be unaware of all the trouble that occurred over the past few days when one of the nation’s largest marathons contracted an app to be put together to allow racers to be tracked in real-time by spectators and fans. So even if you’re not from the DC area, the lessons learned from this expand well beyond the DC area. They should serve as a warning to other sporting event organizers, sports technology folks, and really – developers in general – as a text book example of how not to do things.
Oh, and for referenced – this is indeed the same app I mentioned last week, were I largely gave it a thumbs down in my review. If only I knew what was to come, I would have found a few more thumbs.
In the beginning…
If you turn back the clock to previous years, the Marine Corps Marathon like many other organizations used a combination of an online results site with e-mail and text alerts. While the online site sorta functioned, the e-mail and text alerts had a multi-year history of failures. We’re not just talking one minor incident, but year after years of disappointment. One year virtually nothing fired correctly, and the next folks got alerts 10 times over…hours late.
So, perhaps born out of those experience the folks at RaceMate likely approached the Marine Corps Marathon (MCM) to build an application for them that would allow runners to be tracked in real-time using race-provided splits, as well as allow even greater tracking depth if the runner wore a cell phone running the app (pretty much exact location of the runner).
From a monetary standpoint what likely occurred is that the RaceMate folks probably offered to build the app and run the service for either a low cost or without cost, in exchange for non-stop front page coverage on the Marine Corps website. With the cost of the app at $2 per phone, and the number of folks following runners well into the hundreds of thousands (both at home and on the course), the math says even if a small percentage downloaded – they’d turn a handy profit. Especially when you consider that the foundation of the application would be designed to quickly adapt to other marathons in the future.
All in all – sounds like a win-win, right?
Then came the dark clouds…
The first sign of trouble was the announcement that MCM was changing their model on the texting capabilities. Instead of offering it as a free service, they would now charge if you wanted to track multiple runners. Given tracking multiple folks is a time consuming effort – having text messages is really the easiest – especially if you’re out on the course trying to watch for your friend to come by. So this was essentially strike one.
There was also the strange statistic that the Marine Corps Marathon folks were touting via RaceMate that they expected 46% of the racers to carry smart phones during the race. That’s 1 in every 2 people. Quite frankly, that’s just crazy. I did some informal counting while watching runners go by…and only almost nobody carrying them while racing.
The next bit of trouble came when the first version of the app was released. It didn’t take long for users (including myself) to notice that the app lacked any of the functionality that the Marine Corps Marathon organizers were promising. In fact – it was missing the only thing that really mattered: Athlete Tracking. Given Apple’s App Store approval process, it can be upwards of a week or more before an app update is approved. Of course, it didn’t that the app didn’t do what it promised to do: track users. This is an of itself is a clear violation of Apple’s App Store guidelines and rules.
However, fear not – just 36 hours prior to the start of the race, the RaceMate folks released an update to the app – including the mythical tracking.
Of course, as normally occurs with App Updates, it’s not something that automatically happens for a user. A user must go into the App Store and approve/deny updates – hardly something that would likely happen for the majority of the users in the 36 hours prior to the big moment.
The calm before the storm…or…did the lights go out?
With the App out, maybe all would work out. Maybe the marathon technology gods would shine down upon DC just like the sunny weather. Or maybe…maybe we didn’t even get to the start.
See, one of the most common mantra’s of marathon training is: Getting to the starting line healthy is the most important item.
And in this case, the RaceMate app failed at that goal. See, the servers themselves had already crashed under the pre-marathon load the night before the start of the race. And were so crushed they couldn’t keep them up, thus nothing at all worked come time for the starting gun to go off.
In short: The boat never left the dock. The app was dead on arrival.
And by mid-day the app was pulled from the App Store altogether.
In comes the fire and fury
If you think that the comments on their Facebook page were furious – you should have heard the comments out on the course. No matter where I went I overhead folks upset with the App. After all – the Marine Corps Marathon folks had heavily pushed the app as the main way to track runners.
Though, I think the funniest line I heard all day was when one family asked a guy trying to work the app:
Hopeful person: “Oh wow…did you just get it to work, are you tracking someone now?”
Funny Guy: “No, nothing at all…but it is working quite well at showing where I am standing. So at least I tracked someone today…even if it was only me.”
For a while, the Marine Corps Marathon folks were trying to delete the angry Facebook comments from their page – but the flood seemed to outpace them. There were also plenty of comments on the RaceMate page, here’s a few quality ones:
By late Monday morning the RaceMate folks were forced to refund every purchase of the app, for both iPhone and Android users:
So what can sports technology companies learn from this?
While the app could have been a huge success – and likely will go onto become a success at other marathons after some updates, there are many things that make me question some of the decisions being made:
1) RaceMate got lucky in that their app was approved just 36 hours prior to the race start. That was in effect no luckier than going to a Vegas slot machine and having it payout. The magic Apple machine could have easily approved that the Tuesday afterwards instead. Which gets to the main point: There should be no reason why a company should be releasing the core of the functionality 36 hours prior. This should be weeks or months prior.
2) The app was missing core functionality up until the last minute. For an app that has so much potential, it lacked it. Race information, maps, tracking information, spectator information, etc…– all information that could have been included this past summer…not just a few weeks prior.
3) The app depended on ‘their servers’. For reasons that continue to astound me – sports companies in particular try to run the server infrastructure themselves (I’m looking at you Ironman.com live event tracking). The reality of sporting event traffic though dictates that you have significant spikes in traffic, and then huge dead time. This means that spending money for the server infrastructure doesn’t make sense because you don’t get a valid return of investment. Instead, a far better model for these types of loads is cloud computing where you pay for the bandwidth, processor cycles and hard drive space – instead of the servers themselves. Companies like Amazon and Microsoft are the leaders in this space and make this incredibly affordable – even for both the smallest and largest of companies. By offering elastic solutions, they’re able to scale to the largest loads on the Internet – far beyond an event like the Marine Corps Marathon. Remember, in the scale of Internet events – the MCM isn’t even on the radar as a blip compared to larger scale events.
4) Look at the app as a supplement to event race day coverage – not a substitute for it. The Marine Corps Marathon organizers saw this as a way to force folks to pay for text tracking. While expecting users to pay for the App is reasonable given the extra functionality, expecting one to pay for text tracking is not reasonable. In this case though – their aim was likely to reduce the strain on the system by diverting folks to the App. They probably hoped to distribute their eggs amongst baskets, but in reality – they only bet on a bad basket.
5) Load testing apparently didn’t happen. Regardless of whether you use internal or external servers – there’s no excuse for not completing load testing, both of expected load – and well beyond expected load. As one who designs IT systems for high capacities, that’s a standard part of any project – and also one of the easiest to accomplish. Either it works, or it doesn’t. If it doesn’t – then you redesign (or re-develop) the system, or you add capacity. ‘Hoping’ it works when it goes live doesn’t really work in my experience.
In summary, there are many lessons to be learned here. And these lessons apply to the largest of sports technology companies, as well as the smallest. Attention to core computing project tasks and project milestones would have realistically solved the majority of these issues. And by the same token – races need to hold their vendors accountable, and if they aren’t delivering – they need to pull the plug earlier…not just leave folks stranded on race morning.
How funny is this and yet an app unrelated to the MCM, Runkeeper, actually released something like this and for the most part seems to work, albeit with an additional cost. Well maybe next year they will charge you for using your own phone to track yourself and friends
I think all race organizers should learn from this. After trying to track runners at various races for the last 5 years, one thing is for sure. Very few races have enough band width to support the demand on thier systems during race day. I don’t know how many times I have stood at the 10K point waiting for a runner only to get the 5K split on texted to me after the runner passes at the 10K mark.
Ray, it was actually MCM that was deleting comments from their Facebook page. Any comment or question that you posted about the app was deleted. Anything negative you had to say about the app was deleted. If you had a question about why there was no texting service – it was deleted. Even people who had unrelated problems – people whos final times were not showing in the results page – those questions were deleted.
Huge FAIL on the part of MCM mgmt for this. This is no way to treat a customer base.
FWIW RaceMate at least left the negative comments up and provided a response eventually.
I just dont understand how MCM could have made this decision for an unproven app and an unproven company. The “chip->database” link seems to work since I’m told the Results page was updated in real time and seemed to work. How hard can it be to spend a year focusing on the “database->public” part of that? It does not seem that hard.
Good post – point #3 is especially well taken!
semper fi… let’s get back to racing and having your time written on your bib when you finish. That’s all you need 🙂
I am surprised at how many races do not offer text message tracking, especially long ones like Mdot, where it would be _very_ helpful to the spectators. We paid enough. In this day and age, it should be seamless and part of the base package to send a text message within a very short time frame (single digit seconds) of crossing a split mat at anything more than a local 5k. If you use a chip, SMS tracking should be included. It can’t be that difficult to do a chip>email/SMS lookup>send even with thousands of hits in a small time frame. I am surprised that real time tracking is not ubiquitous.
There are free (instamapper) and paid (accutracking) java apps which run on bottom end Boost/nextel phones (and many smartphones) which allow realtime tracking and geofence SMS.
I hope they don’t can the app, but take their knocks and rise from the ashes better for the experience. I want this at my events.
Ouch. I hope the NYC Marathon app this weekend works out better. It’s done by Participant Sports Inc. No clue what their setup is. But in the past, the NYC Marathon text or email notifications have never worked. Always severely delayed.
there’s a sucker born every day….
I’m seeing a trend here…race organization companies seeing how powerful social media is. And actually responding to it. It’s a good thing
On another note, did anyone else feel the water/aid stations were wayyy to far apart? I was thirsty and then dehydrated from the half on….
Great post! I didn’t purchase the app, but there is a similar one for NYC this weekend and I’m telling my friends and family to track me online and not bother with the app.
+1 ForestGlen! There was NO water from just before mile 19 until just after mile 22. 3+ miles with no water so late in the race?? Obviously, not paying enough attention to the course map was on me. I guess I just assumed there would be water every 1-2 miles at the end, BECAUSE EVERY MARATHON I’VE EVER RUN DOES IT. To hell with the smart phone app; I would have liked some water on those darn bridges!
You know, MCM actually brought one of the app developers from Mobomo onstage at an evening event that they conducted for first-time marathoners on the Friday before the marathon.
Had the guy appeared at the finish line, he probably would have met a very grim end at the hands of hundreds of angry people armed with banana peels…
Interesting today a search of itunes for 26.2 shows this same app as #1.