How to learn from the Marine Corps Marathon App Fiasco Failboat

For those of you outside the DC area, you may be unaware of all the trouble that occurred over the past few days when one of the nation’s largest marathons contracted an app to be put together to allow racers to be tracked in real-time by spectators and fans.  So even if you’re not from the DC area, the lessons learned from this expand well beyond the DC area.  They should serve as a warning to other sporting event organizers, sports technology folks, and really – developers in general – as a text book example of how not to do things.

Oh, and for referenced – this is indeed the same app I mentioned last week, were I largely gave it a thumbs down in my review.  If only I knew what was to come, I would have found a few more thumbs.

In the beginning…

If you turn back the clock to previous years, the Marine Corps Marathon like many other organizations used a combination of an online results site with e-mail and text alerts.  While the online site sorta functioned, the e-mail and text alerts had a multi-year history of failures.  We’re not just talking one minor incident, but year after years of disappointment.  One year virtually nothing fired correctly, and the next folks got alerts 10 times over…hours late.

So, perhaps born out of those experience the folks at RaceMate likely approached the Marine Corps Marathon (MCM) to build an application for them that would allow runners to be tracked in real-time using race-provided splits, as well as allow even greater tracking depth if the runner wore a cell phone running the app (pretty much exact location of the runner).

From a monetary standpoint what likely occurred is that the RaceMate folks probably offered to build the app and run the service for either a low cost or without cost, in exchange for non-stop front page coverage on the Marine Corps website.  With the cost of the app at $2 per phone, and the number of folks following runners well into the hundreds of thousands (both at home and on the course), the math says even if a small percentage downloaded – they’d turn a handy profit.  Especially when you consider that the foundation of the application would be designed to quickly adapt to other marathons in the future.

All in all – sounds like a win-win, right?

Then came the dark clouds…

The first sign of trouble was the announcement that MCM was changing their model on the texting capabilities.  Instead of offering it as a free service, they would now charge if you wanted to track multiple runners.  Given tracking multiple folks is a time consuming effort – having text messages is really the easiest – especially if you’re out on the course trying to watch for your friend to come by.  So this was essentially strike one.

There was also the strange statistic that the Marine Corps Marathon folks were touting via RaceMate that they expected 46% of the racers to carry smart phones during the race.  That’s 1 in every 2 people.  Quite frankly, that’s just crazy.  I did some informal counting while watching runners go by…and only almost nobody carrying them while racing.

The next bit of trouble came when the first version of the app was released.  It didn’t take long for users (including myself) to notice that the app lacked any of the functionality that the Marine Corps Marathon organizers were promising.  In fact – it was missing the only thing that really mattered: Athlete Tracking.  Given Apple’s App Store approval process, it can be upwards of a week or more before an app update is approved.  Of course, it didn’t that the app didn’t do what it promised to do: track users.  This is an of itself is a clear violation of Apple’s App Store guidelines and rules.

However, fear not – just 36 hours prior to the start of the race, the RaceMate folks released an update to the app – including the mythical tracking. 

Of course, as normally occurs with App Updates, it’s not something that automatically happens for a user.  A user must go into the App Store and approve/deny updates – hardly something that would likely happen for the majority of the users in the 36 hours prior to the big moment.

The calm before the storm…or…did the lights go out?

With the App out, maybe all would work out.  Maybe the marathon technology gods would shine down upon DC just like the sunny weather.  Or maybe…maybe we didn’t even get to the start.

See, one of the most common mantra’s of marathon training is: Getting to the starting line healthy is the most important item.

And in this case, the RaceMate app failed at that goal.  See, the servers themselves had already crashed under the pre-marathon load the night before the start of the race.  And were so crushed they couldn’t keep them up, thus nothing at all worked come time for the starting gun to go off.

In short: The boat never left the dock.  The app was dead on arrival.

And by mid-day the app was pulled from the App Store altogether.

In comes the fire and fury

If you think that the comments on their Facebook page were furious – you should have heard the comments out on the course.  No matter where I went I overhead folks upset with the App.  After all – the Marine Corps Marathon folks had heavily pushed the app as the main way to track runners.

Though, I think the funniest line I heard all day was when one family asked a guy trying to work the app:

Hopeful person: “Oh wow…did you just get it to work, are you tracking someone now?” 
Funny Guy:  “No, nothing at all…but it is working quite well at showing where I am standing.  So at least I tracked someone today…even if it was only me.” 

For a while, the Marine Corps Marathon folks were trying to delete the angry Facebook comments from their page – but the flood seemed to outpace them.  There were also plenty of comments on the RaceMate page, here’s a few quality ones:

By late Monday morning the RaceMate folks were forced to refund every purchase of the app, for both iPhone and Android users:

So what can sports technology companies learn from this?

Lessons Learned

While the app could have been a huge success – and likely will go onto become a success at other marathons after some updates, there are many things that make me question some of the decisions being made:

1) RaceMate got lucky in that their app was approved just 36 hours prior to the race start.  That was in effect no luckier than going to a Vegas slot machine and having it payout.  The magic Apple machine could have easily approved that the Tuesday afterwards instead.  Which gets to the main point: There should be no reason why a company should be releasing the core of the functionality 36 hours prior.  This should be weeks or months prior.

2) The app was missing core functionality up until the last minute.  For an app that has so much potential, it lacked it. Race information, maps, tracking information, spectator information, etc…– all information that could have been included this past summer…not just a few weeks prior.

3) The app depended on ‘their servers’.  For reasons that continue to astound me – sports companies in particular try to run the server infrastructure themselves (I’m looking at you Ironman.com live event tracking).  The reality of sporting event traffic though dictates that you have significant spikes in traffic, and then huge dead time.  This means that spending money for the server infrastructure doesn’t make sense because you don’t get a valid return of investment.  Instead, a far better model for these types of loads is cloud computing where you pay for the bandwidth, processor cycles and hard drive space – instead of the servers themselves.  Companies like Amazon and Microsoft are the leaders in this space and make this incredibly affordable – even for both the smallest and largest of companies.  By offering elastic solutions, they’re able to scale to the largest loads on the Internet – far beyond an event like the Marine Corps Marathon.  Remember, in the scale of Internet events – the MCM isn’t even on the radar as a blip compared to larger scale events.

4) Look at the app as a supplement to event race day coverage – not a substitute for it.  The Marine Corps Marathon organizers saw this as a way to force folks to pay for text tracking.  While expecting users to pay for the App is reasonable given the extra functionality, expecting one to pay for text tracking is not reasonable.  In this case though – their aim was likely to reduce the strain on the system by diverting folks to the App.  They probably hoped to distribute their eggs amongst baskets, but in reality – they only bet on a bad basket.

5) Load testing apparently didn’t happen.  Regardless of whether you use internal or external servers – there’s no excuse for not completing load testing, both of expected load – and well beyond expected load.  As one who designs IT systems for high capacities, that’s a standard part of any project – and also one of the easiest to accomplish.  Either it works, or it doesn’t.  If it doesn’t – then you redesign (or re-develop) the system, or you add capacity.  ‘Hoping’ it works when it goes live doesn’t really work in my experience.

In summary, there are many lessons to be learned here.  And these lessons apply to the largest of sports technology companies, as well as the smallest.  Attention to core computing project tasks and project milestones would have realistically solved the majority of these issues.  And by the same token – races need to hold their vendors accountable, and if they aren’t delivering – they need to pull the plug earlier…not just leave folks stranded on race morning.