No, Zwift Racing Wasn’t Hacked. Yet. Sorta. Let Me Explain.

In the computer security world there are a pile of conferences where security folks get together and present various sessions. I use ‘security folks’ as a loose term that can broadly cover everybody from IT professionals working for a non-profit like Red Cross, to government security peoples, to folks with less altruistic goals. These conferences have been around a while, and are generally considered good for the IT security community – assuming things like security bug disclosures are done properly (the concept of giving a company reasonable time to fix the bug before you talk about it).

One of the most well-known conferences from a lore standpoint is Def Con, but there are also many other huge ones such as BlackHat, SANS, and RSA, and other vendor-specific ones like BlueHat (run by Microsoft for Microsoft technologies) or government-specific ones. Again, in general the goal of these summits is to learn about security and improve security practices.

This past Sunday at Def Con (considered one of the more rambunctious events on the circuit) a presentation was given around Zwift and ‘hacking’ it – titled “Cheating in eSports: How to Cheat at Virtual Cycling Using USB Hacks”. Now one has to understand that while in the ‘mainstream’ the term ‘hacking’ is usually akin to ‘breaking’, in the computer world, the term ‘hacking’ is often a bit more nebulous. Sometimes used interchangeably with ‘tweaking’ or ‘optimizing’, and sometimes used in the less ideal variant such as ‘credit cards were hacked’. So one has to take any usage of that term with a bit of sanity check to see what’s going on.

In this case, the presentation was given by Brad Dixon (with support from Mike Zusman), security researchers with the consulting firm Carve Systems. This company has historically done penetration testing for other organizations (pen testing is trying to see if you can break into a system), but has switched in recent years to a more holistic security consulting approach where they do pen testing and then assist companies in making the fixes. More or less this is run of the mill security company stuff, nothing too crazy.

In this case though, two of the employees there are also avid cyclists and wanted to see where what they could do from a Zwift standpoint security-wise.

(Preemptive note: There are far easier ways to cheat at Zwift such as entering your weight or height incorrectly, or mis-calibrating your trainer. I’ve outlined them all in my past piece on the topic here in January. This post is focused on this specific security presentation.)

(Secondary note: There are now financial and career reasons to cheat in Zwift – including UCI sanctioned events, anyone telling themselves otherwise is thinking it’s still 2018. So we’re going to use the very real baseline that people are and will continue to cheat in Zwift where real stuff is on the line beyond bragging rights.)

(Tertiary note: This entire piece could be re-written without Zwift, and with ‘Bkool Racing’ instead. It applies equally. Minus the fact that nobody is actually using any of those platforms for major competitions. Thus when you’re the elephant in the room, it’s appropriate to still call you an elephant.)

(Quaternary note: What I discuss here is well-known in the trainer industry, this won’t surprise anyone. What it might do this is spur on developments to actually start securing this stuff, instead of just continuing to stick one’s head in the sand and think it won’t matter. If people are presenting Zwift hacks at Def Con, it’s time to actually address these issues. Nobody is better positioned than Zwift to do that.)

The ‘Hack’:

I had a chance last week before the conference to do a video conference with Brad and Mike and understand their presentation a bit more. We walked through the technology and where their ‘hack’ fit into things. I’m using the term ‘hack’ here in more of the ‘tweaking’ variant, than the ‘breaking’ side of things.

And in fact, one of the things they noted right at the top of the call was that when they started their research into Zwift they made the decision early on to “not screw people”, in Brad’s words. Specifically, that meant that they never competed in a Zwift race/event with the hack, nor did they save any rides. For clarity, this doesn’t minimize the effectiveness of it, it just simply means it didn’t meaningfully impact others.

Still, the obvious end-goal here is cheating in races. After all, that’s sorta the whole point. And in his presentation at Def Con, he points out what may be obvious to us, but might not be obvious to security researches in the room: “Cyclists are the best cheaters… Tom Brady should learn from these people”, and he goes on to discuss some of the history of cheating in cycling, be it doping or otherwise.

(Side note: It’s actually a fascinating part of the presentation with some crazy Tour de France cheats that have occurred over the last 100 years, it’s in the first few minutes in the YouTube video a bit further below).

And he’s right. It’s hard to come up with another sport where cheating (except perhaps bodybuilding) is as widespread and deeply engrained in over a century of athletics, at all event levels from major and minor competitions, and from pro to amateur. All with unrelenting mainstream media attention. Still, that’s beside the point.

What was their hack? They inserted code between the trainer/power meter and Zwift via USB, effectively boosting the power values that Zwift received.  Except, it was slightly more complex than that (and, also slightly less complex too). More specifically, they created a small device that you plug your normal ANT+ USB stick into (it’s off the shelf hardware with custom code atop it), which intercepts your ANT+ values and manipulates them with a tweaked value to Zwift on your computer:

(Note: All the slides/diagrams you see here are from their presentation)

So in effect, let’s take the most popular trainer out there and what they used for their demo: The Wahoo KICKR.

What they’ve done is inserted themselves in between Zwift and the KICKR, modifying those numbers before Zwift sees it. So let’s say you’re pedaling at 200w, what their little device does is tell Zwift that you’re really doing 220w, or 250w, or anything you want.

They’ve configured an Xbox controller to modify power, cadence, and heart rate. But they’ve also got simple scripts that do other actions. They showed two core ways to control their platform:

EPO Mode: Boosts your performance with a set multiplier
Slacker Mode: Automatically rides for you, and then fakes your HR and cadence based on terrain (so you can sit on the couch)

However, one of the biggest keys here is that the hack correctly replicates the source trainer from an appearances standpoint. Meaning, it looks exactly like a real legit Wahoo KICKR to Zwift. More specifically, this means that they re-passed all of the nuanced parameters of a KICKR, tweaking only the power metrics. That’s a core difference to using something as basic as the industry tool ANT+ simulator (used for testing by companies), which doesn’t really emulate something like a Wahoo KICKR or Tacx NEO – it just emulates a generic power device.

With their hack as is today, there are some requirements though:

A) Only works with Mac/PC (because USB), and not with Android/Apple TV/iOS (totally wireless, no ANT+)
B) You’ve gotta own the hardware, meaning it’s really only good for at-home events
C) Only applicable over ANT+

Now that last one might imply this is an ANT+ vulnerability. But it’s really more simplistic and could be achieved on both ANT+ or Bluetooth Smart. They just started with ANT+ because it was an easier starting point given how much of the code was already out there and developed. But technically speaking both protocols would operate the same way.

But there are also some challenges with the hack as well. As they noted, they didn’t use it in racing – because they didn’t want to mess with other people’s results.  That’s OK though, because we don’t need race performances to know where the hack breaks down a bit. It doesn’t do any of the following:

A) Doesn’t emulate a specific historical/past ride on the course
B) Doesn’t properly emulate the nuances of a human-driven power meter/trainer
C) Doesn’t correctly emulate the nuances of a human’s heart rate

When I say ‘nuances’, I mean that a human pedaling a power meter is constantly shifting their power by a few watts. No matter how perfect you think you are, you’re not actually pedaling exactly 250w every second. Instead, it’s roughly 248w, 253w, 247w, 250w, 255w, etc… (and that’d be a crazy precise person too). As such, algorithms from Zwift and 3rd party site Zwift Power (where results are tabulated) could more easily flag these efforts as they fail to appear like a human’s actual performance. Note that in EPO mode it would have the variability because it’s using your actual power as baseline.

So, in a nutshell, the hack presented as-is, is exclusively for use at home (where you control the environment), with a PC/Mac, and in a case where there might not be a ton of focus on your actual output numbers. Still, it’s a starter point. In the next section we’ll talk about where this goes from here.

However, if you’d like to watch the full presentation on YouTube, check it out below (it’s an audience recording, the official recordings aren’t up yet). Also, here’s their site with all the documentation details/etc

The Next Gen Hack:

Here’s the thing: In the grand scheme of hacks, this hack isn’t super impressive. Not yet. In fact, Keith Wakeham already outlined a more sophisticated hack in terms of resiliency to getting caught back in March. I asked Brad and Mike if they had seen it, and surprisingly they hadn’t. They noted they had started their research back in February before Keith’s video, and given that they came up with a fairly different solution – that seems pretty likely they didn’t know of Keith’s video. Speaking of which, here’s that video from Keith:

While both hacks seem similar – they’re actually fairly different. Specifically the following:

Keith’s hack is ANT+ based and doesn’t re-transmit anything from an existing trainer, but just acts as the sole source of data. Importantly though, it includes slight bits of variation to power/cadence/HR to more accurately emulate a human. He notes he has tried it in races and never triggered any sorts of cheating audits in place today.

 

 

Brad’s hack is ANT+ based but re-transmits existing hardware (trainers/power meters/etc…). It doesn’t include any human variability, but does more accurately replicate the source/original trainer – making it harder for Zwift to detect hardware anomalies.

But what lies ahead is more interesting, and useful. Brad says their next step is to convert over to Bluetooth Smart instead, which means that the entire setup is portable and easily moved to actual Zwift racing venues (such as what we saw here). Specifically, they could create a small bit of hardware that sits in a jersey, or even in a bag somewhere nearby. It wouldn’t matter – it’s all wireless.

Such a device would first connect to the trainer and take over the KICKR’s (or any other trainer’s) Bluetooth Smart connection. Because all of the trainers on the market today only transmit a single Bluetooth Smart connection, that takes away that connection and the trainer would disappear from Zwift (there are some nuances to this, but mostly trivial). But then a split second later the special device would rebroadcast the altered signal to Zwift. Because dropouts and such are totally normal in indoor trainer setups (especially if it were done pre-race), nobody would blink an eye.

At that point there’s a slew of ways the device could function. It could simply operate in the Digital EPO mode and boost your power slightly, just enough to give you an advantage. It would do so entirely silently in the background and nobody would ever know. Or, it could load a previous activity file (perhaps culled from Strava) and replicate that effort with slight tweaks to the power numbers. Seriously, the world is one’s oyster here.

So how does Zwift (or any other indoor trainer platform) protect against this? Well, they have to start taking it seriously.

Sure, Zwift has released their long policy document on cheating – but none of that would stop anything here. Not a single bit of it. Much of this isn’t entirely Zwift’s fault. For example, there’s no end to end encryption or authentication that occurs today from trainers to Zwift, atop via ANT+ or Bluetooth Smart. Just like there isn’t any atop power meters either.

If trainer companies were to implement some form of authorization or encryption (which both ANT+ & Bluetooth Smart already support), that would go a long way to stopping this particular man in the middle type cheat. Of course, that has a higher level of complexity for both trainer companies and app companies. And for the most part there isn’t actually great technical cooperation between Zwift and trainer companies, despite what these sides might say in public. While Zwift holds an annual conference for trainer companies each year at Eurobike, it doesn’t ever extend to technical working groups akin to what we see at the ANT+ Symposium (which Zwift doesn’t attend). Instead it’s more focused on larger business elements of the industry. And there’s nothing wrong with that either – it’s just there’s a gap to fill still.

Going forward:

In a lot of ways, it really comes down to whether or not Zwift believes esports and racing are the future of their platform. Everything they’ve said in the past 6-12 months says they deeply do believe that to be the case. They’ve spent more money than ever before on these efforts, with professional racing being the most heavily promoted aspect of Zwift in 2019, including UCI sanctioned events. They’ve also noted their aspirations to make this an Olympic sport by 2028.

Yes, it’s true that none of these are direct hacks against Zwift as a platform. But that’s sorta akin to saying “We didn’t have encryption on our banking website, so it’s not our fault someone stole your money”. That kind of security mindset stopped in the early 2000’s. Security extends beyond your direct premises, including the API’s that interact with it. In this case, Zwift’s API’s are effectively ANT+ and Bluetooth Smart, and the trainer and power meter companies on the other end. All of which easily fit into a single conference room at your local Holiday Inn (not even a ballroom required, mind you).

And of course, this isn’t just a Zwift issue – it’s an indoor trainer issue. At some point, other legit competitors to Zwift racing will come along (for real, they will, it’s only a matter of time). And they too will face these same underlying issues and attraction for cheating. While each year Zwift holds their Zwift Summit at Eurobike, a night later the rest of the indoor trainer industry (including every trainer manufacturer, many power meter companies, and major trainer apps) have historically also had a gathering, often to discuss more specific technical issues. Yet a few weeks later – everything from both events is mostly forgotten and nothing tangible happens till the next year.

Ultimately, the indoor trainer industry has to take itself seriously as an entity if it wants to be seen seriously by the likes of UCI and the Olympics. Otherwise, the result won’t likely be much different than other aspects of cycling: Full of cheating.

With that – thanks for racing…I mean, reading.

DC Rainmaker:

View Comments (58)

  • The next step will be to create the same type of regular “doping” control for Zwifter who want to participate to official events. On regular basis, they need to zwift at a controlled place (not at home) by Zwift or partners (events, dedicated control) to validate their level compared with their profile.

    NB: I don’t know if it is the result of this fascinating hack event you presented, but we know now that Zwift has been hacked by some countries which ask now to collect local taxes on top of the monthly fee ;-)

  • Early TdF stuff is fascinating. Came across it recently in the Marginal Gains podcast.

    This sounds like gplama’s past life :-)

  • Would requiring dual power sources sidestep this? Power recorded both from the trainer and from a second power source such as Favero pedals or something. Power from the pedals has to match the trainer power within the accepted range of accuracy.

  • I love this category of posts, they're like kryptonite to those of us who know just enough about the underlying systems to get in trouble (like me) :)

  • One fix would be to have the data source devices (power meters/hrm/cadence/etc) digitally sign their values. Each hardware vendor would have their own signing key, and the application could validate the signatures match the presented data and vendor. Signatures would be a new data point, so ignored by older software that did not care.

  • Even without cheating, how valid is a sprint victory recorded by different powermeters? If it is 2 seconds, can a real winner be declared? Or could that race have been decided by the hardware? As I understand even the same kind of powermeter can vary a bit from unit to unit. So even with all connections wired and everybody in the same room and under surveilance, how do you ensure the best athlete wins?

  • It's going to be very difficult to have an ironclad system even with encryption as long as you allow people to ride in their own homes on their own equipment - when the cheater has physical access to the instruments and those overseeing it do not, it's near impossible to fully secure the system. Even if the trainer->software->platform chain was impenetrable and calibration locked down, low-tech hacks like riding an electric assist bike, attaching a belt/chain driven motor instead of a bike or even riding a tandem would do the same thing as the above hack. Systems are only as secure as their weakest link, so beefing up one part really doesn't help unless you address all the others.

    Sanctioned events where you ride their trainers on their computers under the supervision of independent officials would be the only place where this would really be feasible, as everything past the chain/tire could be made untouchable by the competitors. In that circumstance though I'd think adding wired connections (like Kurt has) that could be physically secured would be easier to lock down than layering sophisticated encryption on wireless signals.

    That's not to say encryption is a bad idea, just that it's not a magic bullet that will fix the problem. Cheating has been a big problem in cycling even with armies of officials overseeing everything, so now that we've injected real stakes onto competitions where that's much more difficult to do we're facing a really complicated problem.

    • Agreed, home scenario is already lost. But for public events, two easyish solutions: make every unit broadcast raw ant+ and record everything for later inspection, and add local sd card or just embedded flash to record locally on the trainers and collect these after event for later inspection as well. Add signaturing to local storage to make it more fancy, if possible.

    • Sanctioned indoor rowing competitions on Concept 2 machines have been around for years, but they are done on location using controlled equipment. When tangible benefits are on the line, I see this as the only way moving forward for Zwift. For most of us, it is an irrelevancy, however, and I for one will not spend a single extra dollar to ensure my trainer or accessories are capable of being encrypted, etc. I really could not care less if someone develops code to cheat on a KISS race. After all, nearly everyone wastes time in their lives on inconsequential activity of one form or another.

  • An external dongle as described is certainly not necessary or even the easiest way to do this. In Windows, for example, a reasonably simple WDM Filter driver can insert itself in the USB stack and intercept / modify data coming up from the Ant+ dongle. Lots of examples out there on GitHub etc.

    The use of the Beaglebone does suggest a possible direction. It would be possible for Zwift (or anyone really) to produce a similar small device that would work in the same manner, with the Ant+ dongle plugged into it. But that device would make the data available to Zwift (or any platform) via an encrypted connection. E.g. TLS or SSH over via WiFi or Ethernet. With appropriate private keys.

    Using an embedded platform (e.g. Microchip PIC32MZ or similar) would allow for the software and certificates to be in private non-readable (without scraping and using a scanning electron microscope...) storage. The Ant+ hardware could be added directly on that board.

    Still not 100% non-hackable, but raises the bar substantially. At least until we see TLS or similar built directly into the Power Meter or Trainers.

  • Possibly the simplest fix for live events... Just use a wired connection?

    Now we just need to resurrect Computrainer, and we'll be good to go.

    For home events, I am not sure there is anything that can stop a cheater (motor-doping, actual doping, hacking, rider swapping, etc.)

    • We also make wired trainer. Would not even dream of "begging" Zwift to add our interface though as we are microbes comparatively.

    • The Kinetic R1 actually has a wired connection (as does some of their Smart Control units). Kinetic is basically begging trainer apps like Zwift to implement it. At this point it's mostly sound of silence.

  • I just hope that, if Zwift does add support for encryption, already-purchased trainers will receive updates to support it. But I’d worry the manufacturers used CPUs that were just powerful enough and no more, and thus won’t have the CPU bandwidth to apply fairly expensive encryption. If someone just spent $1200 on a trainer and in 6 months’ time their Zwift profile is missing some sort of “secure data” tick, that’d be pretty terrible.

    • I wouldn't expect such a task would add significant load to the CPU. I also wouldn't expect we'd see trainer companies go terribly far back either.

1 2 3

By continuing to browse the site you are agreeing to the use of cookies