No, Zwift Racing Wasn’t Hacked. Yet. Sorta. Let Me Explain.

In the computer security world there are a pile of conferences where security folks get together and present various sessions. I use ‘security folks’ as a loose term that can broadly cover everybody from IT professionals working for a non-profit like Red Cross, to government security peoples, to folks with less altruistic goals. These conferences have been around a while, and are generally considered good for the IT security community – assuming things like security bug disclosures are done properly (the concept of giving a company reasonable time to fix the bug before you talk about it).

One of the most well-known conferences from a lore standpoint is Def Con, but there are also many other huge ones such as BlackHat, SANS, and RSA, and other vendor-specific ones like BlueHat (run by Microsoft for Microsoft technologies) or government-specific ones. Again, in general the goal of these summits is to learn about security and improve security practices.

This past Sunday at Def Con (considered one of the more rambunctious events on the circuit) a presentation was given around Zwift and ‘hacking’ it – titled “Cheating in eSports: How to Cheat at Virtual Cycling Using USB Hacks”. Now one has to understand that while in the ‘mainstream’ the term ‘hacking’ is usually akin to ‘breaking’, in the computer world, the term ‘hacking’ is often a bit more nebulous. Sometimes used interchangeably with ‘tweaking’ or ‘optimizing’, and sometimes used in the less ideal variant such as ‘credit cards were hacked’. So one has to take any usage of that term with a bit of sanity check to see what’s going on.

In this case, the presentation was given by Brad Dixon (with support from Mike Zusman), security researchers with the consulting firm Carve Systems. This company has historically done penetration testing for other organizations (pen testing is trying to see if you can break into a system), but has switched in recent years to a more holistic security consulting approach where they do pen testing and then assist companies in making the fixes. More or less this is run of the mill security company stuff, nothing too crazy.

In this case though, two of the employees there are also avid cyclists and wanted to see where what they could do from a Zwift standpoint security-wise.

(Preemptive note: There are far easier ways to cheat at Zwift such as entering your weight or height incorrectly, or mis-calibrating your trainer. I’ve outlined them all in my past piece on the topic here in January. This post is focused on this specific security presentation.)

(Secondary note: There are now financial and career reasons to cheat in Zwift – including UCI sanctioned events, anyone telling themselves otherwise is thinking it’s still 2018. So we’re going to use the very real baseline that people are and will continue to cheat in Zwift where real stuff is on the line beyond bragging rights.)

(Tertiary note: This entire piece could be re-written without Zwift, and with ‘Bkool Racing’ instead. It applies equally. Minus the fact that nobody is actually using any of those platforms for major competitions. Thus when you’re the elephant in the room, it’s appropriate to still call you an elephant.)

(Quaternary note: What I discuss here is well-known in the trainer industry, this won’t surprise anyone. What it might do this is spur on developments to actually start securing this stuff, instead of just continuing to stick one’s head in the sand and think it won’t matter. If people are presenting Zwift hacks at Def Con, it’s time to actually address these issues. Nobody is better positioned than Zwift to do that.)

The ‘Hack’:

I had a chance last week before the conference to do a video conference with Brad and Mike and understand their presentation a bit more. We walked through the technology and where their ‘hack’ fit into things. I’m using the term ‘hack’ here in more of the ‘tweaking’ variant, than the ‘breaking’ side of things.

And in fact, one of the things they noted right at the top of the call was that when they started their research into Zwift they made the decision early on to “not screw people”, in Brad’s words. Specifically, that meant that they never competed in a Zwift race/event with the hack, nor did they save any rides. For clarity, this doesn’t minimize the effectiveness of it, it just simply means it didn’t meaningfully impact others.

Still, the obvious end-goal here is cheating in races. After all, that’s sorta the whole point. And in his presentation at Def Con, he points out what may be obvious to us, but might not be obvious to security researches in the room: “Cyclists are the best cheaters… Tom Brady should learn from these people”, and he goes on to discuss some of the history of cheating in cycling, be it doping or otherwise.

(Side note: It’s actually a fascinating part of the presentation with some crazy Tour de France cheats that have occurred over the last 100 years, it’s in the first few minutes in the YouTube video a bit further below).

And he’s right. It’s hard to come up with another sport where cheating (except perhaps bodybuilding) is as widespread and deeply engrained in over a century of athletics, at all event levels from major and minor competitions, and from pro to amateur. All with unrelenting mainstream media attention. Still, that’s beside the point.

What was their hack? They inserted code between the trainer/power meter and Zwift via USB, effectively boosting the power values that Zwift received.  Except, it was slightly more complex than that (and, also slightly less complex too). More specifically, they created a small device that you plug your normal ANT+ USB stick into (it’s off the shelf hardware with custom code atop it), which intercepts your ANT+ values and manipulates them with a tweaked value to Zwift on your computer:

(Note: All the slides/diagrams you see here are from their presentation)

So in effect, let’s take the most popular trainer out there and what they used for their demo: The Wahoo KICKR.

What they’ve done is inserted themselves in between Zwift and the KICKR, modifying those numbers before Zwift sees it. So let’s say you’re pedaling at 200w, what their little device does is tell Zwift that you’re really doing 220w, or 250w, or anything you want.

They’ve configured an Xbox controller to modify power, cadence, and heart rate. But they’ve also got simple scripts that do other actions. They showed two core ways to control their platform:

EPO Mode: Boosts your performance with a set multiplier
Slacker Mode: Automatically rides for you, and then fakes your HR and cadence based on terrain (so you can sit on the couch)

However, one of the biggest keys here is that the hack correctly replicates the source trainer from an appearances standpoint. Meaning, it looks exactly like a real legit Wahoo KICKR to Zwift. More specifically, this means that they re-passed all of the nuanced parameters of a KICKR, tweaking only the power metrics. That’s a core difference to using something as basic as the industry tool ANT+ simulator (used for testing by companies), which doesn’t really emulate something like a Wahoo KICKR or Tacx NEO – it just emulates a generic power device.

With their hack as is today, there are some requirements though:

A) Only works with Mac/PC (because USB), and not with Android/Apple TV/iOS (totally wireless, no ANT+)
B) You’ve gotta own the hardware, meaning it’s really only good for at-home events
C) Only applicable over ANT+

Now that last one might imply this is an ANT+ vulnerability. But it’s really more simplistic and could be achieved on both ANT+ or Bluetooth Smart. They just started with ANT+ because it was an easier starting point given how much of the code was already out there and developed. But technically speaking both protocols would operate the same way.

But there are also some challenges with the hack as well. As they noted, they didn’t use it in racing – because they didn’t want to mess with other people’s results.  That’s OK though, because we don’t need race performances to know where the hack breaks down a bit. It doesn’t do any of the following:

A) Doesn’t emulate a specific historical/past ride on the course
B) Doesn’t properly emulate the nuances of a human-driven power meter/trainer
C) Doesn’t correctly emulate the nuances of a human’s heart rate

When I say ‘nuances’, I mean that a human pedaling a power meter is constantly shifting their power by a few watts. No matter how perfect you think you are, you’re not actually pedaling exactly 250w every second. Instead, it’s roughly 248w, 253w, 247w, 250w, 255w, etc… (and that’d be a crazy precise person too). As such, algorithms from Zwift and 3rd party site Zwift Power (where results are tabulated) could more easily flag these efforts as they fail to appear like a human’s actual performance. Note that in EPO mode it would have the variability because it’s using your actual power as baseline.

So, in a nutshell, the hack presented as-is, is exclusively for use at home (where you control the environment), with a PC/Mac, and in a case where there might not be a ton of focus on your actual output numbers. Still, it’s a starter point. In the next section we’ll talk about where this goes from here.

However, if you’d like to watch the full presentation on YouTube, check it out below (it’s an audience recording, the official recordings aren’t up yet). Also, here’s their site with all the documentation details/etc

The Next Gen Hack:

Here’s the thing: In the grand scheme of hacks, this hack isn’t super impressive. Not yet. In fact, Keith Wakeham already outlined a more sophisticated hack in terms of resiliency to getting caught back in March. I asked Brad and Mike if they had seen it, and surprisingly they hadn’t. They noted they had started their research back in February before Keith’s video, and given that they came up with a fairly different solution – that seems pretty likely they didn’t know of Keith’s video. Speaking of which, here’s that video from Keith:

While both hacks seem similar – they’re actually fairly different. Specifically the following:

Keith’s hack is ANT+ based and doesn’t re-transmit anything from an existing trainer, but just acts as the sole source of data. Importantly though, it includes slight bits of variation to power/cadence/HR to more accurately emulate a human. He notes he has tried it in races and never triggered any sorts of cheating audits in place today.

 

 

Brad’s hack is ANT+ based but re-transmits existing hardware (trainers/power meters/etc…). It doesn’t include any human variability, but does more accurately replicate the source/original trainer – making it harder for Zwift to detect hardware anomalies.

But what lies ahead is more interesting, and useful. Brad says their next step is to convert over to Bluetooth Smart instead, which means that the entire setup is portable and easily moved to actual Zwift racing venues (such as what we saw here). Specifically, they could create a small bit of hardware that sits in a jersey, or even in a bag somewhere nearby. It wouldn’t matter – it’s all wireless.

Such a device would first connect to the trainer and take over the KICKR’s (or any other trainer’s) Bluetooth Smart connection. Because all of the trainers on the market today only transmit a single Bluetooth Smart connection, that takes away that connection and the trainer would disappear from Zwift (there are some nuances to this, but mostly trivial). But then a split second later the special device would rebroadcast the altered signal to Zwift. Because dropouts and such are totally normal in indoor trainer setups (especially if it were done pre-race), nobody would blink an eye.

At that point there’s a slew of ways the device could function. It could simply operate in the Digital EPO mode and boost your power slightly, just enough to give you an advantage. It would do so entirely silently in the background and nobody would ever know. Or, it could load a previous activity file (perhaps culled from Strava) and replicate that effort with slight tweaks to the power numbers. Seriously, the world is one’s oyster here.

So how does Zwift (or any other indoor trainer platform) protect against this? Well, they have to start taking it seriously.

Sure, Zwift has released their long policy document on cheating – but none of that would stop anything here. Not a single bit of it. Much of this isn’t entirely Zwift’s fault. For example, there’s no end to end encryption or authentication that occurs today from trainers to Zwift, atop via ANT+ or Bluetooth Smart. Just like there isn’t any atop power meters either.

If trainer companies were to implement some form of authorization or encryption (which both ANT+ & Bluetooth Smart already support), that would go a long way to stopping this particular man in the middle type cheat. Of course, that has a higher level of complexity for both trainer companies and app companies. And for the most part there isn’t actually great technical cooperation between Zwift and trainer companies, despite what these sides might say in public. While Zwift holds an annual conference for trainer companies each year at Eurobike, it doesn’t ever extend to technical working groups akin to what we see at the ANT+ Symposium (which Zwift doesn’t attend). Instead it’s more focused on larger business elements of the industry. And there’s nothing wrong with that either – it’s just there’s a gap to fill still.

Going forward:

In a lot of ways, it really comes down to whether or not Zwift believes esports and racing are the future of their platform. Everything they’ve said in the past 6-12 months says they deeply do believe that to be the case. They’ve spent more money than ever before on these efforts, with professional racing being the most heavily promoted aspect of Zwift in 2019, including UCI sanctioned events. They’ve also noted their aspirations to make this an Olympic sport by 2028.

Yes, it’s true that none of these are direct hacks against Zwift as a platform. But that’s sorta akin to saying “We didn’t have encryption on our banking website, so it’s not our fault someone stole your money”. That kind of security mindset stopped in the early 2000’s. Security extends beyond your direct premises, including the API’s that interact with it. In this case, Zwift’s API’s are effectively ANT+ and Bluetooth Smart, and the trainer and power meter companies on the other end. All of which easily fit into a single conference room at your local Holiday Inn (not even a ballroom required, mind you).

And of course, this isn’t just a Zwift issue – it’s an indoor trainer issue. At some point, other legit competitors to Zwift racing will come along (for real, they will, it’s only a matter of time). And they too will face these same underlying issues and attraction for cheating. While each year Zwift holds their Zwift Summit at Eurobike, a night later the rest of the indoor trainer industry (including every trainer manufacturer, many power meter companies, and major trainer apps) have historically also had a gathering, often to discuss more specific technical issues. Yet a few weeks later – everything from both events is mostly forgotten and nothing tangible happens till the next year.

Ultimately, the indoor trainer industry has to take itself seriously as an entity if it wants to be seen seriously by the likes of UCI and the Olympics. Otherwise, the result won’t likely be much different than other aspects of cycling: Full of cheating.

With that – thanks for racing…I mean, reading.

DC Rainmaker:

View Comments (58)

  • This is an area that CVRcade is well ahead of the game with its physical equalization. Even if you're using a miscalibrated trainer or cheat you're overstated power becomes your normal power curve. Power curves are then adjusted so that people with different fitness levels can race against each other on a level playing field. The only downside is it takes a number of events for your power curve to settle down to something accurate.

    CVRcade has come a long way since Ray's critical and justly so review earlier in the year.

    • I admittedly don’t know the ins and outs of how it works, but wouldn’t people simply not use any such cheating technology and/or sandbag before a big race to lower the power curve?

      From the CVRcade website:

      “By accessing this special feature, anyone can compete with equal strength against family members, friends, co-workers and even top cyclists, and can WIN by combining the Physical Equalization™ and the use of strategies built into the game — not simply **athletic power and endurance”**

      I mean, the whole point for most of us is being able to compete based specifically on athletic power and endurance. I don’t want to compete against ‘family members, coworkers and friends’, I want to compete against other athletes of the same general level.

      The idea of boosting my performance or penalising Geraint Thomas’ performance so I can beat him in a race just sounds stupid.

  • On an uncalibrated Kickr vs my Neo at a shop ride I managed 40 more watts average over 7 minutes than i would normally do over my Neo... that moves me from a high B/ low A to the top... Zwift racing is just good for a winter speed workout.. take it with a grain of salt.. If I want a real workout on my trainer I'll load up tacx TDA or BigringVR and go do a real video ride with the actual gradients... Zwift's Alpe is so far out to lunch it hurts...

  • I just ran across this and had a chuckle about how one persons "cheat" is another persons pain point or use case. I develop iOS rowing apps, mostly for fun. Two of them, PainSled and RowedBiker, have a "row like a bike" mode where they receive data from Concept2 rowers over BLE or USB and then simulate a cycling power meter so you can "row" with Zwift etc. If I had known that could get me on the stage at defcon, I guess I would have mentioned it!

    As you and others have pointed out, cheating Zwift is not really rocket science at this point, but, to continue the cheating theme, RowedBiker also has an FTP boost function because rowing is less efficient than cycling, I usually run it at 30%. But it also has modes all the way up to 3x because one of my users wanted to Zwift with his young child who couldn't keep up at real wattage. Another cheat that ends up being a legitimate use case, if you arn't racing for real that is.

    RowedBiker also has a cruise control function you can use if you need to grab a drink or want to message your workout group without falling off the back. You know, us rowers don't get to use our hands normally. So that's another cheat/hack, I guess. I DO let you build up and drain off a wattage reserve if you want to pause rowing without "cheating". I'm not sure that anyone but me uses it though!

    Finally, the icing on the cake is another app, Travel Watts, that I threw together so I could do Zwifting on crappy hotel equipment while travelling. It's totally on the honor system. You tell it what wattage you are doing on that Life Cycle or whatever and it tells Zwift. But I won't stop you from leaving it on at 1000w for 3 days so you can get that Tron bike and impress your boy/girlfriend or whatever.

    Another poster mentioned the rowing CRASH-B sprints and how they have to have everyone in a big room to keep it fair. They also use factory-selected ergs for that race and sell them off afterwards, I believe. I do think there are technical ways to enable remote racing competitors, but I'm one of the Zwift users who is just happy to have found a way to stay in shape with a bunch of fun, likeminded folks. Thanks for the site!

    James T.

  • Seems a bit silly that they would inject code when you can just use the ant+ dev tools to supply false sensor data to Zwift. You simply need two ant sticks, one to transmit and one to receive.

  • This is why I have never taken Zwift racing as anything more than an amusing, slightly healthier alternative to playing something like Wii Golf. The idea of dong professional Zwift racing, much less making it an Olympic sport, seems absurd to me. It's not like there's a shortage of real bike racing at the Olympics.

  • Disclaimer - I'm in my mid 45s, a good triathlete cyclist and vaguely competent road cyclist. I do a lot of training for real world events on zwift using my neo. My wife also has a (second) neo and a zwift subscription and she just does the structured workouts/programs as a replacement for the gym/spin classes.

    So for me my big worry here is not that online racing isn't legit (as the philosopher Ms Swift so eloquently said, cheaters gonna cheat), but that any measures to stop it are going to reduce the stability and ease to which I can do my thing. Sure, I know that there may be cheaters in the zwift races I do, but I don't care particularly. I aren't ever going to win, nor am I ever going to come last. But what does really really wind me up is when I get a data drop 45mins into a 50min race, or after a 2.5hr ride then the data doesn't appear in training peaks / strava.

    And I've got the grey hairs to remember copy protection on my floppy disk / Compact disk games in the 80s/90s. Or rather the methods I needed to use to play those pirated games, and the frustration when I'd misplaced the 16digit serial number from the legit game I had actually bought. Same with DVDs/Bluray where after moving from one country to another watching things I'd actually paid for became hard, when the rest of the world with access to teenagers was watching it all for free on the tinterweb.

    So in summary, I reckon that attempts to stop it are futile, and will actually just cause more headaches for the 99.9% that neither care, nor would cheat. The 0.001% that will cheat, will cheat anyway. And so this is all about the 0.09% that won't cheat but are racing the cheats for $ prizes. And so I support myself being drug tested when I'm racing tris or road races as an age grouper (as a 0.9%) but I would say it's madness to drug test every commuter cyclist or family out for a Sunday ride on the canal path. So in sorting out the 'hacking' for e-sports racing don't make it so that the trainers and software is more expensive, less stable, more 'closed system' than it is now. The only reason there is the potential for virtual-sports cycling is because of the mass participation of virtual-cycling.

  • I'm surprised this is a "revelation" somehow since there were existing and still exist tools that can emulate a power meter explicitly for Zwift usage. With all the randomization needed to perform similarly to human.

    Not so long ago it was common to use one just to get quickly the "tron" bike on Zwift where you actually had to ride lots of climb m and lot of Kilometers.

    It's going to be *very* hard to protect Zwift since it's hardware-based external data that is your skill sent to the zwift platform. I'm sure it can be done, but it's a hard problem.

    OTOH if Zwift becomes a serious e-sport platform i image most of the more important events will be held in common locations, on certified hardware where you actually have to perform!

  • It will always be so easy to cheat that this Zwift racing thing is a non sense. They should focus on the efficient training side and on the fun/social side.
    Leave races for racers, and on the road...

  • Zwift just does not seem to have the technical expertise, even with their expensive monthly fee. They cannot even different levels of graphics settings. Someone like Codemasters need to retool their existing assets into a Zwift competitior. Even at $5 a month, they will make a killing.

    Side note: someone also needs to come out with GTA mod for trainers.

  • Just another shout that there could be some use here. My Bushido is up to 20% inaccurate, I use a calibration dope to bring it into line - it would be great calibrate against my Vector digitally. I am also interested in being able to handicap riders to similar abilities for virtual group rides....(like sitting up - or carrying a rucksack)

1 2 3

By continuing to browse the site you are agreeing to the use of cookies