In the computer security world there are a pile of conferences where security folks get together and present various sessions. I use ‘security folks’ as a loose term that can broadly cover everybody from IT professionals working for a non-profit like Red Cross, to government security peoples, to folks with less altruistic goals. These conferences have been around a while, and are generally considered good for the IT security community – assuming things like security bug disclosures are done properly (the concept of giving a company reasonable time to fix the bug before you talk about it).
One of the most well-known conferences from a lore standpoint is Def Con, but there are also many other huge ones such as BlackHat, SANS, and RSA, and other vendor-specific ones like BlueHat (run by Microsoft for Microsoft technologies) or government-specific ones. Again, in general the goal of these summits is to learn about security and improve security practices.
This past Sunday at Def Con (considered one of the more rambunctious events on the circuit) a presentation was given around Zwift and ‘hacking’ it – titled “Cheating in eSports: How to Cheat at Virtual Cycling Using USB Hacks”. Now one has to understand that while in the ‘mainstream’ the term ‘hacking’ is usually akin to ‘breaking’, in the computer world, the term ‘hacking’ is often a bit more nebulous. Sometimes used interchangeably with ‘tweaking’ or ‘optimizing’, and sometimes used in the less ideal variant such as ‘credit cards were hacked’. So one has to take any usage of that term with a bit of sanity check to see what’s going on.
In this case, the presentation was given by Brad Dixon (with support from Mike Zusman), security researchers with the consulting firm Carve Systems. This company has historically done penetration testing for other organizations (pen testing is trying to see if you can break into a system), but has switched in recent years to a more holistic security consulting approach where they do pen testing and then assist companies in making the fixes. More or less this is run of the mill security company stuff, nothing too crazy.
In this case though, two of the employees there are also avid cyclists and wanted to see where what they could do from a Zwift standpoint security-wise.
(Preemptive note: There are far easier ways to cheat at Zwift such as entering your weight or height incorrectly, or mis-calibrating your trainer. I’ve outlined them all in my past piece on the topic here in January. This post is focused on this specific security presentation.)
(Secondary note: There are now financial and career reasons to cheat in Zwift – including UCI sanctioned events, anyone telling themselves otherwise is thinking it’s still 2018. So we’re going to use the very real baseline that people are and will continue to cheat in Zwift where real stuff is on the line beyond bragging rights.)
(Tertiary note: This entire piece could be re-written without Zwift, and with ‘Bkool Racing’ instead. It applies equally. Minus the fact that nobody is actually using any of those platforms for major competitions. Thus when you’re the elephant in the room, it’s appropriate to still call you an elephant.)
(Quaternary note: What I discuss here is well-known in the trainer industry, this won’t surprise anyone. What it might do this is spur on developments to actually start securing this stuff, instead of just continuing to stick one’s head in the sand and think it won’t matter. If people are presenting Zwift hacks at Def Con, it’s time to actually address these issues. Nobody is better positioned than Zwift to do that.)
The ‘Hack’:
I had a chance last week before the conference to do a video conference with Brad and Mike and understand their presentation a bit more. We walked through the technology and where their ‘hack’ fit into things. I’m using the term ‘hack’ here in more of the ‘tweaking’ variant, than the ‘breaking’ side of things.
And in fact, one of the things they noted right at the top of the call was that when they started their research into Zwift they made the decision early on to “not screw people”, in Brad’s words. Specifically, that meant that they never competed in a Zwift race/event with the hack, nor did they save any rides. For clarity, this doesn’t minimize the effectiveness of it, it just simply means it didn’t meaningfully impact others.
Still, the obvious end-goal here is cheating in races. After all, that’s sorta the whole point. And in his presentation at Def Con, he points out what may be obvious to us, but might not be obvious to security researches in the room: “Cyclists are the best cheaters… Tom Brady should learn from these people”, and he goes on to discuss some of the history of cheating in cycling, be it doping or otherwise.
(Side note: It’s actually a fascinating part of the presentation with some crazy Tour de France cheats that have occurred over the last 100 years, it’s in the first few minutes in the YouTube video a bit further below).
And he’s right. It’s hard to come up with another sport where cheating (except perhaps bodybuilding) is as widespread and deeply engrained in over a century of athletics, at all event levels from major and minor competitions, and from pro to amateur. All with unrelenting mainstream media attention. Still, that’s beside the point.
What was their hack? They inserted code between the trainer/power meter and Zwift via USB, effectively boosting the power values that Zwift received. Except, it was slightly more complex than that (and, also slightly less complex too). More specifically, they created a small device that you plug your normal ANT+ USB stick into (it’s off the shelf hardware with custom code atop it), which intercepts your ANT+ values and manipulates them with a tweaked value to Zwift on your computer:
(Note: All the slides/diagrams you see here are from their presentation)
So in effect, let’s take the most popular trainer out there and what they used for their demo: The Wahoo KICKR.
What they’ve done is inserted themselves in between Zwift and the KICKR, modifying those numbers before Zwift sees it. So let’s say you’re pedaling at 200w, what their little device does is tell Zwift that you’re really doing 220w, or 250w, or anything you want.
They’ve configured an Xbox controller to modify power, cadence, and heart rate. But they’ve also got simple scripts that do other actions. They showed two core ways to control their platform:
EPO Mode: Boosts your performance with a set multiplier
Slacker Mode: Automatically rides for you, and then fakes your HR and cadence based on terrain (so you can sit on the couch)
However, one of the biggest keys here is that the hack correctly replicates the source trainer from an appearances standpoint. Meaning, it looks exactly like a real legit Wahoo KICKR to Zwift. More specifically, this means that they re-passed all of the nuanced parameters of a KICKR, tweaking only the power metrics. That’s a core difference to using something as basic as the industry tool ANT+ simulator (used for testing by companies), which doesn’t really emulate something like a Wahoo KICKR or Tacx NEO – it just emulates a generic power device.
With their hack as is today, there are some requirements though:
A) Only works with Mac/PC (because USB), and not with Android/Apple TV/iOS (totally wireless, no ANT+)
B) You’ve gotta own the hardware, meaning it’s really only good for at-home events
C) Only applicable over ANT+
Now that last one might imply this is an ANT+ vulnerability. But it’s really more simplistic and could be achieved on both ANT+ or Bluetooth Smart. They just started with ANT+ because it was an easier starting point given how much of the code was already out there and developed. But technically speaking both protocols would operate the same way.
But there are also some challenges with the hack as well. As they noted, they didn’t use it in racing – because they didn’t want to mess with other people’s results. That’s OK though, because we don’t need race performances to know where the hack breaks down a bit. It doesn’t do any of the following:
A) Doesn’t emulate a specific historical/past ride on the course
B) Doesn’t properly emulate the nuances of a human-driven power meter/trainer
C) Doesn’t correctly emulate the nuances of a human’s heart rate
When I say ‘nuances’, I mean that a human pedaling a power meter is constantly shifting their power by a few watts. No matter how perfect you think you are, you’re not actually pedaling exactly 250w every second. Instead, it’s roughly 248w, 253w, 247w, 250w, 255w, etc… (and that’d be a crazy precise person too). As such, algorithms from Zwift and 3rd party site Zwift Power (where results are tabulated) could more easily flag these efforts as they fail to appear like a human’s actual performance. Note that in EPO mode it would have the variability because it’s using your actual power as baseline.
So, in a nutshell, the hack presented as-is, is exclusively for use at home (where you control the environment), with a PC/Mac, and in a case where there might not be a ton of focus on your actual output numbers. Still, it’s a starter point. In the next section we’ll talk about where this goes from here.
However, if you’d like to watch the full presentation on YouTube, check it out below (it’s an audience recording, the official recordings aren’t up yet). Also, here’s their site with all the documentation details/etc…
The Next Gen Hack:
Here’s the thing: In the grand scheme of hacks, this hack isn’t super impressive. Not yet. In fact, Keith Wakeham already outlined a more sophisticated hack in terms of resiliency to getting caught back in March. I asked Brad and Mike if they had seen it, and surprisingly they hadn’t. They noted they had started their research back in February before Keith’s video, and given that they came up with a fairly different solution – that seems pretty likely they didn’t know of Keith’s video. Speaking of which, here’s that video from Keith:
While both hacks seem similar – they’re actually fairly different. Specifically the following:
Keith’s hack is ANT+ based and doesn’t re-transmit anything from an existing trainer, but just acts as the sole source of data. Importantly though, it includes slight bits of variation to power/cadence/HR to more accurately emulate a human. He notes he has tried it in races and never triggered any sorts of cheating audits in place today.
Brad’s hack is ANT+ based but re-transmits existing hardware (trainers/power meters/etc…). It doesn’t include any human variability, but does more accurately replicate the source/original trainer – making it harder for Zwift to detect hardware anomalies.
But what lies ahead is more interesting, and useful. Brad says their next step is to convert over to Bluetooth Smart instead, which means that the entire setup is portable and easily moved to actual Zwift racing venues (such as what we saw here). Specifically, they could create a small bit of hardware that sits in a jersey, or even in a bag somewhere nearby. It wouldn’t matter – it’s all wireless.
Such a device would first connect to the trainer and take over the KICKR’s (or any other trainer’s) Bluetooth Smart connection. Because all of the trainers on the market today only transmit a single Bluetooth Smart connection, that takes away that connection and the trainer would disappear from Zwift (there are some nuances to this, but mostly trivial). But then a split second later the special device would rebroadcast the altered signal to Zwift. Because dropouts and such are totally normal in indoor trainer setups (especially if it were done pre-race), nobody would blink an eye.
At that point there’s a slew of ways the device could function. It could simply operate in the Digital EPO mode and boost your power slightly, just enough to give you an advantage. It would do so entirely silently in the background and nobody would ever know. Or, it could load a previous activity file (perhaps culled from Strava) and replicate that effort with slight tweaks to the power numbers. Seriously, the world is one’s oyster here.
So how does Zwift (or any other indoor trainer platform) protect against this? Well, they have to start taking it seriously.
Sure, Zwift has released their long policy document on cheating – but none of that would stop anything here. Not a single bit of it. Much of this isn’t entirely Zwift’s fault. For example, there’s no end to end encryption or authentication that occurs today from trainers to Zwift, atop via ANT+ or Bluetooth Smart. Just like there isn’t any atop power meters either.
If trainer companies were to implement some form of authorization or encryption (which both ANT+ & Bluetooth Smart already support), that would go a long way to stopping this particular man in the middle type cheat. Of course, that has a higher level of complexity for both trainer companies and app companies. And for the most part there isn’t actually great technical cooperation between Zwift and trainer companies, despite what these sides might say in public. While Zwift holds an annual conference for trainer companies each year at Eurobike, it doesn’t ever extend to technical working groups akin to what we see at the ANT+ Symposium (which Zwift doesn’t attend). Instead it’s more focused on larger business elements of the industry. And there’s nothing wrong with that either – it’s just there’s a gap to fill still.
Going forward:
In a lot of ways, it really comes down to whether or not Zwift believes esports and racing are the future of their platform. Everything they’ve said in the past 6-12 months says they deeply do believe that to be the case. They’ve spent more money than ever before on these efforts, with professional racing being the most heavily promoted aspect of Zwift in 2019, including UCI sanctioned events. They’ve also noted their aspirations to make this an Olympic sport by 2028.
Yes, it’s true that none of these are direct hacks against Zwift as a platform. But that’s sorta akin to saying “We didn’t have encryption on our banking website, so it’s not our fault someone stole your money”. That kind of security mindset stopped in the early 2000’s. Security extends beyond your direct premises, including the API’s that interact with it. In this case, Zwift’s API’s are effectively ANT+ and Bluetooth Smart, and the trainer and power meter companies on the other end. All of which easily fit into a single conference room at your local Holiday Inn (not even a ballroom required, mind you).
And of course, this isn’t just a Zwift issue – it’s an indoor trainer issue. At some point, other legit competitors to Zwift racing will come along (for real, they will, it’s only a matter of time). And they too will face these same underlying issues and attraction for cheating. While each year Zwift holds their Zwift Summit at Eurobike, a night later the rest of the indoor trainer industry (including every trainer manufacturer, many power meter companies, and major trainer apps) have historically also had a gathering, often to discuss more specific technical issues. Yet a few weeks later – everything from both events is mostly forgotten and nothing tangible happens till the next year.
Ultimately, the indoor trainer industry has to take itself seriously as an entity if it wants to be seen seriously by the likes of UCI and the Olympics. Otherwise, the result won’t likely be much different than other aspects of cycling: Full of cheating.
With that – thanks for racing…I mean, reading.
FOUND THIS POST USEFUL? SUPPORT THE SITE!
Hopefully, you found this post useful. The website is really a labor of love, so please consider becoming a DC RAINMAKER Supporter. This gets you an ad-free experience, and access to our (mostly) bi-monthly behind-the-scenes video series of “Shed Talkin’”.
Support DCRainMaker - Shop on Amazon
Otherwise, perhaps consider using the below link if shopping on Amazon. As an Amazon Associate, I earn from qualifying purchases. It doesn’t cost you anything extra, but your purchases help support this website a lot. It could simply be buying toilet paper, or this pizza oven we use and love.
