Today Zwift has non-announced its latest sanction, against Philipp Diegner of Canyon Esports, for falsifying data as part of their Zwift Racing League pro race series. This would make for the fifth individual in the last 6 months, but most notably – the first male to be publicly suspended under the more robust anti-cheating system geared at pro athletes. All four other cases have involved women, including two just last month. The only other previous male to receive a public racing suspension from Zwift was Cam Jeffers a few years ago, though in an era well before the more advanced and heavily enforced means of capturing cheaters today – and in his case for entirely different reasons.
Ultimately though, the athlete in question had 11 pro race results nullified across two seasons, and received a 6-month suspension of racing on Zwift until July 24th, 2021. This is the first time Zwift has retroactively gone through and looked at past event data for transgressions, and then retroactively nullified those results. As such, Zwift is showing a bit of their hand here in the forward direction of what they plan to do in this arena.
The Case:
The rider’s troubles initially stemmed from a race on January 25th, 2021, specifically the pro-focused Zwift Racing League (Season 2), Men’s Race #3. In that race, the rider finished in 39th position, essentially last place. However, like every other individual that’s been ‘charged’ with a crime, it wasn’t the race itself that got the rider in trouble, but rather, the coverup.
For those that are unfamiliar, pro level races in Zwift require what’s known as dual recording. That means that you need to have Zwift paired to your smart trainer and let that file record like normal. But you *also* need to have your regular bike computer/watch record from your power meter. You thus must indeed own a power meter and a watch/bike computer, which of course, at this level isn’t really an issue. But, what is an issue is if you accidentally pair to the wrong source (such as pairing to your trainer), or, if the battery dies, or you forget to press start, or you delete it, or…any other common day to day issues. If you dual-record enough, eventually things go wrong (seriously, almost nobody out there dual/triple/septuple-records more data than I do…and eventually human or mechanical error happens).
Ostensibly the purpose of this second recording of a second source power device is to ‘prove’ the accuracy of the first device. Or at least, provide a reasonable backstop for the data. If one device says your sprint was 1,300w, and the other says it was 800w, then that demonstrates concern for accuracy – or, perhaps something more sinister. But the true purpose of this secondary recording is it makes falsifying the data immensely more difficult (real-time or post-race). That’s because not only does someone have to falsify one source with one understood signature, but then has to falsify another source with another signature. And then has to ensure the way those two signatures shadow each other during the race actually matches. For example, most crank-based power meters respond far faster than most trainers on sprints, the rate of change is different. But not just sprints, but every little surge in a race – of which there could be a hundred surges of varying intensities, both the ups and the downs. Anyway, we’ll talk about that more another time.
The point here is more simplistic. Every single case to date has revolved around something going wrong with that secondary source recording, and then the athlete trying to tweak that second file to make it appear like a secondary power meter. In most cases, it was simply paired to the trainer instead, or, not started at all.
And in this case, Zwift found that the secondary data source was precisely 2% higher than the original power value from the trainer:
Of course, when someone says “Oh, my power reads 2% higher than my trainer”, we’d all respond with “Sure, that makes perfect sense – it’s higher up in the power meter chain, and thus due to drivetrain losses will be a higher value”. Except what we’re really saying is “Yeah, *about* 2% higher”. In reality, the exact amount will vary from second to second. One second it’ll be 1.5% higher, the next 3% higher, and then if you sprint it might be a second or two ahead, and then be 20% higher for one comparative second, before the next second being back to 1.8% higher as the update/recording/sensor rates all vary slightly from second to second.
Thus, when you blanket 2% increase the values in a file, it’s like using white-out on printed paper text, then using a Sharpie to re-write a few words, followed by trying to pass it off as unchanged. It’s that blindingly obvious.
But the funny thing about computer data is that it basically lasts forever. And over the course of the last 12 months, Zwift has significantly increased their processes for how they catch this sort of thing. Thus, in the report they noted:
About now you’re asking: What the heck is the Three-Sisters calibration test?
In short, it’s a calibration test that pro riders have to do in order to validate their performances. You can read about the entire thing here in the Zwift Ruleset, page 11-12. Or I guess, I could just paste a snippet of it below. Basically, you’re riding a specific Zwift route (called The Three Sisters), which includes three climbs – a short/medium/long one, followed by a sprint. However, you’re riding those specific four sections of the course at maximum all-out intensity. And you’re doing this with the dual-recording system outlined above…oh, and you’re recording a video the entire time of yourself.
The goal of this is to basically create a baseline of your human performance. But more cleverly, it’s creating a baseline of your equipment, and establishing a signature of that. And proving that you rode it. And sure enough, the rider actually falsified the data in that original baseline test:
So in total, 5 races plus the calibration test were falsified. It’s unclear why he selected to fabricate the secondary power source on those five races, but not on the other 6 that he competed in. Nonetheless, all of the racer’s results for both seasons were disqualified, specifically the following races:
Season 1: 8 (Results: 64th, 2nd, 72nd, 27th, 14th, 30th, 71st, 81st)
Season 2: 3 (Results: 81st, 89th, 39th)
These are already reflected within the race results on Zwift’s site.
Upon being presented the information, the athlete initially denied the data tampering, but then changed their mind and admitted wrongdoing:
As a result, the athlete has been banned from Zwift races until July 24th. The athlete still can utilize Zwift for non-racing, as well as race outside of Zwift – as this ban is not in connection with the UCI as the UCI is not overseeing this event.
Additionally, as was apparently announced 14 days ago and nobody noticed on Canyon Esports’ team press site, the team has terminated their relationship with the athlete:
“As of 9 am 8th February 2021, rider Philipp Diegner will no longer be part of the Canyon Esports team. This decision has been made during an investigation by ZADA following Race No. 3 of ZRL Premier League. Evidence was brought to light that was previously unknown to Canyon Esports and it’s management staff. As per the team’s internal protocols, a decision was made to terminate the rider’s contract and part ways. Philipp had previously been suspended from race duties while the team and ZADA’s investigation was ongoing. At the time of publishing this notice, ZADA and Zwift have made no further announcement other than the disqualification from Race 3.
Canyon Esports had been investigating in good faith whether the reported irregularities had occurred involuntarily during the exporting process. However, before we could conclude that investigation we were presented with additional evidence which undermined the necessity to continue. Following further conversations with the rider and ZADA, the team was left with a hard but simple decision to end the relationship.”
Rhys Howell, the Team Manager also added in the statement:
“We are an incredibly close-knit team, so losing a rider is like losing a limb. Personally, I can only describe my feelings as heartbroken. However, I did not hesitate for one second to make the necessary decision to terminate our agreement with the rider in question. Our team is more than any single rider alone and we believe firmly in transparency and a clean sport. There can be no deviations from that belief. Our sport relies on trust and a team like ours is founded upon it. We will now look at how we can avoid such situations in the future and I have reiterated to all our senior and development riders that they can and should always come to me first if they are struggling. I hope this episode will be but a lone footnote in the exciting story of our team.”
At this time, the Canyon Esports team doesn’t show a replacement for their 6th rider, having removed the image and name from their roster page.
As an interesting side note, the rider posted considerable dual-recording analysis sets of data to ZwiftPower here. What’s notable about this is that these dual-recording sets all showed a Wahoo KICKR 18 on them, which is far more easily ‘tweaked’ to create an incorrect (inflated) power value than a newer KICKR V5/2020, as the newer unit automatically and continuously recalibrates back to known factory calibration values within a minute or two of you trying to do a re-calibration. And while we don’t know if the athlete did tweak the calibration value on the KICKR 18, we do know the output values on their baseline Zwift tests were tweaked, but I don’t know why you’d do that except to hide other values.
In fact, even more fascinatingly, it appears the athlete actually posted the data sets in question there, and you can see quite clearly that the 2% increase in wattage between these two labeled ‘Infocrank’ data sets – which appear identical.
In any case, this does continue though to bring up the never-ending question on when on earth Zwift will simply bake dual-recording into the game itself. After all, they’re already recording quite a bit of this in log files. And the same goes for simply warning athletes when they’re using an incorrect power source (per specific race rules), such as using a power meter instead of a trainer – or if they’re missing their heart rate strap. All of these cases would never have happened if so.
Again – there’s zero reason why any athlete, pro or amateur weeknight mom or dad racer, should have to deal with uploading multiple files manually to ZwiftPower or elsewhere.
Having asked Zwift this repeatedly in the past, they’ve said it was a development priority type thing. And given this only impacted a small number of pros, it wasn’t high on the list. Which, I fundamentally disagree with. After all, Zwift is spending millions of dollars each month on pro-level racing (inclusive of production costs, staff costs, etc…) – why can’t they spend a few more dev hours to record the secondary power meter source? This isn’t hard. It’s not remotely hard, and it’d benefit far more than just the pros. It’d benefit all the non-pros that also use ZwiftPower to upload dual-recording files to prove their data state.
And it’d benefit all those who want to see the bar for cheating in Zwift considerably raised.
Going Forward:
Now, as interesting as it may be to out people that cheat – my interest here actually lies elsewhere: The technological aspects of it. Specifically, the technological aspects of catching cheaters, and the vast complexities required there (both trying to cheat, and then catching the cheaters). In fact, it’s an area I’m going to talk a bit about in some upcoming post or two, covering both what Zwift is doing, and how that might translate beyond that in a UCI world.
Part of this stemmed from a recent Twitter observation I made a few weeks back, that all of the people Zwift had caught via this newer system were female. I noted that statistically that was an exceedingly slim likelihood. The resultant of that tweet was swiftly a lot of e-mails from Zwift, but more notably a conversation with their team of data scientists that actually look at these cases, and the processes that they go through that ultimately culminate in sanctions like today’s.
However, at the other end, I’ve also talked with some of those representing athletes that have been accused of cheating by Zwift. And whether or not they’re innocent or guilty, I think there are some elements of the picture that Zwift portrays that are fuzzy at best, and conflicting at worst. Aspects that don’t seem to align as clearly with well established protocols of the UCI or others for what rights an athlete has in this process if they’re being ‘charged’ with a crime. Because ultimately, like today, that athlete’s name (and thus career) is being impacted. As is their ability to monetize that career.
So within that, we’ll dive into why there are 160+ case numbers yet only 5 ‘convictions’, as well as how Zwift separates what’s ‘willful’ cheating from ‘bad sensor data’, both in the data analysis side but also the human accusation follow-up side. And then, to the extent that’s viable, I’m going to show just how difficult it is to get away with cheating here at this level – especially using the methods that all five of these individuals have used…which frankly, are the least smart ways to go about cheating on Zwift. The goal is not to show all the things Zwift does to capture cheaters, but rather, to examine whether their methods are both enough, and fair (procedurally and technically) to those being accused.
And finally – why is it that all of us who Zwift with any frequency know there are cheaters everywhere, but none of them are being caught? Either willfully or just due to bad trainers/data. When and how will Zwift start to address more than just the official pro series events? Yup, I’ll cover that too.
Until then, thanks for reading!
FOUND THIS POST USEFUL? SUPPORT THE SITE!
Hopefully, you found this post useful. The website is really a labor of love, so please consider becoming a DC RAINMAKER Supporter. This gets you an ad-free experience, and access to our (mostly) bi-monthly behind-the-scenes video series of “Shed Talkin’”.
Support DCRainMaker - Shop on Amazon
Otherwise, perhaps consider using the below link if shopping on Amazon. As an Amazon Associate, I earn from qualifying purchases. It doesn’t cost you anything extra, but your purchases help support this website a lot. It could simply be buying toilet paper, or this pizza oven we use and love.
Not the first male to be suspended for it but the first to be publicly named for it
Thanks!
According to which criteria are the names published or not published?
Seems arbitrary!
The current line in the sand is only pro series races would be published, where participants specifically went through the entry criteria outlined here: link to zwift.com
Whereas Joe or Jane Random cheating on a weeknight race isn’t to the level yet of outing.
I am quite familiar with this list and also with the rules.
Nevertheless, as the comment of “Zwift” says, not the first male but the first publicly announced, seems arbitrary.
Especially since such an announcement can damage a career, especially if it was a technical problem.
I suspect we’re also getting to the point here of them being more confident in their methods. Meaning, a year ago, they were less confident in ruining a career due to a technical mistake than now. We’re seeing the cadence of these cases pick-up, and in all of these cases the evidence was so damning.
After all, all of these cases are actually that the cover-up is worse than the crime. Meaning, setting aside the quirks of this most recent case, all the previous four women were for faking the secondary power source data due to race-day file issues. Not because they set out to cheat on race day. Crap happened (mostly human error), and they didn’t have the backup race file to send in. At that point alone, it’s just a mere DQ. No biggie. But then they went off and faked race data to try and cover it up (or, so says Zwift). That’s what got them in trouble.
This one is a twist on that same concept, except for the fake that it wasn’t just one race. But two seasons worth + the baseline calibration test, and as a result that calls into question what exactly one might be wanting to hide beyond that. But ultimately, this individual got in trouble for (and admitting to) faking the secondary files. They didn’t get in trouble for anything beyond that.
please contact me via mail for more
Yup, will do! Sifting through this mornings e-mail still. 🙂
Why not be able to pair your pm at login and have it silently recorded in the background? Or like TR does with power match. Isn’t that just the opposite, where trainer power would be in the background? Seems too easy not to be able to record (and broadcast as necessary) two power sources.
Yup, exactly. This isn’t hard, especially since some of this is already happening with Zwift’s logging files.
Zwifts still hasn’t managed L/R recording after reporting it in beta…. and then their is their broken physics engine… at 6’9″ 205lbs I’ve seen a literal kid pass me on the tempus fugit route doing 80w while I was doing 280w…
I have zero faith that they’ll ever be able to make their platform fair.
I completely agree about the physics (aerodynamics) for tall riders. I have the same kind of experience on tempus fugit or other flat TT courses, I am doing 400+ watts with about 4 w/kg and get matched or beaten by riders doing sub 250 watts with the same w/kg.
From what I have found Zwift is using a calculation for aerodynamics based on a linear “cda” estimation with weight and height input, this heavily favors smaller riders.
This makes it completely impossible to race at comparable level on Zwfit as you do in the real world for tall and big riders.
Won’t be fair until they also account for the real world elevation of a rider in the speed to power calculations. For example, my grotto of mild discomfort is at 5700’ elevation, and thus I give up roughly 7% of power production relative to riders at/near sea level.
This remains an absurd complaint. Your body presumably is acclimated to the higher elevation, and you push the power you push. Whereas if someone who doesn’t live up by you comes up to visit they will see a massive drop in their power. So what you are really saying is that you could generate more power at sea level, if you were there briefly so you didn’t lose the benefit of your training at altitude. So what? You live where you live. It’s a lot hotter in, say, Texas, than San Diego. Should they reduce Texans’ wattage to account for the fact if they are outside they’ll have their legs sapped by the heat? What about folks who live near the shore and have tons of wind to deal with all the time? Heck — my pain cave may be hotter than yours, and maybe I can’t cool it anymore. Should I get a wattage bump to account for the fact that if I had better A/C or more room for fans I would generate more power? The list of environmental conditions Zwift could in theory account for is long. But the reality is that we all throw down the watts we throw down when we toe the line.
Now you are getting into simulating the world. It’s like solving world hunger. There are too many variables that affect each individual differently, and what about compound effects? There is no way to simulate everything with any level of confidence or predictability. Much like caffeine has different effects on people, what about an athletes mental aspect, or their diet? Did they carb up properly? If so when, and what did they eat? are their metabolisms the same? etc… The only thing that can be done is to simulate key factors that put athletes on an even playing field with predictable outcomes. It may not simulate “real world” in terms of complexity and variability, but good enough to reliably predict the outcome of an event. Now I’m not saying that existing factors do not require any work to improve them. What I am saying is you cannot simulate the world.
One more thing to expand on my comment above. When something is simulated….somebody has to develop or come up with a rule set (“if this, then…etc.), and as I have noted above, there are way too many variables when you are talking about simulating the world. If you want to get into artificial intelligence and predictability for rule sets that don’t exist, then that is another subject. Not everything is as simple as it looks.
I agree with your ultimate stance of “whatcha gonna do?”, but it is not an absurd complaint.
“Your body presumably is acclimated to the higher elevation”
Acclimation diminishes the altitude effect. It does not erase it.
“So what you are really saying is that you could generate more power at sea level, if you were there briefly so you didn’t lose the benefit of your training at altitude”
He would be more powerful at sea level, not matter how long he stayed.
“Should they reduce Texans’ wattage to account for the fact if they are outside they’ll have their legs sapped by the heat?” Not sure what you mean.
“What about folks who live near the shore and have tons of wind to deal with all the time?”
What does this have to do with Zwift?
“Heck — my pain cave may be hotter than yours, and maybe I can’t cool it anymore.”
This is exactly the point. You can compensate for heat and humidity with an AC unit and a big fan…pretty mundane things. There will always be someone with a better Zwift environment, sure, but you can make yours good enough with a few bucks…unless you live at altitude.
“The list of environmental conditions Zwift could in theory account for is long.”
And altitude is by far the significant item on that list.
MJC IV — I’m not familiar enough with the science to question your contention that he would be a bit stronger forever at sea level. Training Peaks has a page on this that seems to suggest that acclimated he’s losing about 7% versus sea level, whereas someone not acclimated is giving up more like 12% So that suggests his real “loss” compared to living permanently at sea level is about 2%. To which I say ‘big deal,” particularly as I am not sure he wouldn’t lose some of that if he no longer had the benefit or working out at altitude. (The TP page doesn’t get into this.)
As for the rest of your comments, I mean that he is complaining that his personal environmental conditions aren’t taken into account. So I would like to know where he draws the line. Why not give people in Texas a bonus under the assumption that if they were in their real world they’d have to fight heat that someone in San Diego won’t. Ditto for people who routinely deal with a lot of wind. These things impact them in the real world, so why not account for them on Zwift? Wind doesn’t affect a pain cave, although it affects outdoor training and thus how strong you are indoors (although it may actually make you stronger). But living where it’s very hot will definitely impact the temp in your pain cave, and you may not be able to fully compensate. Nevertheless, the reason I flagged these in general is to sarcastically make the point that you can’t model everyone’s environment. So you end up with whatever wattage you generate.
@You Are High
The article you’re referring to suggests that an acclimated resident of 6000′ produces 93% of their sea-level power = 7% loss. I’m not sure how you’re arriving at 2% loss.
link to trainingpeaks.com
I don’t think Zwift should have altitude compensation, mostly because that just introduces “location cheating”, but I am sympathetic to Zwifters who live at altitude. The power losses are real, and they are significant.
How many of those zooming at 80 watts sit up ? They might present more into the wind than I. I have 4 inches of drop. I am somewhat aggressive and just watch folks smoke climbs and charge away at the flats.
Isn’t it possible they are on a group ride and tethered to someone riding faster, or on one of those group rides that go whacky? I see lots of KOM contenders which when you examine them are on very low wattage.
Could you not compensate for altitude by introducing more oxygen into your indoor riding environment?
It’s about $10 for a can of oxygen that only has about 2000 breaths in it, so it’s fairly expensive. Buying larger sizes might reduce the overall cost though.
NCAA and other organizations have adopted adjustments to times for track athletes to account for the affect of altitude discrepancy between times. Here’s a Google search to help get you started: ncaa track altitude conversion calculator
In the end, when you are comparing athlete A and B performance over the same range, but their localized difference causes a difference in performance, you can and should adjust to compensate. I other words, if both athletes with the same performance apart were to compete side by side after removing the localization difference, who would you place your bet on?
I live in Denver, CO about 5280′ elevation above sea level. I used to run here and could max out a mile in 7 minutes. I went on a work trip to San Diego for a week and ran a PR mile by 30 seconds without trying (I thought I was just running regular and a normal workout) but I also felt better at the end, like I could have run more. The effects accumulate and affect performance but also endurance, and I think they get compounded for the longer an event.
Nice reporting. I didn’t want to go too deeply into the ways to cheat but it’s hard to do in a way that’s undetectable, especially when Zwift has race files going back a year or so.
Hmm, historical data is an added wrinkle, but one could gradually raise their numbers over time to draw less suspicion. As for not going to deeply into ways to cheat, I’d be interested in hearing your and Ray’s thoughts on the detectability of a particularly simple method…
Gradually raising numbers would make it more difficult. But I think the bigger thing that dual-recording does is significantly raise the complexity bar to get right.
Every trainer/power meter/etc as quirky nuances that at Zwift’s scale they can start to tease out (even to the firmware level), not so much for black and white purposes, but to start identifying flags and scoring for whether or not a given data set could be considered more or less trustworthy.
The amount of data stored in .FIT files is incredible, and we haven’t even seen the tip of the iceberg, let alone what’s below the surface.
Zwift’s engineering department is a bit of a joke. They seem to relish spending resources on irrelevant nonsense while stubbornly neglecting core areas. That plus a steadfast refusal to do any testing whatsoever.
They seem to fancy themselves as some kind of e-sports powerhouse. At least they could allocate a few hours to built-in dual power recording, as you said.
Nik, you hit the nail on the head. Somehow I get the impression that Zwift are marketing specialists who dabble in software development. It is beyond ridiculous how fundamental things can remain broken/inoperable for months, even years, but those same programmers will work on tangential features that very few people actually care about. They have no concept of prioritizing bugs based on impact to userbase and instead chase down extremely trivial bugs that impacts a TINY fraction of their userbase.
Honestly, I wish Garmin, Nike, Apple…anyone…would compete in this sector or just buy Zwift outright and remove the incompetence.
Could the reluctance to adding the second source to zwift itself be related to the limitations on some devices running zwift? Apple TV is the one that comes to mind and it’s limited Bluetooth channels and no ant support.
Indeed, there is a limitation there, but that could be handled via the Zwift Companion app, as offloading is done today (today, it’s one or the other, not a blend), but there’s no reason the secondary/backup channel couldn’t be recorded on the companion app if need be.
all that can happen is that more cheaters get better. its not like if what zwift is doing was magic. at the end of the day you can, indeed, just fake some additional power and make it look like its coming from the power meter just fine. if you or i were cheating we’d have faked the data much better than this. in fact if i were i’d just fake it live, as if it came from the powermeter directly, its easier and safer. hell, im sure some of these pms can be reverse engineered to even just modify the firmware and sell the cheater, and that some people are already working on that .. if its not already done.
in fact, i assume many do cheat “properly” today and have for a long time, and that this one person was just dumber than average and got caught for it. which means the next equally dumb one won’t make the same mistake.
Yeah, but in the mean time, a carefully dosed stream of Zwift cheating scandal is working miracles for Zwift racing’s public awareness. I’m sure that I am far from the only one who only ever hears that high profile Zwift racing exists when there is a scandal, and I’ve been a Zwift user since Neo 1 days.
And then there are the normal “real world” ways of cheating PEDs, Hidden motor, etc. I think as Ray mentioned somewhere in an earlier reply, it is about raising the hurdle to cheat. The answer really is as easy as doing the double verification and tending to any new major issues that pop up over time.
Really pleased to see you giving this some focus Ray, it can only be a good thing for cleaner racing.
There are clearly many riders on the platform who’s performances in game are at odds with their real world performances/pedigree, yet thus far little seems to be done to investigate and weed them out. Here’s hoping that you shining a light on it motivates Zwift/ZADA to get their act together before the fledgling sport loses all credibility.
Hi Ray,
Great topic that you choose to talk/write about regularly and not an easy one to communicate on without at least some push back, criticism, trolling and more.
Thank you for your continued work in keeping sports technology honest AND interesting.
Have a great day!
Why not having a rpi with multiple ant key python script with libant/libusb that grab the two sources over two different USB ant sticks and rebroadcast the exact same frame with an increase of 2% if we are over 90% FTP, that would be completely undetectable even to the three sisters tests….
I think that was Shane that was saying, but we really need some sort of TLS encryption over those protocol, be it in the transport or in the encapsulation…
Can probably do a POC and put it in Github but that would help the cheater more than showcasing the exploit.
I’d argue trying to boil the ocean fails. You’re never going to catch every last variant. Never.
However, as each month/etc goes on, the level of difficulty increases to getting away with it. Scripted aspects like that are easy to pick out. Adding in slight variability makes it harder to detect (and likely, today, would go undetected). But then you add in things like firmware versioning, manuf ID’s, etc… All steadily increasing in difficulty to just script and be done.
But, that immediately falls apart at the next level – which is trainers that don’t allow calibration. For example, the Tacx NEO series or KICKR Bike, they don’t allow it. It’s physically not possible. Thus, it instantly kills the inflated power vector, no matter what you do with your power meter. And of course, Zwift racing for higher end races is absolutely moving towards using trainers of that technology.
Eventually you get to the point where yes, if someone wanted to engineer a completely separate firmware – then sure, the world is your oyster. But it’s the same ‘we can’t do anything unless we do everything’ approach. We can indeed do things to weed out low and mid-level cheaters, and that’s before we even apply some of the advanced data aspects that Robert Chung has talked about before.
Some excellent points… ultimately, it may have to come down to a matter of, for “paid” events (or events contributing to a payout season), only certain, “unalterable” trainers, with built-in external sensor recording (or a common secured app that records from both simultaneously) can/should be the only ones allowed.
This unfortunately eliminates many existing platform combinations, but just like any other sport, as you “move up the chain” things get stricter (you can’t just put whatever you want onto a Nascar track, it’s highly restricted and regimented)…
So for devices with updateable firmware, part of the “race protocol” would be to receive an encrypted signature hash from the firmware proving it’s an acceptable revision.
The data files would then create an MD5 type file with that same hash upon “race completion”, thus making the source from the approved “systems” secured (within any reasonable technological argument) from alteration, and then it exports the file for submission.
The firmware would prevent the recalibration of the device (and could even potentially have a “physical tampering” monitor so that if a given ratio range of “power/speed/force application (internal resistance) is found “out of range” from the factory basic calibration acceptable wear ranges, the results are considered negated.
Of course all this requires purchasing new trainers that support the dual recording as Ray suggested, as well as stricter programming/dev requirements, but at the end of the day, that’s the “cost of doing racing” business. For Joe Schmoe trying to lose his keg back to a 6 pack at home, he won’t need to buy the $300-upcharge “racing edition”.
For those wanting to race at a competitive level (or maybe just wanting the extra combined datafile features, etc)… that’s probably the same as a mid-level race entry fee, again, pay to play. And if you want to race, but not have a “racing edition” that’s fine, you just aren’t eligible for prizes and are “asterisked” in any leaderboards as a non-certified result, which can be turned on/off for listing purposes to see the “in-competition” leaderboards versus the “everyone who gave it a shot” leaderboard (so you can see how you’d compare).
Really not a challenge technically at all, to raise the bar enough that “big races” are made truly hard to cheat (data-management-wise)… but will require investments from vendors, and probably “bump up” investments from participants, I used $300, that’s actually extremely excessive, the parts and programming to do all this wouldn’t really add more than about $5 to a devices cost (and most of that’s just to recoup the programming efforts up-front)… so bump it up $50 and it’s still pure profit at that point.
Trick is getting all participating vendors to support the “race standards”, but same thing, pay to play, don’t have to, just can’t be in the official leaderboards for your participants.
> Scripted aspects like that are easy to pick out. Adding in slight variability makes it harder to detect (and likely, today, would go undetected).
You’re misunderstanding what they’re saying. Not harder to detect, impossible.
They’re proposing to man-in-the-middle the rather simple and non-secure protocol that is ANT+, and rebroadcast each frame with only the power value altered. It would indeed be dead simple and completely non-detectible since you ARE using the original data sources with all their built-in variation (so none additional needed), but simply scale them both. In real-time.
Only encryption at protocol level and a shit ton of DRM could defeat that method.
Ahh, sorry, yes, I was mentally thinking he was saying to simply take Source 1 and add 2% automatically with man in the middle. Which solves real-time, but doesn’t make it any harder to detect, however, as noted, adding the variability would break it today.
Yes, a man in the middle is harder to solve for today, but I think it’s all about raising the bar – one step at a time.
Note: ANT+ actually can do encryption. I’m not aware of any sport usages today, but it’s been in the spec for a gazillion years. I remember sitting through presentations about it a long, long, long time ago at the ANT+ Symposium. It briefly reared it’s head again a few years back with some TdF stuff that popped up, then went quiet.
But – and this is key: That’d require Zwift to actually care enough to engage the manufs. In talking with manufs, they aren’t really opposed to this level of stepping it up. After all, it gives good reason for people to potentially buy new hardware to have an ‘E-Racing’ supported version or something.
Actually you’re right I made the mistake of thinking adding encryption would break all existing things but it could of course be used as a simple toggle where a device normally broadcasts straight but can be flipped into a (initially presumably supported only by Zwift) “secure E-racing mode” or similar.
Honestly shouldn’t be hard to implement (I’m just a student but dabble in protocol level microcontroller firmware stuff, it ain’t rocket science) and could be rolled out as firmware update for the masses. Charge $100 for it, call it Pro Mode, and I’m sure the market of non-pros would lap it up and more and more regular races require it.
I brought this up last week in our Zwift cycling group, at which point there was little to be found about it other than buried on Canyons own site. Thanks for bringing it to wider attention, a good deep dive into it.
I would expect that with the use of AI they could easily pick out those workouts/races that are suspicious and have a human double check to call out cheaters. With AI it could be done across Zwift, not just pro events.
It seems pretty dumb to use straight offset for faking data. Could at least do a reverse curve fit from a known set and add a random salt to race file. :facepalm:
The requirement for secondary recording is prohibitive to people on lower incomes requiring power meters in addition to the existing cost of bike, trainer, subscription. It would also create a barrier to uptake by people who just fancy trying it out, (this make it all too complicated, involved and intense).
Have consideration for others and higher levels of participation as opposed to this elitist perspective.
You’re misunderstanding what is being discussed – and the target segment.
The requirement for dual-recording already exists. In fact, the requirement as it is today is exactly what you’re supposedly against. It requires you go out and buy a second head unit/bike computer to record the data. It requires you to have more complication to record the file, ensuring you’re paired to the right device, and then save and upload that file to a second location.
What I and others are asking is why on earth Zwift doesn’t just do this in the app itself, reducing the need to buy a second computer. Make it as simple as pairing any other source.
Nobody is saying everyone needs to go out and buy all this extra stuff to use Zwift, or even race on Zwift. They are saying though that if you’re a pro, who near-100% already has a power meter as part of being a pro, then yes, you need to leverage that.
Looking forward to it. Zwift are putting a lot in to showing the world that they are acting, but nowhere near enough in to actually developing the platform to help in the first place. I can’t remember the last time there was an update to improve racing on the platform for any aspect, let alone reducing the capability to cheat.
Sorry, i misunderstood. I assumed you were expecting everyone who fancied trying a race, should go out and buy a power meter. I agree this is fine for pro level but should be avoided for everyone else so as to maintain the inclusiveness and appeal to all,( present and future users).
I was under the impression the Canyon team had Infocrank as a sponsor. So I’m more confused why he wasn’t already dual recording.
I have dual recorded for quite a while. Not for the reasons mentioned in this article but because I wanted to know the difference between the power meter and my Kickr. And I use the power meter on the road anyway.
Now, however, I have switched to a Kickr Bike. So my 4iiii doesn’t fit on there. Would I need to buy pedals to join these Zwift races (if, which is very unlikely, I would ever be fit enough).
Am I the only one thinking “why even bother cheating” looking at his (cheated) placements: 64th, 2nd, 72nd, 27th, 14th, 30th, 71st, 81st, 81st, 89th, 39th?
Because you are not reading it right, Ray has failed to mention that those results include a Team Time Trial, which Canyon won and without Diegner racing, may have been DQd for not fielding enough riders or not having 3 to finish. Him towing the team along has had a significant impace ton them winning the premier league races.
The results are across the line, in a TTT there are time delays so the Canyon team may win the event but having started 10 minutes after everyone else, will appear 81st or 89th, when in reality, may be 3rd or 4th.
Scratching his result should DQ the team.
This was my first thought…last in a race…why bother?
My second thought…really? Just add 2%? Seems like it would be not too difficult to randomize within a range.
“My second thought…really? Just add 2%? Seems like it would be not too difficult to randomize within a range.”
Because it’s silly easy to tweak a KICKR18’s calibration value. It’s much harder to tweak a power meter. Tweak the KICKR18’s baseline, then duplicate the file and add 2%. No idea if that’s what happened, but that’s the only logical reason that I could think for doing all the items listed here.
Can Zwift tell if someone increases the scale factor on the power meter meant to adjust left/right discrepancy?If you just increased that you would have a randomized difference. You see a lot of pro zwifters running Vector pedals as secondary…
I’m pretty sure that would actually show up on the .FIT file on a Garmin device. Not 100%, but reasonably certain due to the way Garmin sets it.
Interesting. The funny thing is, there is no current limit on dual % variance so you can just mis-calibrate your trainer and then who cares what the PM says. Obviously it doesn’t look great but the results stand.
I guess that leaves the ant+/BT intercept method that you mentioned in a previous article as the gold standard. Ill be interested to read more. I love racing at this level but the spectre of cheating is always there at the moment.
Wouldnt it be possible to fake the power from a power meter by adjusting the slope?
This is also very easy to do on a Quarq via their app, I dont know about other brands though
This is yet another issue seemingly ignored by Zwift/ZADA, we are regularly seeing riders who’s secondary recording is over 5% out across all power ranges yet for some unknown reason their results are allowed to stand. I don’t know about you Jordan, but for me it undermines the whole point of dual recording of it doesn’t matter what that data says!
> It’s much harder to tweak a power meter.
My 4iiii allows scaling reported value greatly (0.5-2.0 I think). So should be dead easy as well (assuming that fact isn’t reported elsewhere in resulting capture, but should be easy enough to scrub compared to making data up wholesale.
In a post-pandemic world (whenever that may be), does Zwift racing continue at elite/sub-elite level? As a spectator, I would much rather watch the actual peleton go over the Alps than watch a bunch of avatars bounce around. I can see virtual racing opening up avenues for 2nd tier/developmental teams as it has to be cheaper to buy even the most fancy trainer for a team than fly a team, support staff and equipment to where-ever. This isn’t to say people don’t cheat in every level of sport, but spectators tend to careless if it is not the top names.
Semi-related, I keep trying to talk my bike-loving father into a smart trainer as his winters aren’t conducive to riding but if it takes a rocket science degree to get everything set up – from physically get the bike mounted to then getting all the sensors correctly connected – he will never do it.
Hi Ray
Any idea/thought about how ZADA is set up i.e is it zwift employees or is it setup completely outside of Zwifts control. With zwifts sponsorship of Legion of LA, which is also racing in the Pro Racing League it seems there is a conflivt of interest as it would look extremely bad if one of their riders were caught cheating on zwift while being sponsored by them. In my opinion they line has already being crossed with Legion having access to in game equipment they have not earned (disc wheels) despite the zwift ruleset stating this is not allowed in appendix c (Code of conduct). It is a minor rule break but the fact that they are allowed to do it while everybody else would be dq’ed really affects the perceived fairness of this series for someone who is actually racing it as well.
A slightly frustasted Zwift racer
Right now it’s a bit half and half. Meaning, that as I understand the ZADA people are not Zwift employees but are indeed paid by Zwift.
I do agree with your points though, and they are all things I plan to discuss more deeply with Zwift in some upcoming pieces.
I’ve been pretty surprised by just how much feedback I’ve gotten from teams today – all of them basically saying variants of the same thing: They want increased scrutiny, not less. And notably, that want consistency in scrutiny. And clear outlines on what the process should look like once an accusation happens (a message I’ve heard multiple times from past accused Zwift athletes).
Last night’s race if you go to the dual recordings, there are riders in the top 10 who had turbo trainers reading well over 100w higher on the sprints. This massively influenced the result. Legion rider one of them.
If the power meter reads higher on a short sprint, pedal or crank based it is not changing the in game speed but the extra 10% from the trainer should trigger a DQ. Will Zada do that?
Is there a 10% DQ rule? I’d think that’d be challenging to support, at least at the 1-second and 2-second rates, maybe at the 3-5s level.
(Perhaps there is, I haven’t re-read all the rules)
Thanks for the Reply and its good to hear about the widespread support from the teams and that it will be brought to zwifts attention by someone they may actually listen to.
In regards to dual recording differences as far as i know there are no set limit, as ZADA is rightly not informing everybody about what will cause a more thorough investigation or a dq. As Chris noted there are however at least 2 riders in the top 10 having very big positive difference in sprinting using there trainer with one being 18 % higher for 15 seconds and another 9 % higher, while also seeing a 7 % difference for his best 20 minute effort as well. This seems to be way more than what should be expected by racers and equipment on this level, and even if it caused by a malfunction like low battery voltage on the PM, it still at least raises some questions about these performances.
Glad you have been receiving similar feedback from multiple (frustrated) sources Ray.
There definitely needs to be more scrutiny of key events (ZRL Premier League and the top division of the Community League in particular) and those racing fairly will embrace it.
I have seen Zwift inconsistently apply the rules, amend them after the fact to avoid having to sanction/prevent riders from racing, and seemingly overlook obviously suspect data and performances. This is undermining any faith in the system held by some of the teams and riders who have helped build racing on Zwift.
Any riders who are seeking to race at the highest levels on Zwift should be open to full scrutiny, and if question marks about their performance persist they should be required by Zwift to undergo performance testing in an independent lab. Dual recordings and Three Sisters tests on Zwift are pointless given how easy it is to cheat them.
So what exactly would a suspension for a non-pro-series racer look like? Outright ban from Zwift? Or is their account somehow locked from joining any event tagged as a race?
(This stems from the pro level suspensions allowing the riders to still take part in community races…I think?)
I haven’t used zwift since August. Have the addressed every day cheating in normal racing yet?
Ray, when are you going to talk about Sticky Watts?
It’s on my fuzzy radar, but honestly a bit further behind this. One ocean at a time, sorry!
Part of the issue (development impact) would be if you required dual recording across all platforms. Could be a lot harder on (for example) iPhone, Android and Apple TV.
If they limited it to desktop (PC and Mac) it shouldn’t be that difficult.
Phillip was my coach for a year back in 2017. I also participated in the Zwift Study he did back in 2017 (results published on ZwiftInsider).
I am in complete and utter shock at this. Phillip was kind, gracious, humble and very generous with his time for the entire year or so that I worked with him. I was stunned to find out he was accused of cheating, and my mind immediately went to ‘must be a mistake’ – up until I read that he had admitted modifying the data files. Completely stunned.
I was surprised as well. As you noted, he’s been around the sports tech block quite a bit, and all doing good work. 🙁
Excellent article DCR for me its not just about the cheatings or rather should i use the word deceptions I’m a 66 year old heart patience and race 4 -6 times a week in C grade really should be D grade and have done so for the last 4 years. And i learnt a long time ago not to go with the front riders as anywhere up to 50% in some cases they consists of unregistered ZP riders or no HRM,S or there sandbaggers all of which gets me further to to the front at the finish . thanks so much .really informative . Dennis Nel .
What was the reason for power discrepancy in those few races? Was it power drop outs from PM? Was tampering just fixing dropouts or editing power to be close enough on average with Kickr? I’d like to check it out but this data is no longer available.
MJC IV — By math done poorly! I somehow did 12-7 and got 2 instead of 5. Oops!
I also worked the numbers from memory, so looking at the article again lets me spell out what I was trying to say. At 6000′ it says you perform at 92.7% of your sea level performance if you are acclimated. It also says that you perform at 88.9% if you are not. So that means that only 3.8% of the difference would survive if you stayed at sea level long enough to lose the benefits of being acclimated to high altitude. This obviously is almost twice the 2% I said, which is some awful math (although it also a good bit lower than the rounding gave me).
I recognize that the answer probably isn’t linear. My overall point is simply that not all of the benefits of acclimated training will last forever. The number is surely lower than 7%; I just don’t have the knowledge to explain exactly how much lower.
I am not unsympathetic to the frustration that must come with being at altitude, but it’s just absurd in my view for those folks to think this should be anywhere on Zwift’s priority list. Not only are they surely a small set of overall users, but, as you note, location cheating becomes an issue. Not to mention a million other issues that create arguably unfair results. For example, I accidentally raced using my trainer yesterday instead of my pedals. The trainer was calibrated the previous evening and the pedals before the ride. I dual record. I left A LOT of watts behind by doing that. If we are concerned about comparing apples to apples, at least in racing, Zwift really needs to control for where your PM is located, and also how to force calibration before races. Of course every PM is a bit different as they have margins of error, but the location issue is real, as is the “oops, I forgot to calibrate” issue.
I did a 30min hard effort on Zwift recently where my wahoo18 power was 291AVG and my 4iii on shimano crank was 334avg. After back and forth with wahoo they said the difference was reasonable after reviewing both files suggesting chain tension, difference in PM location, difference of 2-4% in PM themselves etc….said that would all add up to a difference in reading like this. I just use zwift to train and fun and a 50 yr old masters hack but interesting example re idea data should be within +/-2%.
Just wanted to thank DC Rainmaker for taking adult, sophisticated views of Zwift and sharing them here on this well-trafficked site. This stuff tends to get buried or lost in the noise on Zwift’s own forums.
Wow, Zwift is better than the UCI to find cheaters !!!
I was part of this study back in 2017. Check out the author.
link to zwiftinsider.com
I was wondering what the statistics of some ‘legit’ dual-recorded .fit files would look like, so I ran some code:
link to github.com
Throughout all analyzed file pairs, it looks like the following values can be all over the place and therefore are no good indication whether a file pair is legit or not:
Mean difference in power (between zwift power and external power);
Mean absolute difference in power (between zwift power and external power).
Of course you could argue that if any of these values is particularly large, one of your power sources is miscalibrated. But that would hardly qualify as cheating.
As noted in both the Berger and the Diegner case, a value that is relatively characteristic of a dual recording is the distribution of the difference values between the two data sets (the “residual distribution”).
When the Berger and the Diegner data were still online on ZwiftPower, the residual distribution – just by looking at the original time series on zwiftpower – did seem to correspond with what you typically see in a legit recording, as opposed to what they write in the Board Decision, at least for Berger (point 1a, ii).
These files have now disappeared from zwiftpower unfortunately, otherwise it would have been interesting to see what the residual distribution really looks like. In the Berger case, its variance definitely did not appear to be “0”.
Another point that is hard to understand, if you go through the length of modifying a fit file, why would you not just add some noise to the data? See bottom of the notebook.
It could be that Zwift have some additional evidence for tampering that they do not want to release for good reason.
As you say, large parts of this discussion could be solved very easily by baking the dual recording straight into Zwift.
Great article (as usual). Small thing: towards the end, you have a typo in “…noted that statically that was…” that should be statistically.
Thanks!
Thank you Ray for the education. I had no idea how Zwift validated race results. Seems to me that double recording is kind of an antiquated/problematic way of doing things considering the state of technology. If they, Zwift, do not have complete control over hardware and software it may be the best way to validate. I wondered if Zwift`s entry into the smart bike market may be an attempt at this. Thanks again for the great content.