Behind the Scenes: How I choose which products to review

IMG_9911

Over the last couple of weeks I’ve got a few variations on the following question (for a variety of products):

“Will you be reviewing the Bryton Rider 60 any time soon? If not please can you say why? I realize it’s too much to expect you to review everything but the Rider 60 looks to be better than the Garmin 800 series. I would be very interested in your review.”

So, I figured I’d take a second and talk through how I decide which products to review in general, as well as in this specific case.

First up is noting that there’s a heck of a lot of products on the market that fall into the purview of things I’d be interested in reviewing.  In general, that landscape is “sports technology”, where the ‘technology’ part usually implies some sort of electronics chip or functionality within it.  It isn’t however 100% exclusive to that.  There are little things that interest me that fall slightly outside of that.  For example, the RedShift triathlon quick release aerobars or some of the swim safety devices.  But for 95-98% of reviews, there’s a gadget aspect to it.

I generally shy away from things like bike components (wheels/frames/etc…), because I know almost nothing about those.  As long as my bike mostly rolls forward and doesn’t make too many funky sounds – I’m happy.  So I don’t have too much interest in those things.

So assuming it’s a sports tech gadget, the next piece is actually looking at whether it’s relevant or not in the market.  This comes in two pieces.  First I look at how new the device is.  Obviously, if it’s about to be announced or just announced – then it ranks higher for review.  To that end, I still get roughly once a month review requests for some much older (4-5 year old) Suunto and Polar products that sorta pre-date most of my reviews.  In this case, while I have a tiny bit of hands-on time with some of these devices, I know they simply aren’t competitive in today’s market based on features alone.

Thus assuming it’s new(ish), then I generally do a bit of a gut-check on how competitive it is.  If the unit is so badly non-competitive on specs alone, and if it doesn’t have broad appeal – I’ll probably skip it.  My reasoning is simple: I’ve only got so many hours in the day (err…night), and if there’s something that’s of little interest and is looking like it sucks, it seems silly to waste time on it.  However, if the unit has broad appeal (i.e. Garmin, Polar, Suunto, etc…) but sucks, I’ll still likely write a review on it because there’s significant interest.  Sorta like the Samsung Gear Fit review this past week.

I do, as part of that, look at units that I believe may be ‘breakout’ units.  So even if the company isn’t well known (or known at all), if I can look at it and see massive interest or potential interest – that’s huge.

But what exactly is broad appeal?  Well, that actually gets to the heart of the original question up top on the Bryton unit.  And to that, one probably needs to understand the rough geographic readership of the blog.  In general, y’all are split almost perfectly between being 50% North American, and 50% everywhere else.  That ‘everywhere else’ is heavily weighted towards Western Europe, Australia, New Zealand, South Africa…and Brazil.  I was hoping Google Analytics would make a prettier (more colorful) picture than the below…but that’s all I could make it do.

image

So when I write I generally want the content to benefit as many people as possible.  In the case of the Bryton unit, they don’t sell within the US.  Thus, that removes a massive chunk of my audience immediately.  Despite that, I often review or talk about devices that are either US only or non-US only.  But in most of those cases, the company is planning to release globally relatively shortly thereafter.  Take for example some of the activity trackers like certain FitBit models or the Basis watch.  US only initially, and then global distribution.

Finally – and this one is important – I look at how past reviews by the company have fared.  By ‘fared’, I’m primarily looking at interest from you as readers.  And that’s the one that gets the Bryton units in a bit of a snag.  Last summer I wrote about the Bryton Cardio 60.  As I noted in my review, it’s actually a pretty good little unit.  Hardware wise it’s fairly solid.  Software-wise a little more shaky, but overall not a bad unit.

The challenge though is that the Cardio 60 has sustained so little interest that it ranks as one of my lowest ‘utilized’ reviews of all time (below even posts I’ve written 5-6 years ago).  Interest is a combination of ongoing page views, comments, and discussion about the review out on the interwebs.

And that’s the rub.  Given I’ve got stacks and stacks of hardware to review that I know has a significantly higher and more global interest, I focus on those first.  Said differently and much more succinctly: I basically just review what y’all want.

With that – have a great weekend!  And for those racing in Boston – Good Luck!  I’ll be there cheering on The Girl and all the other runners who run by.

-

Side note: Want to read a bunch of other (more photogenic) DCR behind the scenes posts?  Swing on over to this tag that lists them all!

43 Comments

  1. Tosin Akinmusuru

    Aka "i do what i want"... Drops mic, walks of stage.

    Reply
    • Drew replied

      This would fit in the "Summary and Final Thoughts" section. ;) Thanks Ray!

      Reply
  2. Chris

    Do you know how awesome of an April Fools Day post that title could have made?

    Reply
  3. Matt

    Good luck to the Girl and to my many friends running Boston as well.

    Reply
  4. Asaf

    Ray, I think it's worth saying your work is great and essential for many athletes. Trying to cover every little topic in the sport technology world is impossible by one person. Actually, I think this is also your advantage (the one person thing) - the way you and only you integrate and compare different products in great depth across companies/time/technologies etc.

    Maybe it's worth considering, if at least some traffic to your site is left unanswered, to offer a new separated platform (e.g., forums/Wikipedia style review in which people can write and edit texts, etc.) for those who wish. It won't have your quality seal nor will it be accurate as the polished reviews you write, but at least people can communicate their Q&As.

    Reply
    • Rainmaker replied

      I have considered forums, though, I fear that I wouldn't quite have the time to manage them properly. A Wiki is actually something I've been given a lot of thought to recently, as it might reduce time dedicated to answering common questions on review (which, on average take 2-4 hours a day).

      I've also been pondering some method of elevating a handful of readers that have responded frequently and with significant interest on certain topics. For example, reader Frank Young has done that on some of the activity tracker posts (Vivofit, Loop). He's answers almost every question with detailed, fair, helpful and technically accurate answers. There are others in other reviews doing similar things, but that's just the first example off the top of my mind.

      Reply
  5. Great post Ray, I get similar questions all the time about shoes, but would have to run myself into the ground to review them all. You have to pick and choose and my criteria are very similar. I review what I think my audience is most interested in, and avoid things that might injure me or make my runs less enjoyable.

    Looks like you need to build your audience in the Sahara desert :)

    Reply
    • Mike replied

      or stuff you have to pay for.

      Reply
    • Peter Larson replied

      I buy a fair number of the shoes I review, and most that I don't buy get donated to other runners or Soles for Souls after I write a review. Shoes aren't like fitness electronics, you can't really return them to the brand after they are worn.

      Reply
  6. Thanks for what you do, You do it very well!

    We trust your judgement, keep them coming.

    Have a great weekend!

    Reply
  7. 6co2000

    Ray, just so you know. I have read your website a couple of time while connected from a hotel in Chad. N;Djamena. In Central Africa. I guess I must have been the only one ever... but it s not showing... !!!!
    :-)
    6co

    Reply
    • Rainmaker replied

      Yeah, I think if there's less than a few thousand readers from a country, it all showed the same. :-/

      Reader Brian up above offered to make a map, and I've taken him up on his offer. I'll definitely publish it - looking forward to it!

      Reply
  8. Ray,

    Really appreciate your work. Looking forward to meeting you in June! Keep up the good work.

    Lew

    @lewsview

    Reply
  9. Duane

    I think I'm in good company among those who really enjoy your reviews and what you do here, but the review activity does see a bit thinner lately. It could be that I skip over running gear and mostly tune into bike stuff. Maybe there isn't much of that lately, but I can tell you that my heart always leaps a bit when I see in my feed reader that you have a post. Keep up the good reviews.

    Reply
    • Rainmaker replied

      Yeah, things go in waves depending on season. Right now it's fairly run heavy, whereas the fall is very bike-heavy. Just mostly the launch timeframes.

      That said, I expect May/Early June to be very bike heavy:

      Garmin Edge 1000
      Polar V650
      Mio 505
      Sigma Rox 10

      To name a few, plus some other things bike-related in the next few weeks (smaller stuff, like the heart-rate helmet I've been using all winter).

      Reply
  10. Brian

    Can you export the reader counts by country? I can make it prettier for you. LOL

    Reply
  11. morey000

    Ahh- I see you included an image of your 'behind the scenes' product review team.

    Reply
    • Rainmaker replied

      Nah, that's just my team that focuses on answering Samsung employees commenting as regular people in yesterday's post.

      Reply
    • L Wilson replied

      Good one Ray!
      From what I saw you may need a larger staff...

      Reply
  12. Ben

    So happy you just informed me of the Bryton stuff. I use Garmin, because it's all there is really; if you want ANT and GPS. But Garmin plays a lot of consume games, taking full advantage of its market domination.

    I guess I'm sad to hear that you, having to triage your time, must do reviews that reflect popularity. Can I toss a vote out there for the occasional "charity" review of something interesting? Or, anything that might take market share from Garmin.

    Garmin's insistence on not using Bluetooth until recently, the high prices for hardware, the way-too-early releases, the abrupt abandonment of google maps (since rescinded), and intentions of competing with Strava, etc etc, etc, it's a company that pains me to give money to. The instant they have solid competition, I'm jumping ship!

    Reply
    • Rainmaker replied

      Oh, I definitely continue to look at any product (big or small) that does bleeding edge or cool/unique stuff. The Mio unit from a few weeks back is actually a lot like Bryton in that it's not available in the US. The key difference is that the new Mio unit threw-down the gauntlet on new and innovative features that nobody else is doing (even Garmin). Ultimately I want to highlight innovation.

      Reply
    • Ben replied

      Holy crap, that Mio is the BOMB. that should have Garmin execs worried (I doubt it, there's only hubris there). The instant I can get a Mio, I am, finally, I can skip a song from a BT enabled head unit? Welcome to 2008! See texts from my wife without reaching back into my jersey?

      Another example of Garmin incompetence: the infamous GPS drift on the 500. The 500 is the best device they make so long as you don't mind constant, but minor inaccuracy.

      Anyway, no matter, Ray I love ya man. Thanks for everything. Sorry to have forgotten that you're a sucker for technology and well made gadgets.

      Reply
  13. Rob Forshaw

    Good luck to the girl on Monday! I'll be posted up cheering on the runners on heartbreak hill!

    Reply
  14. Patrick

    Hi Ray, read this post with interest. I'm interested in buying a bike computer for the begining of the season here and am hesitating between a Garmin 510 and the new Sigma Rox 10. I know a review of the sigma is coming, do you know when ? Your expert advice would make a difference. Or can you give me a quick advice in the meantime ? Thanks

    Reply
    • Rainmaker replied

      My goal is May. I've got limited bike time over the next 2 weeks due to travel, so realistically that puts me in May for Sigma.

      Reply
  15. Stephen-H

    Great explanation and makes perfect sense! Thanks so much for review and posting!

    Reply
  16. Maybe I'm just a giant idiot, but, I know there are other products out there, but I know my Garmin products so well, I almost would never consider making a jump.

    Cool to see how you figure out what you're going to review - very very interesting and thorough process!

    Reply
  17. Mike

    My concern is that most of the product reviewed are for a certain segment of runners i.e. those with a lot of disposable income. Now people can argue that the cost of a Garmin 620 is the same as an Xbox or 1/4 to 1/2 the price of a marathon and you get better value, but that still doesn't matter to the set who can't afford either.
    I also worry about the impact of the giant companies vs smaller companies. The clever training deal has me worried as well about absolute objectivity.

    Reply
    • Nathan replied

      Ray has reviewed the Soleus devices. These are affordable. And I have to say, some of the lower end Garmin devices are pretty well priced compared to 910 and 620.

      Reply
    • Dr. D replied

      @Mike, Ray's reviews covers a wide cost spectrum. From the $100 Soleus watch right up to the $400 Garmin 620.

      His neutrality is what a tonne of readers admire the most. When we use Clever Training (or Amazon), we are supporting what he does. It is a great formula that brings you and I objective, impartial, and thorough reviews for 'free'.

      I continue to share/recommend his reviews as they are far better than anything I could share with my colleagues/fellow runners.

      @Ray - best of luck to The Girl (and all Runners) at Boston....

      Reply
  18. Scott

    Ray, Thanks for all the great reviews. Your site is the place to go when I am looking for new gear. That said I would love to see the watches and bike computers broken into two seperate areas in your comparison tool, and maybe add some existing device that you might not have reviewed Ex. SRM PC7, PC8.

    Reply
    • Rainmaker replied

      Yeah, I've been contemplating that actually. The only challenge is that sometimes people are trying to decide between something like the FR910XT, and an Edge 500. Which compete with each other very well.

      Reply
  19. Ando Ellison

    Ray, as always thanks for your content. Some interesting tidbits from around the interwebs:

    Looks like Nike's out of the fitness wearable hardware market:
    link to gizmodo.com

    The world's first surf GPS watch!:
    link to ripcurl.com

    Interesting idea, but I think the market will be somewhat limited. I personally don't like having anything but boardies, leash and board when I'm in the water.

    Reply
  20. You have at least one grateful reader in Japan.

    Reply
  21. Carson McQuarrie

    I would really like to see the Garmin hand held GPS units reviewed like their Rino Series and such. Is there a reason why you haven't done those?

    Reply
    • Rainmaker replied

      I'd love to, but at the moment it would just spread myself too thin. Also, I don't get out to hike very much right now, so it'd be hard to get in the right time to appropriate test it.

      Reply
    • Carson McQuarrie replied

      Ya I hear ya, I imagine you're pretty busy. Well consider it for the future. Hand-held gps units I think the world would greatly appreciate reviews on since they also can be quite confusing and I can't find anyone that's done thorough reviews on them like you would, if you did them. Every day is a good day to start hiking in the mountains :) Take care and thanks for all the posts about triathlons. I am just starting to get into them and will be doing my first in July in Squamish B.C.

      Carson McQuarrie

      Mountain Equipment Co-Op

      Reply
  22. James

    Ray, thanks for the background on how you choose the products to review. I bought the 910xt based on the review, and use the 910xt page frequently for support.

    Reply
  23. Paul D

    So what you're saying is that those interested in the Bryton need to hit 100's of times to boost their popularity!?!

    In all seriousness, it looks like there would be great interest in a type of 'in the pipeline' page to let people know what you're working on (or thinking of working on - heck, maybe we could vote on it). It could save you in turn getting lots of a 'when will you review X' emails to answer. The full blown Wiki would of course be the holy grail.

    Reply
    • Rainmaker replied

      Such a thing exists...it's called The Queue: link to dcrainmaker.com

      Except, without the voting part. ;)

      The challenge I have (even with The Queue) is that my constraints aren't always easy to get around, and that really limits what I can publish. For example, the next three weeks I'm on the road travelling for work. I think I'm home perhaps 1-2 days, that's it. Thus, getting bike-focused reviews is really tough (not going to happen). But that's hard to reflect in The Queue. And because I don't really much control my work travel schedule, it's really hard to predict how that will impact things (and that ignore just regular work workload).

      In other cases, I'm waiting on a company for finalization of a product. For example, Polar and the V800 and V650. A review won't go out till those are fully locked.

      That said, perhaps some middle-ground of basically estimated targets for different reviews.

      Reply
  24. Carson McQuarrie

    Hey Ray,

    I am wandering if you can help me make a decision on which route to go with in terms of purchasing products. Here's a breif background: I am getting into triathlons (will be doing my first one this July) and plan on continuing in the future cause I really enjoy training and data analyzing (not that I do it right now though). I currently have a Forerunner 301 from years ago but it's served me well. I also really enjoy hiking, rock climbing, skiing, mountaineering etc. so I really enjoy the way-point features and backtracking feature. I'm from Vancouver B.C.

    My dilemma: I am looking at either getting the Ambit 2 or Forerunner 910xt. I am unsure if I'll regret not having the details that come from the 910xt for triathlon training in the future, and on the other side I'm not sure if it will serve me as well as the Ambit 2 in terms of navigation and route finding.

    What's your opinion on getting one of these watches or getting the Forerunner 910xt and then a second handheld gps to serve my back-country navigation features?

    On another note, just an idea for your product reviews, I think something like a brief description for each of the features that you list at the end of each review when you compare products. Something like when people drag the curser over a feature a description appears because some features mention under product comparisons are confusing like the virtual racer feature.

    Thanks for all you've done so far!

    Carson :)

    Reply
  25. Carson McQuarrie

    Any thought on reviewing Suunto's other sport watches like the M5, M2 etc?

    Reply
    • Rainmaker replied

      No, not at this time. Given they aren't quite as competitive these days and I'm rarely asked to review them - it's tough to fit them in.

      Reply

Post a Comment

Your email address will not be published. Required fields are marked.
If you would like a profile picture, simply register at Gravatar, which works here on DCR and across the web.

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>