After reading Kevin Pace’s Hot or Not? Consumer Reports post — and the great comments it’s generated — our friend Bob Markovich over at Consumer Reports dropped us a line to respond. Read on past the jump to read his note we received last week:
Let me begin by saying you’ve got a great site with some knowledgeable and passionate contributors. As Consumer Reports’ Home & Yard Editor, I spend much of my time with our testers and am typically the first to know what they’ve found out about the latest cordless drills, mowers, leaf blowers, and other homeowner tools. We’re also eager to find out what we’re doing right, how we can get better, and what we need to improve.
Several comments on our testing and reliability surveys went beyond simply tools. Because I’m also involved with planning our coverage for washers, grills, vacuums, deck stains, and a slew of other non-tool products, I thought I’d answer some broader questions, clear up a misconception or two, and tell you where we’re working to improve:
We don’t rate products based on price. Nor do we include our survey results about brand reliability in our Ratings performance scores, which are based purely on how models fared over weeks or months of testing (not just one “one week”). But we DO consider price and (when available) brand reliability for our Recommended and CR Best Buy models to highlight the ones that combine a good brand-repair history with the most performance and value. Our cordless-drill coverage (November 2008) included top picks for both general and heavier-duty use, with CR Best Buys in each category.
Long-term durability tests? Not usually, but we give products a workout. We do durability testing for treadmills and we test exterior paints for a full three years outdoors 24/7, since, for finishes, longer life is everything. We’ve also found ways to do life tests for car batteries and wear tests for flooring quickly and efficiently. But with today’s faster-paced model turnover, longer tests for some products would probably mean they’d be off the shelves by the time we were done. So we work to make our tests challenging yet real-world. For drills, that involves driving thousands of lag screws, among other tasks. Those tests have proven tough enough to kill “heavy-duty” lithium batteries, fry clutches, and melt the solder in contractor-grade models. Our continuous tests are also getting more tested models in continuously updated Ratings far more quickly.
Our brand-repair surveys aren’t self-selecting or warranty-based. We invite millions of subscribers via regular mail and online to answer questions designed to prevent skewed data based on a product’s age and usage as well as a brand’s market share. Data are based on models that were repaired or had a serious problem—not on warranty claims. Our minimum sample size is 100 cases for each full year in the analysis (most cover at least 3 years). We also verify that no one is stuffing the ballot box with multiple questionnaires. In contrast, user reviews ARE self-selecting and tend to reflect extremes on the satisfaction scale—and you don’t always know whether the writer is an actual consumer or a company shill. But those reviews are an increasingly relevant leg on the stool, which is why we’re looking to grow them on our site, with the above caveats.
We do disclose how we test, but there’s room for more. We include a Guide To The Ratings right next to the chart in Consumer Reports magazine product reports and on Consumer Reports online. We’ve begun highlighting our tests and testers far more in our reports and are working on more online test videos aimed at the newest and/or most touted products. As for which features are most important, we include the ones readers tell us they want, among others. For drills, those include two batteries and quick charging, multiple speed ranges, an adjustable clutch, and a ½-inch chuck. We could do more on heavy-duty features and how they figure into pricing. But as we’ve seen, those don’t always guarantee a better drill just as “pro-style” doesn’t guarantee a better stove.
And of course, our buy-what-we-test, take-no-ads policy makes it easier to call things as we see them. But there’s always room for improvement. I hope this helps.
Whether Bob’s response changes your “hot,” “not,” or “warm” opinion, it does prove one fact: Consumer Reports reads Toolmonger and heard your cry.