Standardization. In theory, it’s a good idea as it relates to consumer research. It certainly helps with efficiency (we’ll talk about that one 
another time ?). Researchers are happy when they can compare the results of several studies because things are done the same way. That’s fine if you’re testing the very same thing over and over again, and assumes the world around you isn’t in constant change. But it is.
People change. Societies change. Life experiences change us in an instant (Orlando will be forever changed after the Pulse incident). Product expectations change. And hopefully, your product offerings are changing too.
The challenge for researchers is to understand when to keep things the same, and when to change.
For the record, I’m an advocate of asking many of these types of questions in the same way to cross-compare – things like overall satisfaction for a service, or overall rating and purchase intent for a product.
However, it may be necessary to change the “internal” ratings when evaluating new product ideas and executions to better reflect consumers’ overall experience. After all, “new” product ideas indicate that something is different about the product, and “new” survey or research questions should be added to better understand how consumers feel about the differences. Otherwise, you’re comparing things that aren’t comparable- apples to oranges.
Here’s a case in point. We recently were a part of research where a new product was a preference winner, but an internal ratings loser compared to current product. What? How can that be? This can happen when there isn’t a standardized question being asked that encompasses what consumers experienced. While the product fell short in terms of some of the standardized questions (and likely needs more development in these areas), it provided consumers with a paradigm shift in other areas. In this case, the test product had a much greater sensory appeal than consumers realized was possible in this product category. They wanted to tell us, but couldn’t find the right “standardized” closed-ended question to do so. Overall rating or preference is all they had. Thank goodness for open-ended voluntaries!
In summary, when designing a questionnaire for new product evaluation, it’s important to:
- Keep standardized “overall” type of questions. They are helpful in cross-comparison and the actual numbers have a significance to those of us with a history using this rating.
- Make sure to include some new questions that you believe “zero in” on some of the benefits unique to the product being evaluated.
- Don’t forget to include those open-ended questions – they may not be efficient, but they are incredibly useful! Consumers will try to express themselves through the ratings, but if they can’t, the voluntaries are priceless.
- A few discussions can shed a lot of light on any confusing data. One-on-one discussions are still the gold standard in my book. When you’re not clear on what the data means, slowing down and having a relaxed conversation with consumers can clear up most things.
-Sandy Clear, President