14 Nov 2012
At Allegiance, we are often asked, “Which scale should I use on my customer experience study?” In this three part series, we will look at some of the analytical differences between scales that can be attributed to only the scale itself.
Allegiance clients use a variety of rating scales for different reasons. In some cases a simple yes/no, 2-point scale is appropriate. Other clients are interested in an 11-point scale to stay consistent with the Net Promoter Score methodology. Having been in this business for a long time, I can honestly say that I have seen every scale in between.
As part of our broader effort to maintain leadership in the customer intelligence space, Allegiance conducts monthly tracking studies in a variety of industries to gauge the quality of customer experiences on a macro-level. Our tracking studies provide us with the opportunity to see outside of our client’s efforts. Furthermore, we take the opportunity with every tracking study to add experimental questions and test innovative approaches and hypotheses.
Using our tracking study, we took the opportunity to ask the same question twice, using different scales. That is, we asked the standard Likelihood to Recommend question using the standard 0-10 scale. And earlier in the survey, we asked the same question using a 5 category scale with answers labeled ‘Very unlikely’ through ‘Very likely.’ All respondents answered both questions, one question at the beginning of the survey and one at the end.
For some clients, the average score is the primary measurement of interest. When using a statistical technique like regression, the average score is also important as a baseline for opportunities to make an impact. The chart below shows a time-trend chart of the average scores over 34 months. The lines are plotted on two axes to ensure the scales maintain relative positioning. We assigned the 5pt scale to the numbers 1-5.
In this case, seeing is believing. The two scales maintain a high degree of consistency over the entire period. Peaks match with peaks and troughs match with troughs. I have to admit; even I was surprised at the remarkable degree of parallelism when I saw the two lines overlaid.
Cohen1 gives the following guidelines for the effect size of Pearson Correlation in the social sciences. The correlation between the two scales over time is 0.78, a large effect. In other words, they are highly correlated. In summary, average scores are remarkably similar between 5pt and 11pt scales over time. In fact, a clever analyst could easily translate the average scores between the two by using a simple formula. I’ll leave it as an exercise to the reader to figure out the formula for this one. Stay tuned for more to come in the next installment.
1 Jacob Cohen (1988) Statistical Power Analysis for the Behavioral Sciences (second ed.). Lawrence Erlbaum Associates. Kyle LaMalfa, Best Practices Manager and Loyalty Expert, Allegiance