Allegiance Blog

We all know the “oh man, am I stuffed” feeling we get after eating our favorite dinner dish.  And when we send out a survey, we hope to get back the data that gives us rich, tasty and analyzable answers.  The problem is, if we don’t properly cook the survey, no will want to finish what’s on their plate.  It just tastes terrible!

Needless to say, the goal of any survey builder or analyst is to get respondents to clean their plate or, in this case, finish the conversation. After all, isn’t a survey just a conversation?  You ask engaging questions and expect the respondent to answer with the intent of giving you a full, satisfying response. Here are some basic best practices for keeping respondents engaged in the dinner/survey conversation:

1. Keep it as short as possible

Your survey needs to capture the information you need. But the longer you stretch it out, the higher the risk of the “I’m tired of this survey, I’m done” syndrome, more commonly known as “drop out.” 

2. Warm them up

Most of us aren’t able to dive into a Shakespearean play the moment we step out of bed in the morning.  Don’t dive into the difficult questions right off the bat.  Start by taking them through the experience you are surveying them for.   Get them thinking about their interaction with you.

3. Keep your questions brief and clear

No one likes to have to read a survey question twice because it’s too long or because they don’t understand what you’re asking.  Avoid using compound and ambiguous questions. “Peter Piper picked a peck of pickled peppers; A peck of pickled peppers Peter Piper picked; If Peter Piper picked a peck of pickled peppers, where’s the peck of pickled peppers Peter Piper picked?”  Um, yeah, not a good survey question.

4. Keep your list of answer options short

A respondent can experience fatigue not just by a marathon-like survey, but also by too many answer options to select from in one question.  Too many options can cause a respondent to select just the first or last option and not read your entire list of options. 

5. Keep it interesting

Though data collection is serious business, and you need to ensure you are gathering valuable, actionable information, remember that you’ll get better data if your respondent enjoys taking your survey.  That doesn’t mean you have to “fluff it up,” but make sure it’s relevant to your respondent.

6. And finally. . . . be considerate, keep your respondent informed

Tell them why they are taking the survey, what you discovered and what kind of actions you’re going to take with the results of the analysis.
 
While this article has a bit of a lighter side to it, I hope it conveys the seriousness of creating a good survey.  Contemplate and plan a better survey to keep your respondents participating in the conversation.

Have you ever noticed that more people start your online surveys than finish them? It’s like they start clicking through the answers and then suddenly they have a heart attack and it’s curtains!

What is happening to those hapless survey takers?

Life insurance actuaries, the Census Bureau, and the Social Security Administration use a tool to understand mortality: life tables. Life tables measure the odds of dying at a particular age. Age 0 to 1 has a higher risk of death than age 1 to 5. Past age 5, our odds of death steadily climb.

One might assume that a similar pattern would be true for online surveys, and that the odds of abandoning a survey increase with each increasing page. So, I decided to check it out. Do survey abandonment curves match similar patterns to human mortality tables? I used one of Allegiance’s client surveys to examine the trends.

In the chart below I used one axis to plot the number of survey takers who had completed each page. On the secondary axis, the red line, I plotted the odds of abandonment. The odds of abandonment is the number of survey takers lost to page x+1 divided by the survey takers remaining at page x.

Turns out, survey abandonment curves are not quite as smooth as human mortality curves. Survey abandonment curves are spiky. It’s like the Grim Reaper inhabits certain pages but not others.

Clearly, one can see where the problem pages are, where abandonment spikes. It turns out that this survey has a bunch of open-ended questions on pages 2, 3 and 10. Answering open-ended questions is a lot of work. Rather than work, people abandon the survey altogether.

There is no way this client will get rid of their open-ended questions. There is nothing I can do to make the questions less prone to abandonment. However, what we can do is re-order the pages and put the low-abandonment pages at the front of the survey and the high-abandonment pages at the back. The thought is that with more survey takers sticking around for longer, we may get to gather more data.

We can measure the increased data the same way the census bureau measures “life-years.” I’ll call this measurement “page-completes.” If we accumulate more page-completes by the end of the survey then we’ve made an improvement. Below is a simulation of re-ordering the pages to put the high-abandonment pages at the end.

Sure, the same number of respondents made it to the end of the survey, 1910. However by re-ordering the survey, the page-completes grow from 28,828 before to 32,319 after. Re-ordering may help me capture more survey data overall.

Kyle LaMalfa, Best Practices Manager and Loyalty Expert, Allegiance

Without getting into deep research, it seems to me that the average marketing manager should be able to put together a sensible survey simply by using some common sense. Somehow, this is not happening as often as I would expect. My speculation is that people are so hungry for feedback on so many items that they can’t resist asking their customers for feedback on all of them. The result of this is that survey abandonment goes up proportionate to the length of the survey and the demographic of the recipient.  And, you wind up with skewed results, since certain classes of respondents, as a group, are more inclined to abandon than others. Here’s a recent experience of mine to make my point.

I recently stayed at a hotel in southern Utah. Two days after my stay, I received a survey request from them. I like this particular hotel chain, so I had no problem opening the survey and giving them my feedback. They had a nice little progress bar on the screen so I knew exactly how far into the survey I had gone. After three or four pages of multiple ranking pages, however, I was still only 40% complete. The next page had 15 ranking questions on everything from their toiletries, to the beds, to the TVs, etc. I bailed out of the survey.

There are two issues in how the hotel should have designed their survey:

1.  The hotel knew who I was and from my profile, should already have known whether or not I was a frequent business traveler or a pleasure/family traveler. Knowing that, they should realize that getting frequent survey responses from me would be very valuable to their business, but also knowing that I am a business person with very little time, they should ask, at most, no more than 5 questions. They could have easily asked me 5 questions out of a set of 20 and by doing this randomly across all their business travelers, still have received the feedback they desired. This would especially be true since their abandonment rate would probably drop by a factor of two or three.

2.  Even if the hotel did not know I was a business traveler, they still should have done the same process outlined above because hotel stays are generally a repeated service. This means unlike, for instance, a car purchase, you are likely to repeat business with them more often than once every few years. Common sense says that recipients of surveys who have made higher dollar, more infrequent purchases will be more likely to tolerate a longer survey. If you are a provider of a more frequent service, you want to design a survey that is quick and easy for the recipient to take so that you will get feedback EVERY time you deliver that service.

So, use common sense when surveying. Understand your recipient. Spread the feedback items across the audience, especially when the sample size and frequencies are high. Know your key goals and cut questions that are not absolutely necessary to meet them.

Terence Fugazzi, VP Demand Marketing, Allegiance