You know the expression “Garbage In, Garbage Out?” Well, that applies to marketing research as well. One of the key pieces in any research project is the questionnaire. Whether it is in the form of a survey, a moderator’s guide or a one-on-one interview, the questions you ask are a critical success factor for developing insights and delivering meaningful, actionable results. So, if you aren’t asking the right questions, you won’t be able to meet the research objectives effectively.
David F. Harris, author of the book The Complete Guide to Writing Questionnaires: How to get Better Information for Better Decisions, has led multiple workshops at L&E where he shared his concerns about questionnaire quality and offered tips to attendees to improve their research quality.
1. KEEP IT SHORT: Longer surveys lead to incompletes, higher incentives, and lower response rates. And yet, marketing researchers routinely design surveys that take 20, 30, or even 45 minutes to complete. Focus on what’s needed to make the decision and limit your questions to gathering only that information.
2. KEEP IT SIMPLE: Surveys that are too complex are almost as bad as surveys that are too long. Avoid questions that are too wordy, response lists with too many choices, and matrix questions with too many entries. All of these make the survey more difficult to complete, and result in questionable data.
3. ASK QUESTIONS THAT YOUR RESPONDENTS CAN ANSWER: It is critical that the questions asked are within the expertise of the respondent. For example, don’t try to ask someone why they bought a product if they were not the decision maker. Either adjust your questions to what the respondent knows or change your sampling.
4. DO QUALITATIVE RESEARCH TO HEAR THE VOC: Qualitative research is invaluable in determining the correct questions to ask of a population. Not only do you learn what language they are comfortable using, but you can explore what is important about a particular experience, as well the metrics that they use to evaluate the experience. Businesses often resist conducting qualitative to prepare for quantitative due to the added expense and time, but it is far better to invest in qualitative research than to conduct a quantitative project with bad results.
5. AVOID LEADING QUESTIONS: Several years ago, a Pew Research survey asked people whether they would: “favor or oppose taking military action in Iraq to end Saddam Hussein’s rule,” 68% were in favor and 25% were opposed to military action. However, when the question was asked “favor or oppose taking military action in Iraq to end Saddam Hussein’s rule even if it meant that U.S. forces might suffer thousands of casualties,” the results changed: only 43% said they favored military action while 48% said they opposed it. By adding the possibility of U.S. casualties, the authors influenced the outcome of the results.
6. AVOID DOUBLE NEGATIVES: You wouldn’t never do this, right? You would? You would what? See the problem? Double negatives are confusing. Just don’t do it!
7. OPEN OR CLOSED? IT DEPENDS: Closed questions (those where the scale is spelled out) are usually easier for respondents to answer, but if you don’t have the right response categories – and all of them – you might miss important data. Moreover, don’t make the mistake of assuming everyone will write in their responses in the “other” response – they just don’t. So, if you don’t know what they respondents might say, go with open-end questions.
8. SHOULD YOU FORCE A CHOICE? THE ROLE OF NO OPINION: Forcing a choice means just that – developing a question and response that forces the respondent to have an opinion. Here’s an example: Are you for or against gun control? Yes or No? Gun control is a very complex issue, and all respondents might not be prepared to answer yes or no. In some cases, it is better to include a “no opinion” or “don’t know” response because it allows for more accurate data.
9. NUMERIC ANSWERS – TO LIST OR NOT TO LIST: It is always easier for respondents to select an answer from a list, but you have to be sure you can develop a list with the correct responses in the correct base. If you are not sure, just collect the numeric data.
10. MAINTAIN LOGICAL ORDER: There are two issues addressed by maintaining logical question order. Again, it makes it easier for respondents to complete the survey. Second, it helps you collect more accurate data. If you want to ask about headaches in the last 30 days as well as the most recent headache experienced, ask all of your questions related to headaches in the past thirty days and then the questions about the most recent headache. Bouncing back and forth between the two time periods may confuse your respondents!
11. ASK ONE QUESTION AT A TIME: How would you answer this question:
› Is orange juice nutritious and delicious? Yes or No
What do you answer if you think it is nutritious, but you don’t like the taste of orange juice? Ask the questions separately: Is orange juice nutritious? Is orange juice delicious?
12. USE FORMATTING TO IMPROVE READABILITY: Italics, bold, underlines – anything you can do to make the survey easier to understand will make it a better experience for respondents and help them give you more accurate responses.
13. WHEN ASKING BEHAVIOR FREQUENCY, SPELL IT OUT: How would you answer this question:
› How frequently do you shop for groceries: Very Frequently, Frequently, Infrequently, or Very Infrequently?
Betty goes grocery shopping once per week, and that seems to be less than her friends, so she responds “very infrequently.” Joe, on the other hand, also goes shopping once per week, but he lives alone and feels that he should be able to go less frequently, so he responds “very frequently”. Same frequency, different response. Be sure respondents are going to be able to respond with accurate metrics.
14. USE BALANCED SCALES: One of the most frequently used scales is “Excellent, Very Good, Good, Fair, Poor.” This scale might work very well in grading students, but it doesn’t work as well for marketing research because it is not balanced: there are three positive responses, one so-so response, and one negative response. This scale biases the responses to the positive. A better scale would be “Very Good, Good, Fair, Poor, Very Poor” which delivers balanced response options.
15. PRETEST THE QUESTIONNAIRE: Before you field the survey with the total sample, conduct a pretest with qualified respondents. Have them take the survey and then ask them how they answered, how they felt as they answered, and how they might improve the survey. Even if you are an expert on the topic, this is the only way to begin to make sure the questions work.
Developing strong questions is both an art and a science. There is a lot of academic research on question development that you can refer to, including Pew Research, the University of Chicago – National Opinion Research Center (NORC), the American Association of Public Opinion Research (AAPOR), and many others. In addition to becoming a student of questions yourself, following these suggestions will go a long way to improving your questions – and the resulting answers.
Want to learn more about how to ask better questions? Sign up for our upcoming webinar on May 13, 2016 with David Harris, How to Write Better Screeners to Get the Right Respondents.
You can also view videos of David’s past workshops by visiting our webinars page of our website.