Questionnaire Design and Scale Development
Chapter 5 QUESTIONNAIRE DESIGN AND SCALE DEVELOPMENT
Naresh K. Malhotra, Georgia Institute of Technology
Introduction
This chapter describes the importance of a questionnaire and presents the process for developing questionnaires and observational forms. Guidelines for questionnaire construction are provided at each stage of the process. In addition, commonly used scaling techniques and use of questionnaires in experimentation are discussed and the design of observational forms is presented.1
What Is a Questionnaire?
A questionnaire is a formalized set of questions for obtaining information from respondents. The overriding objective is to translate the researcher’s information needs into a set of specific questions that respondents are willing and able to answer. While this may seem straightforward, questions may yield very different and unanticipated responses. For example, how would you answer the following question: "Which State is larger, California or Texas?" Would you answer based on population or area?
Why Is a Questionnaire Important?
A questionnaire is the main means of collecting quantitative primary data. A questionnaire enables quantitative data to be collected in a standardized way so that the data are internally consistent and coherent for analysis. Imagine how difficult it would be to analyze the data of a national survey conducted by 40 different interviewers if the questions had not been asked in a standard way, that is, if the interviewers had asked different questions using different
176
176
wording and order. A questionnaire ensures standardization and comparability of the data across interviewers, increases speed and accuracy of recording, and facilitates data processing.
Questionnaire Design Process
No scientific principles guarantee an optimal or ideal questionnaire. Questionnaire design is as much an art as it is a science. The creativity, skill, and experience of the researcher play a major role in the end design. However, several guidelines are available to assist researchers in the questionnaire development process and to help them avoid major mistakes. The guidelines to support questionnaire design are shown as a series of steps.
What Information Is Needed?
The first step in questionnaire design is to specify the information needed. A continual review of the earlier stages of the research project, particularly the specific components of the problem, the research questions, and the hypotheses, will help keep the questionnaire focused. Questionnaires should also be designed with the target respondents in mind, taking into account their educational level and experience. The language used and the context of the questions must all be familiar to the respondents. Questions that are appropriate for college students may not be appropriate for those with only a high school education. Questionnaires that fail to keep in mind the characteristics of the respondents, particularly their educational level and experience, lead to a high incidence of “uncertain” or “no opinion” responses.
How Should Individual Questions Be Framed?
The researcher must determine what should be included in each question. This involves a determination of whether a question is necessary and whether more than one question is needed to obtain the information in an unambiguous way.
177
177
Is the Question Necessary?
Before including a question, the researcher should ask, "How will I use these data?" Questions that may be nice to know but that don 't directly address the research problem should be eliminated. There are exceptions to this rule. Filler questions may be added to disguise the purpose or sponsorship of the project. For example, in brand studies, a researcher may include questions about the full range of competing brands so that the respondents won 't know who is sponsoring the study. Early in the interviewing process when the researcher is attempting to build a relationship with respondents and capture their attention, a few easy-to-answer neutral questions may be helpful. At times, certain questions may also be repeated for the purpose of assessing reliability or validity.
Are Several Questions Needed Instead of One?
In some cases, two questions are better than one. However, asking two questions in one is not the solution. Consider the following question:
“Do you think JC Penney offers better variety and prices than other department stores stores? ' ' (Incorrect)
A “yes” answer will presumably be clear, but what if the answer is “no”? Does this mean that the respondent thinks that JC Penney does not offer better variety, that it does not offer better prices, or that it neither offers better variety nor better prices? Such a question is called a doublebarreled question because two or more questions are combined into one. To avoid confusion, these questions should be asked separately,
178
178
“Do you think JC Penney offers better variety than other department stores? ' ' “Do you think JC Penney offers better prices than other department stores? ' ' (Correct)
Are the Respondents Able to Answer the Question?
Respondents may not always be able to answer the questions posed to them. Researchers can help them overcome this limitation by keeping in mind the reasons people typically cannot answer a question: They may not be informed or they may not remember. Respondents are often asked to answer questions they are uninformed about. A husband may not be informed about monthly expenses for groceries and department store purchases if it is the wife who makes these purchases, or vice versa. Despite the fact that they are uninformed, respondents may provide answers but those answers will be misleading. If it is suspected that many respondents may be uninformed about the topic, “don 't know” should be added to the list of response alternatives. This option has been found to reduce the number of uninformed responses without reducing overall response rate. Many common experiences or practices are difficult to remember. Can you remember the brand name of the shirt you are wearing, what you had for lunch a week ago, or what you were doing a month ago today? Further, do you know how many gallons of soft drinks you consumed during the last four weeks? When making estimates about product consumption levels, in particular, research has found that consumers dramatically overestimate usage. Questions can be designed to aid recall or they can be unaided, depending on the research objectives. For example, unaided recall of cereal commercials could be measured by questions such as, “What brands of cereals do you remember being advertised last night on TV?” A question that employs aided recall attempts to stimulate the respondent 's memory by providing cues related to the event
179
179
of interest. The aided recall approach would list a number of cereal brands and then ask, “Which of these brands were advertised last night on TV?” One of the risks of presenting cues is that they may bias responses and make a respondent unduly sensitive to a topic, thus distorting their answers.
Are the Respondents Willing to Answer the Question?
Even if respondents are able to answer a particular question, they may be unwilling to do so. Refusal to answer a question may be due to a variety of circumstances. The respondent may feel there 's simply too much effort involved, or that the information requested is too sensitive. While most individuals are willing to participate in a survey, this sense of cooperation may vanish if the questions require too much effort to answer. Suppose the researcher is interested in determining from which departments in a store the respondent purchased merchandise on the most recent shopping trip. This information can be obtained in at least two ways. The researcher could ask the respondent to list all the items purchased on her most recent shopping trip, or the researcher could provide a list of departments and ask the respondent to check the applicable ones.
Please list all the departments from which you purchased merchandise on your most recent shopping trip to a department store. (Incorrect)
In the list that follows, please check all the departments from which you purchased merchandise on your most recent shopping trip to a department store. 1. 2. Women 's dresses Men 's apparel ____ ____
180
180
3. 4. . . . 16. 17.
Children 's apparel Cosmetics
____ ____
Jewelry Other (please specify)
____ ____ (Correct)
The second option is preferable because it requires less effort from respondents.
Information of a personal or highly sensitive nature may be difficult to obtain from respondents. Examples of sensitive topics include money, family life, political and religious beliefs, and involvement in accidents or crimes. The respondents may be embarrassed to answer such questions because accurate responses may threaten their prestige or self-image. To increase the likelihood of obtaining sensitive information, such topics should be placed at the end of the questionnaire. By then, rapport has been created and legitimacy of the project established, making respondents more willing to give information. Where appropriate, sensitive information should be obtained in the form of response categories rather than asking for specific figures. While respondents may refuse to answer the question
What is your household 's annual income? ________________________________ (Incorrect)
181
181
They may be willing to check the appropriate income category. A better way of obtaining information on income is to ask:
Which one of the following categories best describes your household’s annual income? under $25,000 __ __ __ $25,001-$50,000 $50,001-$75,000 over $75,000
What Should Be the Structure of the Question?
A question may be unstructured or structured. In the following sections, we define unstructured questions and discuss their advantages and limitations. This is followed by a discussion of the popular forms of structured questions: multiple-choice, dichotomous, and scales.
Unstructured Questions
Unstructured questions are open-ended questions that respondents answer in their own words. They are also referred to as free-response or free-answer questions. Open-ended questions are good as first questions on a topic. They enable the respondents to express general attitudes and opinions that can help the researcher interpret their responses to structured questions. Open-ended questions allow the respondent to express their attitudes or opinions without the bias associated with restricting responses to predefined alternatives. Thus, they can be useful in identifying underlying, motivations, beliefs, and attitudes. Analysis of the verbatim
182
182
comments provides a rich context for interpreting later questions. Unstructured questions are useful in exploratory research. The disadvantages of unstructured questions relate to recording error, data coding, and the added complexity of analysis. In personal or telephone interviews, successfully recording verbatim comments depends entirely on the recording skills of the interviewer. Interviewer bias is introduced as decisions are made regarding whether to record answers verbatim or write down only the main points. Tape recorders should be used if verbatim reporting is important. In general, open-ended questions are useful in exploratory research and as opening questions. However, in a large survey, the complexity of recording, tabulation, and analysis outweighs their advantages.
Structured Questions
Structured questions specify the set of responses as well as their format. A structured question may offer multiple-choices, or a scale.
Multiple-Choice Questions
In multiple-choice questions, the researcher provides a choice of answers, and respondents are asked to select one or more of the alternatives given. Consider the following question: Do you intend to travel overseas within the next six months? _____ Definitely will not travel _____ Probably will not travel _____ Undecided _____ Probably will travel _____ Definitely will travel
183
183
Two concerns in designing multiple-choice questions are (1) the number of alternatives that should be included and (2) order or position bias. Multiple choice questions should include choices that cover the full range of possible alternatives. The alternatives should be mutually exclusive and collectively exhaustive. An “other (please specify)” category should be included where appropriate. Instructions should clearly indicate whether the respondent is to choose only one alternative or select all that apply. (For example, “Please indicate all the brands of cereals that you have consumed in the past week.”) As the list of choices increases, the questions become more difficult to answer. When the alternative list becomes long, the researcher should consider using more than one question to simplify the workload for respondents. Order or position bias is the respondents ' tendency to check an alternative merely because it occupies a certain position in a list. Alternatives that appear at the beginning and, to a lesser degree, at the end of a list have a tendency to be selected most often. When questions relate to numeric values (quantities or prices), there is a tendency to select the central value on the list. Order bias can be controlled by preparing several forms of the questionnaire with changes in the order of the alternatives from form to form. Unless the alternatives represent ordered categories, each alternative should appear once in each of the extreme positions, once in the middle, and once somewhere in between. Multiple choice questions are easier for respondents to answer. They are also easier to analyze and tabulate than open-ended questions. Interviewer bias is also reduced, given that these types of questions work very well in self-administered conditions. Respondent cooperation in general is improved if the majority of the questions are structured. Multiple-choice questions are not without disadvantages. It is difficult to develop effective multiple-choice options. Often, exploratory research must be conducted using open184
184
ended questions to identify the appropriate response options. When large numbers of respondents check the “other (please specify)” category, it indicates that the alternative list may be seriously flawed. The list of options itself also introduces bias. Given their importance, scales are discussed in the next section.
What Type of Scales Should Be Used?
An itemized rating scale has a number or a brief description associated with each response category. The categories are typically arranged in some logical order, and the respondents are required to select the categories that best describe their reactions to whatever is being rated. Itemized rating scales are the most widely used scales in marketing. We will briefly describe the use of Likert scales. For a discussion of other type of scales such as semantic differential and Stapel scales please refer to Malhotra (2004) and Malhotra and Peterson (2006).
Likert Scale
Named after its developer, Rensis Likert, the Likert scale is one of the most widely used itemized scales. The end-points of a Likert scale are typically “strongly disagree” and “strongly agree.” The respondents are asked to indicate their degree of agreement by checking one of five response categories. The following example shows how a Likert scale was used in a retailing study.
Strongly disagree
Disagree
Neither agree nor disagree
Agree Strongly agree
1. JC Penney sells high quality merchandise. 2. JC Penney has poor in-store
185
1
2X
3
4
5
185
service. 3. I like to shop at JC Penney.
1 1
2X 2
3 3X
4 4
5 5
The data are typically treated as interval scale. When using this approach to determine the total score for each respondent on each store, it is important to use a consistent scoring procedure so that a high (or low) score consistently reflects a favorable response. This requires that the categories assigned to the negative statements by the respondents be scored by reversing the scale. Note that for a negative statement, an agreement reflects an unfavorable response, whereas for a positive statement, agreement represents a favorable response. Accordingly, a “strongly agree” response to a favorable statement and a “strongly disagree” response to an unfavorable statement would both receive scores of five.
In the scale shown above, if a higher score is to denote a more favorable attitude, the scoring of item 2 will be reversed while analyzing the data. Each respondent 's total score for each store is calculated. A respondent will have the most favorable attitude toward the store with the highest score. The Likert scale has several advantages. It is easy for the researcher to construct and administer this scale, and it is easy for the respondent to understand. Therefore, it is suitable for mail, telephone, personal, or electronic interviews. Several variants of the Likert scale are commonly used in marketing that vary the number of scale points (for example, 7 or 9 points) as well as the descriptors (for example, importance, familiarity) and other characteristics. The major disadvantage of the Likert scale is that it takes longer to complete than other itemized rating scales. Respondents have to read the entire statement rather than a short phrase.
186
186
Likert Scale Decisions The researcher must make six major decisions when constructing Likert type of scales. 1. The number of scale categories to use 2. Balanced versus unbalanced scale 3. Odd or even number of categories 4. Forced versus nonforced choice
Number of Scale Categories
Two conflicting considerations are involved in deciding the number of scale categories. The greater the number of scale categories, the finer the discrimination among stimulus objects that is possible. On the other hand, most respondents cannot handle more than a few categories. Traditional guidelines suggest that the appropriate number of categories should be seven plus or minus two: between five and nine. Yet, there is no single optimal number of categories.
Balanced Versus Unbalanced Scales
In a balanced scale, the number of favorable and unfavorable categories is equal; in an unbalanced scale, the number is unequal. In general, the scale should be balanced in order to obtain objective data. However, if the distribution of responses is likely to be skewed, either positively or negatively, an unbalanced scale with more categories in the direction of skewness may be appropriate. If an unbalanced scale is used, the nature and degree of unbalance in the scale should be taken into account in data analysis.
Odd or Even Number of Categories
With an odd number of categories, the middle scale position is generally designated as neutral or impartial. The presence, position, and labeling of a neutral category can have a significant influence on the response. The Likert scale is a balanced rating scale with an odd
187
187
number of categories and a neutral point. The decision to use an odd or even number of categories depends on whether some of the respondents may be neutral on the response being measured. If a neutral or indifferent response is possible from at least some of the respondents, an odd number of categories should be used.
Forced Versus Nonforced Scales
On forced rating scales, the respondents are forced to express an opinion, because a “no opinion” option is not provided. In such a case, respondents without an opinion may mark the middle scale position. If a sufficient proportion of the respondents do not have opinions on the topic, marking the middle position will distort measures of central tendency and variance. In situations where the respondents are expected to have no opinion, as opposed to simply being reluctant to disclose it, the accuracy of data may be improved by a nonforced scale that includes a “no opinion” category.
How Should the Question Be Worded?
Translating the information needed into clearly worded questions that are easily understood is the most difficult aspect of questionnaire development. Poorly worded questions can confuse or mislead respondents, leading to nonresponse or response error. Poorly worded questions can also frustrate the respondents to the point that they refuse to answer those questions or items. This is referred to as item nonresponse and leads to nonresponse error. If respondents interpret questions differently than intended by the researcher, serious bias can occur, leading to response error. To avoid problems in question wording, we offer five guidelines: (1) define the issue, (2) use ordinary words, (3) avoid ambiguous words, (4) avoid leading questions, and (5) use positive and negative statements.
188
188
Define the Issue
Questions should always clearly define the issue being addressed. Beginning journalists are told to define the issue in terms of who, what, when, where, why, and way (the six Ws). These, particularly, who, what, when, and where, can also serve as guidelines for defining the issue in a question. Consider the following question:
Which brand of bath soap do you use? (Incorrect)
On the surface, this may seem to be a well-defined question, but we may reach a different conclusion when we examine it under the microscope of who, what, when, and where.
The W 's Who
Defining the Question The Respondent It is not clear whether this question relates to the individual respondent or the respondent 's total household.
What
The Brand of Bath soap It is unclear how the respondent is to answer this question if more than one brand is used.
189
189
When
Unclear The time frame is not specified in this question. The respondent could interpret it as meaning the bath soap used this morning, this week or over the past year.
Where
At home, at the gym, on the road?
A more clearly defined question might read: Which brand or brands of bath soap have you personally used at home during the past month? In case of more than one brand, please list all the brands that apply. (Correct)
Use Simple Words
Simple, ordinary words that match the vocabulary level of the respondent should be used in a questionnaire. When choosing words, keep in mind that the average person in the United States has a high school education, not a college one. For certain respondent groups, the education level is even lower. Simplicity in wording and a conscious effort to avoid technical jargon should guide questionnaire development. As marketing professionals, it is also important to remember that most respondents do not understand marketing terminology. For example, instead of asking,
Is the distribution of snack foods adequate? (Incorrect) ask,
190
190
Are snack foods readily available when you want to buy them? (Correct)
Use Unambiguous Words
When selecting words for a questionnaire, the questionnaire designer should choose words with only one meaning. This is not an easy task given that a number of words that appear unambiguous can have different meanings to different people. These include usually, normally, frequently, often, regularly, occasionally, and sometimes. Consider the following question:
In a typical month, how often do you go to a movie theater to see a movie? _____ Never _____ Occasionally _____ Sometimes _____ Often _____ Regularly (Incorrect)
The categories of this multiple choice question can have different meanings to different people, leading to response bias. Three respondents who go to movie theaters once a month may check three different categories: occasionally, sometimes, and often. The following is a much better worded question:
191
191
In a typical month, how often do you go to a movie theater to see a movie? _____ Less than once _____ 1 or 2 times _____ 3 or 4 times _____ More than 4 times (Correct)
This question is less ambiguous because each respondent is answering it from a consistent frame of reference. Response categories have been objectively defined, and respondents are no longer free to interpret them in their own way.
Avoid Leading or Biasing Questions
A leading question is one that clues the respondent to what the answer should be, as in the following: Do you think that America should provide financial aid to poor foreign countries when it is not our responsibility to do so? _____ Yes _____ No _____ Don 't know (Incorrect) This question would most likely lead respondents to a “No” answer. The answer would be unduly biased by the phrase “it is not our responsibility to do so.” Therefore, this question would not help determine the preferences of Americans for providing aid to poor foreign countries. A better question would be:
192
192
Do you think that America should provide financial aid to poor foreign countries? _____ Yes _____ No _____ Don 't know (Correct) Words can lead respondents in a particular direction. Identification of the research sponsor can have the same effect. When respondents are made aware of the sponsor, they tend to answer questions about that sponsor in a positive manner. Likewise, the mention of a prestigious or nonprestigious name can bias the response, as in, “Do you agree with the American Dental Association that Colgate is effective in preventing cavities?”
Balance Dual Statements
Many questions, particularly those measuring attitudes and lifestyles, are worded as statements to which respondents indicate their degree of agreement or disagreement using Likert scales. The statements in this type of questions can be worded either positively or negatively. Evidence shows that the responses obtained often depend on the direction of the wording of the questions: whether they are stated positively or negatively. Questions of this type should be balanced by using dual statements, some of which are positive and some negative. Two different questionnaires, which reverse the direction of the questions, could also be used to control for any bias introduced by the positive or negative nature of the statements. An example of dual statements was provided in the summated Likert scale that was designed to measure attitudes toward JC Penney.
What Is the Proper Order of Questions?
193
193
When arranging questions in a proper order, the researcher should consider the opening questions, the type of information sought, difficult questions, and the effect on subsequent questions. Questions should be arranged in a logical order, organized around topic areas. Opening questions set the stage for the remainder of the questionnaire. They serve a variety of purposes. They can introduce the topic, attempt to gain the confidence and cooperation of respondents, or establish the legitimacy of the study. The opening questions should be interesting, simple, and nonthreatening. Questions that ask respondents for their opinions are always good openers because most people like to express their opinions. Some studies require a prescreening of the respondents to ensure that they are eligible to participate in the interview. In these cases, qualifying questions are used as opening questions. Three types of information are obtained from a questionnaire: (1) basic information, (2) classification information, and (3) identification information. Basic information relates directly to the research problem. Classification information consists of socioeconomic and demographic characteristics. It is used to classify the respondents in order to analyze results across different groups. Identification information includes name, address, and telephone number. Identification information may be obtained for a variety of purposes, including verifying that the respondents listed were actually interviewed, remitting promised incentives, and so on. Because basic information is the most important aspect of a study, it should be obtained first, followed by classification and then identification information. Classification and identification information is of a more personal nature. Respondents may resist answering a series of personal questions. Therefore, these types of questions should appear at the end of the questionnaire. Questions that could be perceived as difficult should be placed late in the sequence after a relationship has been established and the respondent is involved in the process. The last
194
194
question of the classification section is typically income information; the respondent 's telephone number is the final item in the identification section for the same reasons. Initial questions can influence questions asked later in a questionnaire. As a rule, a series of questions should start with a general introduction to a topic, followed by specific questions related to the topic. This prevents specific questions from biasing responses to the general questions. Going from general to specific is called the funnel approach, because you begin with broader (more general) questions and then ask narrower (more specific) questions, reflecting the shape of a funnel.2 Although the funnel approach is more commonly used, sometimes the inverted funnel approach may be used when the respondents do not have clearly formulated views about a topic or when they lack a common frame of reference in responding to general questions on the topic. In this approach, the specific questions are asked first followed by more general questions. Questions should be asked in a logical order, organized around topic areas. When switching topics, brief transitional phrases or sentences should be used to help respondents switch their train of thought, for example, “In this section, we ask questions related to your purchase of a new car in the last six months.”
How Should the Questionnaire Be Pretest?
Pretesting refers to testing the questionnaire on a small sample of respondents, usually 15 to 30, to identify and eliminate potential problems. Even the best questionnaire can be improved by pretesting. As a general rule, a questionnaire should not be used in the field study without extensive pretesting. All aspects of the questionnaire, including question content, wording, sequence, form and layout, question difficulty, and instructions should be tested. Additionally, pretesting should be conducted with a subset of the respondent group. The pretest groups should
195
195
be similar to the respondents in terms of their background characteristics, familiarity with the topic, and attitudes and behaviors of interest (see Diamantopoulos, Schlegelmilch, & Reynolds, 1994; Martin & Polivka, 1995). Pretests are best done by personal interviews, even if the actual survey is to be conducted by telephone, mail, or electronically, so that interviewers can observe respondent reactions and attitudes. After the necessary changes have been made, another pretest could be administered using the actual data collection approach, if it is mail, telephone, or electronic. This stage of the pretest will reveal any potential problems in the interviewing method to be used in the actual survey. The pretest should be conducted in an environment and context similar to that of the actual survey. Based on feedback from the pretest, the questionnaire should be edited, and the identified problems corrected. After each significant revision of the questionnaire, another pretest should be conducted, using a different sample of respondents. Pretesting should be continued until no further changes are needed. As a final step, the responses obtained during the pretest should be coded and analyzed. The analysis of pretest responses can serve as a check on the adequacy of the problem definition, and provide insight into the nature of the data as well as analytic techniques that will be required.
Design of Questionnaires for Experimentation
While the most common use of questionnaires is for survey research, questionnaires are also used for experimentation. They may be used to measure the variables of interest before and/or after exposure to the experimental treatment such as a television commercial. The design of a questionnaire for experimentation follows the same principles that we have outlined for survey research.
196
196
Design of Observational Forms
Observational forms are designed to record respondent reaction to new products, advertising, packaging, or some other marketing stimuli. Since there is no questioning of the respondents, the researcher need not be concerned with the psychological impact of the questions and the way they are asked. Observational forms are designed primarily for the field work and the tabulation phase, providing a guide for recording information accurately and to simplify coding, entry, and analysis of data.
Questionnaire Design Checklist
While we have discussed the Dos and Don’ts of questionnaire design throughout this chapter illustrated them with incorrect and correct questions, we present a checklist for appropriate questionnaire design. Information Needed 1. 2. Ensure that the information obtained fully addresses all the components of the problem. Have a clear idea of the target population.
Individual Question Content 1. 2. Is the question necessary? Are several questions needed instead of one to obtain the required information in an unambiguous manner? 3. Do not use double-barreled questions.
Overcoming Inability and Unwillingness to Answer 1. Is the respondent informed?
197
197
2.
If respondents are not likely to be informed, filter questions that measure familiarity, product use, and past experience should be asked before questions about the topics themselves.
3. 4.
Can the respondent remember? Questions that do not provide the respondent with cues can underestimate the actual occurrence of an event.
5. 6. 7. 8.
Can the respondent articulate? Minimize the effort required of the respondents. Make the request for information seem legitimate. Is the information sensitive?
Choosing Question Structure 1. 2. 3. Open-ended questions are useful in exploratory research and as opening questions. Use structured questions whenever possible. In multiple-choice questions, the response alternatives should include the set of all possible choices and should be mutually exclusive. 4. In a dichotomous question, if a substantial proportion of the respondents can be expected to be neutral, include a neutral alternative. 5. Consider the use of the split ballot technique to reduce order bias in dichotomous and multiple-choice questions. 6. If the response alternatives are numerous, consider using more than one question.
Choosing Question Wording 1. 2. Define the issue in terms of who, what, when, where, why, and way (the six Ws). Use ordinary words. Words should match the vocabulary level of the respondents.
198
198
3.
Avoid ambiguous words: usually, normally, frequently, often, regularly, occasionally, sometimes, and so on.
4. 5. 6. 7. 8.
Avoid leading questions that clue the respondent to what the answer should be. Avoid implicit alternatives that are not explicitly expressed in the options. Avoid implicit assumptions. Respondent should not have to make generalizations or compute estimates. Use positive and negative statements.
Determine the Order of Questions 1. 2. 3. The opening questions should be interesting, simple, and nonthreatening. Qualifying questions should serve as the opening questions. Basic information should be obtained first, followed by classification, and, finally, identification information. 4. 5. 6. Difficult, sensitive, or complex questions should be placed late in the sequence. General questions should precede specific questions. Questions should be asked in a logical order.
Pretesting 1. 2. Always pretest Test all aspects of the questionnaire, including question content, wording, sequence, form and layout, question difficulty, and instructions. 3. Use respondents in the pretest that are similar to those who will be included in the actual survey. 4. 5. Begin the pretest by using personal interviews. Conduct the pretest by mail, telephone, or electronically if those methods are to be used in the actual survey.
199
199
6. 7.
Use a variety of interviewers for pretests. The pretest sample size should be small, varying from 15 to 30 respondents for the initial testing.
8.
After each significant revision of the questionnaire, conduct another pretest, using a different sample of respondents.
9.
Code and analyze the responses obtained from the pretest.
Summary
To collect quantitative primary data, a researcher must design a questionnaire or an observational form. A questionnaire must translate the information needed into a set of specific questions. Designing a questionnaire is as much an art as it is a science. We can provide guidelines for development, but no one optimal questionnaire design fits every research need. The process begins by specifying the information needed. The next step is to decide on the content of individual questions. Questions must be written to overcome the respondents ' inability to answer. Respondents may be unable to answer if they are not informed or cannot remember. When too much effort is required, respondents will be unwilling to participate. Questions that attempt to collect sensitive information may also be met with resistance. Questions can be unstructured (open-ended) or structured to a varying degree. Structured questions include multiple-choice questions and scales. The commonly used scales are Likert, semantic differential and Stapel. Determining the wording of each question involves defining the issue, using ordinary words, using unambiguous words, and using dual statements. The issue should be clearly defined in terms of who, what, when, and where. The researcher should avoid leading questions. Once the questions have been worded, the order in which they will appear in the questionnaire
200
200
must be decided. Special consideration should be given to opening questions, type of information, difficult questions, and the effect on subsequent questions. The questions should be arranged in a logical order. The effectiveness of all these design decisions must be assessed in a pretest. Questionnaire design is a very important aspect of marketing research and effort devoted to this task can payoff handsomely in terms of the quality and completeness of the data and the results.
ENDNOTES
1
The material presented here is drawn from the work of Malhotra (2004) and Malhotra and
Peterson (2006). For more information, including detailed references, please consult these books.
2
Rating a brand on specific attributes early in a survey may affect responses to a later overall
brand evaluation (for more information on this, see Bartels, 2002; Bickart, 1993).
201
201
REFERENCES Bartels, L. M. (2002). Question order and declining faith in elections. Public Opinion Quarterly, 66(1), 67–79. Bickart, B. A. (1993). Carryover and backfire effects in marketing research. Journal of Marketing Research, 30, 52–62. Diamantopoulos, A., Schlegelmilch, B. B., & Reynolds, N. (1994). Pretesting in questionnaire design: The impact of respondent characteristics on error detection. Journal of the Market Research Society, 36(October), 295–314. Malhotra, N. K. (2004). Marketing research: An applied orientation (4th ed.). Upper Saddle River, NJ: Prentice Hall -------, & Peterson, M. (2006). Basic marketing research: A decision making approach (2d ed.). Upper Saddle River, NJ: Prentice Hall. Martin, E. & Polivka, A. E. (1995). Diagnostics for redesigning survey questionnaires. Public Opinion Quarterly, 59(4), 547–567.
202
202
References: Bartels, L. M. (2002). Question order and declining faith in elections. Public Opinion Quarterly, 66(1), 67–79. Bickart, B. A. (1993). Carryover and backfire effects in marketing research. Journal of Marketing Research, 30, 52–62. Diamantopoulos, A., Schlegelmilch, B. B., & Reynolds, N. (1994). Pretesting in questionnaire design: The impact of respondent characteristics on error detection. Journal of the Market Research Society, 36(October), 295–314. Malhotra, N. K. (2004). Marketing research: An applied orientation (4th ed.). Upper Saddle River, NJ: Prentice Hall -------, & Peterson, M. (2006). Basic marketing research: A decision making approach (2d ed.). Upper Saddle River, NJ: Prentice Hall. Martin, E. & Polivka, A. E. (1995). Diagnostics for redesigning survey questionnaires. Public Opinion Quarterly, 59(4), 547–567. 202 202
Naresh K. Malhotra, Georgia Institute of Technology
Introduction
This chapter describes the importance of a questionnaire and presents the process for developing questionnaires and observational forms. Guidelines for questionnaire construction are provided at each stage of the process. In addition, commonly used scaling techniques and use of questionnaires in experimentation are discussed and the design of observational forms is presented.1
What Is a Questionnaire?
A questionnaire is a formalized set of questions for obtaining information from respondents. The overriding objective is to translate the researcher’s information needs into a set of specific questions that respondents are willing and able to answer. While this may seem straightforward, questions may yield very different and unanticipated responses. For example, how would you answer the following question: "Which State is larger, California or Texas?" Would you answer based on population or area?
Why Is a Questionnaire Important?
A questionnaire is the main means of collecting quantitative primary data. A questionnaire enables quantitative data to be collected in a standardized way so that the data are internally consistent and coherent for analysis. Imagine how difficult it would be to analyze the data of a national survey conducted by 40 different interviewers if the questions had not been asked in a standard way, that is, if the interviewers had asked different questions using different
176
176
wording and order. A questionnaire ensures standardization and comparability of the data across interviewers, increases speed and accuracy of recording, and facilitates data processing.
Questionnaire Design Process
No scientific principles guarantee an optimal or ideal questionnaire. Questionnaire design is as much an art as it is a science. The creativity, skill, and experience of the researcher play a major role in the end design. However, several guidelines are available to assist researchers in the questionnaire development process and to help them avoid major mistakes. The guidelines to support questionnaire design are shown as a series of steps.
What Information Is Needed?
The first step in questionnaire design is to specify the information needed. A continual review of the earlier stages of the research project, particularly the specific components of the problem, the research questions, and the hypotheses, will help keep the questionnaire focused. Questionnaires should also be designed with the target respondents in mind, taking into account their educational level and experience. The language used and the context of the questions must all be familiar to the respondents. Questions that are appropriate for college students may not be appropriate for those with only a high school education. Questionnaires that fail to keep in mind the characteristics of the respondents, particularly their educational level and experience, lead to a high incidence of “uncertain” or “no opinion” responses.
How Should Individual Questions Be Framed?
The researcher must determine what should be included in each question. This involves a determination of whether a question is necessary and whether more than one question is needed to obtain the information in an unambiguous way.
177
177
Is the Question Necessary?
Before including a question, the researcher should ask, "How will I use these data?" Questions that may be nice to know but that don 't directly address the research problem should be eliminated. There are exceptions to this rule. Filler questions may be added to disguise the purpose or sponsorship of the project. For example, in brand studies, a researcher may include questions about the full range of competing brands so that the respondents won 't know who is sponsoring the study. Early in the interviewing process when the researcher is attempting to build a relationship with respondents and capture their attention, a few easy-to-answer neutral questions may be helpful. At times, certain questions may also be repeated for the purpose of assessing reliability or validity.
Are Several Questions Needed Instead of One?
In some cases, two questions are better than one. However, asking two questions in one is not the solution. Consider the following question:
“Do you think JC Penney offers better variety and prices than other department stores stores? ' ' (Incorrect)
A “yes” answer will presumably be clear, but what if the answer is “no”? Does this mean that the respondent thinks that JC Penney does not offer better variety, that it does not offer better prices, or that it neither offers better variety nor better prices? Such a question is called a doublebarreled question because two or more questions are combined into one. To avoid confusion, these questions should be asked separately,
178
178
“Do you think JC Penney offers better variety than other department stores? ' ' “Do you think JC Penney offers better prices than other department stores? ' ' (Correct)
Are the Respondents Able to Answer the Question?
Respondents may not always be able to answer the questions posed to them. Researchers can help them overcome this limitation by keeping in mind the reasons people typically cannot answer a question: They may not be informed or they may not remember. Respondents are often asked to answer questions they are uninformed about. A husband may not be informed about monthly expenses for groceries and department store purchases if it is the wife who makes these purchases, or vice versa. Despite the fact that they are uninformed, respondents may provide answers but those answers will be misleading. If it is suspected that many respondents may be uninformed about the topic, “don 't know” should be added to the list of response alternatives. This option has been found to reduce the number of uninformed responses without reducing overall response rate. Many common experiences or practices are difficult to remember. Can you remember the brand name of the shirt you are wearing, what you had for lunch a week ago, or what you were doing a month ago today? Further, do you know how many gallons of soft drinks you consumed during the last four weeks? When making estimates about product consumption levels, in particular, research has found that consumers dramatically overestimate usage. Questions can be designed to aid recall or they can be unaided, depending on the research objectives. For example, unaided recall of cereal commercials could be measured by questions such as, “What brands of cereals do you remember being advertised last night on TV?” A question that employs aided recall attempts to stimulate the respondent 's memory by providing cues related to the event
179
179
of interest. The aided recall approach would list a number of cereal brands and then ask, “Which of these brands were advertised last night on TV?” One of the risks of presenting cues is that they may bias responses and make a respondent unduly sensitive to a topic, thus distorting their answers.
Are the Respondents Willing to Answer the Question?
Even if respondents are able to answer a particular question, they may be unwilling to do so. Refusal to answer a question may be due to a variety of circumstances. The respondent may feel there 's simply too much effort involved, or that the information requested is too sensitive. While most individuals are willing to participate in a survey, this sense of cooperation may vanish if the questions require too much effort to answer. Suppose the researcher is interested in determining from which departments in a store the respondent purchased merchandise on the most recent shopping trip. This information can be obtained in at least two ways. The researcher could ask the respondent to list all the items purchased on her most recent shopping trip, or the researcher could provide a list of departments and ask the respondent to check the applicable ones.
Please list all the departments from which you purchased merchandise on your most recent shopping trip to a department store. (Incorrect)
In the list that follows, please check all the departments from which you purchased merchandise on your most recent shopping trip to a department store. 1. 2. Women 's dresses Men 's apparel ____ ____
180
180
3. 4. . . . 16. 17.
Children 's apparel Cosmetics
____ ____
Jewelry Other (please specify)
____ ____ (Correct)
The second option is preferable because it requires less effort from respondents.
Information of a personal or highly sensitive nature may be difficult to obtain from respondents. Examples of sensitive topics include money, family life, political and religious beliefs, and involvement in accidents or crimes. The respondents may be embarrassed to answer such questions because accurate responses may threaten their prestige or self-image. To increase the likelihood of obtaining sensitive information, such topics should be placed at the end of the questionnaire. By then, rapport has been created and legitimacy of the project established, making respondents more willing to give information. Where appropriate, sensitive information should be obtained in the form of response categories rather than asking for specific figures. While respondents may refuse to answer the question
What is your household 's annual income? ________________________________ (Incorrect)
181
181
They may be willing to check the appropriate income category. A better way of obtaining information on income is to ask:
Which one of the following categories best describes your household’s annual income? under $25,000 __ __ __ $25,001-$50,000 $50,001-$75,000 over $75,000
What Should Be the Structure of the Question?
A question may be unstructured or structured. In the following sections, we define unstructured questions and discuss their advantages and limitations. This is followed by a discussion of the popular forms of structured questions: multiple-choice, dichotomous, and scales.
Unstructured Questions
Unstructured questions are open-ended questions that respondents answer in their own words. They are also referred to as free-response or free-answer questions. Open-ended questions are good as first questions on a topic. They enable the respondents to express general attitudes and opinions that can help the researcher interpret their responses to structured questions. Open-ended questions allow the respondent to express their attitudes or opinions without the bias associated with restricting responses to predefined alternatives. Thus, they can be useful in identifying underlying, motivations, beliefs, and attitudes. Analysis of the verbatim
182
182
comments provides a rich context for interpreting later questions. Unstructured questions are useful in exploratory research. The disadvantages of unstructured questions relate to recording error, data coding, and the added complexity of analysis. In personal or telephone interviews, successfully recording verbatim comments depends entirely on the recording skills of the interviewer. Interviewer bias is introduced as decisions are made regarding whether to record answers verbatim or write down only the main points. Tape recorders should be used if verbatim reporting is important. In general, open-ended questions are useful in exploratory research and as opening questions. However, in a large survey, the complexity of recording, tabulation, and analysis outweighs their advantages.
Structured Questions
Structured questions specify the set of responses as well as their format. A structured question may offer multiple-choices, or a scale.
Multiple-Choice Questions
In multiple-choice questions, the researcher provides a choice of answers, and respondents are asked to select one or more of the alternatives given. Consider the following question: Do you intend to travel overseas within the next six months? _____ Definitely will not travel _____ Probably will not travel _____ Undecided _____ Probably will travel _____ Definitely will travel
183
183
Two concerns in designing multiple-choice questions are (1) the number of alternatives that should be included and (2) order or position bias. Multiple choice questions should include choices that cover the full range of possible alternatives. The alternatives should be mutually exclusive and collectively exhaustive. An “other (please specify)” category should be included where appropriate. Instructions should clearly indicate whether the respondent is to choose only one alternative or select all that apply. (For example, “Please indicate all the brands of cereals that you have consumed in the past week.”) As the list of choices increases, the questions become more difficult to answer. When the alternative list becomes long, the researcher should consider using more than one question to simplify the workload for respondents. Order or position bias is the respondents ' tendency to check an alternative merely because it occupies a certain position in a list. Alternatives that appear at the beginning and, to a lesser degree, at the end of a list have a tendency to be selected most often. When questions relate to numeric values (quantities or prices), there is a tendency to select the central value on the list. Order bias can be controlled by preparing several forms of the questionnaire with changes in the order of the alternatives from form to form. Unless the alternatives represent ordered categories, each alternative should appear once in each of the extreme positions, once in the middle, and once somewhere in between. Multiple choice questions are easier for respondents to answer. They are also easier to analyze and tabulate than open-ended questions. Interviewer bias is also reduced, given that these types of questions work very well in self-administered conditions. Respondent cooperation in general is improved if the majority of the questions are structured. Multiple-choice questions are not without disadvantages. It is difficult to develop effective multiple-choice options. Often, exploratory research must be conducted using open184
184
ended questions to identify the appropriate response options. When large numbers of respondents check the “other (please specify)” category, it indicates that the alternative list may be seriously flawed. The list of options itself also introduces bias. Given their importance, scales are discussed in the next section.
What Type of Scales Should Be Used?
An itemized rating scale has a number or a brief description associated with each response category. The categories are typically arranged in some logical order, and the respondents are required to select the categories that best describe their reactions to whatever is being rated. Itemized rating scales are the most widely used scales in marketing. We will briefly describe the use of Likert scales. For a discussion of other type of scales such as semantic differential and Stapel scales please refer to Malhotra (2004) and Malhotra and Peterson (2006).
Likert Scale
Named after its developer, Rensis Likert, the Likert scale is one of the most widely used itemized scales. The end-points of a Likert scale are typically “strongly disagree” and “strongly agree.” The respondents are asked to indicate their degree of agreement by checking one of five response categories. The following example shows how a Likert scale was used in a retailing study.
Strongly disagree
Disagree
Neither agree nor disagree
Agree Strongly agree
1. JC Penney sells high quality merchandise. 2. JC Penney has poor in-store
185
1
2X
3
4
5
185
service. 3. I like to shop at JC Penney.
1 1
2X 2
3 3X
4 4
5 5
The data are typically treated as interval scale. When using this approach to determine the total score for each respondent on each store, it is important to use a consistent scoring procedure so that a high (or low) score consistently reflects a favorable response. This requires that the categories assigned to the negative statements by the respondents be scored by reversing the scale. Note that for a negative statement, an agreement reflects an unfavorable response, whereas for a positive statement, agreement represents a favorable response. Accordingly, a “strongly agree” response to a favorable statement and a “strongly disagree” response to an unfavorable statement would both receive scores of five.
In the scale shown above, if a higher score is to denote a more favorable attitude, the scoring of item 2 will be reversed while analyzing the data. Each respondent 's total score for each store is calculated. A respondent will have the most favorable attitude toward the store with the highest score. The Likert scale has several advantages. It is easy for the researcher to construct and administer this scale, and it is easy for the respondent to understand. Therefore, it is suitable for mail, telephone, personal, or electronic interviews. Several variants of the Likert scale are commonly used in marketing that vary the number of scale points (for example, 7 or 9 points) as well as the descriptors (for example, importance, familiarity) and other characteristics. The major disadvantage of the Likert scale is that it takes longer to complete than other itemized rating scales. Respondents have to read the entire statement rather than a short phrase.
186
186
Likert Scale Decisions The researcher must make six major decisions when constructing Likert type of scales. 1. The number of scale categories to use 2. Balanced versus unbalanced scale 3. Odd or even number of categories 4. Forced versus nonforced choice
Number of Scale Categories
Two conflicting considerations are involved in deciding the number of scale categories. The greater the number of scale categories, the finer the discrimination among stimulus objects that is possible. On the other hand, most respondents cannot handle more than a few categories. Traditional guidelines suggest that the appropriate number of categories should be seven plus or minus two: between five and nine. Yet, there is no single optimal number of categories.
Balanced Versus Unbalanced Scales
In a balanced scale, the number of favorable and unfavorable categories is equal; in an unbalanced scale, the number is unequal. In general, the scale should be balanced in order to obtain objective data. However, if the distribution of responses is likely to be skewed, either positively or negatively, an unbalanced scale with more categories in the direction of skewness may be appropriate. If an unbalanced scale is used, the nature and degree of unbalance in the scale should be taken into account in data analysis.
Odd or Even Number of Categories
With an odd number of categories, the middle scale position is generally designated as neutral or impartial. The presence, position, and labeling of a neutral category can have a significant influence on the response. The Likert scale is a balanced rating scale with an odd
187
187
number of categories and a neutral point. The decision to use an odd or even number of categories depends on whether some of the respondents may be neutral on the response being measured. If a neutral or indifferent response is possible from at least some of the respondents, an odd number of categories should be used.
Forced Versus Nonforced Scales
On forced rating scales, the respondents are forced to express an opinion, because a “no opinion” option is not provided. In such a case, respondents without an opinion may mark the middle scale position. If a sufficient proportion of the respondents do not have opinions on the topic, marking the middle position will distort measures of central tendency and variance. In situations where the respondents are expected to have no opinion, as opposed to simply being reluctant to disclose it, the accuracy of data may be improved by a nonforced scale that includes a “no opinion” category.
How Should the Question Be Worded?
Translating the information needed into clearly worded questions that are easily understood is the most difficult aspect of questionnaire development. Poorly worded questions can confuse or mislead respondents, leading to nonresponse or response error. Poorly worded questions can also frustrate the respondents to the point that they refuse to answer those questions or items. This is referred to as item nonresponse and leads to nonresponse error. If respondents interpret questions differently than intended by the researcher, serious bias can occur, leading to response error. To avoid problems in question wording, we offer five guidelines: (1) define the issue, (2) use ordinary words, (3) avoid ambiguous words, (4) avoid leading questions, and (5) use positive and negative statements.
188
188
Define the Issue
Questions should always clearly define the issue being addressed. Beginning journalists are told to define the issue in terms of who, what, when, where, why, and way (the six Ws). These, particularly, who, what, when, and where, can also serve as guidelines for defining the issue in a question. Consider the following question:
Which brand of bath soap do you use? (Incorrect)
On the surface, this may seem to be a well-defined question, but we may reach a different conclusion when we examine it under the microscope of who, what, when, and where.
The W 's Who
Defining the Question The Respondent It is not clear whether this question relates to the individual respondent or the respondent 's total household.
What
The Brand of Bath soap It is unclear how the respondent is to answer this question if more than one brand is used.
189
189
When
Unclear The time frame is not specified in this question. The respondent could interpret it as meaning the bath soap used this morning, this week or over the past year.
Where
At home, at the gym, on the road?
A more clearly defined question might read: Which brand or brands of bath soap have you personally used at home during the past month? In case of more than one brand, please list all the brands that apply. (Correct)
Use Simple Words
Simple, ordinary words that match the vocabulary level of the respondent should be used in a questionnaire. When choosing words, keep in mind that the average person in the United States has a high school education, not a college one. For certain respondent groups, the education level is even lower. Simplicity in wording and a conscious effort to avoid technical jargon should guide questionnaire development. As marketing professionals, it is also important to remember that most respondents do not understand marketing terminology. For example, instead of asking,
Is the distribution of snack foods adequate? (Incorrect) ask,
190
190
Are snack foods readily available when you want to buy them? (Correct)
Use Unambiguous Words
When selecting words for a questionnaire, the questionnaire designer should choose words with only one meaning. This is not an easy task given that a number of words that appear unambiguous can have different meanings to different people. These include usually, normally, frequently, often, regularly, occasionally, and sometimes. Consider the following question:
In a typical month, how often do you go to a movie theater to see a movie? _____ Never _____ Occasionally _____ Sometimes _____ Often _____ Regularly (Incorrect)
The categories of this multiple choice question can have different meanings to different people, leading to response bias. Three respondents who go to movie theaters once a month may check three different categories: occasionally, sometimes, and often. The following is a much better worded question:
191
191
In a typical month, how often do you go to a movie theater to see a movie? _____ Less than once _____ 1 or 2 times _____ 3 or 4 times _____ More than 4 times (Correct)
This question is less ambiguous because each respondent is answering it from a consistent frame of reference. Response categories have been objectively defined, and respondents are no longer free to interpret them in their own way.
Avoid Leading or Biasing Questions
A leading question is one that clues the respondent to what the answer should be, as in the following: Do you think that America should provide financial aid to poor foreign countries when it is not our responsibility to do so? _____ Yes _____ No _____ Don 't know (Incorrect) This question would most likely lead respondents to a “No” answer. The answer would be unduly biased by the phrase “it is not our responsibility to do so.” Therefore, this question would not help determine the preferences of Americans for providing aid to poor foreign countries. A better question would be:
192
192
Do you think that America should provide financial aid to poor foreign countries? _____ Yes _____ No _____ Don 't know (Correct) Words can lead respondents in a particular direction. Identification of the research sponsor can have the same effect. When respondents are made aware of the sponsor, they tend to answer questions about that sponsor in a positive manner. Likewise, the mention of a prestigious or nonprestigious name can bias the response, as in, “Do you agree with the American Dental Association that Colgate is effective in preventing cavities?”
Balance Dual Statements
Many questions, particularly those measuring attitudes and lifestyles, are worded as statements to which respondents indicate their degree of agreement or disagreement using Likert scales. The statements in this type of questions can be worded either positively or negatively. Evidence shows that the responses obtained often depend on the direction of the wording of the questions: whether they are stated positively or negatively. Questions of this type should be balanced by using dual statements, some of which are positive and some negative. Two different questionnaires, which reverse the direction of the questions, could also be used to control for any bias introduced by the positive or negative nature of the statements. An example of dual statements was provided in the summated Likert scale that was designed to measure attitudes toward JC Penney.
What Is the Proper Order of Questions?
193
193
When arranging questions in a proper order, the researcher should consider the opening questions, the type of information sought, difficult questions, and the effect on subsequent questions. Questions should be arranged in a logical order, organized around topic areas. Opening questions set the stage for the remainder of the questionnaire. They serve a variety of purposes. They can introduce the topic, attempt to gain the confidence and cooperation of respondents, or establish the legitimacy of the study. The opening questions should be interesting, simple, and nonthreatening. Questions that ask respondents for their opinions are always good openers because most people like to express their opinions. Some studies require a prescreening of the respondents to ensure that they are eligible to participate in the interview. In these cases, qualifying questions are used as opening questions. Three types of information are obtained from a questionnaire: (1) basic information, (2) classification information, and (3) identification information. Basic information relates directly to the research problem. Classification information consists of socioeconomic and demographic characteristics. It is used to classify the respondents in order to analyze results across different groups. Identification information includes name, address, and telephone number. Identification information may be obtained for a variety of purposes, including verifying that the respondents listed were actually interviewed, remitting promised incentives, and so on. Because basic information is the most important aspect of a study, it should be obtained first, followed by classification and then identification information. Classification and identification information is of a more personal nature. Respondents may resist answering a series of personal questions. Therefore, these types of questions should appear at the end of the questionnaire. Questions that could be perceived as difficult should be placed late in the sequence after a relationship has been established and the respondent is involved in the process. The last
194
194
question of the classification section is typically income information; the respondent 's telephone number is the final item in the identification section for the same reasons. Initial questions can influence questions asked later in a questionnaire. As a rule, a series of questions should start with a general introduction to a topic, followed by specific questions related to the topic. This prevents specific questions from biasing responses to the general questions. Going from general to specific is called the funnel approach, because you begin with broader (more general) questions and then ask narrower (more specific) questions, reflecting the shape of a funnel.2 Although the funnel approach is more commonly used, sometimes the inverted funnel approach may be used when the respondents do not have clearly formulated views about a topic or when they lack a common frame of reference in responding to general questions on the topic. In this approach, the specific questions are asked first followed by more general questions. Questions should be asked in a logical order, organized around topic areas. When switching topics, brief transitional phrases or sentences should be used to help respondents switch their train of thought, for example, “In this section, we ask questions related to your purchase of a new car in the last six months.”
How Should the Questionnaire Be Pretest?
Pretesting refers to testing the questionnaire on a small sample of respondents, usually 15 to 30, to identify and eliminate potential problems. Even the best questionnaire can be improved by pretesting. As a general rule, a questionnaire should not be used in the field study without extensive pretesting. All aspects of the questionnaire, including question content, wording, sequence, form and layout, question difficulty, and instructions should be tested. Additionally, pretesting should be conducted with a subset of the respondent group. The pretest groups should
195
195
be similar to the respondents in terms of their background characteristics, familiarity with the topic, and attitudes and behaviors of interest (see Diamantopoulos, Schlegelmilch, & Reynolds, 1994; Martin & Polivka, 1995). Pretests are best done by personal interviews, even if the actual survey is to be conducted by telephone, mail, or electronically, so that interviewers can observe respondent reactions and attitudes. After the necessary changes have been made, another pretest could be administered using the actual data collection approach, if it is mail, telephone, or electronic. This stage of the pretest will reveal any potential problems in the interviewing method to be used in the actual survey. The pretest should be conducted in an environment and context similar to that of the actual survey. Based on feedback from the pretest, the questionnaire should be edited, and the identified problems corrected. After each significant revision of the questionnaire, another pretest should be conducted, using a different sample of respondents. Pretesting should be continued until no further changes are needed. As a final step, the responses obtained during the pretest should be coded and analyzed. The analysis of pretest responses can serve as a check on the adequacy of the problem definition, and provide insight into the nature of the data as well as analytic techniques that will be required.
Design of Questionnaires for Experimentation
While the most common use of questionnaires is for survey research, questionnaires are also used for experimentation. They may be used to measure the variables of interest before and/or after exposure to the experimental treatment such as a television commercial. The design of a questionnaire for experimentation follows the same principles that we have outlined for survey research.
196
196
Design of Observational Forms
Observational forms are designed to record respondent reaction to new products, advertising, packaging, or some other marketing stimuli. Since there is no questioning of the respondents, the researcher need not be concerned with the psychological impact of the questions and the way they are asked. Observational forms are designed primarily for the field work and the tabulation phase, providing a guide for recording information accurately and to simplify coding, entry, and analysis of data.
Questionnaire Design Checklist
While we have discussed the Dos and Don’ts of questionnaire design throughout this chapter illustrated them with incorrect and correct questions, we present a checklist for appropriate questionnaire design. Information Needed 1. 2. Ensure that the information obtained fully addresses all the components of the problem. Have a clear idea of the target population.
Individual Question Content 1. 2. Is the question necessary? Are several questions needed instead of one to obtain the required information in an unambiguous manner? 3. Do not use double-barreled questions.
Overcoming Inability and Unwillingness to Answer 1. Is the respondent informed?
197
197
2.
If respondents are not likely to be informed, filter questions that measure familiarity, product use, and past experience should be asked before questions about the topics themselves.
3. 4.
Can the respondent remember? Questions that do not provide the respondent with cues can underestimate the actual occurrence of an event.
5. 6. 7. 8.
Can the respondent articulate? Minimize the effort required of the respondents. Make the request for information seem legitimate. Is the information sensitive?
Choosing Question Structure 1. 2. 3. Open-ended questions are useful in exploratory research and as opening questions. Use structured questions whenever possible. In multiple-choice questions, the response alternatives should include the set of all possible choices and should be mutually exclusive. 4. In a dichotomous question, if a substantial proportion of the respondents can be expected to be neutral, include a neutral alternative. 5. Consider the use of the split ballot technique to reduce order bias in dichotomous and multiple-choice questions. 6. If the response alternatives are numerous, consider using more than one question.
Choosing Question Wording 1. 2. Define the issue in terms of who, what, when, where, why, and way (the six Ws). Use ordinary words. Words should match the vocabulary level of the respondents.
198
198
3.
Avoid ambiguous words: usually, normally, frequently, often, regularly, occasionally, sometimes, and so on.
4. 5. 6. 7. 8.
Avoid leading questions that clue the respondent to what the answer should be. Avoid implicit alternatives that are not explicitly expressed in the options. Avoid implicit assumptions. Respondent should not have to make generalizations or compute estimates. Use positive and negative statements.
Determine the Order of Questions 1. 2. 3. The opening questions should be interesting, simple, and nonthreatening. Qualifying questions should serve as the opening questions. Basic information should be obtained first, followed by classification, and, finally, identification information. 4. 5. 6. Difficult, sensitive, or complex questions should be placed late in the sequence. General questions should precede specific questions. Questions should be asked in a logical order.
Pretesting 1. 2. Always pretest Test all aspects of the questionnaire, including question content, wording, sequence, form and layout, question difficulty, and instructions. 3. Use respondents in the pretest that are similar to those who will be included in the actual survey. 4. 5. Begin the pretest by using personal interviews. Conduct the pretest by mail, telephone, or electronically if those methods are to be used in the actual survey.
199
199
6. 7.
Use a variety of interviewers for pretests. The pretest sample size should be small, varying from 15 to 30 respondents for the initial testing.
8.
After each significant revision of the questionnaire, conduct another pretest, using a different sample of respondents.
9.
Code and analyze the responses obtained from the pretest.
Summary
To collect quantitative primary data, a researcher must design a questionnaire or an observational form. A questionnaire must translate the information needed into a set of specific questions. Designing a questionnaire is as much an art as it is a science. We can provide guidelines for development, but no one optimal questionnaire design fits every research need. The process begins by specifying the information needed. The next step is to decide on the content of individual questions. Questions must be written to overcome the respondents ' inability to answer. Respondents may be unable to answer if they are not informed or cannot remember. When too much effort is required, respondents will be unwilling to participate. Questions that attempt to collect sensitive information may also be met with resistance. Questions can be unstructured (open-ended) or structured to a varying degree. Structured questions include multiple-choice questions and scales. The commonly used scales are Likert, semantic differential and Stapel. Determining the wording of each question involves defining the issue, using ordinary words, using unambiguous words, and using dual statements. The issue should be clearly defined in terms of who, what, when, and where. The researcher should avoid leading questions. Once the questions have been worded, the order in which they will appear in the questionnaire
200
200
must be decided. Special consideration should be given to opening questions, type of information, difficult questions, and the effect on subsequent questions. The questions should be arranged in a logical order. The effectiveness of all these design decisions must be assessed in a pretest. Questionnaire design is a very important aspect of marketing research and effort devoted to this task can payoff handsomely in terms of the quality and completeness of the data and the results.
ENDNOTES
1
The material presented here is drawn from the work of Malhotra (2004) and Malhotra and
Peterson (2006). For more information, including detailed references, please consult these books.
2
Rating a brand on specific attributes early in a survey may affect responses to a later overall
brand evaluation (for more information on this, see Bartels, 2002; Bickart, 1993).
201
201
REFERENCES Bartels, L. M. (2002). Question order and declining faith in elections. Public Opinion Quarterly, 66(1), 67–79. Bickart, B. A. (1993). Carryover and backfire effects in marketing research. Journal of Marketing Research, 30, 52–62. Diamantopoulos, A., Schlegelmilch, B. B., & Reynolds, N. (1994). Pretesting in questionnaire design: The impact of respondent characteristics on error detection. Journal of the Market Research Society, 36(October), 295–314. Malhotra, N. K. (2004). Marketing research: An applied orientation (4th ed.). Upper Saddle River, NJ: Prentice Hall -------, & Peterson, M. (2006). Basic marketing research: A decision making approach (2d ed.). Upper Saddle River, NJ: Prentice Hall. Martin, E. & Polivka, A. E. (1995). Diagnostics for redesigning survey questionnaires. Public Opinion Quarterly, 59(4), 547–567.
202
202
References: Bartels, L. M. (2002). Question order and declining faith in elections. Public Opinion Quarterly, 66(1), 67–79. Bickart, B. A. (1993). Carryover and backfire effects in marketing research. Journal of Marketing Research, 30, 52–62. Diamantopoulos, A., Schlegelmilch, B. B., & Reynolds, N. (1994). Pretesting in questionnaire design: The impact of respondent characteristics on error detection. Journal of the Market Research Society, 36(October), 295–314. Malhotra, N. K. (2004). Marketing research: An applied orientation (4th ed.). Upper Saddle River, NJ: Prentice Hall -------, & Peterson, M. (2006). Basic marketing research: A decision making approach (2d ed.). Upper Saddle River, NJ: Prentice Hall. Martin, E. & Polivka, A. E. (1995). Diagnostics for redesigning survey questionnaires. Public Opinion Quarterly, 59(4), 547–567. 202 202
You May Also Find These Documents Helpful
-
An example of other issues would be the topic of whether or not response time intertwines with officer satisfaction and how this affects crime statistics. This type of qualitative data gathering is a great advantage because it records attitudes, feelings, and behaviors. ("Analyse This!!! Learning To Analyse Qualitative Data", (2008)). As mentioned also opens up new topic areas that were not initially considered. These initial questions are also structured or closed ended questions in which will involve the needed information. Which means the data collected from the answers won’t involve thorough answers that need to be explained. In completion of the questionnaire questions, a small group of similar respondents will take a pretest that will point out any confusion or misunderstanding of question formatting and…
- 1025 Words
- 5 Pages
Better Essays -
There are many advantages and disadvantages of questionnaires and throughout this essay I am going to examine the usefulness of questionnaires in sociological research. I will compare the advantages and disadvantages and find out whether it is can be a useful method to use.…
- 1272 Words
- 6 Pages
Powerful Essays -
Rattray, J., & Jones, M.C. (2007).Essential elements of questionnaire design and development. Journal of Clinical Nursing,16,234-243.…
- 12139 Words
- 67 Pages
Powerful Essays -
Questionnaires can be helpful for a company when the questions are produced correctly and given to the right people.…
- 1058 Words
- 3 Pages
Satisfactory Essays -
The reason for using questionnaire: 1. 2. 3. 4. The result is more objective as respondents are chosen at random. Questions are designed by purpose. A large number of respondents could be reached. Outcome is straightforward and is easy for analysis.…
- 3274 Words
- 14 Pages
Best Essays -
The participants were each asked to fill a questionnaire that has been prepared specifically for this research. Some participants preferred to fill the questionnaire before sampling, some chose to do so after sampling, while a few others were orally asked the questions while sampling was been carried out. It was mandatory that each questionnaire was filled in my presence, so that no important questions were misunderstood or left unanswered.…
- 940 Words
- 4 Pages
Powerful Essays -
Questionnaire Oral - Information is gathered by asking respondents a number of predetermined questions, they can be open or closed. They responses are recorded on a tally sheet.…
- 373 Words
- 2 Pages
Satisfactory Essays -
Survey research “is a measurement process used to collect information during a highly structured interview—sometimes with a human interviewer and other times without” ” (Cooper & Schindler, 2011, p. 242-243).…
- 567 Words
- 3 Pages
Good Essays -
The Assessment Task is due on the date specified by your assessor. Any variations to this…
- 4662 Words
- 26 Pages
Powerful Essays -
Questionnaires are most useful when the views or knowledge of a large number of people need to be obtained or when the people are geographically dispersed, for example, in a company with many branches or offices around the country or around the world. Questionnaires are also appropriate for information systems which will be used by the general public, and where the analyst needs to get a picture of the types of user and usage that the system will need to handle.…
- 320 Words
- 2 Pages
Satisfactory Essays -
Glasow, P. A. (2005, April). Fundamentals of Survey Research Methodology . Retrieved November 28, 2011, from Mitre Product: http://www.mitre.org/work/tech_papers/tech_papers_05/05_0638/05_0638.pdf…
- 2698 Words
- 11 Pages
Best Essays -
The questionnaire (attached as Appendix II), is a fast and cost effective way of gathering primary data from a lot of people, for my study. However I understand that the quality of data emerging from questionnaires is not always great. The responses to questions are limited because the researcher determines the questions that are asked and the range of answers that can be given. I also cannot predict if the respondent has been truly honest and genuine in their responses (Gillham B. 2007).…
- 1106 Words
- 5 Pages
Powerful Essays -
The questionnaire is a suitable method as it gives the student time to formulate the questions clearly and concisely on paper therefore making them easily understood to whom ever is reading. Another advantage of the questionnaire is that it does not take a long time to be completed.…
- 290 Words
- 2 Pages
Satisfactory Essays -
Although questionnaires are generally viewed as high in reliability and low in validity, this does not negate the possibility that questionnaires can have high validity. Questionnaires that search for answers from issues that are clear-cut and straightforward…
- 984 Words
- 4 Pages
Better Essays -
| Questionnaires: Advantages: 1. Can reach large sample 2. Personal influence of researcher in sight 3. Produces Quantitative, reliable and representative data.Dis- advantages: 1. Response rates may be low 2. Answers may be incomplete 3. Data may not be valid or even truthful 4. Cannot be sure who completed the questionnaire 5. Limits possible answers the participants can giveStructured Interviews:Advantages: 1. Higher response rates 2. Can ‘probe’ the participants responses by asking follow-up questions 3. Can assess truthfulness of…
- 693 Words
- 3 Pages
Satisfactory Essays