5

Questionnaire, an Important Tool for Collecting Information

Like in any survey research, a structured interview schedule popularly referred to as “Questionnaire” is an essential tool for collecting data for measuring voting behavior. Questionnaire or interview schedule is a written document in which various questions are written in a clearly worded format on the topics which the survey aims to probe and enquire. For a study aimed at analyzing voting, the questionnaire or the interview schedule will have questions aimed at collecting information about the political choices, likes and dislikes about political parties, and similar such questions. Besides sampling, a good questionnaire is a key to how accurately one can measure voting behavior since the accuracy of this measurement depends largely upon the quality of the data. If the data which have been collected is inaccurate, there is nothing that can be done to improve the accuracy of the measurement of voting behavior. Here the quality of data does not refer to the sample size but to a few things which determine the quality of the answers. The quality of answers in return is determined by the following parameters related to interview schedules or questionnaires:

A structured questionnaire or interview schedule is required for data collection for measuring voting behavior to ensure standardization. It is desirable that in the process of data collection, all the questions should be communicated to most of the respondents in a similar manner and should convey a similar meaning. This is possible only if the questions are written, clearly worded, and simple. One may not need an interview schedule or a questionnaire if the effort for measuring voting behavior is done not by the survey method but by a constituency level study. A standard questionnaire is needed for data collection for measuring voting behavior which involves interviewing a large number of voters, to maintain uniformity in approach and communication. Standardization minimizes errors in survey and ensures that the data collected are credible and free from measurement errors.

Standard and semi-structured questionnaires are used for measuring voting behavior and attitudes in India. The popular belief among people is that election surveys in India are conducted with questionnaires that are neither properly formulated nor have questions that are empirically tested and tried. The election survey questionnaires are designed by polling agencies that suit the information required by the clients and have commercial value for the media. This may be true in some cases, but it is not a correct reflection of the election polling industry in India. The questionnaire/interview schedule used for face-to-face interviews of voters during opinion polls is generally written and systematically formatted. Designing a good questionnaire is an art that is perfected after conducting extensive survey research and development. A good questionnaire is a mix of freshly formulated questions and questions already existing in the knowledge domain. Questionnaire designing is a lengthy and cumbersome process as it involves brainstorming sessions for formulating questions and conducting field trials for checking its efficacy and relevancy. The draft questionnaire design undergoes a series of pre-tests and based on the feedback from the respondents, it is revised and finalized for conducting the survey.

This chapter is divided into three broad sections: The first section explains the basic parameters that need to be followed in framing a good questionnaire for measuring voting behavior and attitudes. The focus would be on providing the broad contours for framing election survey questions with relevant examples and illustrations. The purpose would be to differentiate and distinguish between good and bad questions. A brief sketch of technical survey terms and jargons that should be avoided while designing the questionnaire would also be outlined; the second section deals with the different types of classification of question structures like open ended, close with ordered answer variables, close-ended and unordered answer variables, and partially close-ended questions. There is no thumb rule for selecting any particular type of question structure for an election survey and it depends upon the kind of information that it aims to gather. Examples will be used to show that for a particular kind of information certain question structures are more suitable and best suited. The final section deals with the importance of pre-testing the questionnaire and checking the efficacy of questions before it is finalized for computing the voting behavior of the electorates. It will outline the stages of pre-testing which a questionnaire should ideally undergo and why each stage is necessary for designing a good election survey questionnaire. Pre-test of the questionnaire helps in finding out how the survey questions were understood and responded to by the sampled voters and based on the feedback, inconsistencies, flaws, and errors can be identified and ironed out.

Basics of a Good Questionnaire

The foundation for survey research lay in identifying the problem that motivates a survey in the first place and then translating the problem or idea into good questions that respondents can understand and answer objectively. Social scientists call it operationalizing. That means setting up categories of events or phenomena that can be observed and measured (Salant and Dillman, 1994). A good questionnaire is not one that contains questions written in grammatically correct language nor does it mean a document designed aesthetically and in stylized format. A question is considered good in survey research when it conveys the same meaning and is understood by all the respondents in more or less the same way. A question understood by different people in different ways would certainly be a poor question. The other element that qualifies the question to be good is the ability to get different answers from different people. A question that gets only one kind of response is not considered a good question and in survey parlance it is called “The question was wasted.” If a question asked from the respondents gets more than 95 percent responses of one kind, it is considered a bad question. So while collecting information for measuring voting behavior, the questions which we need to ask should be such which communicate the same meaning to almost all the respondents. A good quality election survey questionnaire can be formulated by the following parameters.

Use of Simple and Common Words

A good questionnaire for an election survey aimed at measuring voting behavior would be one that contains questions written in a simple manner and in the common language. The questions should be such that if they are asked to the voters, they should easily understand it, and there should not be any need for further explanation for those questions. The questions should be written in the language spoken by the people and its wording should not be very formal and bookish. When the question is asked to the voter, it should clearly communicate to the person the intent of what is being asked. The effort should be to use as far as possible simple words and words that are popularly used by people in day-to-day conversation. Normally the questions are formulated in one language, but at times there may be a need for mixing words from other languages due to linguistic acceptability of certain English words in Indian languages which are more commonly used than the native language words. For example, in formulating an election survey questionnaire in Hindi, some English words like MP or MLA or Elections are incorporated as they are more popularly used by common people in Hindi-speaking areas of India than the Hindi words like, Saansad, Vidhayak, or Chunaav.

Length of Questions Should Be Short

There is a general consensus among survey researchers that the questions should be short and lengthy questions should be avoided as far as possible. However, it is contested by some experts who believe lengthy questions are more suitable and in some situations it is very crucial. The argument put forward in favor of lengthy questions is that introducing the questions in a few sentences before the actual question begins helps in placing the respondent in a better situation. The lengthy question also allows the respondent a little more time to think about the answer and provide better responses. Thus long questions are useful and desirable in some situations but it should be used sparingly in the questionnaire. The reasons for avoiding long questions arise: too many lengthy questions make the questionnaire bulky and very time consuming and it leads to respondent fatigue and a higher rate of incomplete interviews during the survey.

However, lengthy questions are useful or rather unavoidable if the election survey questions are to measure the opinion of the voters on politically sensitive issues like religious riots and electoral polarization. The questions on sensitive issues need to be framed in a contextual/situational frame that makes them necessarily lengthy. Such sensitive questions if asked in short would be most likely to be seen as intrusive or embarrassing to the voters and will not solicit the right answers. Thus questions to respondents like accepting liquor or cash for voting for a particular candidate or political party would fall in the category of sensitive questions. If sensitive questions are asked in a direct or straightforward way, it becomes uncomfortable for the voters to answer such questions leading to high non-response rates. It is useful to formulate sensitive questions in a longer format to neutralize the value judgments attached with it like there is nothing good or bad about the question. It also allows the respondent to have a better recall of the situation and provides a little more time to answer the questions. For example:

Inappropriate Formulation: For whom did you vote during the recently concluded Assembly Elections? Did you get any gift or money or similar such thing from any candidate during the recently concluded Assembly Elections?

Appropriate Formulation: While talking to the people in this locality, we found out that some people voted while some people were not able to vote, what about you, were you able to vote or not?

While talking to the people in this locality, we came to know that some voters receive gifts or money from the candidates during the recently concluded Assembly Elections, while some did not receive any gift or any money. What about you, did you receive any gift or money during the recently concluded Assembly Elections?

Similarly, questions on computing the electorate's opinion on socially undesirable issues are also lengthy as they are framed in a desirable context. These questions on socially undesirable behavior or attitudes if addressed directly to the voters and in a short version usually do not yield correct answers as people tend to avoid them. Thus voters may have indulged in some form of malpractices during the election but when asked about it in a straightforward way, they do not give correct answers. So, when writing questions on such issues, care should be taken that the main question is prefaced with a small narrative to tone down its undesirability and awkwardness. This helps in getting a more accurate response compared to what one can get if such questions were asked in a straight forward manner.

Avoid Writing Two Questions in One

While formulating survey questions, one should always avoid writing two questions in one question. For example, Do you like Rahul Gandhi and Sonia Gandhi? This looks like asking a question or expressing opinion on leadership, but the problem in this question is that this question has two questions in one question. While the first question is about the choice of Rahul Gandhi, the other question is about Sonia Gandhi. At times it may be difficult for the respondent to give a clear-cut answer to this type of question. One may not like Rahul Gandhi as a leader but may like the leadership of Sonia Gandhi. If the question is worded like this, it may be very difficult for the respondent to give a clear answer to this question. The respondent does not have the choice of giving a positive answer for one and negative answer for the other. So if the answer to this question is recorded in the affirmative, it implies that the respondent likes both the leaders. On the contrary, if the answer is recorded in the negative it means the respondent neither likes Rahul Gandhi nor Sonia Gandhi which may not be correct. The respondent may like one and dislike the other, but questions of this kind do not help in getting a clear-cut answer from the respondent. Such unclear answers will result in measuring voting behavior which may be unreliable. This is called a “Double-barreled” question that must be completely avoided in any survey questionnaire.

Questions with Clear Time Reference

Election survey questionnaires have some questions that try to measure the opinion of the voters with reference to some time reference. Any question that refers to a time period should clearly mention the exact period whether in days, months, or years depending upon the issues for which the reference is being made, so that different respondents do not interpret the time period in different ways. Thus the last few years can be interpreted by some respondents as one or two years while others may understand it as four or five years. This is against the basic principle of standardization and should be strictly avoided. The question should clearly state as to what is the time period for which this question is being referred and that would be possible only if it is clearly spelled out in the question in exact years. The time reference in the question should be adequate as a very long time recall is not suitable for survey questions. In case of very long time references, the respondent fails to recall clearly the issue or the event that took place a long time ago. Thus a longer time reference becomes a memory testing exercise for the respondent and their responses are often vague or inaccurate.

Inappropriate Formulation: During the last few years how many times has the MLA of your area visited your village/town/city?

Appropriate Formulation: During the last one year how many times has the MLA of your area visited your village/town/city?

The question that is asked in reference to a time period could be more appropriate and accurate if it is asked with a specific time reference or with the help of some landmark or by providing some cues.

Answer Categories Should Follow the Question

While measuring voting behavior, we do ask questions on which it is difficult for the voter to give an answer in either “Yes” or “No” and at times voters give subjective or qualified responses. While formulating such questions, where answers may not be given in either yes or no, it is desirable that some answer categories which are likely to be the response of the voter should be read out along with the question. If the answer categories need to be read out, it should form part of the question. But there is one caution, these answer categories should follow the questions, which is to say that the questions should be written in such a way that first the question should be read out and the answer categories should be read out later. This sequencing helps the respondent in first listening to the question and then choosing the possible answer from various answer categories that was read out.

Inappropriate Formulation: Would you say that you are satisfied to a great extent, to some extent, not much or not at all with the performance of the state government?

Appropriate Formulation: To what extent are you satisfied with the performance of the state government—to a great extent, to some extent, not much or not at all?

Specific Questions Should Be Preferred

While asking questions for measuring voting behavior, specific questions should be preferred over general questions. General and vague questions are prone to inappropriate interpretations, different interpretations resulting in giving different meaning to the question by different voters. If different voters attach different meaning to the same question, it would go against the basic principle of standardization in survey research. When voters will attach different meaning to the same question, they are bound to give different responses to the same questions resulting in analysis being inaccurate. The measurement of voting behavior based on data collected by faulty questions is bound to be questionable. The questions with wide range of interpretations by the voters invoke the risk of higher margins of measurement errors.

Thus a question on the performance of the central government in India would be open for various kinds of interpretation by different voters. Some may interpret the performance in terms of basic amenities provided while others may interpret it on the parameters of handling terror attacks and law and order in the country. In order to minimize the possibility of various interpretations by different respondents for the same question, it is essential that the question should be specific in nature and clear in its intent.

Inappropriate Formulation: How would you rate the performance of the central government—very good, good, not so good or bad?

Appropriate Formulation: How would you rate the performance of the government on the issue of handling the problem of unemployment—very good, good, not so good or bad?

Avoid Double Negative In the Question

While designing an election survey questionnaire, negatively worded questions and double negatives should be avoided as far as possible since it creates confusion not only among the voters but also among the field enumerators. For example, “Voting in elections should not be made compulsory and non-voters should not be punished.” Tell me if you agree or disagree with the following statement? The problem in this question is that the negative answer is the positive response while the positive response is the negative answer.

Though questions with double negative should be avoided, some survey experts feel that a few such questions should be purposively incorporated in the questionnaire as it helps the respondents in being more attentive and also breaks the interview monotony. The double negative question is especially useful for the battery of questions where with one single question more than one question is asked from the respondent one by one without a break. The responses are recorded either in agree or in disagree format (positive or negative). If in such a battery all the statements are written in a positive manner, then the respondent tends to provide one set of answers to all the questions that may be either positive or negative. This is referred to as “Response Set.” Mixing of positive and negative statements in the battery of statements is very useful in breaking this response set of the respondents even at the cost of investigators having to pay little more attention while recording answers to these double negative questions. But the number of such questions in the battery should be few or else it will lead to confusion among the enumerators as well as the respondents.

Prefer Forced Choice Questions

In an election survey, there are questions which measure the opinion and attitudes of the electorate in agree–disagree answer categories. The questions are asked in the form of a statement and the voters respond to it by either agreeing or disagreeing with them. Survey experience reveals that quite often the voters agree or disagree to two statements that are quite contradictory to each other. This is quite common among illiterate voters or those who have low levels of educational attainments. They tend to give one kind of response irrespective of the statements. This in survey parlance is termed as “Acquiescence Bias” and it can be minimized in a survey by providing forced choice questions.

Inappropriate Formulations: Voting should be made compulsory in India—do you agree or disagree?

Only educated people should be allowed to contest elections—do you agree or disagree?

Appropriate Formulation:

I am going to read out two statements, tell me whether you agree with statement 1 or statement 2.

 

Statement 1 Voting should be made compulsory since this will help in increasing the voting turnout in India.
Statement 2 In a democracy like India people should have the freedom to decide whether they would like to vote or not, voting should not be made compulsory.
Statement 1 Only educated people should be allowed to contest elections in India since only educated people can run the government in India.
Statement 2 In a democracy like India everybody should have the freedom to contest elections and there should not be any educational qualification for contesting elections.

Avoid Recency Effects

While asking questions for collection of data, there are questions, where few answer categories are read out and the voter is expected to choose one of those answer categories. The responses are not shown to the voters but are read out by the investigator to which the voters listen. The voter has to choose one of the answer responses. Research over the years has indicated that in such situations, for such questions, there is a tendency amongst the voters of choosing either the first response or the last response which is read out to him. The respondent tends to remember or recall these two types of answer responses more than other responses. In survey research, this is known as Primacy or Recency effects. When there is the tendency of the respondent to pick up the first response more often, it is regarded as the primacy effect, and if the respondent picks up the last answer category, it is known as the recency effect. To minimize this problem, while formulating questions it is desirable that the answer response which is more popular compared to other responses should be placed neither at the beginning nor at the end; such responses are to be placed in the middle of the list of responses. This helps in softening the primacy or the recency effect and minimizing its effect (see Table 5.1).

Example: Who among the following do you think have been the greatest political leader of India?

Table 5.1. Minimizing Primacy and Recency Effect: Basic Rules

images

The first formulation of the questions may invite both primacy and recency effects since the two most popular leaders Mahatma Gandhi and Jawaharlal Nehru have been placed at the beginning and at the end of the list. A large number of respondents would pick up one of these two answer responses since they are popular. Their names would be mentioned either at the beginning or at the end which makes it easier for the voter to remember. There would be only a few who would mention the name of V.P. Singh. We may not get the correct estimate of what proportion of voters consider these leaders as the greatest political leader of India. Analysis of voting behavior and political choices based on such responses would certainly be unreliable. It may be appropriate to place them somewhere in the middle of the list to minimize the primacy and the recency effect on this question. The revised formulation where Mahatma Gandhi and Jawaharlal Nehru have been placed in the middle of the list helps in minimizing the primacy and recency effect on this question. This would help in getting the response closer to the actual reality.

Other Issues

The list of parameters to be followed for designing survey questions as outlined above is a comprehensive list but there are other issues that need to be kept in mind while designing a good questionnaire. An important issue is of providing answer categories of “Do not know” or “No opinion” in all the questions. There would be questions that the respondent would not like to answer due to the following reasons: First the respondent could not understand the questions so did not have an answer, second the respondent understood the question but did not have the awareness/knowledge of the issue, and finally the respondent understood the question and has an opinion on the issue but does not want to express his opinion. In order to capture these kinds of situations, all the questions should have an explicit “Do not know” or “No opinion” answer categories. If we do not leave an option of No Opinion for the voters to mention and for the investigator to record, the proportion of other answers would get inflated resulting in inaccurate measurement of voting behavior.

An important issue in questionnaire designing is sequencing or ordering of the survey questions. Which question is to be asked in the beginning of the questionnaire and which is to be asked at the end should be selected very carefully. As a general rule the first question should be close ended with no more than two or three answer choices. A yes/no format works well. The second or third question should be open ended (Salant and Dillman, 1994). The general agreement among survey practitioners is that sensitive questions should not be asked in the beginning or at the end but should be placed somewhere in the middle of the questionnaire. If possible the questionnaire should be so designed that the respondent has the scope of giving the answer to sensitive questions in a secret manner that may not be heard or known by people present at the time of interview. In election surveys, the voting question is very sensitive and voters want to maintain the secrecy about their vote. Since the interviews are conducted in a face-to-face situation and few people gather around the voters being interviewed, the question on voting should always be asked in such a way which enables voters maintain the secrecy of his vote, and others should not get any hint of which party he or she had voted for.

The voting question is asked with the use of dummy ballot paper and the dummy ballot box. Normally, the names and symbol of all the contesting candidates and parties are listed on the dummy ballot paper which is handed over to the voter when the voting question is asked. They are requested to mark their response on that dummy ballot paper and put it secretly in the dummy ballot box which is sealed to maintain the confidence of the voters being interviewed.

Classification and Structure of Questions

A questionnaire designed for measuring the opinion and attitudes of the voters contains questions that can be broadly classified into opinion questions and factual questions. Thus questions on the performance ratings of Sheila Dixit's government in Delhi and her popularity rankings are opinion questions as they are based on the satisfaction levels of the voters in Delhi. On the other hand, questions like who is the chief minister of Delhi and since how long she has been in power are factual questions. The responses of factual questions are correct or incorrect, whereas the responses of the opinion questions are the views of the voters that cannot be termed as right or wrong. Though there is qualitative difference between these two kinds of questions, both these categories of questions complement each other. Thus election survey collects background data of the electorates like age groups, gender, educational attainments, economic class that are factual information as well as data on electoral politics that are based on opinion and attitudes of the voters. The measurement variables for factual questions are in “Yes or No” or “Correct or Incorrect” format, whereas for the opinion or attitude questions different types of scaling categories are used to measure the varying degrees of responses. Herein below is an illustration of the answer categories popularly used for measuring the response of the voters on opinion questions:

Inappropriate Formulation: Are you satisfied with the performance of Sheila Dixit's government in Delhi? Yes/No

Appropriate Formulation: To what extent are you satisfied with the performance of Sheila Dixit's government in Delhi—to a great extent, to some extent, not much or not at all?

Questions are further differentiated on the basis of its structure into close and open-ended questions and choosing a particular structure for a question depends upon its needs and suitability. As the word denotes, a close-ended question is one in which the choice of answers has been closed and restricted to only a few answer variables. The answer categories are already predecided and the respondent has to pick up one or more answers as per their choice. The respondent has to choose an answer from the menu of answer variables provided in the question and is not given the freedom to pick an answer from outside. In literal terms, the choice of the responses has been closed in advance and there is no choice to add a new response. Close-ended questions are further differentiated into three categories: close-ended with ordered answer variables, close-ended with unordered answer choices, and partially close-ended. On the other hand, an open-ended question is one in which the respondent has the freedom to give an answer of his choice as the answer variables are not provided in the question. Thus the ends of the questions are open and the responses of respondents are recorded in verbatim in the blank space provided beneath the question. A detailed analysis of the structure of questions with its advantages and disadvantages is necessary, as it will allow choosing the right format for designing election survey questionnaires.

Open-ended Questions

An open-ended question does not provide answer variables but allows the respondent the freedom to record answers of their choice. The advantages of using open-ended questions in an election survey questionnaire are as follows: (a) if a researcher intends to obtain information on what is likely to be the most important issue in an election, it may be useful to ask this question in the open-ended format since different voters may give different kinds of responses. Open-ended questions will enable the voters to give the response the way he/she would like to give and the respondent will record it the way the voter will mention. Also it is difficult to anticipate various kinds of responses for such a question. Thus for any issue, if the enquiry is conducted for the first time and a new question is being framed, an open-ended question is a good alternative as making an exhaustive list of answer variables is neither possible nor naive. An open-ended question on an unexplored theme is a better option as it allows pre-testing of the questions through field trials among the respondents and soliciting information from survey experts. Based on the feedback information, the open-ended question can be turned into a close-ended format; (b) the universe of study in an election survey is so varied with differences among the voters on the basis of culture, language, regions, etc., that an open-ended question becomes necessary for collecting information. For example, the opinion of the voters about democracy in India and the way they perceive the meaning of the term is so exhaustive that the whole gamut of information can be captured only by an open-ended question; (c) an open-ended question is useful on issues which are politically or socially sensitive and that often lead to strong polarization or cleavages among the various sections of voters in the society. This format allows the voters to freely express their opinion and provide detailed reasons why they support or oppose an issue. Thus affirmative action for other backward classes in India is a politically sensitive and emotive issue that is opposed by sections of society who do not get the benefits. An open-ended question would give the respondents an opportunity to make a statement and provide a deeper insight on such kind of issues than a close-ended one; (d) open-ended questions are suitable for collecting personal information of the respondents which they can accurately recall and state it precisely and (e) open-ended questions are suitable if a nationwide survey intends to collect some localized information for which preparing an exhaustive list of answer variables would be cumbersome and time-consuming. For example, an open-ended question would be the best format to collect information about the local issues in general elections in India on which the voters make up their mind on whom to vote.

Open-ended questions though quite useful in collecting information in certain contexts also have some inherent drawbacks. The drawbacks are: First, open-ended questions can be time consuming and burdensome for the respondents as they have to think and then express their opinions. For factual questions, they can recall from the past and narrate their experiences easily but if the question is on any issue, then they have to recollect it and form an argument to express their opinion; second, the respondents sometimes feel reluctant to answer open-ended questions as it involves thinking and finding an easy way out by providing irrelevant answers. Thus for open-ended questions, most of the answers do not address the issue directly and are vague in response; third, the coding, entry, and analysis of the open-ended questions take much more time and greater efforts as compared with close-ended questions which can be done quickly and instantly; and finally open-ended questions do not provide accurate measurements of voter's opinion and attitudes as survey experience reveals. The information collected though open-ended questions have been found to be inconsistent and incomparable at times.

Close-ended Questions

Close ended with ordered answer variables is the most common type of questions that are used in surveys aimed at measuring voting behavior in India. This structure of question has a complete range of likely answer variables that are ordered systematically or sequentially. The answer variables provide the respondent the complete variety of answers from which to select one of their choices. Close-ended questions with ordered answer choices tend to be quite specific, less demanding for the respondent, and much easier to code and analyze than open-ended questions (Salant and Dillman, 1994). Close-ended questions with unordered answer variables are questions where choice of answers are not arranged in a continuum but presented in an unordered fashion. This type of question structure is mostly used in surveys where the respondents are asked to rank or prioritize certain issues. Close-ended questions with unordered responses are commonly used in election surveys to find out the popularity ratings of political leaders or ranking the achievements of the incumbent government based on the voter's opinion and feedback. This question structure is more difficult as compared with close-ended questions with ordered answer variables as the voters have to think over the given choices and then rate them accordingly.

A mix of both open-ended and close-ended questions is the partially open-ended question. This question combines the advantages of both the question structures as it provides all the possible ranges of answer variables and also provides space for recording the respondent's answer. This question structure is used for measuring the voter's opinion on certain electoral issues like considerations for forming voting decisions. Survey findings show that the voters mostly tend to select the answer from the existing answer variables in the question and only in some cases it yields new information. But in most of the cases, this question structure does not bring forth any additional information. Partially close-ended questions have the advantage of not forcing respondents into predefined boxes that do not fit the situation and it occasionally generates new information (Salant and Dillman, 1994).

Close-ended questions are the dominant form of question structure that is used in election surveys for measuring the voting behavior and attitudes of the electorate. Though close-ended questions cannot gather exact information even with the best graded answer categories in some situations, it still remains the popular choice of question structure for opinion polls and exit polls. The advantages of using close-ended questions are as follows: First it collects information that are more consistent, credible, and authentic as compared with open-ended questions; second it is easier for the respondents to pick up the answer category of their choice and for the enumerators to record it correctly. The respondent does not need to think in which form to answer the question but only choose from the available answer options; third close-ended questions are less time-consuming to ask and record the responses and also for data coding and processing; and finally close-ended questions with the same answer variables used over a period of time can provide comparability and time series data that could not be possible with open-ended questions.

Evaluating the Survey Questionnaire

The questionnaire for measuring the voting behavior and attitudes of the electorate are evaluated at two stages. The first stage of evaluation takes place while designing and developing the questionnaire and the second stage by conducting field trials for testing its efficacy and relevance. The process of questionnaire designing for survey research has evolved over a period of time through sustained research and practice. This led to the development of some quality standards and protocols that are accepted and followed by opinion polling industry. However, a questionnaire designed keeping in mind all the thumb rules outlined above is still not fool proof and needs to be minutely evaluated and reviewed before it can be finalized for use in an election survey. Thus the questionnaire needs to be evaluated for the following parameters: First whether the questions are relevant and consistent with the stated objectives of the survey; Second are the questions easily understood by the voters and are they willing to answer the questions; and Third whether the field enumerators are able to administer the questionnaire properly within the expected time frame and address the issues it originally intended to enquire. While the questionnaire may have few time-tested questions that might have been used in earlier surveys, the majority of questions are new and freshly written. So a new questionnaire designed is evaluated and reviewed at two stages before it is finalized for a survey.

Stage I Exploration and Development

The initial stage of questionnaire designing for measuring voting behavior is exploratory as new questions are formulated, revised, and reformulated to give it a credible shape. At the early stage of designing the question, conducting a focus group discussion (FGD) with a few voters from the universe of the study is a useful exercise. FGD allows in finding out whether the respondents know or do not know about the issues of survey enquiry, the terms and words they popularly use in their day-to-day life and explore their overall understanding and perspective of the survey topics. The feedback given by the voters on the electoral issues provides a direction on which questions can be designed as the knowledge, experience, and perceptions shared by the voters immensely help in deciding the question structure and answer categories for questions. FGD though qualitative in nature provides information about the survey topics from the respondent's viewpoint and brings better clarity in designing the questionnaire in the preliminary stages.

A good method of evaluating a questionnaire is through the method of cognitive interviewing. Cognitive interviewing is a specialized method of evaluating a questionnaire by understanding the hidden thought process and observable process of the respondents. The purpose is to identify poor questions in the questionnaire that create misinterpretation and confusion among the respondents and correct the response errors. Two types of cognitive interviewing methods that are popularly used are think-aloud interviewing and verbal probing techniques. It is useful for election survey research as the voters give not only an overall feedback about the questionnaire but also specific suggestions on the content and wording of the questions. Thus FGD and cognitive interviewing helps in preparing a draft questionnaire for the survey.

The next stage in the development of questionnaires is review of the instrument by survey experts. Expert opinion on the design of the questionnaire in terms of both technical and quality of content is an essential step. The questionnaire is evaluated by experts on finer points like the length of the questionnaire, the wording and ordering of the questions, the clarity and preciseness of the questions, the suitability of the structure of questions, the choice of answer variables for the questions, instructions for the field investigators, skip patterns used, and mismatch between questions and answer categories. The checklist detailed here is indicative as some experts use a longer list with more exhaustive checkpoints for evaluating and reviewing the questionnaire.

Stage II Pre-testing

The questionnaire developed after exploration and evaluation undergoes pre-testing through field trials. The objectives of testing the questionnaire are to find out the efficacy and validity of questions and the overall flow and consistency of the questionnaire. A pilot survey among a small sample of voters is done with fieldwork procedures similar to full-scale survey to pre-test the questionnaire. The pre-test of questions is done to check the following: First to find out variations in the question and whether any specific question can be asked in a different way; second to find out if the question conveys the same meaning to different respondents and is understood in a similar manner. Every question is intended to convey a particular meaning and it becomes imperative to find out if the respondents are able to understand the import of the question in the same way or differently; third even if the question is easy for the respondent to comprehend, it may not be easy for the respondent to answer that question. The task of answering the question may be difficult for the respondent and the pre-test aims at assessing the “task difficulty” in the question; and finally whether the question raises the curiosity and interest levels of the respondents during the interview.

Apart from the questions that are rigorously tested during the pilot survey, the questionnaire is also pre-tested on the following parameters: First while administering the questionnaire the flow of the interview is smooth and consistent; second the ordering of the questions is done topic wise and clubbed together properly. The placement of easy and light questions as well as serious and difficult ones done correctly; third the clarity of instructions for skip patterns and double level probes and comfort levels of the enumerators while administering the questionnaire; fourth the time frame for completing the interview is reasonable and the questionnaire is neither very lengthy nor too short; fifth any section of the questionnaire that is vulnerable to the field investigator's bias which tends to creep in the questionnaire despite all the care and cautions taken; and finally the interactive levels of the questionnaire and how well it connects with the respondents and motivates them to participate in the survey.

The pre-testing of questionnaires is done through two methods:

  1. Participating Pre-test: In this method of pre-testing, the respondent is informed that the interview is only for testing the questionnaire and it is not a real survey. The advantage of participating in a pre-test is that it allows the enumerators to further probe after reading out the main question from the questionnaire. Thus the questions can be asked in several ways to find out how it is understood and interpreted by the respondents. Thus questions that might convey a different meaning to the respondents can be identified and corrected. However, this method of pre-testing has some major disadvantages. All the questions in the questionnaire cannot be asked with further probes as it will be time consuming and the respondent may not spare so much time. Further the respondent knows that it is not a real survey and may not give proper responses.
  2. Undeclared Pre-test: The respondent is not informed in this method of pre-testing that the purpose of the survey is experimental for evaluation of the questionnaire. The interview of the respondents is conducted formally and completed as if it is a real survey. The advantage of this method is that the completed interviews can be electronically punched and data analyzed to find out the problems in the questionnaire. The data will reveal any discrepancies in the questions as their answer responses would be inconsistent and skewed. The biggest disadvantage of an undeclared pre-test is that the enumerators can just ask questions only from the questionnaire and cannot probe deeper like in a participatory pre-test.