Controlling Non-Sampling Errors

CONTROLLING NON-SAMPLING ERRORS

In practice, market research agencies make all reasonable attempts, within the limits imposed by cost and time constraints, to minimize or at least measure the impact or make some estimate of non-sampling errors and of bias in the sampling procedure. Thus, as far as response errors are concerned, agencies may:

• pilot-test questionnaires in order to check for misunderstandings of questions,

• analyse tendencies to overclaim or underclaim for certain kinds of con­sumer behaviour, for example, the tendency to underclaim the consump­tion of alcohol, or to overclaim television watching,

• use aided-recall techniques (prompted lists) to help respondents remem­ber products that they may have purchased and forgotten about, or radio  programs that they forgot they had listened to,

• use questioning techniques that minimize the effort respondents need to make.

To minimize interviewer error, agencies will often:

• set rigorous training standards for interviewers,

• monitor the process of interviewing by doing ‘back checks’ – calling or telephoning respondents who have already been interviewed to check that the interview was carried out properly, or sending supervisors to accompany interviewers on a regular sample basis,

• computer analyses may be made of questionnaire errors to identify’ inter­viewers who may need retraining or reminding of particular points.

To minimize errors resulting from non-response, agencies do one or more- of several things:

• for interview surveys interviewers may be asked to make a specified number of callbacks if the respondent was not at home on the first call Three or four such callbacks may be made, ideally at different times and days of the week

• interviewers may make an appointment by telephone with the respondent.

• self-completing questionnaires may be left where no contact has been made

• monetary incentives or gifts may sometimes help to improve the response rate,

• interviewers may get a ‘foot-in-the-door’ by having respondents comply with some small request before presenting them with the larger survey,

• non-respondents to a postal survey may be sent interviewers to persuade respondents to complete the questionnaire, or they may be sent further reminders.

Processing errors will be minimized by careful editing and checking of the questionnaires in addition to the use of data entry validation procedures.

Market research agencies will try to minimize bias by using carefully con­structed sample designs that use random procedures wherever possible, or by imposing restrictions on interviewer choices where it is not. These sample designs were described earlier. Biases will still remain, however, and some­times these are known. Thus it may be known that there are too many women in the sample, or too few men aged 20-24, compared with known population proportions. Many agencies will make corrections to the data to adjust for these biases by ‘weighting’ them.

In the real world of market research agencies and their clients it is unfortu­nately true that many clients do not understand or lack interest in the basics of sampling. In consequence many clients do not ask for estimates of bias or calculations of random sampling error. At the same time the agencies feel that to produce calculations, for example of confidence intervals for a large number of variables will only add confusion and perhaps distrust of the data. In consequence, sampling errors are often quietly ignored, and the estimates given are taken to be the ‘truth’. Agencies will instead try to assure their clients that the occurrence and impact of non-sampling errors have been minimized by:

• demonstrating that the procedures for the collection, analysis and report­ing of the results are ‘respectable’, meticulous and thorough,

• showing that the research design features are such as to minimize sources of error within the parameters set by time and cost,

• emphasizing the extent of quality control checks that will uncover, cor­rect and minimize the occurrence of ‘mistakes’,

• making corrections to the resulting data so that known biases are adjusted for.

Beyond these assurances, clients are sometimes given some indication of the extent of random sampling error that remains. Clients may be given ‘read-off tables for groups of products or types of variable, based on the ‘average’ variability for that group or type, given a particular sample size.

_{___}