Response Rates and Survey Validity
There are many sources for the information that businesses need to function effectively. Most management information comes through administrative records compiled in the normal operation of the business, such as financial, personnel, inventory, and sales records. But sound marketing and strategic decisions require additional information on current and prospective clients.
General information of this type can be obtained through government reports, general market analyses, or specific marketing studies conducted on targeted clients. While general marketing studies focusing on demographics can help establish the scope and locality of favorable markets, only feedback from current or potential customers can provide guidance on the suitability of specific products and services.
Information on specific products and services usually needs to be gathered through a "survey" or other similar information-gathering tool specially designed to obtain relevant feedback. Typical survey or survey-related methods are convenience sample surveys, scientific sample surveys, focus groups, and various types of expert panels. Sound management decisions should draw on all sources of available information.
Ideally, statistical surveys based on scientific probability samples should be conducted to obtain information on designated populations. A response rate of about 75 percent or more is a typical target range for a scientific sample. In practice, surveys of this nature are not generally conducted in the business setting because of the high cost and time required. As a result, less costly methods are more generally used. Despite low or very low response rates, customer feedback surveys of various types, such as those administered after meetings and conferences can provide useful information.
While caution should be exercised in extrapolating the reactions of the respondents to the general population, data obtained from a sizeable number of respondents reflect a significant body of information that should not be ignored. For example, if 500 people called and complained about a specific product, this would be a cause for concern. If 500 people out of 5,000 sampled (10 percent response rate) sent back survey forms critical of a specific product, it should also be cause for concern. The opinions of a large number of interested persons have important value regardless of the response rate.
Businesses make frequent use of focus groups too, and these are not representative samples either. They often are comprised of people who have been persuaded to come to the focus group meeting through some sort of rewards system. This technique is useful to obtain information of an in-depth nature in response to guided questions.
Still, the number of viewpoints obtained is much less than a sample survey, even one with a very poor response rate. Focus groups also run the risk of being dominated by one or a few persuasive individuals who can influence a "majority" opinion. The opinions in even a limited survey will be more honest and better reflect individual preferences.
There are ways to verify results from non-scientific samples. Particularly if results seem surprising, small-scale verification tests can be done for relatively limited costs. For example, suppose 5 percent of a conference's 10,000 participants report about the hotel accommodations. Half say that staying in the conference hotels was the most miserable experience of their lives. So, roughly 250 are highly critical. A sample of as few as 100 could be selected from the attendance roster and these persons would be followed up specifically and intensively to obtain their opinions.
Based on the high response rate for this limited group, a rough confidence interval could be set up to verify or reject the validity of the 50 percent negative feelings of the larger group. The more detailed information from the larger group could be used to identify the nature of the problems, but a rough magnitude of the problem could be set in context. Using this technique, one form is sent out to a large number of persons to get fairly detailed information, and a second very small survey is used to validate the most important aspects of the large survey.
Sound decision making should be based on all the information that is available. Sometimes judgment needs to be exercised to put a weight on the value of the information. But, no information should be dismissed out of hand simply because it is not ideal. Frequently, the alternative is to base management decisions on a few casual conversations. This practice suffers from far worse reliability problems.
Marketing information frequently suffers some limitations, and the best information is sometimes subject to different interpretations. The best policy for formulating marketing strategies is to use information from multiple sources, corroborate important findings, and draw especially on those results based on the opinions of large numbers of clients.
©2002 Track Marketing Group