PROS AND CONS OF INTERNET AND MAIL SURVEYS

So you want to do a survey? Perhaps you want to see what your members think of your latest benefits package, or see if your donors are turned off by your president's new haircut. Whatever the case, there are numerous ways to conduct a survey. In this edition of DonorSpeak, we discuss the pros and cons of Internet surveys and mail surveys. In upcoming editions we will cover phone surveys, intercept surveys, and surveys done using an interactive phone call (IVR).

We begin with Internet surveys. Internet surveys are a relatively new methodology for researchers. They began around 1993, but took several years to go mainstream. Today, they are a common and increasingly sophisticated part of many research plans.

It probably goes without saying, but with an Internet survey, your questionnaire is programmed into a web format using survey software and hosted on an Internet server. You invite potential respondents to visit the Internet survey using a postal mailing, an e-mail invitation or online pop-up announcement, after which people access the survey using their web browsers. Since the respondents answer the questions online, you don't need to do any further data entry. This allows you to see the results in real-time, as they are collected.

Very often, charities and institutions will collect e-mail addresses from constituents, enabling them to contact their donors or customers online. This can be a valuable source of names for a survey sponsored by the organization that collected them. Opt-in Internet panels are another excellent source of respondent names.

The pros of online surveys include…

• Fielding costs are typically lower than offline surveys. Online surveys often do not have postal costs, interviewer costs, printing costs or phone calling charges. Email broadcasting is affordable; delivery costs mere pennies per message or less. Organizations with simple questionnaires can set-up and design their surveys very cheaply with the online survey companies that are currently out there.
• Results can be collected very quickly. Fielding can take days instead of weeks or even months. However, be careful not to field a survey too quickly! They should stay open for at least a week to minimize the risk of bias (for example, if working moms are more likely to respond over the weekend and your survey was only open between Monday and Friday, you've just underrepresented a group). Also, just as much time should go into designing an Internet questionnaire as would be spent on a mail or phone questionnaire.
• Internet questionnaires are typically easy to modify in the midst of conducting the survey. Want to add a question or restrict access to certain types of respondents? Not a big deal with most online surveys.
• Results from an online survey can also be reviewed in real-time online. Graphs, tables, cross-tabulations, etc. can all be generated. This makes monitoring an active survey very easy, and allows easy sharing of data between departments and companies. In some instances there is no further reporting needed, eliminating even more time and costs.
• In comparison to mail surveys, Internet surveys allow a researcher to efficiently direct respondents to appropriate questions based on their responses. This tends to increase the completion rate and minimize data cleaning, both of which are good for the research.

Cons for online surveys include…

• Response rates for Internet surveys have dropped consistently since their inception. It is not uncommon for an Internet survey to receive only 2% response.
• Some groups are naturally under-represented on the Internet. For example, only 8% of Americans online at home are seniors, though they make up 16% of the population. Fourteen percent of Americans online with home Internet access are low income ($25k or less), though they make up 19% of the population (source: Harris Poll, Inc.).
• It may not be possible to obtain email addresses for the specific groups you are trying to reach. In such cases it may be necessary to mail links to the survey in postcards or letters. The response rates for mail to Internet surveys are lower than e-mail to Internet surveys, since with mail pieces potential respondents must make the effort to go online and manually type in the link to participate. If they receive an email invitation, they are already online and only need to click the link.
• Internet technology is still young and continuously changing. Smaller organizations may find it difficult to maintain the technological knowledge needed to effectively and securely employ online studies.
• Email address list costs are typically more expensive than the costs of mailing address and phone number lists.
• Security? Well, it is easier to crash an online survey than it is to steal someone's paper survey out of the mail and fill it out. There is also the problem of keeping respondents from answering your survey multiple times in order to get more incentives or affect the results.

In this age of the Internet, mail surveys seem almost ancient. But they are in no way antiquated. With a mail survey, a researcher typically mails their paper survey and has it returned by providing a postage-paid envelope. In-person survey distribution, fax-based survey returning, and other varied methods can be blended in with a mail survey format. After the surveys are returned to the researcher, the data is inputted into a software collection program—either manually or using optical recognition technologies.

The pros of mail surveys include…

• Mail surveys are convenient for respondents; they can complete them when and where they prefer.
• As with Internet surveys, respondents have the opportunity to spend as much time on the survey as they need, which can result in more detailed responses.
• Bulk-rate postage, automated mail preparation, large-quantity printing, and printing paper are all relatively affordable.
• A wide variety of affordable mailing lists are available.
• Researchers have complete control over what is presented to each potential respondent. The appearance of Internet surveys and invitations depends on the computer settings of respondents, and every phone interviewer will act differently. But with mail surveys, you can print and prepare the presentation to your exact preferences.
• This format works well if a large quantity of information needs to be collected, since interviewer time is not being used as the surveys are conducted. Remember to provide the respondents with some sort of motivation for spending time with your survey. The same goes for Internet surveys.

Cons for mail surveys include…

• It takes much more time to conduct a mail survey than it does a phone or Internet survey. Allow extra time for material printing, mailing piece preparation, mail delivery, and data entry.
• Respondents are essentially on their own. There is no interviewer to explain instructions, answer questions, or provide clarification. Question-skipping instructions are prone to human error, and respondents can skip whichever questions they do not want to answer. This can result in high levels of abandoned and incomplete surveys. Because of these factors, mail surveys are typically not used for complex issues.
• Mailed surveys have to compete with other direct mail. Such competition is not exclusive to this format though; email invites must avoid being grouped with spam and phone survey interviewers must compete with telemarketers.
• Mail surveying can involve outsourcing to numerous different vendors (printers, mail processors, etc.), which requires additional project management time and the potential for external delays.

Often an organization's resources and intended audience will dictate what format they use. An organization that has little technological know-how but houses a large direct-mail department may opt for a mail survey. An organization with a wealth of available email addresses but few physical mailing addresses may opt for an Internet survey. In any case, it is important to verify that the format you plan to use will provide solid data. The most convenient methods are not necessarily the best.

When in doubt, opt for the input of professional researchers – even if you plan to conduct the study in-house. Seeking advice from survey researchers is generally not that expensive; many will charge an hourly rate, like a lawyer or accountant. A good researcher may save you both time and money on a project, no matter which methodology you choose.