When measuring the effectiveness of a program, surveys have a number of advantages.

Problem

You want to use surveys to determine if your project is effective in changing attitudes or behaviors, but traditional options for survey data collection are expensive, hard to use, and challenging to manage.

Solution

Use technological tools for survey data collection and analysis, especially when gathering pre- and post-implementation data.

Survey design is crucial in order to obtain the data you need.

Discussion

Surveys are a classic method for data collection. They are flexible, easy to implement, and offer a nearly limitless range of data with reliable results. The data gathered during an effective survey provides a unique opportunity to obtain detailed insight into a program. Because you can gather large amounts of feedback directly from individuals who are affected by the program, surveys act as the finger on the pulse of your project and can measure its strength.

Surveys can be utilized in a number of ways. For example, they can measure change-over-time when used to monitor pre-program conditions against post-program conditions; they can be used to gather extensive data regarding perceptions of participants when it is important to judge a participant’s motivation during a program; and they can ensure the consistent collection of data because all respondents receive exactly the same questions in exactly the same way.

Harnessing Technology

However, traditional surveys can be very costly to administer and analyze. Much of the cost of data collection is in providing staff and hard resources, such as paper copies of a survey. By harnessing the power of technology, you can create, conduct, and review surveys at a much lower cost, while gathering information with a speed and efficiency that you could never achieve using paper surveys. In addition, online data management systems can automatically convert the data into a useful state for analysis.

Online surveys have the following additional benefits:

• Reducing lengthy delays between collecting data and interpreting the results.
• Eliminating the need for data entry costs.
• Removing the problems caused by lost or damaged paper surveys.
• Eliminating problems related to interpreting responses that are written by hand.
• Reducing the time it takes for participants to complete and submit a survey.
• Reducing the amount of consumable materials required to administer the survey.
• Placing the survey in a location where it can easily be found and administered.
• Broadening the audience of the survey while improving response rates.

Technology can also be used to incorporate the benefits of surveys into meetings or feedback sessions such as focus groups. An electronic survey method called an Audience Response System uses wireless keypads which enable attendees to complete surveys from their seats; the data is then immediately compiled into a database. This method is especially useful with large audiences, such as those attending conference presentations. Traditional methods of collecting data from audiences utilize large quantities of paper and pencils and require the patience of people who are ready to move on to the next event.

Limitations of Surveys

While there are many benefits of surveys, they also have some limitations. For instance, a survey only gathers information about the questions asked. In contrast, during an interview, the interviewer can explore important subjects in depth, as they are uncovered. Generally speaking, surveys are effective only when those surveyed have at least a moderate degree of literacy. Although oral surveys are possible, they are usually impractical.

Designing Surveys

To design an effective survey:

• Determine what type of survey you need to send out and to whom.
• Make sure that respondents have the skills, knowledge, and access to the required technology in order to complete the evaluation.
• Collect data from a statistically appropriate sample size. It is not necessary to survey every participant.
• Provide incentives for participants to complete the survey.
• Consider whether you need to provide assistance to your participants in order for them to complete the survey. If, for example, you are performing an evaluation of a childcare facility, you may need to provide childcare during the survey.
• Protect respondents’ privacy when you are asking for sensitive information. When administering online surveys, data can be easily encrypted to provide anonymity.
• Time your survey according to the information you are hoping to gather. Pre-implementation data can be used to guide your program development, whereas post-implementation data can be used to gauge your program’s success and make recommendations for change. Look to your program objectives to help you determine the kind of data you need.

Designing Valid Questions

Good questions are essential to a survey’s success. Questions should be extremely focused to avoid unexpected responses that confound data analysis.

When formulating a question, start by determining the type of answer you wish to receive. If you want a lengthy description or opinion, ask an open-ended question, such as “What did you do today?” If you want a “yes” or “no” response, ask a closed question, such as, “Did you go to the supermarket today?”

Good questions find a way to measure the expected response. To this end, make a list on possible indicators that relate to the type of response you are seeking. For example, if a dentist is trying to determine the quality of a person’s dental hygiene habits, indicators of good dental hygiene might be the frequency of brushing and flossing, the elapsed time since the person’s last dental visit, or the length of time spent brushing one’s teeth.

Once indicators have been determined, you can formulate a valid question. In the case of the dental patient, the cunning dentist inquires, “How many times a week do you floss?” He would not ask, “What is the quality of your home oral hygiene practice?” because it leads only to the most likely response, i.e. “It is fine.”

To ensure that your questions will generate the responses you are seeking, offer a trial version of the survey to a sample of participants. In fact, it is often good practice to run several trials before committing yourself to the final version of the survey. It might even be helpful to ask your participants if they considered other ways to answer each question. If they answer affirmatively, you may need to revise your questions.

Likert Scales

An effective method for obtaining consistent survey responses is to use a Likert scale. A Likert Scale allows a participant to provide feedback that is slightly more expansive than a simple close-ended question, but that is much easier to quantify than a completely open-ended response.

A Likert Scale lists a set of statements (not questions) and provides a 5-point or 6-point scale for which the participant can rate his/her level of agreement or disagreement with the statement. For example:

I was able to locate the information I needed.

• Strongly Agree
• Agree
• Not Sure
• Disagree
• Strongly Disagree

With a 6-point scale, you might choose a response such as “Not Applicable” to ensure that you can allow participants a way to opt out of the question and still quantify their response.

Because there are a limited number of responses, each response can be translated into a numerical value that can be used for easier statistical data analysis. For example, a “Strongly Agree” response may be assigned the value of 1, “Agree” as 2, and so forth. But be careful: if you use numerous Likert scales and cluster them together, participants often settle into a pattern and subsequently circle the same response for different questions. If possible, space them out over the course of the entire survey.