Evaluation isn’t helpful unless you measure the right things.

Problem

You’ve implemented the feedback from your focus groups, but your project is still failing. You are confused because test users say your program is great, but no one is taking advantage of it. You need to know what is working and what isn’t.

Solution

Evaluate your product or program using multiple evaluation techniques (social scientists call this “triangulation”), such as candid data collection, contextual observation, mind mapping, paper prototyping, and microsurveying.

Test users may tell you what you want to hear. Dig deeper to get candid, more useful data.

Discussion

During your focus groups, participants may have told you what they thought you wanted to hear while withholding negative feedback about the program. Using alternative evaluation methods can help you better understand your target audience so that you can revise either the program or its marketing plan.

Candid Data

Many times, your test users want to see you succeed and may neglect to advise you about potential problems with the program. This is especially true if your test users are also your staff members. Gathering more candid data from your test users can tease out information that wasn’t shared during focus groups. Pointing out that you are encoding responses for anonymity will help ensure honest feedback.

To collect candid data, you can use an independent facilitator to interview users about the program. You can also use online surveys and feedback forms to collect and encode data. You can even bring in new users who don’t have a vested interest in the success of the program to give feedback and identify problems.

Observing people in context allows you to see for yourself what works and what doesn’t.

Contextual Observation

Contextual observation allows you to watch users perform tasks required by the program in order to see where they stumble and where they move easily. For example, if a user doesn’t understand where to click in order to advance to the next screen, you can witness him or her trying to locate the correct button. Because observing where users venture when they are unguided is crucial, it’s important not to interject as you observe. Contextual observation may even be helpful in collecting candid data because the user doesn’t have to tell you when he or she is having trouble – you’ll see that for yourself.

Mind maps allow you to learn how your target audience thinks.

Mind Maps

Mind maps are a web of information linked together by common thoughts, much like the way the Internet operates. One thought leads to another, which leads to another, and so on. Instead of using linear “A to Z thinking”, mind maps require a stream-of-consciousness approach. Mind maps are particularly useful when trying to understand the thought processes of your target audience.

To generate a mind map, start with a key concept or image written on the middle of a page. From that central concept, write or draw related concepts and connect them with lines to the central concept. Use arrows, icons, or other visual aids to show links between different elements. If some of the secondary concepts relate to each other, indicate those connections by lines as well. Keep branching out concepts until the idea has been thoroughly explored. Avoid self-editing. If you have an idea as you are mapping, write it down. You can always remove it later.

Paper prototyping allows you to see the progam through the eyes of the users and to understand the functions users want.

Paper Prototyping

Paper prototyping helps you understand your user’s perspective. In this method, you ask potential users to create a prototype user interface on paper. You describe the purpose of the program to the user and let him or her work out his or her idea of an optimal interface. You may choose to provide some initial features or capabilities of the interface, but for the most part you should leave the design process up to the user.

The user will draw and describe features believed to be important when participating in your program. You may even ask him or her to fit their features on to your existing system framework. To do this, you can provide screen prints of your basic interface and ask them to cut out the features they have created and insert them onto the existing template.

When using paper prototyping, keep in mind that your users don’t have the technical expertise to know what is or isn’t possible. Indeed, their mockups will almost certainly lack the refinement of a subject matter expert and may be to some extent unusable in real life. That is irrelevant, however, because the insight you gain with this method is to see the progam through the eyes of the users and to understand the functions that users want. Their ideas – not their finished products – can show you what is important and point you in the right direction. This tool is particularly useful for helping you move from the theoretical stage of your project development into the implementation stage, where you turn your ideas into an actual product.

Microsurveys give you immediate access to vast amounts of data.

Microsurveys

Concise microsurveys at the end of a web page are designed to give users a brief opportunity to send you feedback on what they have just used or read. A common microsurvey is one that asks the user, “Was this information useful to you?” The user clicks “yes” or “no,” and may be allowed to make a comment to supplement his or her response. For example, if you are testing a new system, you can place a microsurvey at the bottom of every page in order to collect feedback from users as they work through the program. This opportunity to give immediate feedback results in your having quick access to a vast amount of useful data.