Preparing for an Evaluation

Preparing for an Evaluation

Preparing for an evaluation involves identifying:
1) those individuals who will use the information generated by the evaluation, and
2) the uses to which they will put the information.
It then requires creating evaluation questions, and selecting an evaluator.

Identify Stakeholders and Primary Intended Users

"Intended users are more likely to use evaluations if they understand and feel ownership of the evaluation process; they are more likely to understand and feel ownership if they've been actively involved." ¹

The first important steps to take in performing an evaluation are to identify all stakeholders and to determine who are the intended primary users of the information to be attained from the evaluation.

Stakeholders are those persons or groups that "have an interest in the project being evaluated or in the results of the evaluation."² Stakeholders in a given project may include, for example: funders, staff, administrators, program participants, collaborating groups and community members.

As many of the identified stakeholders as possible should be involved in early discussions about the evaluation; such involvement increases the likelihood that as many stakeholders as possible will:

• be invested in the evaluation,
  
• be willing to work to make evaluation successful,
  
• contribute their perspectives to the information gathered,
  
• accept the recommendations resulting from the evaluation,
  
• facilitate implementation of recommendations. ³

While it is unlikely that all stakeholders will want to be, or will be able to be, involved throughout the evaluation, it is important that the intended main users of the evaluation results (those who Michael Quinn Patton calls the "intended primary users") be identified and engaged. Patton emphasizes the importance of gathering information for specific individuals who will have the willingness, authority, and ability to put evaluation results to work. Without this focus on specific intended users, he argues, it is too easy to collect information that, while potentially interesting, will never be utilized.

Determine Primary Purposes and Intended Uses of the Evaluation

Once the primary intended users are identified, the evaluation team needs to determine the primary purposes and intended uses of the evaluation for those users. Patton recommends asking users what things they would like to know that would make a difference to what they do. ⁴ He argues that there are three broad categories of things for which users would want information:

1. Rendering judgments: e.g., did this work?, did this program attain its goals?
2. Facilitating improvements: e.g., what are the program's strengths and weaknesses?
3. Generating knowledge: enlightening program staff and the wider public.

It is important to identify in which of these categories of questions the intended users are primarily interested. When you identify this, you identify the primary purpose of the evaluation, and thus can design the evaluation to serve that purpose.

Developing Evaluation Questions

The most important thing in developing evaluation questions is to keep the primary purpose of the evaluation firmly in mind. Is the purpose, for example, to determine whether a particular program accomplished what it was created for, so that the director of the agency can make a decision whether to continue or discontinue the program? Or, alternatively, is the purpose to determine the strengths and weaknesses of that particular program in order for the director not to decide its fate, but rather to seek to improve it? Those two different purposes for gathering information require very different evaluation questions.

In developing evaluation questions, it is important to seek input from multiple sources with different perspectives on what information is relevant to the primary purposes of the evaluation and where to find such information. Kellogg recommends seeking "input from as many perspectives as possible to get a full picture before deciding on questions."⁵ Different stakeholders may have different opinions on how to evaluate, for example, whether a program accomplished the goals for which it was created - it is important to capture and make use of those various opinions.

Selecting an Evaluator

An evaluator can be internal (already affiliated with the program or agency to be evaluated) or external (unafftiliated with the program or agency). There are pros and cons to each option.

An internal evaluator will be less costly, more familiar with the project, the staff and community members, and have access to organizational resources, and opportunities for informal feedback from stakeholders. In addition, any knowledge or insights gained from the evaluation stay in house when the evaluator remains. He or she might, however, lack perspective and/ or technical skills.

An external evaluator will have access to evaluation resources, broader evaluation experience, and will bring a new perspective. He or she might, however, be more costly, have limited knowledge of the project's needs and goals, limited access to project activities, and less ability that an internal evaluator to facilitate implementation of recommendations.

It is possible to set up a partnership whereby an internal evaluator conducts the evaluation and an external evaluator assists with the technical aspects and the gathering of specialized information.⁶ This can be a very effective utilization of the best of both options.

Wherever the evaluator is drawn from, he or she must have certain skills to do the job well. These include "the ability to listen, negotiate, bring together multiple perspectives, analyze the specific sittuation, and assist in developing a design with the evaluation team that will lead to the most useful and important information and final products." ⁶

Footnotes:
1. Michael Quinn Patton, Utilization-Focused Evaluation: The New Century Text, 3rd ed., Sage Publications, 1997, at 22.
2. W.K. Kellogg Foundation Evaluation Handbook (1998) at 48.
3. Id. at 49.
4. Patton, supra note 1, at 32.
5. Kellogg, supra note 2, at 54.
6. Id at 58.
7. Id at 59.