Determine data -collection methods

The Kellogg Handbook argues that "no single evaluation method can document and explain the complexity and richness of a project."⁴ Thus, in making decisions about methodology, there is no one correct approach but instead a range of options to choose among. One is not inherently better than the other, but quantitative methods will provide different information than qualitative; interviews do expose different issues than will tests and assessments or document reviews.

How then to choose the evaluation design? Patton argues in the first place that it is important to involve intended users in methods decisions⁵ . If the design makes sense to the intended users, the credibility of the results will be increased, as will the likelihood of the results being utilized.

After ensuring the involvement of intended users, an initial step in deciding upon a design is to navigate the difficult decision of whether to use quantitative methods (essentially counting and measuring) or qualitative (essentially asking and interpreting how a project functions and what that means to the people involved). One position, advocated by the Kellogg handbook, is essentially that both methodologies have much to offer and should both be used. Qualitative methods can "help the evaluator understand the context of a project;" the context then "sets the framework for a meaningful understanding" of quantitative data. Patton has a more complicated response to the issue, arguing that, in place of quantitative v. qualitative, evaluators should seek simply to create fair/ conscientious designs by presenting both (or multiple) sides of issues, and being aware of, and presenting, the evaluator's own biases.⁶

Patton argues further that neither quantitative nor qualitative can be evaluated as good or bad methods in the abstract; it all depends upon the purpose of the project being evaluated, and the goals of the evaluation. "The more a program aims at individualized outcomes," he posits, "the greater the appropriateness of qualitative methods. The more a program emphasizes common outcomes for all participants, the greater the appropriateness of standardized measures of performance and change."⁷

In addition to the question of whether to approach evaluation from a quantitative or qualitative stance is the question of what data collection method to use. Should you, for instance, employ observation, interviews, written questionairres, tests and assessments, or document review? The Kellogg handbook suggests that in making the decision evaluators must be informed by considerations of: resources available; sensitivity to participants/ respondents; credibility; and importance of the information.⁸

Collecting data

"Collect only the information you are going to use, and use all the information you collect."⁹

Once you have decided upon evaluation questions and the methods to employ, you are ready to collect data. A valuable first step to avoid unnecessary work and to utilize to the fullest work that has come before yours is to examine and learn from existing information about "your target population, community or project."

When you perform the data collecting outlined in your evaluation plan, you should remain open to the possible need to "revise data-collection strategies based on initial analyses. What is working? What is not working? What pieces of data are still missing?"¹⁰ Data collection and methodology design should be flexible, responsive, and iterative, focused constantly on realizing the primary purpose of the evaluation.

Footnotes:
1. Michael Quinn Patton, Utilization Focused Evaluation: The New Century Text, 3d Edition, Sage Publications, at 126.
2. W.K Kellogg Foundation Evaluation Handbook (1998) at 69.
3. Id.
4. Id. at 70.
5. Patton, supra note 1, at 242.
6. Id at 282, citing Egon Guba (1981).
7. Id. at 290.
8. Kellogg, supra note 2, at 71.
9. Id. at 85.
10. Id. at 87.