Knowing What and How Much to Evaluate

Planning the scope of a program evaluation is crucial. Without appropriate planning, the evaluation can be too narrow - not addressing enough questions, off target - addressing relatively unimportant questions, or too broad - addressing many questions without the necessary depth (often at great expense). Evaluators must determine the most important aspects of a program and the most important evaluation questions. Key audiences play an important part in determining the key program aspects or questions. As evaluators develop a full understanding for the program to be evaluated, or object, they may pose other important questions to be addressed.

Once all potential questions are generated, evaluators (often with the assistance of key audiences) select the most important questions based on a combination of factors including:

• Questions deemed important by key audiences (i.e., questions that need to be addressed as required by funding sources)
• Investigation of program aspects that are key to the functioning of the program
• Investigation of program aspects that may involve health and welfare of stakeholders (i.e., Are all participants able to conduct CPR after the training?)
• Questions that are feasible to address

Once these key questions are identified, the evaluators review them with the person(s) requesting the evaluation to make sure that they agree to the scope of the evaluation. In the planning stages of an evaluation, many questions are often generated. It is important to realize that an evaluation rarely addresses all questions that are posed.

Additional suggestions:

• Make sure that the evaluator and key audience(s) agree on the prioritized question(s).
• Evaluators may have important input into what an evaluation should address.
• Don't select evaluation questions just because they're the easiest ones to answer.
• Consider all potential sources of information (there is sometimes readily available information).

Example: A Teacher Center has implemented two different inservice courses on two different types of reading approaches. One approach has a strong track record documented through several research studies. The other is popular with teachers but has no available documentation about its effectiveness. The evaluator chooses to use previous documentation (research studies) to certify the first program as an effective one. She also devotes relatively minor resources to document that participants are implementing the first program as prescribed and that they have the necessary time and materials for successful implementation. She puts additional resources into evaluating the effectiveness of the second program, that becomes a major focus of that year's evaluation.

How Good is Good Enough?: Establishing Criteria for Program Success. 

While it is important to collect accurate, objective information when evaluating programs, most all evaluation efforts require value judgments. Hopefully, these judgments are based on a preset criteria that is agreed upon by the evaluator and or the audience(s).

For example, if a program is implemented to increase Internet research skills, an objective observation procedure may be established. For example, student use can be observed using a checklist and or a test of skills. The agreed upon objective for the program may be arrived at through negotiation between the evaluator and the inservice course instructor (i.e., students will follow 90% of the steps independently, and 100% of the steps with peer assistance).

In some cases, external criteria may be helpful in establishing outcome criteria. These sources may include: already defined program objectives, legal requirements, state guidelines or reference points, or outcome results of competing programs or institutions.

The degree to which evaluations have “research grade” rigor in terms of data collection and analysis is largely dependent on the demands of evaluation audiences, the political context, and the stakes of the evaluation. The evaluator should document in the final report how outcomes were assessed and how the criteria were established.