The Application of the Logic of Evaluation to the Real World
Understanding the four-step logic of evaluation process.
Posted October 17, 2018
Evaluation is the process of determining the merit, worth, or significance of something, and an evaluation is a concrete product of that process. According to Michael Scriven in The Logic of Evaluation, “Professional evaluation is evaluation done in a systematic and objective way with a degree of expertise that requires extensive specific training or learning.” The logic of evaluation is an example of the step-by-step process that can be adapted to help us make judgments on the quality of products, programs, policies or personnel.
Defining the Process
The logic of evaluation is a four-step process that includes: 1) defining the criteria that will be used to evaluate something; 2) setting standards of performance on those criteria; 3) measuring the actual performance; and 4) synthesizing the results to reach an evaluative judgment. What is especially interesting about this process is that people often do it intuitively, but they may skip steps that could lead to misleading conclusions. This article hopes to clarify what this process looks like and how it can be applied to a real-world situation.
The first step of defining the criteria is focused on generating a list of dimensions or elements that would help us capture the performance of a product/program/policy. For example, if we wanted to evaluate a car, some of the criteria we would generate could include the number of people it seats, the miles per gallon (MPG), its crash safety rating, how it looks (its aesthetics), and its price. This is not a comprehensive list of criteria, but it offers a starting point to get us thinking about the dimensions that go into how we evaluate a car. It’s also important to note that not all criteria are equally important, so in the case of the car evaluation we may want to place a lot of importance on the safety criteria, so even if a car is reasonably priced, looks great, gets great MPG, and seats six people, but happens to fail the simplest safety crash tests then it automatically fails as a car since safety is probably one of the most important features.
Once we figure out the criteria, then we have to determine how well something needs to perform on those criteria (i.e. standards). For example, if we use MPG as the criteria, we would also have to determine a standard of performance for that criteria. This means that we have to figure out an acceptable level of performance on that criteria, so would 10 MPG be acceptable? 35 MPG? 50 MPG or even 100 MPG? There are different ways of approaching this process, the simplest way is to set a cutoff level of performance where you decide that due to the price of gas that a car needs to get at least 35 MPG for it to be considered a “good” level of performance on MPG. Another way is to develop a point system where the better the MPG the more points a car gets (e.g. 10 MPG gets 10 points, 100 MPG gets 100 points). You can go through this process with the other criteria, so you can set a price point for how much you want to spend on a car, identify the number of passengers that you want to be able to seat, and maybe even develop a rating system for how good the car looks.
When you finish the criteria identification and standard setting, the fun part of measuring and collecting information starts. In the car example, this would mean that we would start to collect criteria data on the different cars we are considering. You can collect the data directly, so you can put a gallon of gas in a car and drive it around a track to see how far it goes (this can help you determine MPG) or find existing data sources about the car’s performance. Some of the performance data is easy to get to, for example, the number of people that can fit in a car can be determined by looking at the interior of the car. Other information may be hard to collect directly, and you will need to rely on other outside sources. For example, crash safety data would need to be attained from the Insurance Institute for Highway Safety (IIHS). Once all the measurement is completed and the data is collected then the most interesting step in the evaluation process begins.
The synthesis process is where you take the criteria that were identified, the standards of performance that were determined, and the measurements that were collected to come up with a final evaluative conclusion or judgment. In the car example, this would mean that you would compare the data that you collected about each car to the standards of performance that you identified and allocate points to how well they performed on that criteria. For example, you can have a car that gets 40 MPG (40pts out of a 100 points), that looks acceptable (e.g. gets 50pts out of 100 points), performed excellent on the safety crash tests (80 points out of 100 points), but is very expensive (10 points out of 100 points). The total score for that car would be 180 out of a possible 400 points. Another car could get 300 points out of 400, or 50 out 400, depending on how well it performed during the measurement process. Once you look at the point allocation then it would be easy to figure out which is considered the “best” car.
It’s also important to remember that as part of this process you have to determine if these cars passed the most important or critical criteria (e.g. crash test safety). So even if a car performs really well on all the other criteria (100 points on price, 100 points on aesthetics, 100 points on MPG) but fails the safety tests then the car needs to be removed from consideration and judged as a “failed” car.
I hope that this illustration offers some insights into how the logic of evaluation approach can make abstract concepts more explicit and systematic. The logic of evaluation approach can be used in many different situations such as the evaluation of programs like student retention efforts, pregnancy prevention initiatives, and drug rehabilitation interventions. It can also be used in the evaluation of various policies that affect our lives, and, as illustrated by the car example, be used in our regular decision-making process. However, there are added complexities to this process that will be addressed in future articles, for example when evaluating a school program, the principal’s criteria for performance may be very different from what parents think and the process of negotiating between these differences could affect the credibility and usefulness of the final evaluation judgment. We will explore these complexities and the role of evaluation in decision making in the next installment.
Tarek Azzam, Ph.D., is an associate professor at Claremont Graduate University's Master’s in Evaluation & Applied Research Online program.
Scriven, Michael, "The Logic of Evaluation" (2007). OSSA Conference Archive. 138. https://scholar.uwindsor.ca/ossaarchive/OSSA7/papersandcommentaries/138