Verified by 
By Dr. Russell T. Warne (@Russwarne)
Last spring, over 4 million high school students took more than 2 million Advanced Placement (AP) tests. Sponsored by the College Board, AP courses are taught by high school teachers and allow students to earn college credit by passing a test at the end of the school year. According to the College Board, students in AP courses learn more material, are more prepared for college, and finish a bachelor’s degree earlier than non-AP students.
And yet, there are two reasons that make it too simplistic to merely state that the AP program leads to academic benefits. The first is that the AP program—like many educational programs—has many components, and it is often difficult to say which one(s) lead to academic benefits. Second, students often choose whether to take AP courses, and AP students often differ from non-AP students in important ways. This confounds the impact of the program with pre-existing differences between the two groups of students. A recent study that my research team conducted (Warne, Larsen, Anderson, & Odasso, in press) illustrates both points well.
Lots of “Moving Parts”
Educational programs are rarely simple. They often include many “moving parts”—each of which contributes to the program as a whole. It can be difficult for researchers to determine which components of an educational program are effective in increasing learning and which are unimportant. In the AP program, for example, there are 34 tests that cover college-level topics ranging from English, to statistics, to music theory, to world history. Each of these topics has a curriculum and a standardized AP test, and each classroom has a teacher who creates assignments, designs tests, and prepares students for the test. School, district, and state characteristics can also contribute to how a program functions.
In our study of the AP program, my co-authors and I decided to isolate one component of the AP program: the test. We wanted to investigate whether merely enrolling in an AP class led to academic gains for students or whether it was necessary for students to take the AP test. To do this, we used a dataset with over 90,000 high school students in it, and we divided the students into four groups: (a) non-AP students, (b) AP students who did not take the AP test, (c) AP students who took the test but did not pass, and (d) AP students who took and passed the AP test.
We compared the ACT scores of the four groups of students. If the class itself were an active component, then there should be a difference in the ACT scores between groups (a) and (b). If the test were an active component, then group (c) should have a higher ACT score than group (b). We found that the average ACT scores were 21.0 for group (a), 21.3 for group (b), 22.5 for group (c), and 26.5 for group (d). The results indicate that enrolling in the AP class gives students a small benefit (0.3 points on the 36-point ACT sacle), but that taking the test provides bigger benefits (1.2 points).
Pre-Existing Student Differences
On the surface, the results in the previous paragraph would seem to indicate that both the class and the test are effective—although the test clearly has the stronger impact. But the problem with this sort of comparison is that these four groups of students are not equivalent, because students choose whether to enroll in AP courses. Therefore, AP students are different from non-AP students (Sadler & Sonnert, 2010). Psychologists call this self-selection, and it is very common in many branches of psychological research. In fact, there are two levels of self-selection that researchers have to consider when studying the AP program because AP students also have the option of whether to take the test. The reasons for self-selection into the AP program or to take the AP test are often unclear, but administrative policies (like a state’s decision to pay the AP fee for students or a school’s decision to offer a particular AP class) and student characteristics (such as how prepared a student is for the test) seem to matter.
To handle self-selection, it is necessary to control for pre-existing student differences. Previous researchers (e.g., Long, Conger, & Iatarola, 2012; Sadler & Sonnert, 2010) have controlled for variables in which the groups of students differ. For example, AP students tend to have higher high school grade-point averages (GPAs), are often more motivated, and are less likely to come from low-income homes. Without controlling for variables like these, the impact of the AP program would be confounded with differences between student groups.
In our study, we controlled for about 70 student variables, including sex, racial/ethnic group, grade-point average, academic achievement test scores, and whether the child was in special education. We then compared the four groups’ ACT scores after making these statistical adjustments. Non-AP students had an average ACT score of 21.4. AP students who did not take the AP test had an average ACT score of 21.2. AP students who did not pass the AP test had an average ACT score of 21.7. Finally, AP students who passed the AP test had an average ACT score of 24.6.
Comparing these results to the unadjusted results shows the importance of making these adjustments. The benefit of taking the AP class evaporates (as shown in the score difference between the first two groups), and the impact of taking the test drops from 1.2 points to 0.5 points—over 50%. Without controlling for pre-existing differences, the AP program looks much more effective than it really is. In the end, my coauthors and I reached that the AP test is an important component of the AP program. Also, students who pass the test receive more benefits than their classmates who do not take the AP test, or who take the test and do not pass it.
Conclusion
This study is an example of the challenges of evaluating and understanding educational programs. The many components of some educational programs and the pre-existing differences among groups of students present challenges to educational psychologists. But, the challenges are not insurmountable. I invite readers to investigate the original article, or to watch a video I made about the study. Both are available (for free) at the URLs below:
Full article: http://www.tandfonline.com/doi/pdf/10.1080/00220671.2014.917253
Video summary: https://www.youtube.com/watch?v=aViwSlcu2F0
References
Sadler, P. M., & Sonnert, G. (2010). High school Adanced Placement and success in college coursework in the sciences. In P. M. Sadler, G. Sonnert, R. H. Tai & K. Klopfenstein (Eds.), AP: A critical examination of the Advanced Placement Program (pp. 119-137). Cambridge, MA: Harvard Education Press.
Long, M. C., Conger, D., & Iatarola, P. (2012). Effects of high school course-taking on secondary and postsecondary success. American Educational Research Journal, 49, 285-322. doi:10.3102/0002831211431952
Warne, R. T., Larsen, R., Anderson, B., & Johnson, A. O. (in press). The impact of participation in the Advanced Placement program on students' college admissions test scores. The Journal of Educational Research. doi:10.1080/00220671.2014.917253
