Data That Tells Us When a Strategy Is Working: Instruction
Part I: Educators can use data to determine which instruction is sticking.
Posted Nov 23, 2020
Most educators are eager to analyze and then act on the data they see (van der Meij, 2008). Data forms a key component when determining which strategies are effective and in what ways. For example, if a teacher implements a new writing program, his or her data (from traditional assessment, formative assessment, observation, student survey feedback, parent survey feedback, projects, etc.) might reveal:
- Overall students are not performing significantly better, but their performance in Reading Informational Text has increased by 51 percent.
- 42 percent of students are enjoying the new reading program “much more” than the old program.
- 27 percent of parents note “some confusion” as to what is supposed to take place at home in relation to students’ reading.
- Assessments reveal a 21 percent increase in students’ responses to questions requiring higher-order thinking, and observation reveals a similar trend.
You can imagine how this feedback could shape the teacher’s use of the reading program, her communication with parents, and more to produce greater student achievement. However, teachers cannot implement the best strategies for students if they do not understand the data on whether or not the instruction is ‘sticking’ or not.
In national studies at districts known for strong data use, teachers had difficulty with question-posing and data comprehension, and teachers correctly interpreted given data only 48 percent of the time (U.S. Department of Education Office of Planning, Evaluation and Policy Development, 2009, 2011). Other research supports this trend: educators use data to determine which strategies are working and how, but they do not always understand the data they use. In fact, many educators incorrectly analyzing student data do not realize they do not understand the data. For example, in a study of 211 educators of varied roles and backgrounds, those who perceived their own data analysis skills as “very proficient” answered data analysis-based questions correctly only 27 percent of the time (Rankin, 2013).
Teachers and other educators are highly skilled (American Institutes for Research, 2013), well-educated (Papay, Harvard Graduate School of Education, 2007), and intelligent (Hurley, 2012); educators should not be wrongly blamed for struggling with data use. Though improved professional development and staff support can always help, a large culprit in data misunderstandings is: even educational data that looks simple is actually more complex.
Consider the following pitfalls that can occur when an educator tries to determine whether or not a new strategy is working:
- If an assessment’s domains or clusters differ in difficulty, the domain where the highest score was earned does not necessarily indicate performance in that area was better than performance in other areas.
- If the same survey is given to students in grades 3 through 6, the same answers might mean different things from one grade level to the next due to students’ varying ability to understand questions and response options (and if the survey is modified by grade level, other discrepancies can exist).
- If one measure indicates students scored Proficient on one Common Core State Standard (CCSS) and another measure indicates students scored Not Proficient on a different CCSS, the two scores cannot be evenly compared.
- If local writing samples are not scored with vertically scaled scoring practices and rubrics, then results cannot be evenly compared from one grade level to the next (for the same year) or from one year to the next (for the same grade level).
Since education data is hard to understand, we need to make it easier for educators to use it with fidelity so they can get true feedback on strategies’ effectiveness.
American Institutes for Research (AIR). (2013). Most teachers "highly qualified" under NCLB standards, but teacher qualifications lag in many high poverty and high minority schools. Retrieved from http://www.air.org/reports-products/index.cfm?fa=viewContent&content_id=417
Hurley, D. (2012, April 22). Can you build a (better brain?) The New York Times, MM38.
Papay, J., Harvard Graduate School of Education (2007). Aspen Institute datasheet: The teaching workforce. Washington, DC: The Aspen Institute.
Rankin, J. G. (2013). Over-the-counter data’s impact on educators’ data analysis accuracy. ProQuest Dissertations and Theses, 3575082. Retrieved from http://pqdtopen.proquest.com/doc/1459258514.html?FMT=ABS
U.S. Department of Education Office of Planning, Evaluation and Policy Development (2009). Implementing data-informed decision making in schools: Teacher access, supports and use. U.S. Department of Education (ERIC Document Reproduction Service No. ED504191)
U.S. Department of Education Office of Planning, Evaluation and Policy Development (2011). Teachers' ability to use data to inform instruction: Challenges and supports. United States Department of Education (ERIC Document Reproduction Service No. ED516494)