Our evolving approach to ELA data analysis

by Ryan McCarty

One thing that has struck me ever since I joined the team at ANet is that we are truly a learning organization. We constantly ask ourselves how we can better serve our school partners and refine our resources.

At the moment, we’re honing our approach to ELA item analysis and planning. It’s an exciting time to be involved in this work because there’s a sense of belonging to a larger community of educators who embrace the challenge of the Common Core State Standards. We’ve been examining ELA performance data, analyzing what’s working in our schools, and collaborating with literacy experts so that our partner schools can better help students develop the advanced literacy skills they need for college and career.

Why has our approach changed?

A decade ago, ANet thought about ELA data in much the same as we did math data—as discrete skills tied to specific standards. Our approach reflected the conventional thinking in the literacy field at time. Reading strategies were treated largely the same regardless of the type of text. The conventional wisdom was that we could best support readers by making texts, and reading, easier.

The Common Core Standards have helped shift the conversation by putting an emphasis on the “range, quality and complexity of texts.” The standards were influenced by research (cited in Appendix A of the CCSS) showing that what differentiated the students who excelled on reading assessments from those who struggled was not their performance on items aligned to particular skills, it was their ability to read complex texts. As a result, the CCSS devote Standard 10 to the need for students to “read and comprehend complex texts independently and proficiently.”

In addition, many literacy experts now argue that it’s not helpful to draw conclusions about students’ instructional needs based on solely on analysis of multiple choice data tied to specific skills (e.g. main idea, making connections etc.). Reading is a complex language activity that requires students to do several things at once, and there are many reasons why students may struggle beyond the skill or standard tagged to a test item. As literacy expert Tim Shanahan states, “When a kid can’t answer a main idea question, it doesn’t mean the main idea portion of his brain is broken.” The inability to read the text fluently, for instance, or difficulty using context to determine the meaning of academic vocabulary could also cause students to miss a “main idea” question.

How has our Approach changed?

The Central Importance of the Text

Prompted by the results of an audit we commissioned from Student Achievement Partners, an organization founded by creators of the Common Core State Standards, we came to some important realizations about our approach to ELA analysis and planning. Though our analysis process included reading the test passage first, mastery was still being judged in terms of individual standards rather than viewing reading more holistically. This caused us to narrow our focus too quickly, losing the connection to the text. 

This was problematic because finding the central idea of an excerpt from a literary text like the novel Watership Down, for example, is very different from finding the central idea of an article about light pollution. We realized that digging into the complexities of a test passage prior to analysis could empower teachers to then look across the multiple items and standards that comprise an assessment to gain deeper insight into what students understood and where they struggled.

Looking at Multiple Sources of Data

By inviting expert partners to challenge our thinking, ANet has gained a deeper understanding that, while multiple choice items as a set help us understand student comprehension of text, we shouldn’t draw hard and fast conclusions about what’s needed based on item analysis alone. We now use student work to gain more insight about what students can do and what meaning they are making from text, as opposed to focusing on where they struggle.

Analyzing the text, looking at student work, and examining multiple choice items enable us to triangulate on student understanding. Each facet contributes to our understanding, allowing us to draw stronger inferences about what students need next for instruction.

Join us on the journey

The evolution of ANet’s approach to ELA data analysis has been a long and worthwhile journey, driven by our commitment to continue to learn and grow as an organization. In the coming months, we’ll share some of the innovative work we’re doing with our partner schools around ELA data analysis. You’ll hear first-hand accounts and see artifacts from the people who work alongside school leaders every day. We’re excited to share what we’ve learned so far on our journey, and our path forward in the months to come.

Ryan (@RyanP_McCarty) is a coach with ANet Massachusetts.