Searching for the Cause of Higher NAEP Scores in Urban School Districts

The 2013 National Assessment of Educational Progress scores are in, and according to the Trial Urban District Assessment (TUDA), there are some important findings in several urban districts across the United States. While a few districts evidenced score declines, a significant number of urban districts showed significant gains.

NAEP data: Statistically significant educational gains in urban school districts

TUDA’s executive summary states that, “Students in eight districts scored higher in 2013 than in 2011 in at least one subject and grade combination, with some districts making gains across multiple subjects and/or grades.” Of the latter, Washington, D.C. public schools had a particularly strong result, showing overall gains in both fourth and eighth graders in both subjects, though their overall scores still fared lower than large city averages.

According to Michael Casserly, executive director of the research and policy group Council of Great City Schools, the gains were statistically, educationally, and likely politically “significant,” noting that large cities have dedicated great energy into education reform and that it is likely these efforts have paid off. Education Secretary Arne Duncan believes this progress is “encouraging,” though he notes that there is still work to do to close achievement gaps.

Cause of higher test scores is difficult to pinpoint

Parents, teachers, and educational policy researchers are looking for the source of these continued improvements in order to apply them to nationwide educational reform. Some point to the significant gains in the District of Columbia as evidence that various controversial educational reforms in the district have worked, including firing low-performing teachers and quickly adopting the Common Core Standards.

It is important to note that the DC charter schools are not included in the TUDA data, though the Washington Post argues that district information shows, “black students in the city’s public charter schools overall did better than black students in the traditional public school system.”

“Haphazard” school reform implementation

What, then, should educators thank for the significant gains in DC and other urban schools? According to the LA Times, “Researchers say it’s impossible to ferret out the reasons because the implementation of school reforms tends to be haphazard, overly broad, and seldom assessed.”

They point out that test scores in the L.A. Unified School District rose despite increased class sizes and without teacher evaluation being tied to testing results, but that it is entirely possible these gains might be related to old policy changes that are starting to pay off. It is even possible that such gains aren’t tied to educational policy changes at all — instead, they may be linked to factors like increased education levels among the students’ parents.

Educational gain data and “statistical noise”

Some researchers suggest restraint when handling educational gain data. Economics professor Dale Ballou cites the difficulties of standardized testing and the significant “statistical noise” that arises when educational gain scores are calculated. Any testing is subject to outside influences such as distractions, emotional state, or luck. Ballou explains that these create statistical noise for each level of testing.

When the difference between two test scores is compared, this increases the statistical noise in the data. While statisticians are well-trained in reducing this noise and have created procedures to limit it, such noise can become significant when looking at smaller and smaller portions of nationwide testing data.

More significant: Steady overall gains or recent test score increases?

Other critics note that several of these districts have shown gains all along, and that these recent increases mean relatively little overall. Edweek author Stephen Sawchuk also warns against using the exam’s data as support for various educational reforms like test-based accountability or specific interventions, noting that data from NAEP exams has been used to support both one conclusion — and its exact opposite.

While it’s impossible to tie individual district policies to overall gains, these significant gains in urban areas are worthy of note and further study.  Strong correlations between policy moves and testing gains indicate areas worthy of further research to establish causation between district policy changes and overall educational progress.

Monica Fuglei is a graduate of the University of Nebraska in Omaha and a current adjunct faculty member of Arapahoe Community College in Colorado, where she teaches composition and creative writing.