StickWithANose

Last month, I pointed readers toward a hack-tacular piece of research by noted Hoover Institute troll and Stanford economist Caroline Hoxby that purports to demonstrate that charter schools out-perform traditional public schools. I knew then that it was only a matter of time before somebody fact-checked her “research” and found the same old errors that plague her work in generally.  Like her spectacularly bad research on the Milwaukee school choice program, Hoxby is working backwards from pre-determined conclusions.

A recent report on New York City charter schools found achievement results at the charters to be better than comparison traditional schools. But that report relies on a flawed statistical analysis, according to a new review. The report is How New York City’s Charter Schools Affect Achievement and was written by Caroline Hoxby, Sonali Murarka, and Jenny Kang. When it was released in late September, it was enthusiastically and uncritically embraced by charter advocates as well as media outlets. The Washington Post offered an editorial titled, “Charter Success. Poor children learn. Teachers unions are not pleased.” [...]

The New York Daily News was no less effusive: “It’s official. From this day forward, those who battle New York’s charter school movement stand conclusively on notice that they are fighting to block thousands of children from getting superior educations.”

Because of the declared importance of the new report, we asked Professor Sean Reardon to carefully examine the report’s strengths and weaknesses for the Think Tank Review Project and write a review that would help others use the study in a sensible way. Reardon, like the report’s lead author Hoxby, is a professor at Stanford University. He is an expert on research methodology…

Reardon points out that the report’s key findings are grounded in an unsound analysis — an inappropriate set of statistical models — and that the report’s authors never provide crucial information that would allow readers to more thoroughly evaluate “its methods, results, or generalizability.”

Reardon’s review notes these shortcomings in the report:

- In measuring the effects of charter schooling on students in grades 4 through 12, the study relies on statistical models that include test scores from the previous year, measured after the admission lotteries take place. Yet because of that timing, those scores could be affected by whether students attend a charter school. As a consequence, the statistical models “destroy the benefits of the randomization” that is a strength of the study’s design. (The use of a different model makes the results for students in grades K-3 more credible, he notes.)

- The report’s claims regarding the cumulative effects of attending a New York City charter school from kindergarten through eighth grade are based on an inappropriate extrapolation.

- It uses a weaker criterion for statistical significance than is conventionally used in social science research (0.05), referring to p-values of roughly 0.15 as “marginally statistically significant”.

- The report describes the variation in charter school effects across schools in a way that may distort the true distribution of effects by omitting many ineffective charter schools from the distribution.
Reardon explains that, as a result of the flaws in the report’s statistical analysis, the report “likely overstates the effects of New York City charter schools on students’ cumulative achievement, though it is not possible — given the information missing from the report — to precisely quantify the extent of overestimation.” This, as well as the lack of detailed information in the report to assess the extent of that bias, make it impossible for readers to know whether the report’s estimated charter school effects are in fact valid.

There is also one limitation that seems to have escaped everybody involved: selection bias. If I’m reading this correctly, Hoxby is arguing that the lottery system in NYC acts as a randomization tool that allows the study to be conducted clinically, but the assumption one has to make is that the parents and families that go out of their way to participate in and navigate the complexities of the lottery process are a self-selecting group with specific characteristics that set them apart from others in the same community. [See Ladd for one example] Even so, it is still clear that all of the fanfare and acclaim thrown about in news media was unjustified and that the research itself is geared toward finding a pre-determined conclusion that contradicts the findings of the CREDO study often cited here.

According to Professor Kevin Welner, director of the University of Colorado at Boulder’s Education and the Public Interest Center (EPIC): “Readers of this review will understand that, while Hoxby’s charter school study is a contribution, it has significant flaws and limitations. Unfortunately, the editorial reaction of otherwise-respectable media outlets trumpeted the New York City findings as the final and faultless word on charter school performance. In fact, the study used inappropriate methods that overstate the performance of the charter schools it studied.”

Welner notes that the Think Tank Review Project also recently reviewed another charter school study, released in June by Stanford’s CREDO policy center. That study encompassed 65-70% of the nation’s charter schools. “Our review pointed out a number of limitations but also noted the relative strength and comprehensiveness of the data set, the solid analytic approaches of the CREDO researchers, and the important fact that the CREDO results were consistent with a large body of research showing charter schools overall to be performing no better than (and perhaps worse than) traditional public schools,” Welner says. But he added that “the CREDO and Hoxby reports used different designs and covered different schools. They are not directly comparable, nor are we able to say which is ‘better.’ Neither report is definitive or without notable weaknesses.”

This episode is not unique to Hoxby; it is a problem that is inherent to our current political milieu marked by an increasing number of well-funded, partisan political think-tanks springing up all over the place. When a large study like CREDO comes out, think-tanks and their marketing machines spring to action to produce “evidence” supporting pre-existing policy and ensuring that our lazy, under-staffed news media publish breathless articles for wider consumption. It is the antithesis of social science, and we need to label it as such. The larger problem that researchers in all fields must now face is the politicization of evidence.