StickWithANose

I noted earlier in the week that the machine had kicked into gear to discredit Diane Ravitch after her very public conversion from being a big supporter of NCLB and charters to a critic. To her credit, Ravitch’s conversion was one based on the growing body of evidence that the accountability-charter juggernaut is both destructive and unsustainable. However, for the think tank machine which once praised her, she is now the enemy and must be silenced or discredited.

The Quick and the Ed has been my go to place for watching this unfold, and Bill Tucker’s post “Ravitch: Just a Tad Defensive?” is a textbook case of misinformation. Accusing her of being “selective with evidence”, Tucker demonstrates classic Jungian projection by… what for it… being extremely selective with evidence in order to muddy the water in his favor. Tucker uses Ravitch’s reference to the CREDO national study as an opportunity to bring up CREDO’s recent study of NYC charters [pdf] that offers a mixed bag of results for school level comparisons:

• In reading, 29% of charter schools out-perform traditional public schools / 59% perform the same / 12% under-perform
• In math, 51% charter schools out-perform traditional public schools / 33% perform the same / 16% under-perform

It is obvious from the results that charters have good scores for math and average reading scores, however these differences in outcomes is most likely the result of math gains made in the early elementary years in which “drill and kill” instruction benefits the kind of standardized assessments used in these studies. [1][2] To further bolster his argument, Tucker then turns to every think tank troll’s favorite economist Caroline Hoxby. Hoxby’s recent non-peer-reviewed study found that NYC charter were wildly successful. The problem with Hoxby’s work is that she has never met a school choice or charter school model that she wasn’t willing to shill for no matter how hard she has to massage the numbers. For years, Hoxby’s study of Milwaukee’s school choice plan was the go to citation for all manner of think tank and media pundits. Of course, the supposed miracle in Milwaukee was nothing of the sort, and Hoxby’s study was smartly critiqued by scholars who have to actually demonstrate the validity of their work to their peers. Likewise, Hoxby’s NYC study received huge amounts of attention in popular media but, as before, lacked much to be desired.

In his review, Reardon observes that the report “has the potential to add usefully to the growing body of evidence regarding the effectiveness of charter schools.” New York charter schools’ use of randomized lotteries to admit students to charter schools offers the possibility that the study of those schools can roughly approximate laboratory conditions.

But Reardon points out that the report’s key findings are grounded in an unsound analysis — an inappropriate set of statistical models — and that the report’s authors never provide crucial information that would allow readers to more thoroughly evaluate “its methods, results, or generalizability.”

Reardon’s review notes these shortcomings in the report:

- In measuring the effects of charter schooling on students in grades 4 through 12, the study relies on statistical models that include test scores from the previous year, measured after the admission lotteries take place. Yet because of that timing, those scores could be affected by whether students attend a charter school. As a consequence, the statistical models “destroy the benefits of the randomization” that is a strength of the study’s design. (The use of a different model makes the results for students in grades K-3 more credible, he notes.)
- The report’s claims regarding the cumulative effects of attending a New York City charter school from kindergarten through eighth grade are based on an inappropriate extrapolation.
- It uses a weaker criterion for statistical significance than is conventionally used in social science research (0.05), referring to p-values of roughly 0.15 as “marginally statistically significant”.

The report describes the variation in charter school effects across schools in a way that may distort the true distribution of effects by omitting many ineffective charter schools from the distribution.Reardon explains that, as a result of the flaws in the report’s statistical analysis, the report “likely overstates the effects of New York City charter schools on students’ cumulative achievement, though it is not possible — given the information missing from the report — to precisely quantify the extent of overestimation.” This, as well as the lack of detailed information in the report to assess the extent of that bias, make it impossible for readers to know whether the report’s estimated charter school effects are in fact valid.

For those not in the academy, these kinds of critiques are integral to scholarship and are the reason why scholars value peer-reviewed studies over the ideological drivel coming out of think tanks and, in this case, NBER.

The key point is that there is simply no evidence that the strong push for growing the number of charter schools as a means of closing achievement gaps and reforming public schooling is an effective approach. The only other national study done on charter school efficacy was conducted by the RAND corporation with funds provided by the Bill & Melinda Gates Foundation. When that study reported that charters are no more effective than traditional public schools, it conveniently disappeared down the memory hole without so much as an acknowledgment in popular media. If it had found significant differences, the Gates media machine would have ensured that the RAND study would have been splashed all over the New York Times and Washington Post like Hoxby’s study.

More importantly, these two national studies confirm trends emerging from more localized studies that pre-date them: there is limited to no evidence that charters out-perform traditional public schools [1][2][3][4] and there is a large body of evidence that charters increase student stratification along racial and social class line [1][2][3][4]. Of course, this matters not to the folks over at the Quick and the Ed. As Upton Sinclair noted, it is difficult to get a man to understand something when his salary depends on his not understanding it. Tucker is paid to shill for the edu-philanthropists that fund the Education Sector not for the students being funneled into segregated corporate testing facilities.