Detailed criticism of IFF study of DC public schools

I haven't read the report yet (Quality Schools, Every School, Every Child, Every Neighborhood), so I haven't felt comfortable commenting, although it is true that based on the dictum in Graham Allison's Essence of Decision that "where you stand depends on where you sit," the fact that the Illinois Facility Fund is a real estate organization focused on charter schools enablement leads to the supposition that they will draw certain conclusions.

The Teachers and Parents for Real Education Reform blog has criticism of the current report, as well as previous entries about the process. See "IFF Study Released to Criticism of Research Methodology."

From the entry:

The IFF report’s methodological problems can be summarized as follows:

• The researchers created a zero-sum game in which schools are divided into four “Tiers” based on how much better or worse their DC-CAS scores are from the mean score. Each school is deemed to have “performing seats” or “non-performing seats” depending on whether students’ DC-CAS scores are higher or lower than the mean. Under this methodology, there will always be winners and losers relative to other schools.

• When schools are compared, no account is made of the socio-economic background of the students or the number of special education students, or English language learners among them. Differences in DC-CAS scores are compared under an assumption that students are all equally equipped and supported to do well. The differences in achievement and whether the trend is upward, downward or stationary is credited entirely to the school.

• Within each of 39 neighborhood clusters, when schools get different DC-CAS results, the researchers assume that the students are the same and therefore, the difference is a credit to the school. No attempt is made to look at possible differences in socio-economic background of the students who select one school over another, which might be different than the neighborhood overall. ...

• The researchers acknowledged that standardized test scores are not the best way to measure the quality of a school, and they wished they had other ways describe differences in quality, but used the only measure that they had at their disposal. Notwithstanding their admission, this study is the most extreme example of reducing the quality of a school only to its test score we have ever encountered. No other factors were used. Most reputable research examining school improvement efforts, for example that of the highly esteemed Consortium on Chicago Schools Research, has never taken this reductionist approach.

• The IFF researchers failed to consider recent history in DCPS, before making their recommendations. The track-record of school turnarounds in DCPS [by charter school organizations and others] since 2008 has been an embarrassment. Outside management firms have been brought in to run Dunbar, Anacostia, and Coolidge high schools. Two have abandoned their partners. None of them have achieved significantly better results. School consolidations have led to explosive results at Hart MS and elsewhere. The national report card for national charter chains has not been good. In other words, there is no silver bullet contained in changing the management of schools. Nevertheless, the folks at IFF are wedded to this recommendation as their bias. It will be resisted in DC for good reason.

• The IFF also seems to ignore the national research that shows a mixed track record of school turnarounds and national charter chains, and overwhelming national research that demonstrates that standardized test scores tend to mirror the socio-economic background of the students taking the test. It is one thing to make every effort to counter this reality with programs and strategies that mitigate the effects of poverty. It is quite another to structure a study premised on the assumption that any difference in test scores is the result of good or bad teaching alone, or as they term it – “performing seats.” The simplistic design of this study flat-out ignores the national research and it promotes an unfair and punitive set of assumptions in response to the data.