Today we welcome our first guest post here on the blog—an analytical commentary by Minnesota native and longtime education policy wonk Dan Wick, who left the late lamented Twin Cities nonprofit Charter School Partners a couple of years ago to pursue graduate studies at Princeton.
Wick drafted his post in response to one I penned in the wake of a couple of local judicial decisions in twin school integration proceedings that some have perceived to be, in essence, Trojan horses containing frontal attacks on parental rights to school choice. In my piece, I suggested that Myron Orfield, the controversial University of Minnesota professor who has churned out stacks of anti-choice, anti-charter research, is in the habit of tossing out scholar-speak that intimidates critics while not actually proving his points.
Well, turns out Wick can perform a regression analysis, too, as well as see through one. What follows is Wick’s compulsively readable critique—complete with footnotes—of the 2013 study in which Orfield claims to show that charter schools are failing impoverished Twin Cities students.
Forthwith:
Argumentum Verbosium Pt. II: On “All Things Being Equal”
Statistical jargon is an awful combination of boring and inaccessible. Myron Orfield’s Institute for Metropolitan Opportunity “Failed Promises” analysis uses this jargon to try to prove that charter schools are a bad option for Twin Cities’ families. In simple terms, I will attempt to explain why this is intuitively and statistically incorrect.
What on earth is regression analysis and how can I use it to state my policy conclusions with infallible certainty?
Central to Mr. Orfield’s argument against charter schools is his regression analysis that shows that charter schools do not perform as well as traditional public schools. This is what Mr. Orfield is doing when he says that, “The new results imply that all else equal, proficiency rates were 11.2 percentage points lower for math and 5.9 percentage points lower for reading in charter elementary schools than in traditional elementary schools.”[1]
An earlier version of the report uses statistical phrases like this to justify bold statements such as, “The findings made it clear that, at that time, charter schools offered a poor choice to low-income students and students of color—one between low-performing public schools and charters that fared even worse.”[2] But does the first statistical statement allow us to draw this conclusion? This is what we will try to untangle.
Let’s think about two categories of schools: suburban schools and urban schools. We see that the suburban schools have higher proficiency rates. Most people, Mr. Orfield included, will rightly say, “Of course, because the schools in Edina serve fewer kids in poverty. This doesn’t prove those schools are better.”
All regression analysis does is solve this problem of different characteristics using fancy math. Mr. Orfield is claiming that he has identified the major differences between these groups of schools, including poverty, and “controlled” for them. If a study accounts for all of the differences that might influence academic performance it achieves a state of statistical nirvana known as “all things being equal.” For complex public policy issues, nirvana is very hard to achieve.
The Truth of Regression: Garbage in, Garbage Out
In order to make bold claims like Mr. Orfield’s, we need to be 100% sure that we have identified literally ALL of the fundamental differences between the schools that might influence academic performance. So the questions we must ask of Mr. Orfield are: have you captured all of these reasons? Are all things really equal based on your model? Even with no statistical knowledge, anyone can evaluate if the assumptions behind the regression model make sense based on their experiences in education.
As listed in the study, the IMO claims the differences between the schools in the metro area come down to:[3]
- Student Poverty
- Race
- Special Education Needs
- Limited Language Abilities
- Student Mobility Rates
- School Size
- If it’s a Charter School
- If it’s a Choice is Yours School
Identifying major factors that explain school performance outside of this list discredits Mr. Orfield’s conclusions. Fancy math can’t overcome bad intuition about why schools perform the way they do.
All things are not equal in the IMO analysis:
The IMO model leaves out several critical differences between schools across the metro. For example, urban charter schools serve students from more distressed communities than their suburban public school and charter school peers. The model neglected to account for the fact that Edina looks much different than North Minneapolis. We would expect urban schools to deal with more psychological and emotional trauma in their student body among other problems. This could lower their test scores when compared to suburban schools.
I expect Mr. Orfield will claim that he “controls for poverty” in his analysis. However, he hasn’t accounted for fundamental differences in the communities that go deeper than the percent of students in poverty that a school serves.[4] The model gives suburban schools too much credit for their results when they have a huge systematic advantage over urban charter schools in performance.[5] This is only one of many possible reasons Mr. Orfield should not claim, “all things are equal” in his analysis—much less that he knows what is best for low-income students of color.
What assumptions does the IMO make about parents and their school choices?
Mr. Orfield’s model relies on the assumption that parents have a feasible choice between all of the schools in the metro area through programs like the Choice is Yours. The voodoo of statistics is saying what would happen if every school served exactly the “same” kids. Sometimes they would perform better, sometimes worse, but metro-wide, traditional public schools would get a few more of this hypothetically identical group kids to be proficient on tests than charter schools would.
So what does this mean for a real parent sitting in North Minneapolis making a high stakes, one chance decision for their child?
It says that you should consider schools from Wayzata to Mendota Heights and that among all these choices; traditional public schools perform a little better. The model doesn’t care if it isn’t feasible for you to bus your kid on a 2-hour commute to the suburbs every day. If you want to go to a school in your neighborhood, something many parents want, Mr. Orfield’s model has even less useful information. It claims that the traditional public schools up the street perform like the average of all the traditional public schools in the metro area. It does not tell me anything about the unique context of my neighborhood because it’s assuming my neighborhood covers 11 counties.
This intuition likely rubs a lot of people the wrong way. Mr. Orfield’s model—and its big assumptions—does not reflect people’s experiences on the ground. The fact that he uses “regression analysis” does not make his argument stronger. The reality is that his faulty assumptions about parent choice also infect the statistical analysis with bias.
Bottom line, the IMO study is an unhelpful simplification of a complex education debate. Its ideological commitment to metro-wide integration causes statistical bias and lacks common sense. Its limited conclusions do not justify the bold policy proclamations of its authors. There is a reason this hasn’t been published anywhere other than the IMO website and it shouldn’t be given credence in the education debate in Minnesota.
Dan Wick is a graduating MPA student from the Woodrow Wilson School of Public Affairs at Princeton. Feel free to contact him at [email protected] with questions, comments, or if you want to give him a job. Prior to graduate school he was an Associate with Charter School Partners and lifelong Minnesota resident.
[1]Charter Schools in the Twin Cities: 2013 Update. Institute for Metropolitan Opportunity, 2013. p.8
[2] Update of “Failed Promises: Assessing Charter Schools in the Twin Cities,” IRP, 2012, p.1
[3] IMO 2013 update, p. 8
[4] In the 2008 IMO Study, Orfield Acknowledges: “Rural poverty differs significantly from metropolitan poverty in important ways, including family structure and race. Poor people in rural areas also do not typically experience the same set of disadvantages associated with concentration of poverty frequently encountered in metropolitan neighborhoods. Failing to control for this could bias the measured relationship between charter schools and performance.” P. 22. This would also certainly hold true between the suburbs and urban areas, but it is not controlled for in the published results.
[5] Jargon Alert: The statistical concept here is called fixed effects. When analyzing student performance across the metro it is important to account for shared experiences between groups of schools that might influence their results systematically. For example, schools across the metro receive different levels of funding. Because suburban schools use property taxes to supplement their funding it gives them a large resource advantage. This advantage is not reflected in the model and results in biased results.
Beth Hawkins is an journalist in residence at Education Post. This was republished with permission from her personal blog, BethHawkins.org.
