When I took a full-time education policy job in 2010, I was not current on the latest research. I’d received my Masters in Public Policy, with a focus in education, a full 15 years earlier. Back then I paid close attention in my classes with Professors Jim Hines and Tom Loveless. It was widely accepted in scholarly literature that “inputs” such as money and teacher qualifications had not improved the quality of American public schools. Like many left-of-center observers, I found this scholarly conclusion disappointing, but it appeared robust, withstanding repeated attempts at re-analysis from the Coleman Report in 1966 to a 1986 survey by Eric Hanushek who is now at Stanford.
Twitter, of all things, provided me with news to the contrary in August of 2014. Mark Weber, a sincere reform skeptic and public school teacher in New Jersey, goes by the Twitter handle “Jersey Jazzman,” and is a part-time doctoral student at the Rutgers University Graduate School of Education. Weber pointed me to publications written by one of his professors, a fellow named Bruce Baker. Although I had never previously heard of Baker, Weber was not the only person who recommended him. Ben Spielberg, who graduated from my undergraduate alma mater told me flatly that Baker’s “research is legitimate.” Spielberg, Weber, and other reform skeptics cited Baker often, and indeed Baker was described by AEI’s Rick Hess as the 40th-most cited education scholar in America. Even better, it seemed that Baker was willing to engage folks I knew to be smart and careful, such as Ulrich Boser at the Center for American Progress and Rebecca Sibilia at EdBuild.
I thus read Baker’s research with an open mind.
Baker’s papers blew me away. They totally reversed the narrative. For instance, Baker pointed me to a 2012 piece he wrote called “Does Money Matter in Education,” which concluded that school spending is important and impactful for students.
This conclusions was the opposite of the consensus in academia when I had been a student in the 1990s. How had the prior research been so wrong? What had happened in the previous 15 years? Well, Baker cited Northwestern University’s Larry Hedges, who re-reviewed Hanushek’s 1986 survey of evidence using “quality control measures” to exclude some studies and change some interpretations. According to Baker, this settled the matter: “by the early 2000s, the cloud of uncertainty conjured by Hanushek in 1986 had largely lifted in the aftermath of the various, more rigorous studies that followed.”
I was surprised, but frankly relieved. As I wrote in response to Baker at the time, “Thank heavens. Someone who actually talks evidence.” Shortly thereafter, I read another piece from Baker regarding implementation of high-stakes testing, and frankly his analysis was solid. I assumed that this level of analysis was typical of Baker’s work, and was further relieved that a high-profile reform skeptic was taking the time to do careful research. As I wrote to him, “Bruce, your facts & analysis R best I’ve seen on UR side. Wish AFT/NEA pushed you, not smears.” I circulated Baker’s work to elevate that approach.
But wait, something smells fishy
The first clues that something was fishy came as I dove deeper into Baker’s body of work. The highly respected Ulrich Boser had written a report on waste and inefficiency in school spending, and Baker had written a rebuttal. Baker’s rebuttal was, as I wrote to him, “More strident, less compelling than UR usual.” I was being delicate; Baker’s rebuttal was full of personal insults and exclamation points. Disappointing for an alleged scholar.
Then, I read a Baker critique of Mathematica policy research regarding the effectiveness of KIPP charter schools. Baker’s critique was terrible, a long list of hand-waving attacks that seemed to call into question the very possibility of actual empirical research in education. As I wrote to him, the methodology of his approach seemed like that of climate denialists, whose attacks often are a “kritik” of the very idea of research.
Things got worse still when I started to read Baker’s work about teachers’ unions, a subject about which I had substantial personal exposure from visiting state legislators in places where unions were active. As I wrote to Baker in response to a blog of his on the subject, his thumb appeared to be on the scale of the internal workings of his models. His methodology on unions was so sloppy it seemed deliberate.
Boy, was I a sucker
In 2015, Rucker Johnson and others published an NBER analysis of the impacts of school spending. The NBER report was broadly sympathetic to Baker’s 2012 claims that money can matter, so I read the report with interest.
Wait a minute . . . the 2015 NBER report, entirely focused on the question of “does money matter in education,” did not once mention the Bruce Baker publication from 3 years earlier with the title “Does Money Matter in Education?”
That seemed odd. Even more odd, the NBER paper referred to studies from 1995 and 1996 that showed school spending doesn’t lead to better results.
Wait, what? Wasn’t that the period of time that Baker reviewed, when he wrote that the “cloud of uncertainty” created by Hanushek in 1986 had lifted based on subsequent work? Why didn’t Baker mention those 1995 and 1996 studies by other scholars?
With my antennae finally up, I dug into Baker’s 2012 claims more fully. As it turns out, Baker omitted so much context from his report that his conclusion borders on outright mendacity. For instance, Baker chooses not to mention that Hanushek wrote several peer-reviewed rebuttals to Hedges’ work, including that they engaged in “statistical manipulations … to overturn prevailing conclusions,” and that they “misinterpret the implications of their analysis [and,] through a series of analytical choices, systematically bias their results toward the conclusions they are seeking.” Baker wrote a conclusion that “uncertainty” created by Hanushek “lifted” after 1986, without even deigning to mention that Hanushek didn’t agree? Baker’s presentation of this conclusion was so skewed that later scholars on the exact same subject didn’t even mention Baker’s paper?
Giving Baker a taste of his own medicine
Although Baker’s attacks on Ulrich Boser and others have been filled with insults and sloppiness, we do not need to go to that level to properly assess Baker’s voice in the education reform debate. After all, serious education researchers tend to not even mention Baker. Instead, Baker’s arguments should be taken for what they are: raw advocacy, rather than academic research. Thus, when he makes fair points, they can be discussed on the merits. But no self-respecting writer should ever cite Bruce Baker’s conclusions without carefully reviewing all of his hyperlinks and details, and also doing a quick Google search to see if he’s omitting crucial information.
If I had done my research, I would have known this. Dropout Nation editor RiShawn Biddle pointed out four years ago that Baker’s so-called “analyses” are designed to achieve his intended results by making subjective and one-sided decisions about what to include and what to ignore. This may be expected for expert witnesses at trials, but is disturbing for someone who pretends to be an academic, and is not transparent that he gets paid for reports by parties with a direct financial stake in his outcomes. Indeed, going forward, jurists, litigators, and policy-makers should be careful when Baker is called in as an expert witness in contentious proceedings.
The problem with Baker was further underscored in a 2011 tape-recorded conversation in which Baker said he would play with data, manipulate the questions he asked, and “pull things in and out” of his models “to tell the most compelling story” in exchange for a substantial research grant. This telephone conversation, including Baker’s own partially exculpatory comments, appears in full at about the 3-minute mark of this video clip. [Baker offers a rather lengthy explanation and defense of what happened.]
I am disappointed in the folks who encouraged me to read Baker, but I’m even more disappointed in myself for being so gullible. None of this automatically invalidates Baker’s conclusions, but most of his research suffers the same kinds of glaring deficiencies just mentioned regarding his 2012 Shanker Institute paper. Some day, someone may decide to write a point-by-point review of Baker’s editorials, but for now the main point is to take his sweeping anti-reform conclusions with a heaping of salt.