Rejecting "Evidence-Based Practice"

There are many words and phrases I'd love to toss out of our educational vocabulary. "Rigor" is, of course, a big one. "Flipped Classroom" is another - unless we're going to really try the old Fred Astaire-Royal Wedding thing. "Special Needs," "Special Education," and "SpEd" are others because it is way beyond time to embrace Universal Design for Learning, full inclusion, IEPs for everyone, and Toolbelt Theory for all. But here I am with one more... the dreaded "Evidence Based Practice."

"Evidence Based Practice" sounds so good, sounds so authoritative, sounds so "medical." It is, after all, medical. The term was "Evidence Based Medicine," and like the "Gold Standard" research promoted by the AERA and the US Department of Education, it sounds great to the uneducated, but remains not just impossible to transfer to the field of education, but undesirable.

I heard a medical researcher on NPR this week describing how a highly effective counter-treatment to the side effects of antibiotics couldn't really spread because it could not be evaluated the FDA "gold standard" way, and it reminded me of how evaluation systems stymie innovation everywhere.

But even if these research and practice paradigms worked fully in medicine... we still could not, and should not, import them. First, it is not possible to do a "double blind" trial in education, the practitioner surely knows what "treatment" is being used, and so do most students (we often might as well give out "Control Group" T-shirts) so the "Gold Standard" promoted by AERAis simply fraudulent. Then there's the undesirable - we are educators, it is malpractice for us to withhold interventions from students when we believe they will help.

And so if the methods of collecting our "evidence" are both fraudulent and irresponsible... where does that leave "Evidence Based Practice"?

It leaves it as the tool of those who want to preserve - to harden - the industrial-process nature of education, and as the tool of hucksters with products to sell, like researcher-salesmen Slavin and Marzano. As Peter Smagorinsky says, in terms of how the US Department of Education operates:

 "Consider the ways universities operate. Universities initially compete for students, but once they’re enrolled, that game is over. After enrollments are secured, universities compete for prestige, and the game is not zero-sum: The more prestigious universities, the better the nation is served. The U.S. university system is not centrally organized, so each campus competes in its own way. Public universities are bound by state regents systems, at times intrusively; but for the most part are entrusted to be the institution they want to be. Innovations are very public and available as models for others to consider. Indeed, the publishing imperative forces good work out into the open and contributes to the status of the institution as others seek to adopt it. This approach works well for universities, but public school teachers are not entrusted with the same opportunity.

"The competition and innovation that local control would promote would enable teachers to exercise judgment, an opportunity that is rarely available under the current testing regime. Just as importantly, it would help to shape the incentive system within which education takes place. Presently, young people looking for a field of endeavor read daily attacks on teachers and schools, and see a punitive, assembly-line approach to teaching that follows from standardized curriculum, instruction, and assessment imposed by Washington bureaucrats with no teaching experience. Replace that image with a workplace where teachers’ ideas matter and their innovations are appreciated as dynamic and worthy of admiration. Which environment would you decide to enter if you were a talented young person looking for a rewarding career? Most teachers I know say that they want to teach because they want to make a difference. It’s hard to make a difference when the system rewards those who make everything the same." - Peter Smagorinsky, Distinguished Research Professor of English Education at The University of Georgia and included in Valerie Strauss' Washington Post blog The Answer Sheet, March 28, 2012

I am not against "evidence," we all collect "evidence" to support our actions or to suggest change in our actions every day, but I am also for innovation and creation and experimentation every day, especially in schools. For example, I have never repeated the same "class" - not even when teaching the same thing three times in a row. I learn from each presentation and then adapt, that's my evidence based practice, you know, "well, that didn't work," but there is something else creating the change - the "audience," the interactions change every time - and I can not let "evidence" from a previous experience interfere with the need to innovate, improvise, based on the needs currently before me.

This is the problem with "Evidence Based Practice," with "Common Core," with any enforced "National Standards," all limit our possibilities by enforcing "what is" as Gary Stager noted on Twitter: "All capitulations to national standards reduce your ability to make decisions as a teacher." All capitulations to the "scientific evidence" requirement prevent innovation.

What we need instead in education is a nation, nations, of experimental researchers working constantly with Grounded Theory based Action Research. All teachers as innovative experimenters, all schools as laboratories of what might be possible. This is essential, because otherwise we are, to use the metaphor for this week, simply re-arranging the deck chairs on the Titanic because we saw them arranged differently on the Central America.

In other words, our current praxis, our commitment to "Evidence Based Research," is utilizing fraudulent research paradigms to investigate micro-changes in a system developed not through science or research at all, but as a Social Darwinistic effort to prevent social change which might threaten power structures. [See the Five Part series - Designed to Fail.]

Grounded Action

"If a grounded theory is one that, as Strauss (1990) contends, “…is inductively derived from the study of the phenomenon it represents," then grounded action could be described as “…one that is inductively derived from the study of the phenomenon it represents for the purpose of creating and applying practical solutions to social systems” (Simmons & Gregory, 2003)."

Putting together Grounded Theory [link is to a pdf], the observation-based data collection concept which accepts that the real world is more powerful than the laboratory for social structures,  with Action Research, the continually adaptive method of applying developing knowledge to the solution of problems in addition to the study of them, allows us to form a circle which moves us - and more importantly - our students forward. Forward into a world wholly unrelated to the Henry Barnard - Ellwood Cubberley structure of education which remains intact even when "flipped" or "flattened" or privatized.

"Operational theory is systematically generated from and grounded in explanatory theory. The operational theory provides a grounded theoretical foothold for action planning and implementation."

You learn to see your classroom anew. You observe all the possible options. That becomes your data floor. You make changes, observing the results, constantly observing, recording, and adapting to the inflow of information. You are educators. You are not going to trap a control group of children into a failed reading system for 5 months in order to prove that system works - you are going to keep searching for solutions for those kids - but you are going to do it like researchers... recording your observations, your changing efforts, and your successes and failures, and then sharing that with the world of education - wholly and freely.

And if we all work on invention, and we all share what we're doing, and we're all building on what we're doing, we can stop buying third rate adaptations and begin to grasp the possible future.

Which has to be our goal. To move rapidly into the future, where our students already reside. And we're not going to get there buying ten-year-old crap, that we collected "evidence" for five-years ago.

- Ira Socol

Measurement and the Overpromise

Measurement has its problems.

Listening to the American "Main Stream Media" speak about education today, or reading most of what passes for educational research, is much like waking up too early on a weekend morning and watching infomercials.

@arneduncan?

There's the "prosperity theology," 'just give two years of your life to Teach for America and God will smile on your CV for all time.' There's "weight loss," 'lose hundreds of pounds of poverty with our KIPP program.' There's "get rich on government money," 'just become a charter school operator!' And then there are dozens of miracle cures for what ails us, the 'Marzano Magic Learning Machine will cure boredom and everybody gets smart.' The 'Slavin Super Reading Pill,' will solve illiteracy. Or the "Harvard Lecture" will make all your students upper class success stories. Or just ridding yourself of unsightly unions will cure all.

And just like those infomercials, anyone with half a functioning brain knows this is all nonsense, and that the peddlers, wearing their wealth (Bill Gates, Eli Broad) or their connections (Michelle Rhee, Wendy Kopp) or their PhDs (Marzano, Slavin, the entire Harvard Education faculty) like the white coats of hair restoration tonic hucksters, are self-serving phonies. And yet...

"Education Reform" has been very
good for Our Miss Broomstick

And yet "we," as a society, are fully suckered in. And the fault lies not just in the work of the evil rich - Gates, Broad, Koch, Bloomberg - but in our very belief in scientific management.

Since the beginnings of the Industrial Revolution "we" (the "western" human) have been obsessed with measurement. We measure "production" and "gross domestic product" and "per capita income" and the height, weight, and intelligence of our children, and we compare these against the measurements of others.

Actually, it goes back a bit further, to the Protestant Reformation, and the belief that humans and communities might "measure up" to the perfections described by Calvin and Luther, or not. And to the idea that wealth (responsibly handled) indicated God's love and approval.

The problem with measurement is that it does three very negative things: (1) It creates false comparisons against a fiction - "the average human" and "the average human experience." The child born in the village in rural Kenya is made to line up against Bill Gate's children, on a scale created by Bill Gates. Thus that child is "not white enough," "not Protestant enough," "does not read enough books," and simply lacks "computer time." (2) It ties us firmly to the past - we can only measure against a known, thus measurement itself binds us to prior function and blocks future dreams. The measurer wants a faster horse, he/she cannot conceive of another method of transport. And (3) measurement limits what a society thinks is important. A local bad businessman measures European tax rates against those in the US and insults me by damning European society (no, I won't come back). He focuses on what Americans consider valuable (not supporting their neighbors and children) rather than what Europeans might (supporting their neighbors and children) and thus comes to the set of decisions which put people like Paul Ryan, John Boehner, Scott Walker, Chris Christie, and Rick Snyder in power, which, in turn, probably terminates the American future.

And of course, "measurers" become "fixers." No one "measures up," an idea which has kept preachers employed for 500 years, or 5,000. So the "fix" is needed. "Oh my, I'm not as rich as my neighbor, I need a fix." "Oh my, my child is not as smart as my neighbor's kid, I need a fix." "Oh my, our children do not interpret alphabetical symbols as quickly as those in Scarsdale (on average), we need a fix."

The "fix" is, of course, joined to the overpromise. With so many "fixes" competing for money and attention (and "attention" leads to money), every "Hooked on Phonics" huckster, every aggressive PhD candidate, every professor with dollar signs in his or her eyes, every politician desperate for a CNN or FoxNews soundbite, claims that "their fix" is "the fix" - God has smiled on them and made them the conduit of the perfect truth.

any difference between these two videos? ("fixers" at work)

And each of these fixes limits, and ties us to the past, and preserves the status quo. Dr. Robert Slavin has never spent a minute considering (publicly) what "reading" means. He just wants kids to spend their entire childhood chasing his grandchildren, so his grandchildren stay on top (those other kids cannot possibly catch up unless Bob's grandchildren stop moving forward, since they've begun so far ahead). Bill Gates has never spent a minute considering (publicly) what "education" means. Rather, he sells a system which lists his kids as the 'best in the world' and measures all children by what's going on in that big house outside Seattle.

build your own Bill Gates house, and your kids can grow up to be like his...

Thus, we as a society don't consider how we might re-imagine reading so it is no longer the elite activity of the Gutenberg/Reformation era. We don't consider how we might re-imagine education so it is no longer the industrial process crafted by 19th century Social Darwinists. We don't consider how we might re-design our social structure to eliminate the poverty which destroys opportunity. Instead we run in circles chasing the fictional... that "average" human which does all the things important in the past pretty well.

Measurement has its problems. Measuring ensures that most will be "left behind." Measuring ensures that destructive competition will rule. Measuring blocks truly new ideas. Measuring rates the status quo as the best we can do.

So maybe, just maybe, we should stop doing it.

- Ira Socol

The Problem with Hypotheses

In what I would call "an informal study" I have ranked the "atmosphere" in various high school hallways around the area in which I live. Yes, it is a "convenience sample," consisting of schools I happened to be visiting. No, I had no particular method of recording observations. No, I will not provide a long bibliography of sources cited. Yes, I have developed my own way of coding the "data" and my own way of ranking the "school atmosphere."

And yet, my study has one thing I think is essential for validity. I began this study with no hypothesis. I had no idea, no theory, no assumptions in mind when I began visiting these schools. Thus, unlike almost every study of education in the United States I have read in the past five years, there is no "confirmation bias" in my study, because their was nothing to confirm.

In my first semester of graduate school, in my first research methods class, we read a study that Dr. Robert Slavin of Johns Hopkins University had done on the reading project he sells. In fact, almost everyone connected with the study had a very strong financial incentive to make the product, Success for All, look good.

My professors thought this was a great study. It followed "all the rules" in Scientific Research in Education1, the "bible" of "gold standard" educational research created by top AERA professors for the Bush Administration.

But I, and a number of other students, found the study to be, as research, worthless. First, as I noted, it was conducted entirely by people with a financial interest in proving the program's success. Second, it combined dozens of changes in school operations with the introduction of Slavin's product, including things like free food and massive tutoring efforts - how could anyone possibly suggest which part of those changes produced any effect? Third, it refused to look at any "side effects" - what happened to students in other areas when most of the school day became devoted to chanting sentences?

But, the more I thought about it, the biggest problem was the hypothesis. Slavin, of course, set out to "prove" that his product worked. That was his hypothesis. And, shock of shocks! he confirmed his belief.

We constantly teach hypothesis in our schools. It is as much the lifeblood of almost every Science Fair as the fact that parents do the project. But does hypothesis help or hurt our science?

Let's go way back to a very famous example. Galileo hypothesized, via Copernicus, that the planets circled the sun. And, yes, his "experiments" (observations) confirmed this. But when Jesuit researchers looked up they noted that what Galileo was saying could not possibly fit with their observation.

The problem is, of course, that the planets orbit the sun, but not in circles. At Galileo's trial both sides were "right" about celestial mechanics and both sides were quite wrong. Galileo had the idea but not the details, the Jesuits had the details but not the idea. Both had gotten to the wrong place through what we now call "confirmation bias." And confirmation bias is a direct result of our commitment to hypothesis.

Hypothesis should not be first. The first question when we study something needs to be "what is happening?" The problem is - a problem most social science research "leaders" gloss over - is that when someone like Slavin goes out to "prove" something "works," every question he asks, every bit of data he collects, every measure he uses, is, at least in part, designed to prove his hypothesis. Slavin measures short term reading test gains, not for example, student interest in literature. The tests he uses measure components of reading, not gains in subject interest. He does not compare "his" schools to others with extensive tutoring. He uses statistical models which presume that the human experience can be "averaged." It is not that he is lying. He is not. It is not that he falsely manipulates data. He does not. But his study (studies) are fatally flawed from the inception because the intent is not to observe but to confirm.

It would be no different if I ran a study to prove that Success for All was a ridiculous waste of school money. My questions, my measures, my statistical analysis would be selected to confirm that.

Is there another way? 

When people ask my hypothesis I tend to say, "I have no idea."  This didn't go over big on my "practicum" project with the faculty, but I tried hard to stick to it, even when measurement structures were imposed on my by the federal grant. On my dissertation, I began with the question, "how did this system develop?" and then, "why did this system develop?" If I had begun with "America's schools were designed to limit opportunity," I would - for example, have read both Horace Mann and William Shearer in very different ways. But by beginning with non-leading questions, I was free to hear these men as they wished to be heard, not through the results of 50 years or a century later.

So, could you, a researcher, walk into a school without a hypothesis? Could you clear your mind of your years of training and just "see"?

How might that change educational research?

- Ira Socol

Oh, the results of that "informal" study?

My top five "best," "healthiest" environments - corridors seemed under control, kids were polite, there was little or no bullying, kids and adults interacted well, kids seemed to arrive in class after passing times not unduly stressed...
1. Black River Public School (a small, urban, fairly diverse, non-profit charter)
2. Godfrey-Lee Middle School and Lee High School (an impoverished urban school)
3. Holland Christian High School (a small Christian Reformed Church school)
4. Hamilton High School (a large rural/suburban public school)
5. Reeths-Puffer High School (a large suburban/rural public school)
My bottom five (opposite of above)
5. Zeeland West High School (a large suburban/rural public school)
4. North Muskegon High/Middle School (a small wealthy suburban public school)
3. Holland High School (a large urban public school)
2. Zeeland East High School (a large suburban public school)
1. West Ottawa High School (a large suburban/rural public school)

What's important here is not the ratings, informal and, as with all research, highly subjective. What's important is that this reveals that, at least to me, the questions we ask about size, income level, management might be the wrong questions. If I had gone out to compare large to small, urban to suburban, non-public to public, I suspect what I would have "seen" would have changed, and my very hypothetical question would have altered my results.

1. I have called Scientific Research in Education "the most destructive book of the last decade," so, there is some bias there.