What Might Not Work: A Response to Ben Goldacre's "Building Evidence Into Education"

The randomised control trial (RCT) is a kind of field experiment that can be used in real-life situations to test the relative effectiveness of specific treatments, procedures or programmes. RCTs have been widely used (especially in medicine) since the late 1940s. Galashiels-born Archie Cochrane was a key figure in establishing their acceptance and development in the medical field after 1970 and they are still regularly used and seen as the most demanding test of efficacy of any treatment (especially when combined with a double blind in which even the practitioner doesn't know which treatment has been offered to each randomly assigned subject).

In an article last  month, urging wider adoption of the randomised control trial in UK education, Ben Goldacre was re-opening a fairly old debate. His enthusiastic approach to making RCTs relevant to education in 2013 emphasises the question "What works?" The question echoes the title of the USA's "What Works Clearing House" set up for the educational field in 2002 following a policy push by George W. Bush. Bush had been persuaded to make RCTs the dominant research instrument for developing new approaches to teaching and his legacy has persisted under the umbrella of the Institute of Educational Sciences (http://ies.ed.gov) and the George W. Bush Institute (http://www.bushcenter.org/george-w-bush-institute)

Goldacre wants today's teachers and researchers to adopt the same question in the UK and to use RCTs to test and confirm their answers. By taking part in RCTs themselves and by referring to evidence derived from them teachers will, he believes, increase the learning of their pupils and students and simultaneously increase their own professional standing and independence. In becoming more evidence-based, he suggests, teachers will become more like doctors in terms of scientific authority and teaching competence.

His recent and non-technical briefing "Building Evidence Into Education" (http://media.education.gov.uk/assets/files/ben goldacre paper.pdf) is based on a longer document "Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials" co-written with the Behavioural Insights Team in the Cabinet Office and David Torgeson Director of the University of York Trials Unit. (https://www.gov.uk/government/publications/test-learn-adapt-developing-public-policy-with-randomised-controlled-trials) Most of what follows is a response aimed at readers of Goldacre's shorter paper. I would like to offer something that, at least, puts up a cautionary note to anyone convinced by the shorter Goldacre paper that RCTs could bring about the results that he hopes for:

"we all expect doctors to be able to make informed decisions about which treatment is best, using the best currently available evidence. I think teachers could one day be in the same position" Goldacre (2013) p7

I would start by agreeing that "What works?" is a simple and appealing question. It is "common sense" and the more immediate or urgent the answer, the more focused such a question can be. A fire in the kitchen, a raging fever, a stopped heart, a violently out-of-control 11 year old in a classroom, a blank inability to understand percentages… all these situations call for acceptable solutions that are tried, tested, defensible and well established as effective within the resource limits of the context.

In the very often contentious context of teaching, however, it is harder to conceptualise the research equivalent of a medical cure or of protection from immediate physical danger when we think about "what works?" Pupils are not in school because of their illness, deficiency or vulnerability. Educational realities and educational goals are very much more diverse and open-ended and some important educational purposes (especially the long term purposes that most teachers, parents and politicians would want to consider) are impervious to the testing possible in RCTs as they are generally conducted.

Some of these purposes are acknowledged, some are not, but many are so deeply held that the conflicts between them can generate anger and conflict. Some goals are long term (I want my child to go to University), some short (I want the chance to take my child abroad when flights are affordable, so missing a week's lessons is acceptable). Some focus on personal development and intrinsic reward, others focus on long-term career and extrinsic reward. An educational practice that "works" for one set of people or purposes can be ruination for the ambitions or immediate needs of another set. We hope that without offering a long and still not-comprehensive string of real examples, it should be clear that asking "What works" immediately opens the floodgates to more disruptive questions like "Works for who?" "Works to whose disadvantage?" and "works to what ends?" To insist that "it's obvious" is simply to short-circuit the debate and insist on your own values or to accept the apparent majority values.

Asking "what works" is especially a problem in the face of education's character as a long-term (some say lifetime) progression. The research literature on science teaching in higher education, for example, provides plenty of indications that even successful (It Worked!) A Level students embark on their undergraduate studies with what seem like perverse conceptions of basic processes. What worked for getting those high grades at A Level has become an impediment to understanding the greater complexities at University. Similarly, it is a commonplace to note that some outstandingly creative or successful adults would have been or were actually recorded as failures in school learning.

These two points – the disputed nature of educational goals and the long and not well-understood development of individuals through education and into adulthood reduce the apparent value of RCTs in education considerably, especially when considered in relation to their costs. These points do not rule RCTs out altogether. No intelligently undertaken research can be a waste of time (even if its use of resources might be profligate) but promoting them as a special gold standard that will draw other methodologies and the status of teaching along with them is, in my view, perverse.

None of what I have said so far would challenge the valuable idea of research-informed teaching. A lot of practical classroom research goes on already. (I conducted some while I was a practising teacher). Teacher training, despite efforts of the Department for Education under a succession of Secretaries of State, has kept productive links between teachers and researchers over many years. Teachers undertake research degrees and researchers involve themselves in the design and evaluation of teaching innovation. A succession of teacher-research programmes have come and gone, always dependent on meagre funding and lack of Government commitment. To the great shame of all, most of the results of this research has been kept away from teachers with most academic libraries and databases closed to them. The British Education Index (www.bei.ac.uk) – a huge catalogue of all the educational research in the British Isles over more than 50 years has never been funded to allow free access to the very profession that needs it most. In Goldacre's paper a reader will find no reference to "What Already Exists?" (but which could soon be lost), nor to recommendations from the past to make more digital content accessible to teachers and other participants in the professional world of education (see, for two examples of initiatives left to wither through lack of political will: (http://www.leeds.ac.uk/educol/documents/00003809.htm and http://www.tlrp.org/dspace/handle/123456789/1620 ). I wonder if Goldacre is aware of the huge efforts that have already been made to invest in educational endeavour that is "evidence-based" or "research-informed" or whatever else you want to call it. His paper seems to give the impression that there is a blank canvas just waiting to be filled with enlightened quantitative scientific research.

The awkward truth is that serious research is not wholly welcome in Government. It does tend to yield unwelcome knowledge about the costs and the demands of education that would really benefit children in general and it does disclose some of the less welcome news about the inseparably awkward social and political consequences of national educational practices and processes.

In general terms I agree with Goldacre that teachers, researchers, and teacher researchers have common cause in resisting ideological control from central government. The best possible research would be a teacher's best source of professional development and a strong stimulus for personal improvement as a teacher. However, RCTs need to be understood as just one way of accumulating knowledge about teaching, not as a gold standard.

A more balanced review of RCT practice was presented at the 2010 Conference of the Society for Research on Educational Effectiveness (SREE) by Savitha Moorthy, Raquel Sanchez, and Fannie Tseng. (pdf download: http://www.eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED513344)The SREE is broadly sympathetic to the cause-and-effect tradition of educational research espoused by Goldacre. The caveats included in this paper provide a useful balance to the "myths" described by Goldacre. From my perspective they still don’t address the more fundamental questions that supporters of the RCT approach have not answered.

The promotion of RCTs by a Minister like Michael Gove would  yield a narrowing stultification of research and a futile search for simplistic recipe knowledge. There is no certainty in teaching, just a gradual and intelligent gathering of wisdom using as many valid approaches as possible.

References

Goldacre, Ben (2013) Building Evidence Into Education London Department for Education Website 19pp (http://media.education.gov.uk/assets/files/ben goldacre paper.pdf)

Haynes, Laura; Service, Owain; Goldacre, Ben; Torgerson, David (2012) Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials Cabinet Office London 35pp (https://www.gov.uk/government/publications/test-learn-adapt-developing-public-policy-with-randomised-controlled-trials)