Ei-iE

The OECD’s Programme for International Student Assessment or PISA has been running for fifteen years. It is not the only international survey of student achievement. There are others such as TIMSS (Trends in International Mathematics and Science Study) and PIRLS (Progress in International Reading Literacy Study) but it is the one to which policy makers and the media now give the greatest attention. It is also the most controversial- certainly amongst academics and researchers.

Last year an open letter was signed by a number of academics, who were critical that they had been excluded in decision making related to PISA. They were also critical because PISA surveyed three ‘literacies’ – reading, numeracy and science -- which they believed painted too narrow a picture of what was important in education systems.

As a key member of TUAC, EI has been attending some of the governance meetings for PISA and has been engaged in these debates. EI has consistently said that it finds the policy messages arising from PISA extremely important, including the message that non-selective comprehensive public education systems provide the best conditions for educational success. However, PISA’s performance “league tables” obscure these important policy messages by creating a media frenzy around the relative positions of individual countries within the tables. The irony of the tables is that by OECD’s own admission, with the exception of the few outlier countries at either end of the tables, the position of countries has little meaning but it does create a great deal of often unhelpful noise.

In order to provide a forum for debate around PISA and encourage the use of its data to constructively inform education policy and research, OECD recently ran a conference in Oslo in November 2015 that was co-sponsored by the Norwegian Government: ‘What can PISA tell us?’

Norway’s EI affiliate, UEN, attended and EI’s Deputy General Secretary, Haldis Holst, and Senior Consultant, John Bangs, were speakers. A range of speakers were fielded from the academic community including Ekhart Klieme of the German Institute for International Educational Research, Irwin Kirsch from ETS, the University of Oslo’s Kirsti Kiette, David Rutkowski, and Karen Hammerness, and William H Schmidt from Michigan State University.

Many valuable insights and findings were shared by these researchers. I’m going to highlight those of Kleime and Hammerness here.

Kleime presented on “What does PISA Measure” and began by pointing out that PISA’s main purpose is about building indicators for policy not for research. PISA is not a study of educational effectiveness per se because it is cross-sectional data. He recommended design enhancements for PISA, including an allowance for classroom-based, longitudinal data to cross-check school effectiveness.

Hammerness looked at the “teacher dimension” for those countries where the PISA scores are tops (e.g., Finland, Hong Kong) demonstrate a mixture of the following kinds of “good practices” in relation to teacher policies: selectivity; high quality teacher preparation; teaching as a collaborative practice in the schools; professional standards underpinning teaching; teacher learning as a continuum; and opportunities for teacher leadership in schools. Such policies were aligned, coherent and stable over time.

Both Haldis Holst and I emphasised that the issues arising from PISA were not about its reliability and validity but about how it was used and presented, (eg through its performance tables and the media response). Haldis said researchers should challenge the OECD when they believed their research was being misinterpreted. I reminded that EI had in fact presented an alternative to PISA’s performance tables in a study by Professor Peter Mortimore.

I also emphasised that the most important thing for policy makers to do was to engage with the issues coming out of PISA such as social class, equity and the need to involve teachers and their unions in education policy making.

As Haldis reminded the conference, PISA can inform policy not determine it. ‘In real life…the individual child cannot be aggregated. She is unique, not a random sample.’