Ei-iE

By Steve Klees, University of Maryland

To experiment with the possible privatisation of its primary education system, Liberia initiated the Partnership Schools of Liberia (PSL) which turned over the management of 93 public schools to eight private contractors.  This has been a very controversial experiment, both in Liberia and worldwide.  A randomized controlled trial (RCT) study was set up comparing the PSL schools with matched public schools, and the report on the first year results of the RCT was recently published.  A variety of blogs, op eds, and tweets have discussed their findings, including one on this site.  I have just completed a paper critiquing the RCT study and the PSL, and I report on my results here.  I thought the RCT study was carefully and competently done, but I have some major disagreements with some of the analyses and the interpretation of findings.

Did Test Scores Increase by 60 Percent?

This is the key and most disseminated finding of the RCT study.  But is it valid?  This result is based on a regression analysis of post-test scores on whether students are in the PSL schools and a few control variables.  Notably absent was any control for a student’s pretest score.  This is an extremely unusual procedure.  In almost all RCTs, the impact of the treatment is measured by the gain in the post-test score over the pre-test score.  It turns out that the pre-test score in the PSL schools was significantly higher than in the government schools.  The RCT study argues that this was due to late pre-testing and that the higher score reflects that the PSL schools were already doing better due to the intervention.  But this is speculation based on very little evidence.  When they do control for pre-test, the effect of the PSL is considerably reduced to about 45%, on average, across math and English.  I believe most researchers would say these are the figures that should be used.

Another way to look at the impact of the PSL on test scores is to look directly at the raw test data.  The test data in the analyses above does not use the raw test data but weights it according to item response theory.  If we examine the impact of the PSL on the raw test data, we find that the impact of the PSL is even lower, on average about 35%.  Moreover, the relatively low effect in standard deviation terms makes it likely that neither the experimental nor control groups are learning much

What are the Reasons for the Gain in Test Scores?

The RCT study claims that about one-quarter of the gain in test scores was due to reduced teacher absenteeism resulting from better management of the PSL schools.  I argue that the statistical procedure used is completely invalid.  I contend that any gain in test scores has little, if anything, to do with the private management of schools.  Instead, the gains in test scores are most likely the result of five policy changes that could easily be implemented in regular public schools without any need for outsourcing to the private sector.

First, PSL schools were allowed to increase school hours, on average, by 22%, with one of the few effective contractors increasing contact hours by 70%.  Second, PSL schools were allowed to deviate from the primary school curriculum and emphasize instruction in math and English, thus teaching to the test.  Third, PSL schools were permitted to have lower class sizes than regular public schools.  Fourth, PSL schools were allowed to replace many existing classroom teachers with much better trained, higher paid, recent graduates of teacher training institutions that had been focusing on math and English instruction.  Finally, the private contractors were able to afford to supply more of basic inputs like textbooks, chalk, and pens and pencils.

Are the PSL Results Generalizable and Sustainable?

The RCT study is careful to say that the PSL is neither sustainable nor generalizable, and I agree.  PSL schools were subsidized to double the $50/student average expenditure on primary school students, and most private contractors supplemented that by very large amounts, up to an additional $700 to $1000 per student.  Moreover, the schools in the PSL were not at all a random sample of schools in Liberia.  Both experimental and control schools were purposefully selected to be closer to the capital, Monrovia, close to paved roads, have more students and classrooms, as well as access to better infrastructure like water and latrines or toilets.  Both were also allowed to have many more teachers per school than average.  All this makes the value of the PSL and this RCT study for Liberian policy-makers very questionable.

Cost and Cost-Effectiveness

Given the very high expenditures and relatively low test score gains, the RCT study concludes that the experiment is far from cost-effective.  I concur and add an examination of PSL expenditures.  The PSL and its RCT are very expensive.  I add together the $50 per student supplement received, the higher salaries paid to PSL teachers, the cost of running the RCT, and the expenditures added by the private contractors.  In total, the PSL is likely to have cost over $25 million for the three-year period!  This is a very expensive experiment, and this does not include the substantial additional costs that will be incurred with the planned expansion of the PSL.

Conclusions

Is it worth it?  My conclusion is that the PSL is a waste of resources.  There is a huge international literature that clearly answers the question of whether private schools are better than public schools.  They are not.  With similar students, private and public schools show similar achievement levels.   Moreover, as this paper shows, the test score gains of the PSL schools are very likely due to changes that could easily be made in regular public schools.

The PSL seems to have been initiated more for ideological reasons than for its potential effectiveness or cost-effectiveness.  Early into the experiment, Minister of Education Werner said to PSL’s critics:

While I believe it holds great potential, my team and I are clear that the program will not be scaled significantly until the data shows it works…. Judge us on the data – data on whether PSL schools deliver better learning outcomes for children.

Yet halfway through the RCT, without any data available, Minister Werner announced that he was going to expand the project.  That decision stimulated controversy within and outside Liberia, and the Minister was forced to recant.  But then, in June of this year, he reiterated his plans to go ahead with more than doubling the number of PSL schools, again before any of the RCT study results were available.

The Minister seems to have little interest in data.  Early into the PSL, the University of Wisconsin was commissioned by Education International and Action Aid to undertake a qualitative study to complement the RCT and get inside the black box of the RCT to find out more about how and why it was working – but the Minister refused to allow the study to be conducted.

Minister Werner has indicated that Liberia has the potential in the medium term to double primary school expenditures per pupil from $50 to $100.  These additional resources could go a long way towards improving government schools and student learning without the need to privatize Liberia’s educational system.